Patent ID: 12204518

DETAILED DESCRIPTION

In the following description of various illustrative embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which is shown, by way of illustration, various embodiments in which aspects of the disclosure may be practiced. It is to be understood that other embodiments may be utilized, and structural and functional modifications may be made, without departing from the scope of the present disclosure.

It is noted that various connections between elements are discussed in the following description. It is noted that these connections are general and, unless specified otherwise, may be direct or indirect, wired or wireless, and that the specification is not intended to be limiting in this respect.

FIGS.1A and1Bdepict an illustrative computing environment for providing and maintaining efficient and effective processing of sets of work items in enterprise computing environments by optimizing distributed and parallelized batch data processing in accordance with one or more example embodiments. Referring toFIG.1A, computing environment100may include one or more computer systems. For example, computing environment100may include a distributed data processing computing platform110, enterprise computing infrastructure120, a work queue database130, an administrator computing device140, an enterprise user computing device150, and a remote user computing device160.

As illustrated in greater detail below, distributed data processing computing platform110may include one or more computing devices configured to perform one or more of the functions described herein. For example, distributed data processing computing platform110may include one or more computers (e.g., laptop computers, desktop computers, servers, server blades, or the like).

Enterprise computing infrastructure120may include one or more computing devices and/or other computer components (e.g., processors, memories, communication interfaces). In addition, and as illustrated in greater detail below, enterprise computing infrastructure120may be configured to provide various enterprise and/or back-office computing functions for an organization, such as a financial institution. For example, enterprise computing infrastructure120may include various servers and/or databases that host, execute, and/or store various enterprise resources (e.g., enterprise applications, enterprise databases, enterprise information). For instance, enterprise computing infrastructure120may include various servers and/or databases that store and/or otherwise maintain account information, such as financial account information including account balances, transaction history, account owner information, and/or other information. In addition, enterprise computing infrastructure120may host, execute, and/or store one or more enterprise applications that process and/or otherwise execute transactions on specific accounts based on commands and/or other information received from other computer systems included in computing environment100. Additionally or alternatively, enterprise computing infrastructure120may load data from distributed data processing computing platform110, manipulate and/or otherwise process such data, and return modified data and/or other data to distributed data processing computing platform110and/or to other computer systems included in computing environment100.

Work queue database130may include one or more computing devices and/or other computer components (e.g., processors, memories, communication interfaces). In addition, and as illustrated in greater detail below, work queue database130may be configured to store information associated with various work items and/or other information. In some instances, such information may be organized into various data tables and/or sub-databases maintained by work queue database130. Additionally, work queue database130may store and/or execute one or more database engines which may manage the information stored by work queue database130, enable access to such information, and/or perform other functions.

Administrator computing device140may be a personal computing device (e.g., desktop computer, laptop computer) or mobile computing device (e.g., smartphone, tablet). In addition, administrator computing device140may be linked to and/or used by an administrative user (who may, e.g., be a network administrator of an organization operating distributed data processing computing platform110).

Enterprise user computing device150may be a personal computing device (e.g., desktop computer, laptop computer) or mobile computing device (e.g., smartphone, tablet). In addition, enterprise user computing device150may be linked to and/or used by an enterprise user (who may, e.g., be an employee or other affiliate of an organization operating distributed data processing computing platform110).

Remote user computing device160may be a personal computing device (e.g., desktop computer, laptop computer) or mobile computing device (e.g., smartphone, tablet). In addition, remote user computing device160may be linked to and/or used by a remote user (who may, e.g., be an employee or other affiliate of an organization operating distributed data processing computing platform110and who may be located outside of an enterprise network associated with the organization).

Computing environment100also may include one or more networks, which may interconnect one or more of distributed data processing computing platform110, enterprise computing infrastructure120, work queue database130, administrator computing device140, enterprise user computing device150, and remote user computing device160. For example, computing environment100may include a private network170(which may, e.g., interconnect distributed data processing computing platform110, enterprise computing infrastructure120, work queue database130, administrator computing device140, enterprise user computing device150, and/or one or more other systems which may be associated with an organization, such as a financial institution) and public network180(which may, e.g., interconnect remote user computing device160with private network170and/or one or more other systems, public networks, sub-networks, and/or the like).

In one or more arrangements, work queue database130, administrator computing device140, enterprise user computing device150, remote user computing device160, and/or the other systems included in computing environment100may be any type of computing device capable of receiving a user interface, receiving input via the user interface, and communicating the received input to one or more other computing devices. For example, work queue database130, administrator computing device140, enterprise user computing device150, remote user computing device160, and/or the other systems included in computing environment100may, in some instances, be and/or include server computers, desktop computers, laptop computers, tablet computers, smart phones, or the like that may include one or more processors, memories, communication interfaces, storage devices, and/or other components. As noted above, and as illustrated in greater detail below, any and/or all of distributed data processing computing platform110, enterprise computing infrastructure120, work queue database130, administrator computing device140, enterprise user computing device150, and remote user computing device160may, in some instances, be special-purpose computing devices configured to perform specific functions.

Referring toFIG.1B, distributed data processing computing platform110may include one or more processor(s)111, memory(s)112, and communication interface(s)113. A data bus may interconnect processor111, memory112, and communication interface113. Communication interface113may be a network interface configured to support communication between distributed data processing computing platform110and one or more networks (e.g., network170, network180, or the like). Memory112may include one or more program modules and/or processing engines having instructions that when executed by processor111cause distributed data processing computing platform110to perform one or more functions described herein and/or one or more databases that may store and/or otherwise maintain information which may be used by such program modules, processing engines, and/or processor111. In some instances, the one or more program modules, processing engines, and/or databases may be stored by and/or maintained in different memory units of distributed data processing computing platform110and/or by different computing devices that may form and/or otherwise make up distributed data processing computing platform110. For example, memory112may have, store, and/or include a distributed data processing module112a, a distributed data processing database112b, a machine learning engine112c, and a plurality of workers112d-1,112d-2,112d-n.

Distributed data processing module112amay have instructions that direct and/or cause distributed data processing computing platform110to optimize distributed and parallelized batch data processing, as discussed in greater detail below. Distributed data processing database112bmay store information used by distributed data processing module112aand/or distributed data processing computing platform110in optimizing distributed and parallelized batch data processing. Machine learning engine112cmay perform and/or provide one or more machine learning and/or artificial intelligence functions and/or services, as illustrated in greater detail below. Each worker of the plurality of workers112d-1,112d-2,112d-nmay retrieve and process work items and/or perform other functions, as illustrated in greater detail below.

FIGS.2A-2Cdepict an illustrative event sequence for providing and maintaining efficient and effective processing of sets of work items in enterprise computing environments by optimizing distributed and parallelized batch data processing in accordance with one or more example embodiments. Referring toFIG.2A, at step201, work queue database130may load an initial work queue. For instance, work queue database130may receive initial groups of work items from various other computing devices and/or systems (e.g., enterprise computing infrastructure120, administrator computing device140, enterprise user computing device150, and/or remote user computing device160), tag and/or otherwise process the work items, and/or load the work items into one or more data tables that may form the initial work queue. Each work item may, for instance, be a record in the database that instructs distributed data processing computing platform110or another computer system to do a specific piece or sequence of processing work. In some instances, a work item may include one or more commands to be executed (e.g., by distributed data processing computing platform110and/or one or more workers) and/or embedded data to be processed. In this way, work queue database130may maintain a current work queue that can be filled from other systems and/or devices and may operate as a task list that includes functions and/or processing work to be completed and/or assigned to different workers.

At step202, distributed data processing computing platform110may receive initial configuration settings from administrator computing device140. Such initial configuration settings may, for instance, define various parameters that can be used by different workers that may operate on distributed data processing computing platform110. For instance, one or more of these initial configuration settings may define a configurable number of work items that each worker should request from work queue database130in a given instance.

At step203, distributed data processing computing platform110may initialize one or more workers. For example, at step203, distributed data processing computing platform110may initialize at least two processing workers, and initializing the at least two processing workers may include initializing a first processing worker (e.g., worker112d-1) and a second processing worker (e.g., worker112d-2).

At step204, distributed data processing computing platform110may cause worker112d-1to query work queue database130and process work items. For example, at step204, distributed data processing computing platform110may cause the first processing worker (e.g., worker112d-1) to perform a first query on a work queue database (e.g., work queue database130), where performing the first query on the work queue database includes reading and locking a first set of work items identified in the work queue database. For instance, worker112d-1may read and lock a first set of work items identified in the work queue database (e.g., work queue database130) based on instructions and/or commands received from distributed data processing computing platform110. In addition, distributed data processing computing platform110may cause the first processing worker (e.g., worker112d-1) to initiate parallel processing of the first set of work items, where initiating parallel processing of the first set of work items includes: processing a first subset of the first set of work items in parallel; and upon completion of processing each work item of the first subset of the first set of work items, marking the corresponding work item as completed in the work queue database (e.g., work queue database130). For instance, worker112d-1may process a first subset of the first set of work items in parallel, and upon completion of processing each work item of the first subset of the first set of work items, mark the corresponding work item as completed in the work queue database (e.g., work queue database130) based on instructions and/or commands received from distributed data processing computing platform110.

In marking the corresponding work item as completed in the work queue database (e.g., work queue database130), worker112d-1may read data records that form or otherwise make up the work item, mark such data records as reserved, and/or write a timestamp to the database (or, e.g., cause such a timestamp to be recorded in the database or with a database engine associated with the database). Additionally or alternatively, in retrieving work items from the work queue database (e.g., work queue database130), worker112d-1may take a configurable number of work items from a processing queue maintained in and/or associated with the work queue database (e.g., work queue database130). For example, distributed data processing computing platform110and/or worker112d-1may be configured to read the queue associated with work queue database130, grab and/or reserve twenty pieces of work (e.g., work items), and begin processing such work items. For instance, distributed data processing computing platform110and/or worker112d-1may be configured to process four items simultaneously in parallel (e.g., if the worker and/or the platform have sufficient bandwidth, at the time, to support parallel processing). After distributed data processing computing platform110and/or worker112d-1splits the retrieved work items and begins processing them in parallel, and at the conclusion of processing a given work item, distributed data processing computing platform110and/or worker112d-1may mark the corresponding data record as completed in the database (e.g., work queue database130). For example, in processing the set of work items retrieved from the database, distributed data processing computing platform110and/or worker112d-1may write data to work queue database130and/or enterprise computing infrastructure120. When distributed data processing computing platform110and/or worker112d-1completes processing that set of work items, distributed data processing computing platform110and/or worker112d-1may go back to the queue and request additional work items from work queue database130. If there are no additional work items to process at that time, distributed data processing computing platform110and/or worker112d-1may periodically ping the queue and wait for more work items to be added to the queue.

In some embodiments, performing the first query on the work queue database may include causing a database engine associated with the work queue database to lock the first set of work items upon read. For example, in performing the first query on the work queue database (e.g., work queue database130), distributed data processing computing platform110and/or worker112d-1may cause a database engine associated with the work queue database (e.g., work queue database130) to lock the first set of work items upon read. By locking the first set of work items upon read, distributed data processing computing platform110and/or worker112d-1may ensure that other workers (e.g., worker112d-2) do not grab the same work items as the first worker (e.g., worker112d-1).

In some embodiments, performing the first query on the work queue database may include selecting the first set of work items based on one or more priority parameters. For example, in performing the first query on the work queue database (e.g., work queue database130), distributed data processing computing platform110and/or worker112d-1may select the first set of work items based on one or more priority parameters. Such priority parameters may, for instance, specify that certain types of work items should be selected and/or processed before other types of work items. For instance, the priority parameters may specify that ‘critical’ items should be processed before ‘high priority’ items, which should be processed before ‘medium priority’ items, which should be processed before ‘low priority’ items.

In some embodiments, performing the first query on the work queue database may include writing, in the work queue database, a first timestamp indicating when the first set of work items were read from the work queue database. For example, in performing the first query on the work queue database (e.g., work queue database130), distributed data processing computing platform110and/or worker112d-1may write, in the work queue database (e.g., work queue database130), a first timestamp indicating when the first set of work items were read from the work queue database (e.g.,130). This timestamp may, for instance, enable tracking of processing times for various work items, which in turn may enable various metrics to be computed and/or captured as well as for expired work items to be passed to alternative workers for processing.

In some embodiments, the first timestamp may be tracked by a database engine associated with the work queue database to provide expired items to one or more other processing workers. For example, the first timestamp written by distributed data processing computing platform110and/or worker112d-1in the work queue database (e.g., work queue database130) may be tracked by a database engine associated with the work queue database (e.g., work queue database130) to provide expired items to one or more other processing workers. For instance, the database engine may determine that a given item is expired if a predetermined amount of time (e.g., five minutes) has elapsed since the timestamp was written (and, e.g., the work item was read and locked by a given worker) and the item has not been marked completed. In these instances, the database engine may operate on the assumption that if the item has not been marked completed by the worker that read and locked the item within the predetermined amount of time (e.g., five minutes), that the worker has failed and that the item will not be processed. Thus, in these instances, the database engine may unlock the item so that the item becomes available again for processing by one or more other workers (e.g., worker112d-2). In particular, once the item is unlocked and available again, another processing worker (e.g., worker112d-2) may read and lock the item (and subsequently processed the item) even though the item was previously read and locked by the first processing worker (e.g., worker112d-1).

Referring toFIG.2B, at step205, distributed data processing computing platform110may cause worker112d-2to query work queue database130and process work items. For example, at step205, distributed data processing computing platform110may cause the second processing worker (e.g., worker112d-2) to perform a second query on the work queue database (e.g., work queue database130), where performing the second query on the work queue database includes reading and locking a second set of work items identified in the work queue database. For instance, worker112d-2may read and lock a second set of work items identified in the work queue database (e.g., work queue database130) based on instructions and/or commands received from distributed data processing computing platform110. In addition, distributed data processing computing platform110may cause the second processing worker (e.g., worker112d-2) to initiate parallel processing of the second set of work items, where initiating parallel processing of the second set of work items includes: processing a first subset of the second set of work items in parallel; and upon completion of processing each work item of the first subset of the second set of work items, marking the corresponding work item as completed in the work queue database (e.g., work queue database130). For instance, worker112d-2may process a first subset of the second set of work items in parallel, and upon completion of processing each work item of the first subset of the second set of work items, mark the corresponding work item as completed in the work queue database (e.g., work queue database130) based on instructions and/or commands received from distributed data processing computing platform110.

In processing the work items and marking work items complete, worker112d-2may execute functions and/or perform steps similar to those described above with respect to worker112d-1. For instance, worker112d-2may read data records that form or otherwise make up the work item, mark such data records as reserved, and/or write a timestamp to the database (or, e.g., cause such a timestamp to be recorded in the database or with a database engine associated with the database). Additionally or alternatively, in retrieving work items from the work queue database (e.g., work queue database130), worker112d-2may take a configurable number of work items from the processing queue maintained in and/or associated with the work queue database (e.g., work queue database130). For example, distributed data processing computing platform110and/or worker112d-2may be configured to read the queue associated with work queue database130, grab and/or reserve twenty pieces of work (e.g., work items), and begin processing such work items. Worker112d-2may retrieve such work items by executing a command to ‘select’ the top N database records (e.g., the top 20 database records) ‘where’ such records are not locked. Because the database engine associated with work queue database130processes requests sequentially, this retrieval query will ensure that the second worker (e.g., worker112d-2) does not get any of the non-expired records taken by the first worker (e.g., worker112d-1) since each worker may command the database engine associated with work queue database130to lock records upon read. As a result, the second worker (e.g., worker112d-2) might not even see the records that the first worker (e.g., worker112d-1) is processing, because the first worker's records were locked and the database engine essentially makes those records invisible to the second worker.

Like the first worker (e.g., worker112d-1), the second worker (e.g., worker112d-2) may be configured to process four items simultaneously in parallel (e.g., if the worker and/or the platform have sufficient bandwidth, at the time, to support parallel processing). After distributed data processing computing platform110and/or worker112d-2splits the retrieved work items and begins processing them in parallel, and at the conclusion of processing a given work item, distributed data processing computing platform110and/or worker112d-2may mark the corresponding data record as completed in the database (e.g., work queue database130). For example, in processing the set of work items retrieved from the database, distributed data processing computing platform110and/or worker112d-2may write data to work queue database130and/or enterprise computing infrastructure120. When distributed data processing computing platform110and/or worker112d-2completes processing that set of work items, distributed data processing computing platform110and/or worker112d-2may go back to the queue and request additional work items from work queue database130. If there are no additional work items to process at that time, distributed data processing computing platform110and/or worker112d-2may periodically ping the queue and wait for more work items to be added to the queue.

In some embodiments, performing the second query on the work queue database may include reading at least one work item that was read and locked by the first processing worker. For example, in performing the second query on the work queue database (e.g., work queue database130) at step205, distributed data processing computing platform110and/or worker112d-2may read at least one work item that was read and locked by the first processing worker (e.g., worker112d-1). This may, for instance, occur because the timestamp(s) of the at least one work item has expired and the database engine associated with work queue database130is once again returning the at least one work item for processing. As introduced in examples discussed above, if work items are initially read and locked but not processed within a predetermined amount of time after being read and locked, these work items may then become available for processing by another worker. The ‘locked’ status of these work items is maintained by and expires with the database engine associated with work queue database130, and so if a lock on some work items has expired when the second worker (e.g., worker112d-2) queries the database, the database engine may allow those expired and now-unlocked work items to be read and locked by the second worker. In these instances, it may be assumed that the first worker (e.g., worker112d-1) failed to process those work items is not going to be able to process them (e.g., because the worker failed or an error occurred from which the worker cannot recover). Additionally or alternatively, the first worker (e.g., worker112d-1) may be configured such that if it comes back online and starts looking for work again, the first worker will go back to work queue database130and request new work items for processing. The risk of duplicative efforts thus may be greatly reduced, if not eliminated, since the second worker might only end up with work items initially taken by the first worker in instances where the first worker has most likely failed.

At step206, distributed data processing computing platform110may monitor the workload of the various processing workers (e.g., worker112d-1, worker112d-2) and/or the work queue database (e.g., work queue database130). For example, at step206, distributed data processing computing platform110may monitor a pending workload in the work queue database (e.g., work queue database130). This may, for instance, include querying work queue database130, interrogating the database engine associated with work queue database130, and/or interrogating one or more hardware host systems associated with work queue database130.

In some embodiments, monitoring the pending workload in the work queue database may include requesting one or more size values from the work queue database. For example, in monitoring the pending workload in the work queue database (e.g., work queue database130), distributed data processing computing platform110may request one or more size values from the work queue database (e.g., work queue database130). For instance, distributed data processing computing platform110may query and/or retrieve size values indicating the size(s) of one or more tables, columns, rows, or the like associated with records that are waiting to be processed, without locking such records.

At step207, distributed data processing computing platform110may dynamically scale one or more worker processors. For example, at step207, based on monitoring the pending workload in the work queue database (e.g., work queue database130), distributed data processing computing platform110may dynamically scale an availability level of processing workers (e.g., worker112d-1, worker112d-2, worker112d-n). In some embodiments, dynamically scaling the availability level of processing workers may include initializing at least one additional processing worker. For example, in dynamically scaling the availability level of processing workers, distributed data processing computing platform110may initialize at least one additional processing worker. This may, for instance, include commanding and/or controlling a hypervisor to boot up a new virtual machine or other container that may host such a worker. In some embodiments, dynamically scaling the availability level of processing workers may include destroying at least one processing worker. For example, in dynamically scaling the availability level of processing workers, distributed data processing computing platform110may destroy at least one processing worker (e.g., by killing and/or otherwise deactivating one or more of worker112d-1, worker112d-2, worker112d-n).

At step208, distributed data processing computing platform110may generate one or more reports. Such reports may, for instance, include statistical information indicating how many worker items were processed in a given period of time by distributed data processing computing platform110and/or by individual workers, how many errors were encountered, how many items remain in work queue database130, and/or other information. Distributed data processing computing platform110may send and/or otherwise provide such reports to one or more other systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160), which may cause these other systems and/or devices to display such reports and/or interact with the information included in them.

Referring toFIG.2C, at step209, distributed data processing computing platform110may send the reports to one or more other systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160). In some instances, by sending the reports to one or more other systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160), distributed data processing computing platform110may cause such systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160) to generate, display, and/or otherwise present one or more graphical user interfaces that include information associated with work item processing. For instance, distributed data processing computing platform110may cause such systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160) to generate, display, and/or otherwise present a graphical user interface similar to graphical user interface300, which is illustrated inFIG.3. As seen inFIG.3, graphical user interface300may include text and/or other information indicating a high workload in the work queue database (e.g., “Current processing times are below optimal levels. 1 additional worker processor is being initialized to handle the current workload.”). Additionally or alternatively, distributed data processing computing platform110may cause such systems and/or devices (e.g., administrator computing device140, enterprise user computing device150, remote user computing device160) to generate, display, and/or otherwise present a graphical user interface similar to graphical user interface400, which is illustrated inFIG.4. As seen inFIG.4, graphical user interface400may include text and/or other information indicating a processor failure (e.g., “Processing Worker A has failed in completing processing of Work Item Set X. Processing Worker B is picking up the dropped work items for completion.”).

At step210, distributed data processing computing platform110may initialize a monitoring process. For example, at step210, distributed data processing computing platform110may initialize a monitoring process configured to monitor a pending workload in a work queue database (e.g., work queue database130).

At step211, distributed data processing computing platform110may cause the monitoring process to create records associated with the processing of work items in work queue database130. For example, at step211, distributed data processing computing platform110may cause a monitoring process to create one or more historical records. Such historical records may for instance, include information denoting current and/or prior data processing status of various work items. For example, distributed data processing computing platform110may cause the monitoring process to query the work queue database (e.g., work queue database130) and create one or more historical records indicative of a workload processing status associated with one or more processing workers (e.g., worker112d-1, worker112d-2), where the one or more processing workers (e.g., worker112d-1, worker112d-2) are configured to process work items identified in the work queue database (e.g., work queue database130) based on one or more processing parameters.

At step212, distributed data processing computing platform110may dynamically optimize itself and/or one or more other systems. For example, at step212, distributed data processing computing platform110may dynamically optimize one or more processing parameters based on the one or more historical records. For instance, distributed data processing computing platform110may adjust and/or otherwise optimize one or more processing parameters used by distributed data processing computing platform110and/or its associated workers (e.g., worker112d-1, worker112d-2, worker112d-n) based on the one or more historical records created at step211. In some instances, in dynamically optimizing the one or more processing parameters based on the one or more historical records, distributed data processing computing platform110may tune one or more processing parameters based on one or more machine learning models and/or algorithms. For example, distributed data processing computing platform110may use learning engine112cto tune the configurable number of work items retrieved by each processing worker (e.g., worker112d-1, worker112d-2, worker112d-n) based on the one or more historical records.

In some embodiments, after initializing the monitoring process and causing the monitoring process to query the work queue database and create one or more historical records and/or in dynamically optimizing the one or more processing parameters, distributed data processing computing platform110may identify one or more new parameter values for the one or more processing parameters associated with the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) based on the one or more historical records. For instance, distributed data processing computing platform110may identify such new parameter values (such as, e.g., the configurable number of work items, priority parameters, and/or other configuration parameters similar to those discussed in examples above) using learning engine112cto process the one or more historical records.

Subsequently, distributed data processing computing platform110may configure the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) based on the one or more new parameter values identified for the one or more processing parameters associated with the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n). For example, in configuring the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n), distributed data processing computing platform110may launch new instances of processing workers based on the new parameter values, modify settings and/or otherwise reconfigure existing processing workers based on the new parameter values, and/or otherwise change operational features based on the new parameter values.

In some embodiments, each processing worker of the one or more processing workers may be configured to read a set of work items identified in the work queue database and process the set of work items in parallel. For example, each processing worker of the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) may be configured to read a set of work items identified in the work queue database (e.g., work queue database130) and process the set of work items in parallel, as illustrated in the examples discussed above.

In some embodiments, each processing worker of the one or more processing workers may be configured to write a timestamp into the work queue database upon reading the set of work items, and the timestamp may indicate when the set of work items were read from the work queue database. For example, each processing worker of the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) may be configured to write a timestamp into the work queue database (e.g., work queue database130) upon reading the set of work items, and the timestamp may indicate when the set of work items were read from the work queue database (e.g., work queue database130), as illustrated in the examples discussed above.

In some embodiments, each processing worker of the one or more processing workers may be configured to lock the set of work items upon reading the set of work items. For example, each processing worker of the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) may be configured to lock the set of work items upon reading the set of work items, as illustrated in the examples discussed above.

In some embodiments, identifying the one or more new parameter values for the one or more processing parameters associated with the one or more processing workers may include: inputting, into at least one machine learning model, data selected from the one or more historical records; and receiving, from the machine learning model, the one or more new parameter values for the one or more processing parameters associated with the one or more processing workers. For example, in identifying the one or more new parameter values for the one or more processing parameters associated with the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n), distributed data processing computing platform110may input, into at least one machine learning model (which may, e.g., be stored by and/or executed on machine learning engine112c), data selected from the one or more historical records. Subsequently, distributed data processing computing platform110may receive, from the machine learning model, the one or more new parameter values for the one or more processing parameters associated with the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n). For instance, in receiving the one or more new parameter values for the one or more processing parameters associated with the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n) from the machine learning model, distributed data processing computing platform110may receive new values for the configurable number of work items reserved by each worker when requesting new work items from work queue database130, the expiration time used by the database engine associated with work queue database130(which may, e.g., control how long the database engine may wait to release locked—but incomplete—work items to another worker), priority settings, and/or other settings.

In some embodiments, configuring the one or more processing workers may include adjusting a number of work items requested by each processing worker of the one or more processing workers. For example, in configuring the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n), distributed data processing computing platform110may adjust a number of work items requested by each processing worker of the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n). This may, for instance, modify the number of work items reserved by each worker when requesting new work items from work queue database130.

In some embodiments, configuring the one or more processing workers may include adjusting one or more priority settings used by each processing worker of the one or more processing workers. For example, in configuring the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n), distributed data processing computing platform110may adjust one or more priority settings used by each processing worker of the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n). This may, for instance, modify settings that define higher priority work items for relatively earlier processing and lower priority items for relatively later processing.

In some embodiments, configuring the one or more processing workers may include adjusting one or more expiration time values of a database engine associated with the work queue database. For in configuring the one or more processing workers (e.g., worker112d-1, worker112d-2, worker112d-n), distributed data processing computing platform110may adjust one or more expiration time values of a database engine associated with the work queue database (e.g., work queue database130). For instance, distributed data processing computing platform110may modify expiration time value(s) of the database engine that responds to queries from the processing workers. This may, for instance, modify settings that control how long the database engine of work queue database130waits to release locked, incomplete work items to another worker for processing.

FIG.5depicts an illustrative method for providing and maintaining efficient and effective processing of sets of work items in enterprise computing environments by optimizing distributed and parallelized batch data processing in accordance with one or more example embodiments. Referring toFIG.5, at step505, a computing platform comprising at least one hardware processor, a communication interface, and memory may initialize at least two processing workers, and initializing the at least two processing workers may include initializing a first processing worker and a second processing worker. At step510, the computing platform may cause the first processing worker to: perform a first query on a work queue database, where performing the first query on the work queue database includes reading and locking a first set of work items identified in the work queue database; and initiate parallel processing of the first set of work items, where initiating parallel processing of the first set of work items includes: processing a first subset of the first set of work items in parallel; and upon completion of processing each work item of the first subset of the first set of work items, marking the corresponding work item as completed in the work queue database.

At step515, the computing platform may cause the second processing worker to: perform a second query on the work queue database, where performing the second query on the work queue database includes reading and locking a second set of work items identified in the work queue database; and initiate parallel processing of the second set of work items, where initiating parallel processing of the second set of work items includes: processing a first subset of the second set of work items in parallel; and upon completion of processing each work item of the first subset of the second set of work items, marking the corresponding work item as completed in the work queue database. At step520, the computing platform may monitor a pending workload in the work queue database. At step525, based on monitoring the pending workload in the work queue database, the computing platform may dynamically scale an availability level of processing workers.

FIG.6depicts another illustrative method for providing and maintaining efficient and effective processing of sets of work items in enterprise computing environments by optimizing distributed and parallelized batch data processing in accordance with one or more example embodiments. Referring toFIG.6, at step605, a computing platform comprising at least one hardware processor, a communication interface, and memory may initialize a monitoring process configured to monitor a pending workload in a work queue database. At step610, the computing platform may cause the monitoring process to query the work queue database and create one or more historical records indicative of a workload processing status associated with one or more processing workers, where the one or more processing workers may be configured to process work items identified in the work queue database based on one or more processing parameters. At step615, the computing platform may identify one or more new parameter values for the one or more processing parameters associated with the one or more processing workers based on the one or more historical records. At step620, the computing platform may configure the one or more processing workers based on the one or more new parameter values identified for the one or more processing parameters associated with the one or more processing workers.

One or more aspects of the disclosure may be embodied in computer-usable data or computer-executable instructions, such as in one or more program modules, executed by one or more computers or other devices to perform the operations described herein. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types when executed by one or more processors in a computer or other data processing device. The computer-executable instructions may be stored as computer-readable instructions on a computer-readable medium such as a hard disk, optical disk, removable storage media, solid-state memory, RAM, and the like. The functionality of the program modules may be combined or distributed as desired in various embodiments. In addition, the functionality may be embodied in whole or in part in firmware or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGA), and the like. Particular data structures may be used to more effectively implement one or more aspects of the disclosure, and such data structures are contemplated to be within the scope of computer executable instructions and computer-usable data described herein.

Various aspects described herein may be embodied as a method, an apparatus, or as one or more computer-readable media storing computer-executable instructions. Accordingly, those aspects may take the form of an entirely hardware embodiment, an entirely software embodiment, an entirely firmware embodiment, or an embodiment combining software, hardware, and firmware aspects in any combination. In addition, various signals representing data or events as described herein may be transferred between a source and a destination in the form of light or electromagnetic waves traveling through signal-conducting media such as metal wires, optical fibers, or wireless transmission media (e.g., air or space). In general, the one or more computer-readable media may be and/or include one or more non-transitory computer-readable media.

As described herein, the various methods and acts may be operative across one or more computing servers and one or more networks. The functionality may be distributed in any manner, or may be located in a single computing device (e.g., a server, a client computer, and the like). For example, in alternative embodiments, one or more of the computing platforms discussed above may be combined into a single computing platform, and the various functions of each computing platform may be performed by the single computing platform. In such arrangements, any and/or all of the above-discussed communications between computing platforms may correspond to data being accessed, moved, modified, updated, and/or otherwise used by the single computing platform. Additionally or alternatively, one or more of the computing platforms discussed above may be implemented in one or more virtual machines that are provided by one or more physical computing devices. In such arrangements, the various functions of each computing platform may be performed by the one or more virtual machines, and any and/or all of the above-discussed communications between computing platforms may correspond to data being accessed, moved, modified, updated, and/or otherwise used by the one or more virtual machines.

Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, and variations within the scope and spirit of the appended claims will occur to persons of ordinary skill in the art from a review of this disclosure. For example, one or more of the steps depicted in the illustrative figures may be performed in other than the recited order, and one or more depicted steps may be optional in accordance with aspects of the disclosure.