Systems and methods for micro-batch processing of data

This disclosure relates to micro-batch processing of data. Micro-batch processing of data may be accomplished by receiving data conveying information pertaining to operation of client computing platforms. For a general time duration, the data may be added to a general queue. The data in the general queue may be processed in memory in accordance with a general job. For one or more specific time durations, the data may be added to one or more specific queues based on the client computing platform to which the data pertains. The data in the one or more specific queues may be processed in memory in accordance with one or more specific jobs. One or more errors in processing the data may be detected. The data corresponding to the detected error may be added to a skip queue.

FIELD

This disclosure relates to systems and methods for micro-batch processing of data.

BACKGROUND

Data received from client computing platforms may be processed to extract information within the data. Processing data may include multiple operations on the data. Processing large amounts of data may be unwieldly and time consuming. If errors occur while processing large amounts of data, it may be difficult to locate the cause/source of the errors.

SUMMARY

This disclosure relates to micro-batch processing of data. Micro-batch processing of data may be accomplished by receiving data conveying information pertaining to operation of client computing platforms. For a general time duration, the data may be added to a general queue. The data in the general queue may be processed in memory in accordance with a general job. For one or more specific time durations, the data may be added to one or more specific queues based on the client computing platform to which the data pertains. The data in the one or more specific queues may be processed in memory in accordance with one or more specific jobs. One or more errors in processing the data may be detected. The data corresponding to the detected error may be added to a skip queue.

A system for micro-batch processing of data may include one or more of physical processors, and/or other components. The physical processor(s) may be configured by machine-readable instructions. Executing the machine-readable instructions may cause the physical processor(s) to facilitate micro-batch processing of data. The machine-readable instructions may include one or more computer program components. The computer program components may include one or more of a receive data component, a general queue component, a general job component, a specific queue component, a specific job component, an error component, and/or other computer program components.

The receive data component may be configured to receive one or more data conveying information pertaining to operation of client computing platforms. In some implementations, client computing platforms may include one or more of a camera, a mobile device, a desktop device, and/or other client computing platforms. One or more data may pertain to a client computing platform. For example, one or more data may include first data pertaining to a first client computing platform, and/or other data. In some implementations, information pertaining to operation of client computing platforms may include information pertaining to operation of one or more cameras communicating with the client computing platforms.

The general queue component may be configured to add one or more data to one or more general queues. One or more general queues may correspond to one or general time durations. For example, a general queue may correspond to a general time duration. Adding one or more data to a general queue may include adding one or more data to the general queue for the general time duration. In some implementations, the general queue component may be configured to add one or more data to one or more general queues based on the client computing platform to which the data pertains.

The general job component may be configured to process the data in one or more general queues in memory in accordance with one or more general jobs. One or more general queues may correspond to one or more general jobs. For example, a general queue may correspond to a general job. Processing the data in one or more general queues may include processing the data in the general queue in accordance with the general job. In some implementations, one or more general jobs may include one or more metadata operations on the data in the one or more general queues.

The specific queue component may be configured to add one or more data to one or more specific queues based on the client computing platform to which the data pertains. One or more specific queues may include one or more specific queues corresponding to one or more client computing platforms. One or more specific queues may correspond to one or more specific time durations. For example, one or more specific queues may include a first specific queue corresponding to the first client computing platform and/or other specific queues. The first specific queue may correspond to a first specific time duration. Adding one or more data to one or more specific queues may include adding the first data to the first specific queue for the first specific time duration.

The specific job component may be configured to process the data in one or more specific queues in memory in accordance with one or more specific jobs. One or more specific queues may correspond to one or more specific jobs. For example, the first specific queue may correspond to a first specific job. Processing the data in one or more specific queues may include processing the first data in accordance with the first specific job.

The error component may be configured to detect one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs. In some implementations, detecting one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs may include detecting one or more unexpected stops in processing the data in accordance with one or more general jobs and/or one or more specific jobs. In some implementations, detecting one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs may include detecting one or more unexpected values in one or more data.

Responsive to detection of one or more errors, the error component may be configured to add the data corresponding to the detected error to one or more skip queues. In some implementations, the error component may be configured to generate one or more notifications based on detection of one or more errors in processing the data in accordance with the general job and/or the one or more specific jobs. In some implementations, the error component may be configured to provide access to one or more skip queues. Access to one or more skip queues may be provided such that one or more data corresponding to one or more detected errors is assessable via one or more skip queue queries.

In some implementations, the error component may be configured to detect an unavailability of one or more general jobs and/or one or more specific jobs to process the data. Responsive to detection of the unavailability of one or more general jobs and/or one or more specific jobs to process the data, the error component may be configured to add the data corresponding to the detected unavailability to one or more retry queues.

DETAILED DESCRIPTION

FIG. 1illustrates system10for micro-batch processing of data. Micro-batch processing of data may be accomplished by receiving data conveying information pertaining to operation of client computing platforms. For a general time duration, the data may be added to a general queue. The data in the general queue may be processed in memory in accordance with a general job. For one or more specific time durations, the data may be added to one or more specific queues based on the client computing platform to which the data pertains. The data in the one or more specific queues may be processed in memory in accordance with one or more specific jobs. One or more errors in processing the data may be detected. The data corresponding to the detected error may be added to a skip queue.

System10may include one or more server11, processor12, electronic storage13, client computing platform(s)14, external resource(s), and/or other components. Components of system10may be connected via one or more networks16, for example communications networks.

Server11may include one or more of processor12, electronic storage13, and/or other components. Processor12may be configured to provide information processing capabilities in system10. As such, processor12may comprise one or more of a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. Processor12may be configured to execute one or more machine readable instructions100to facilitate micro-batch processing of data. Machine-readable instructions100may include one or more computer program components. Machine-readable instructions100may include one or more of receive data component102, general queue component104, general job component106, specific queue component108, specific job component110, error component112, and/or other computer program components.

Electronic storage13may include electronic storage media that electronically stores information. Electronic storage13may store software algorithms, information determined by processor12, information received remotely, and/or other information that enables system10to function properly. For example, electronic storage13may store information relating to data conveying information pertaining to operation of client computing platforms, client computing platforms, general queues, general jobs, specific queues, specific jobs, skip queues, retry queues, and/or other information.

One or more features and/or functions of server11may be configured to facilitate micro-batch processing of data conveying information pertaining to operation of client computing platforms and/or other types of processing. It is noted that although the implementations disclosed herein are directed to micro-batch processing of data conveying information pertaining to operation of client computing platforms, one or more other implementations of system10and/or server11may be configured for other types of processing. By way of non-limiting example, types of data may include one or more of enterprise data, business intelligence data, data used to drive internal and/or commercial applications, and/or other types of data.

Receive data component102may be configured to receive one or more data conveying information pertaining to operation of client computing platform(s)14. Receive data component102may receive one or more data conveying information pertaining to operation of client computing platform(s)14directly from one or more client computing platform(s)14and/or indirectly (e.g., via an intermediate computing device, via a data collection service, etc.). Computing devices/services sending data to server11/receive data component102may be referred to as data sources. One or more data may pertain to a client computing platform. For example, one or more data may include first data pertaining to a first client computing platform, and/or other data.

Client computing platform(s)14may refer to computing devices that may generate and/or communicate information about their operation (e.g., modes of operation, activities of user interface, buttons pushed, features activated, etc.) to server11via a communications medium (e.g., network16, etc.). In some implementations, client computing platform(s)14may include one or more of a camera, a mobile device, a desktop device, and/or other client computing platforms. One or more of a camera, a mobile device, a desktop device, and/or other client computing platforms may generate and store information about their operation in one or more data and/or communicate information about their operation in one or more data to server11via a communications medium (e.g., network16, etc.). Data conveying information pertaining to operation of client computing platform(s)14may include different information and/or be stored in different formats based on client computing platform(s)14. For example, data conveying information pertaining to operation of client computing platform(s) may include information specific to client computing platform(s)14and/or be stored in format(s) specific to client computing platform(s)14(e.g., different formats for logs indicating operation of client computing platform(s)14, etc.).

Information pertaining to operation of client computing platform(s)14may include information pertaining operation of one or more hardware components and/or one or more software of client computing platform(s)14. For example, information pertaining to operation of a camera may include information pertaining to operation of one or more hardware components of the camera (e.g., lens size, focus, capture speed, hardware based control of image/video capture, etc.) and/or one or more software (e.g., driver version, software based control of image/video capture, metadata, etc.) of the camera. Information pertaining to operation of a camera may be stored in a camera log. Information pertaining to operation of a mobile device or a desktop device may include information pertaining to operation of one or more hardware components of the mobile/desktop device (e.g., hardware specification, hardware usage, etc.) and/or one or more software (e.g., software version, software based manipulation of image/video, etc.) of the mobile/desktop device. Information pertaining to operation of a mobile device or a desktop device may be stored in a log of mobile/desktop device operation and/or a log of software operation.

In some implementations, client computing platform(s)14may communicate with one or more cameras. Information pertaining to operation of client computing platform(s)14may include information pertaining to operation of one or more cameras communicating with client computing platform(s). For example, information pertaining to operation of client computing platform(s)14may include information pertaining to operation (e.g., modes of operation, activities of user interface, buttons pushed, features activated, etc.) of the one or more cameras. In some implementations, information pertaining to operation of client computing platform(s)14may include metadata generated by and/or from the one or more cameras.

FIG. 3-4illustrate examples of client computing platforms14connected to a network for micro-batch processing of data. InFIG. 3, server11may be connected to network16. Client computing platforms14connected to network16may include camera(s)14a, mobile device(s)14b, desktop device(s)14c, and other device(s)14d. Camera(s)14a, mobile device(s)14b, desktop device(s)14c, and/or other device(s)14dmay generate and/or communicate information about their operation to server11via network16.

InFIG. 4, server11may be connected to network16. Client computing platforms14connected to network16may include camera(s)14a, mobile device(s)14bcommunicating with camera(s)14e, desktop device(s)14ccommunicating with camera(s)14f, and/or other device(s)14dcommunicating with camera(s)14g. Camera(s)14a, mobile device(s)14b, desktop device(s)14c, and/or other device(s)14dmay generate and/or communicate information about their operation to server11via network16. Mobile device(s)14b, desktop device(s)14c, and/or other device(s)14dmay generate and/or communicate information about operation of camera(s)14e,14f,14g, respectively, to server11via network16.

General queue component104may be configured to add one or more data to one or more general queues. One or more general queues may correspond to one or general time durations. For example, a general queue may correspond to a general time duration. Adding one or more data to a general queue may include adding one or more data to the general queue for the general time duration.

For example, the general time duration may include a few seconds (e.g., 2 seconds, etc.) or a few minutes (2 minutes, etc.) and general queue component104may batch a stream of data in the general queue by adding data to the general queue for the few seconds or the few minutes. Different general queues may correspond to the same general time duration or different general time durations. For example, a first general queue and a second general queue may correspond to a general time duration of 10 seconds, or a first general queue may correspond to a first general time duration of 5 seconds and a second general queue may correspond to a second general time duration of 60 seconds. Other durations for one or more general time durations are contemplated.

In some implementations, general queue component104may be configured to add one or more data to one or more general queues based on the client computing platform to which the data pertains. For example, data pertaining to certain client computing platforms may be added to one or more general queues. Data pertaining to certain client computing platforms may not be added to one or more general queues. For example, data pertaining to a certain computing platform may not contain information necessary to perform one or more operations corresponding to a general queue. General queue component104may filter such data so that they are not added to the general queue.

General job component106may be configured to process the data in one or more general queues in memory in accordance with one or more general jobs. One or more general queues may correspond to one or more general jobs. For example, a general queue may correspond to a general job. Processing the data in one or more general queues may include processing the data in the general queue in accordance with the general job.

A general job may refer to an operation performed on data regardless of client computing platform(s)14to which the data pertains. A general job may extract information from the data in one or more general queues, transform information from the data in one or more general queues, and/or add information to the data in one or more general queues. For example, a general job may include an IP-to-Geolocation operation, a serial number correlation operation, a user agent operation, a personally identifiable information anonymizing operation, a binary to human-readable format translation operation, a write to disk operation, or other general jobs. In some implementations, one or more general jobs may include one or more metadata operations on the data in one or more general queues.

An IP-to-Geolocation operation may obtain from the data the IP address(es) of the client computing platform(s)14from which data was sent and use the IP address(es) to determine the location(s) of the client computing platform(s)14. A serial number correlation operation may obtain from the data the serial number(s) of the client computing platform(s)14and use the serial number(s) to determine hardware and/or software information regarding the client computing platform(s)14. A user agent operation may obtain user agent information from the data and use the user agent information to determine hardware and/or software information regarding the client computing platform(s)14. A personally identifiable information anonymizing operation may identify personally identifiable information (PII) in the data and change PII into non-identifying information). A binary to human-readable format translation operation may convert data in binary format (e.g., camera log, etc.) into a human-readable format (e.g., JSON, XML, etc.). A write to disk operation may write the data in memory to one or more data storage (e.g., HDFS storage, etc.). Other general jobs are contemplated.

General job component106may process the data in one or more general queues to include data lineage information with the data. Data lineage information may indicate a history of processes performed on the data. The history of processes performed on the data may include information relating to time, result, other processes, and/or other information relating to the data. For example, general job component106may process the data in one or more general queues to include information relating to (1) time of when one or more general jobs for the data began and/or ended, (2) whether one or more general jobs for the data was successful, (3) whether one or more general jobs for the data failed, (4) whether one or more general jobs for the data will be repeated, and/or other data lineage information. Data lineage information may be embedded into the data or be separately stored from the data (e.g., metadata passed along with the data, etc.).

Specific queue component108may be configured to add one or more data to one or more specific queues based on the client computing platform to which the data pertains. One or more specific queues may correspond to one or more client computing platforms. One or more specific queues may correspond to one or more specific time durations. For example, one or more specific queues may include a first specific queue corresponding to the first client computing platform and/or other specific queues. The first specific queue may correspond to a first specific time duration. Adding one or more data to one or more specific queues may include adding the first data to the first specific queue for the first specific time duration.

For example, the first specific queue time duration may include a few seconds (e.g., 2 seconds, etc.) or a few minutes (2 minutes, etc.) and specific queue component108may batch a stream of data in the first specific queue by adding data to the first specific queue for the few seconds or the few minutes. Different specific queues may correspond to the same specific time duration or different specific time durations. For example, a first specific queue and a second specific queue may correspond to a specific time duration of 10 seconds, or a first specific queue may correspond to a first specific time duration of 5 seconds and a second specific queue may correspond to a second specific time duration of 60 seconds. One or more general time durations and one or more specific time durations may be of the same or different durations. Other durations for one or more specific time durations are contemplated.

Specific job component110may be configured to process the data in one or more specific queues in memory in accordance with one or more specific jobs. One or more specific queues may correspond to one or more specific jobs. For example, the first specific queue may correspond to a first specific job. Processing the data in one or more specific queues may include processing the first data in the first queue in accordance with the first specific job.

A specific job may refer to an operation performed on data based on client computing platform(s)14to which the data pertains. A specific job may extract information from the data in one or more specific queues, transform information from the data in one or more specific queues, and/or add information to the data in one or more specific queues. For example, a specific job may include an IP-to-Geolocation operation specific to client computing platform(s)14, a serial number correlation operation specific to client computing platform(s)14, a user agent operation specific to client computing platform(s)14, a personally identifiable information anonymizing operation specific to client computing platform(s)14, a binary to human-readable format translation operation specific to client computing platform(s)14, a write to disk operation specific to client computing platform(s)14, or other specific jobs. In some implementations, one or more specific jobs may include one or more metadata operations on the data in one or more specific queues.

A specific job may include a data conversion operation specific to client computing platform(s)14. A data conversion operation specific to client computing platform(s)14may convert data (containing different information based on client computing platform(s)14and/or stored in different formats based on client computing platform(s)14) into a general binary format, which may be converted into a human-readable format via a general binary to human-readable format translation operation. Other specific jobs are contemplated.

Specific job component110may process the data in one or more specific queues to include data lineage information with the data. Data lineage information may indicate a history of processes performed on the data. The history of processes performed on the data may include information relating to time, result, other processes, and/or other information relating to the data. For example, specific job component110may process the data in one or more specific queues to include information relating to (1) time of when one or more specific jobs for the data began and/or ended, (2) whether one or more specific jobs for the data was successful, (3) whether one or more specific jobs for the data failed, (4) whether one or more specific jobs for the data will be repeated, and/or other data lineage information. Data lineage information may be embedded into the data or be separately stored from the data (e.g., metadata passed along with the data, etc.).

Error component112may be configured to detect one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs. One or more general jobs and/or one or more specific jobs may indicate an error in processing the data to error component112. For example, one or more general jobs and/or one or more specific jobs may indicate an error in processing the data via one or more flags and error component112may be configured to detect one or more flags. Other methods of detecting errors are contemplated.

In some implementations, detecting one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs may include detecting one or more unexpected stops in processing the data in accordance with one or more general jobs and/or one or more specific jobs. For example, error component112may detect that one or more general jobs and/or one or more specific jobs stopped processing the data before completing their respective jobs.

In some implementations, detecting one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs may include detecting one or more unexpected values in one or more data. For example, error component112may detect a blank or obsolete value in the data. Error component112may detect an inappropriate/inconsistent value in the data. For example, error component112may detect a character value when a numeric value is expected (or vice versa) and/or inconsistencies between related values in the data. In some implementations, detecting one or more errors in processing the data in accordance with one or more general jobs and/or one or more specific jobs may include detecting ranges of error in one or more unexpected values in one or more data.

Responsive to detection of one or more errors, error component112may be configured to add the data corresponding to the detected error to one or more skip queues. One or more skip queues may correspond to one or more general jobs and/or one or more specific jobs. For example, in response to detecting one or more errors in accordance with a general job, error component112may add the data corresponding to the detected error to a common skip queue and/or a skip que corresponding to the general job. In response to detecting one or more errors in accordance with a specific job, error component112may add the data corresponding to the detected error to a common skip queue and/or a skip que corresponding to the specific job. Data corresponding to the detected error may include pre-processed data, post-processed data, information relating to the error, and/or other information. In some implementations, adding the data corresponding to the detected error to one or more skip queues may be based on the range of error.

In some implementations, error component112may be configured to generate one or more notifications based on detection of one or more errors in processing the data in accordance with one or more generals job and/or one or more specific jobs. The notification may indicate that one or more generals job and/or one or more specific jobs have failed to processed the data/some part(s) of the data. In some implementations, a notification may include an electronic message that is transferred to one or more users. In some implementations, a notification may include a message that is displayed on a user interface.

In some implementations, error component112may be configured to provide access to one or more skip queues. Access to one or more skip queues may be provided such that one or more data corresponding to one or more detected errors is assessable via one or more skip queue queries. For example, a skip queue query may provide access to one or more data corresponding to one or more detected errors added to a common skip queue. A skip queue query may provide access to one or more data corresponding to one or more detected errors added to one or more general skip queues. A skip queue query may provide access to one or more data corresponding to one or more detected errors added to one or more general skip queues. A skip queue query may provide access to one or more data corresponding to one or more detected error added to one or more different types of skip queues.

In some implementations, error component112may be configured to detect an unavailability of one or more general jobs and/or one or more specific jobs to process the data. Responsive to detection of the unavailability of one or more general jobs and/or one or more specific jobs to process the data, error component112may be configured to add the data corresponding to the detected unavailability to one or more retry queues. One or more retry queues may correspond to one or more general jobs and/or one or more specific jobs. For example, in response to detecting an unavailability of a general job, error component112may add the data corresponding to the detected unavailability to a common retry queue and/or a retry que corresponding to the general job. In response to detecting an unavailability of a specific job, error component112may add the data corresponding to the detected unavailability to a common retry queue and/or a retry que corresponding to the specific job. Data added to one or more retry queues may be processed by the corresponding general job/specific job after a time duration and/or after the availability of the general job/specific job has been detected.

FIGS. 5-7illustrate exemplary scenarios of micro-batch processing of data. Other scenarios of micro-batch processing of data are contemplated. InFIG. 5, data source(s)501may send data conveying information pertaining to operation of client computing platforms. For a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue A502), the data may be added to general queue A502. After the general time duration, the data added to general queue A502may be processed in accordance with general job A503. The processed data may be added to specific queue A504a, specific queue B504b, and/or specific queue C504cfor one or more specific time durations (e.g., a common specific time duration or specific time durations corresponding to specific queue A504a, specific queue B504b, specific queue C504c) based on the client computing platform to which the data pertains. After the one or more specific time durations, the data added to specific queue A504a, specific queue B504b, and/or specific queue C504cmay be processed in accordance with specific job A505a, specific job B505b, and/or specific job C505c, respectively. The processed data may be added to general queue B506for a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue B506). After the general time duration, the data added to general queue B506may be processed in accordance with general job B507. Processing of data may contain other queues and/or jobs.

InFIG. 6, data source(s)501may send data conveying information pertaining to operation of client computing platforms. For one or more specific time durations (e.g., a common specific time duration or specific time durations corresponding to specific queue A504a, specific queue B504b, specific queue C504c), the data may be added to specific queue A504a, specific queue B504b, and/or specific queue C504cbased on the client computing platform to which the data pertains. After the one or more specific time durations, the data added to specific queue A504a, specific queue B504b, and/or specific queue C504cmay be processed in accordance with specific job A505a, specific job B505b, and/or specific job C505c, respectively. The processed data may be added to general queue A502for a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue A502). After the general time duration, the data added to general queue A502may be processed in accordance with general job A503. The processed data may be added to general queue B506for a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue B506). After the general time duration, the data added to general queue B506may be processed in accordance with general job B507. Processing of data may contain other queues and/or jobs.

InFIG. 7, data source(s)501may send data conveying information pertaining to operation of client computing platforms. For a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue A502), the data may be added to general queue A502based on the client computing platform to which the data pertains. For a specific time duration (e.g., a common specific time duration or specific time durations corresponding to specific queue A504a), the data may be added to specific queue A504abased on the client computing platform to which the data pertains. After the general time duration, the data added to general queue A502may be processed in accordance with general job A503. After the specific time duration, the data added to specific queue A504amay be processed in accordance with specific job A505a. The data processed in accordance with general job A503may be added to specific queue B504band/or specific queue C504cfor one or more specific time duration (e.g., a common specific time duration or specific time durations corresponding to specific queue B504band/or specific queue C504C) and the data may be processed in accordance with specific job B505band/or specific job C505C, respectively. The processed data may be added to general queue B506for a general time duration (e.g., a common general time duration or a general time duration corresponding to general queue B506). After the general time duration, the data added to general queue B506may be processed in accordance with general job B507. Processing of data may contain other queues and/or jobs.

Although server11, client computing platform(s)14, and external resource(s)15are shown to be connected to network16inFIG. 1, any communication medium/link may be used to facilitate interaction between any components of system10. One or more components of system10may communicate with each other through hard-wired communication, wireless communication, or both. One or more components of system10may communicate with each other through multiple networks and/or multiple wired/wireless communications. By way of non-limiting example, wireless communication may include one or more of radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other wireless communication. Other types of communications are contemplated by the present disclosure.

In some implementations, network16may include the Internet and/or may employ other communications technologies and/or protocols. By way of non-limiting example, network16may employ communication technologies including one or more of Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, and/or other communication technologies. By way of non-limiting example, network16may employ networking protocols including one or more of multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), User Datagram Protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), and/or other networking protocols.

Information exchanged over network16may be represented using formats including one or more of hypertext markup language (HTML), extensible markup language (XML), and/or other formats. One or more exchanges of information between components of system10may be encrypted using encryption technologies including one or more of secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), and/or other encryption technologies. In some implementations, one or more components of system10may use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above. It will be appreciated that this is not intended to be limiting and that the components of system10may be operatively linked via some other communication media.

External resource(s)15may include sources of information, hosts, and/or other entities outside of system10, external entities participating with system10, and/or other resources. In some implementations, some or all of the functionality attributed herein to system10may be provided by external resource(s)15.

Although processor12is shown inFIG. 1as a single entity, this is for illustrative purposes only. In some implementations, processor12may comprise a plurality of processing units. These processing units may be physically located within the same device, or processor12may represent processing functionality of a plurality of devices operating in coordination. Processor12may be configured to execute one or more components by software; hardware; firmware; some combination of software, hardware, and/or firmware; and/or other mechanisms for configuring processing capabilities on processor12.

It should be appreciated that although computer components are illustrated inFIG. 1as being co-located within a single processing unit, in implementations in which processor12comprises multiple processing units, one or more of computer program components may be located remotely from the other computer program components.

The electronic storage media of electronic storage13may be provided integrally (i.e., substantially non-removable) with one or more components of system10and/or removable storage that is connectable to one or more components of system10via, for example, a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage13may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EPROM, EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage13may be a separate component within system10, or electronic storage13may be provided integrally with one or more other components of system10(e.g., processor12). Although electronic storage13is shown inFIG. 1as a single entity, this is for illustrative purposes only. In some implementations, electronic storage13may comprise a plurality of storage units. These storage units may be physically located within the same device, or electronic storage13may represent storage functionality of a plurality of devices operating in coordination.

One or more features and/or functions attributed to server11may be provided at a client-end application that may be installed on client computing platform(s)14, and/or vice versa. In some implementations, one or more features and/or functions attributed to server11may be configured as a cloud-based application that a user may access via client computing platform(s)14. In some implementations, an application providing one or more features and/or functions attributed herein to server11may be configured to be part client-end and part cloud based. Further, it is noted that the one or more client computing platforms, servers, and/or other machines executing a client-based application and/or a cloud-based application may include one or more of one or more processors, electronic storage configured to store and/or execute program code corresponding to the processes described herein, and/or other components.

By way of non-limiting example, individual ones of one or more computing platforms122may include machine-readable instructions comprising the same or similar components as machine-readable instructions106of server102. Computing platforms122may be configured to execute locally the same or similar components of machine-readable instructions106

FIG. 2illustrates method200for micro-batch processing of data. The operations of method200presented below are intended to be illustrative. In some implementations, method200may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. In some implementations, two or more of the operations may occur substantially simultaneously.

Referring toFIG. 2and method200, at operation201, data conveying information pertaining to operation of client computing platforms may be received. In some implementations, operation201may be performed by a processor component the same as or similar to receive data component102(shown inFIG. 1and described herein).

At operation202, the data may be added to a general queue for a general time duration. In some implementations, operation202may be performed by a processor component the same as or similar to general queue component104(shown inFIG. 1and described herein).

At operation203, the data in the general queue may be processed in memory in accordance with a general job. In some implementations, operation203may be performed by a processor component the same as or similar to general job component106(shown inFIG. 1and described herein).

At operation204, the data may be added to one or more specific job queues for one or more specific time durations. In some implementations, operation204may be performed by a processor component the same as or similar to specific queue component108(shown inFIG. 1and described herein).

At operation205, the data in the one or more specific job queues may be processed in memory in accordance with one or more specific jobs. In some implementations, operation205may be performed by a processor component the same as or similar to specific job component110(shown inFIG. 1and described herein).

At operation206, an error in processing the data in accordance with the general job or the one or more specific jobs may be detected. In some implementations, operation206may be performed by a processor component the same as or similar to error component112(shown inFIG. 1and described herein).

At operation207, responsive to detection of the error, the data corresponding to the detected error may be added to a skip queue. In some implementations, operation207may be performed by a processor component the same as or similar to error component112(shown inFIG. 1and described herein).