Big data propagation agent framework

A data distribution computer system. The system comprises a data store, a non-transitory memory, a processor, and a data propagation application stored in the non-transitory memory. When executed by the processor, the application initiates execution of a plurality of data propagation agents in a restricted runtime environment and provides information to the agents identifying data that is available in the data store via a discovery application programming interface (API). The application further provides data from the data store to the agents via a data access API, where the data propagation application manages the scheduling of the agents and restricts the data in the data store that is propagated to the agents based on an access policy for each agent.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

BACKGROUND

Modern communication and computer systems make collection and mining of large volumes of data, which may sometimes be referred to as “big data,” practical. The big data may pertain to any of a variety of items such as a large number of customer accounts, a large number of financial transactions, a large number of students, a large number of healthcare patients, a large number of medical events, and the like. The big data may be analyzed or “mined” to extract valuable information. Working with big data, creates challenges for traditional computing methods and architectures.

SUMMARY

In an embodiment, a data distribution computer system is disclosed. The system comprises a data store, a non-transitory memory, a processor, and a data propagation application stored in the non-transitory memory. When executed by the processor, the data propagation application initiates execution of a plurality of data propagation agents in a restricted runtime environment, provides information to the agents identifying data that is available in the data store via a discovery application programming interface (API), and provides data from the data store to the agents via a data access API. The data propagation application manages the scheduling of the agents and restricts the data in the data store that is propagated to the agents based on an access policy for each agent.

In another embodiment, a method of propagating data from a data store is disclosed. The method comprises receiving by an agent framework application a first request to execute a first number of instances of a first data propagation agent in an execution sandbox, wherein the agent framework application executes on a computer system and, responsive to receiving the first request, instantiating a number of instances of the first data propagation agent to run in an agent framework sandbox by the agent framework application that is less than the first number, whereby a parallelism of the first data propagation agent is restricted by the agent framework application. The method further comprises, in response to receiving by the agent framework application a request from an instance of a second data propagation agent to access an item of data stored in a data store, transmitting an access request rejection message by the agent framework application to the instance of the second data propagation agent on the grounds that the second data propagation agent is restricted from access to the item of data. The method further comprises providing a data-as-a-service interface to data propagation agents for receiving data propagated from the data store by the agent framework application, where the data provided by the agent framework application comprises a marked text document and scheduling execution of instances of data propagation agents to manage and maintain availability of data-as-a-service access to the data store.

In yet another embodiment, a federated data distribution computer system is disclosed. The system comprises a data store, a non-transitory memory, a processor, a data propagation application stored in the non-transitory memory. When executed by the processor, the data propagation application instantiates a plurality of data propagation agents in a restricted runtime environment, restricting the execution parallelism of the agents based on maintaining availability of data-as-a-service access to the data store, registers instantiated agents in an agent registry, and schedules execution of the agents listed in the agent registry. The application further establishes a communication connection with the data store, promotes discovery of data contained in the data store by extending a discovery application programming interface (API) to the agents, and brokers requests from the agents to read data from the data store, wherein the agents have no communication connection with the data store, and the data propagation application restricts access of agents based on a per-agent access policy. The application further reads data from the data store via the communication connection and provides the data read from the data store as a marked text document.

DETAILED DESCRIPTION

A federated data distribution computer system is taught herein. A large data store is desirably accessed by each of a flexible number of independent and disparate consumers of data without blocking or delaying access of other data consumers to the data store. Further, the data access mechanism desirably need not be modified to support new data consumers; instead new data consumers conform themselves to the defined services and interfaces of the data access mechanism, and the data access mechanism enforces data access restrictions and policy on the data consumers.

For example, a data propagation application (e.g., a data access mechanism) registers data access agents in an agent registry. The data propagation application instantiates the agents based on desires of a data consumer and access restriction policies and schedules the agents for execution in a restricted runtime environment or a sandbox. The data propagation application may modulate a parallelism requested by the data consumers and/or the agents to provide equitable access of the data consumers and/or the agents to the data store. Equitable access of data consumers to the data store may be defined as providing an about equal throughput of data propagation per unit of time for each data consumer. Alternatively, equitable access may be defined as providing about equal throughput of data propagation per unit of time for data consumers belonging to a same service level. Other definitions of equitable access are also contemplated. For example, equitable access may be defined in terms of data request average queue depths or in terms of data request average pendency times of agents.

A number of agent instances per data consumer may be restricted by the data propagation application to try to achieve the equitable access among data consumers. The number of agent instances allowed by the data propagation application for data consumers may vary depending on a time varying load placed on the system by the data consumers. The data propagation application may command data consumers to terminate one or more of its agents as a load on the system grows or may itself terminate agents.

The data propagation application may send notifications to data consumers when spare system processing capacity is available and that additional instances of agents may be requested by data consumers. In an embodiment, the system may periodically reevaluate the system processing capacity and reduce or expand agent instances, for example every second, every ten seconds, every minute, or on some other periodic basis.

As an example, a data consumer may request10instances of its agent to be run in parallel, and the data propagation application may only instantiate5instances of that agent, in order to avoid stalling the access to data of other currently executing agents. Additionally the data propagation application schedules execution of the instances of the agents to provide equitable access by all data consumers and/or agents to the data store. The data propagation application may schedule instances of the agents to approach the goal of data consumers equitably sharing system resources and access to the data store as defined above, for example granting more or less execution duty cycles to agents of a data consumer based on the needs of other data consumers at that time. Alternatively, the data propagation application may schedule execution of the instances of agents to best fulfill service level agreements (SLAs) negotiated between the data consumers and the data propagation application on registration of the agents. The data propagation application may adapt the scheduling of agent instances on an on-going basis, for example every second, every ten seconds, every minute, or some other periodic basis.

The registration of agents is performed according to an application programming interface (API) extended by the data propagation application. The data consumer may identify how many parallel instances of the agent are desired to be instantiated in the API call that registers an agent. The data consumer and/or agent calls a discovery API extended by the data propagation application to identify types or classes of data available from the data store. The agent may then request data from one or more of the data classes via a data access API extended by the data propagation application. The data propagation application restricts the data types that an agent accesses based on an access policy defined for that specific agent and/or data consumer. The access policy of different agents may be different from each other, for example based on a service tier negotiated between the data propagation application and the data consumer that registers the agent. The data consumers may be associated with remote computer systems not controlled by the data propagation application. The data propagation application establishes data communication links with the data store and does not provide direct communication links between the agents and the data store. The data propagation application may be said to broker access of the agents and/or data consumers to the data store.

The federated data distribution computer system described in brief above can achieve a fair and balanced shared data use environment for loosely coupled, unaffiliated data consumers via their agents. The system can dynamically support the addition of new agents and/or data consumers without redesign, redevelopment, or retesting of the data propagation application and without substantive perturbation of the experience of more senior agents and/or data consumers. Further, the system can protect the integrity of the data encapsulated in the data store, for example by executing the agents in a controlled runtime environment or sandbox. As such, the federated data distribution computer system comprising the data propagation application taught by the present disclosure improves known computing machines and/or computer systems.

Turning now toFIG. 1, a federated data distribution system100is described. In an embodiment, the system100comprises a data propagation agent framework102, a data store104, a plurality of service delivery platforms (SDPs)106, and a network108. The data propagation agent framework102provides access to the data in the data store104to the SDPs106via data propagation agents112that execute in a runtime sandbox110within the agent framework102. The network108comprises one or more public networks, one or more private networks, or a combination thereof. The SDPs106need not be affiliated with each other, coordinate with each other, or be aware of each other.

In an embodiment, the data propagation agent framework102comprises an agent configuration manager116, a data discovery component118, a data access broker120, an agent scheduler122, an agent registry124, and the already mentioned runtime sandbox110. The collectivity of the computer artifacts110,116,118,120,122, and124may be said to constitute a data propagation application or an agent framework application. It is understood that in some embodiments the functionality ascribed to two or more of the computer artifacts110,116,118,120,122, and124may be combined in a single computer artifact. Additionally, it is understood that one or more of the computer artifacts110,116,118,120,122, and124may be provided by two or more separate components. Computer artifacts110,116,118,120,122, and124execute on one or more computer systems. Computer systems are described further hereinafter. The data propagation agent framework102and/or a data propagation application and/or an agent framework application may be said to support federated data distribution to the agents112.

The agents112may execute in the runtime sandbox110but are not formally components or constituents of the framework102. Rather the framework102is adopted to receive executable files or images associated with the agents112from calls to an API provided by the framework102(i.e., calls invoked by the SDPs106to register and/or execute the agents112) and execute the agents112in the runtime sandbox110. In some contexts, the data propagation agent framework102may be said to provide a data-as-a-service functionality to the SDPs106. In an embodiment, the SDPs106may provide their agents112to the data propagation agent framework102in a Java archive (JAR) file.

The SDP106may be referred to in some contexts as data consumers. The agents112retrieve data from the data store104via the framework102, optionally format the shema-less data provided to the agents112by the framework102, and sends the data to the corresponding SDP106. The agents112and/or the SDPs106may store the retrieved data consumer data stores114, and the SDPs106may process or mine the data by reading out of the consumer data stores114. Each SDP106may develop and register distinct and/or different agents112(i.e., different from the agents112developed by other, different SDPs106) with the framework102. A single SDP106may develop and register two or more distinct and/or different agents112with the framework102, for example different agents112to access different data in the data store104of interest to the same SDP106.

An SDP106may access the data store104, via its data propagation agent112executing in the runtime sandbox110of the data propagation agent framework102, to perform data mining functions. For example, the SDP106may analyze data regarding purchasing and/or content consumption of users and/or subscribers of a wireless communication system to evaluate the desirability of paying to present a specific advertisement or other information content in a display window of a wireless communication device of a user (e.g., targeted advertising). A user with a history of having browsed on-line wilderness trail descriptions may be more disposed to act on an advertisement for hiking boots than another user with no such history. Notwithstanding this specific use case, it is understood that the advantages of the federated data distribution system100described herein are not limited to a targeted advertisement application. For example, the data store104may comprise a large database of health records, and the SDPs106may mine the data store104to perform independent epidemiological investigations.

The data discovery component118provides an API that provides SDPs106and/or the agents112a controlled, structured mechanism for determining the kind of data that is accessible to them from the data store104. The data discovery component118may provide information about the types of data available from the data store104. The data discovery component118may provide an API that supports a request message to discover data collection types, a request message to discover data collections, a request message to discover data key types, a request message to discover data keys, and other request messages. Different SDPs106may be allowed to access different data and/or categories of data, and hence the same discovery calls executed by different SDPs106may return different results to different SDPs106. For example, a first SDP106may be allowed to access a broader or more extensive selection of data in the data store104than a second SDP106, for example when the first SDP106has negotiated with the framework102to enjoy a higher class of service than did the second SDP106.

The agent registry124provides an API that allows SDPs106to register and to provide an execution handle for one or more data propagation agents112. The API may provide an argument for defining how many parallel instances of its agent112the SDP106wishes to be executed concurrently as well as other execution mode preferences. For example, the API may allow an SDP106to select how the framework102should handle the case when the requested number of parallel instances of the agent112cannot be scheduled to execute concurrently: for example, to wait to schedule running any instances of a particular agent112until all the requested parallel instances of the particular agent112can be run concurrently or to schedule fewer than requested but as many instances of the particular agent112as possible. The agent configuration manager116may act on the registration information contained in the agent registry124to instantiate agents112and to configure the services and/or privileges available to each agent112in the runtime sandbox110. In part, the agent configuration manager116may configure data store104access policies for each agent112and/or execution privileges for each agent112.

The agent scheduler122schedules data propagation agents112to run based on policy and/or service level agreements to achieve equitable access to the data store104. The agent scheduler122may access a list of agent instances maintained by the agent registry124to know when and what agents112to execute.

The data access broker120establishes communication links with the data store104via the network108. When a data propagation agent112executes a data access API extended by the data access broker120, the data access broker120requests the identified data from the data store104, subject to any access constraints defined by policy or otherwise for the subject agent112. The data returned to the agents112may be said to be schema-less data, in the sense that no specific structure may be imposed by the agents112on the format of data returned to the agents112. Said in other words, the agents112are not enabled to define the structure of the data that is returned to them. The data may be returned to the agents112as a marked text document, for example as a JavaScript object notation (JSON) document. Because the agents112do not establish (are not allowed by the framework102to establish) communication links to the data store104, the risk that the agents112might block access of others to the data store104, corrupt data in the data store104, or access data in the data store104to which they are not entitled is avoided. The agents112are restricted from direct access to the APIs of the data store104, for example the agents112are restricted from direct access to representational state transfer (REST) APIs of the data store104. The restriction of the agents112from access to the APIs of the data store104may be referred to in some contexts as decoupling the agents from the data store104.

Turning now toFIG. 2, a method200is described. At block202, an agent framework application receives a first request to execute a first number of instances of a first data propagation agent in an execution sandbox, wherein the agent framework application executes on a computer system. For example, the agent configuration manager116, the agent registry124, or the agent scheduler122receives the first request. At block204, responsive to receiving the first request, the agent framework application instantiates a number of instances of the first data propagation agent to run in an agent framework sandbox that is less than the first number, whereby a parallelism of the first data propagation agent is restricted by the agent framework application. For example, the agent configuration manager116or the agent scheduler122instantiates the instances of the first data propagation agent. At block206, in response to receiving by the agent framework application a request from an instance of a second data propagation agent to access an item of data stored in a data store, the agent framework application transmits an access request rejection message to the instance of the second data propagation agent on the grounds that the second data propagation agent is restricted from access to the item of data. For example, the agent configuration manager116, the agent scheduler122, or the agent registry124transmits the rejection message.

At block208, the agent framework application provides a data-as-a-service interface to data propagation agents for receiving data propagated from the data store, where the data provided by the agent framework application comprises a marked text document. For example, the data discovery component118and/or the data access broker120provide the data-as-a-service interface to the data propagation agents. At block210, the agent framework application schedules execution of instances of data propagation agents to manage and maintain availability of data-as-a-service access to the data store. For example, the agent scheduler122schedules execution of instances of data propagation agents.