Patent ID: 8688850

Claim:
A method for inter-site communication of data streams in a cooperative data stream processing system, the method comprising: identifying a plurality of distributed sites, each site comprising one or more nodes and independently hosting on the nodes jobs derived from inquiries to process continuous dynamic streams of data; and using a processor located on one or more of the plurality of distributed sites to perform the steps of: breaking each job into a plurality of processing elements, each processing element comprising a portion of a data stream processing application and the processing elements for a given job utilizing a data stream flow comprising the continuous dynamic streams of data that is communicated through the processing elements of that given job to mine the continuous dynamic streams of data; grouping the processing elements into subjobs; executing the subjobs on nodes comprising computing systems within two or more of the plurality of distributed sites; identifying tunneling processing elements in a given pair of the distributed sites containing nodes on which processing elements derived from the given job are executing, each tunnel processing element comprising an executable application compatible with the processing elements derived from the given job and executing on its distributed site to maintain the data stream flow communicated through those processing elements; establishing at least two tunnels between the given pair of nodes, each node located on one of the distributed sites by establishing communication between the tunneling processing elements located on each node in the given pair of nodes across a network connection between the given pair of sites to exchange the data stream flow between the tunneling processing elements and the processing elements, the tunnel associated with the given job by creating a distributed plan comprising definitions of required tunneling flows comprising: an identification of an originating site in the pair of sites and a destination site in the pair of sites; an identification of originating site application processing elements to be involved in data stream flow transfer and an identification of destination site application processing elements to be involved in data stream flow transfer; an identification of data stream flows to be exchanged through the tunnel; security and privacy labels associated with each data stream flow to be exchanged; type mappings associated with each data stream flow to be exchanged; and a priority associated with each data stream flow to be exchanged; using the tunnels to route the data stream flow between the given pair of sites and through processing elements of the given job as required to process the given job by mining the continuous dynamic streams of data; and transforming the data stream flows to compensate for heterogeneity between the pair of sites, the heterogeneity between the pair of sites comprising security, data type and data semantic differences between the pair of sites.