Identification and visualization of data set relationships in online library systems

An apparatus includes a processor to: receive multiple normalized metadata portions based on metadata portions originating from vendor devices storing data sets of a distributed online library system; compare the multiple pieces of information between pairs of normalized metadata portions to identify at least one pair of identical portions of data; analyze the pieces of information of normalized metadata portions corresponding to an identified pair of identical portions of data to determine if there is a dependency relationship between each portion of data of the pair and another identical portion of data stored within another device; and in response to there being such a pair of dependency relationships, generate a visualization that includes a combination of graphical elements depicting the pair of dependency relationships, and transmit the visualization to the client device to enable a visual presentation of the visualization.

BACKGROUND

Curators of online library systems created for such entities as universities, businesses and/or government agencies, typically seek to provide their users (e.g., personnel associated with such entities) with access to a wide variety of pieces of data from a wide variety of sources to enable their users to obtain a very complete view of whatever subject they may research, including information from different sources to enable at least some degree of fact checking, and/or including opposing opinions to provide wider perspectives. In so doing, such curators often enter into licensing agreements with multiple providers of data in an effort to broaden the sources of data that become part of their online library system. However, curators may subsequently find themselves caught by surprise by the introduction of various functional problems into their library systems by such efforts. More specifically, instances in which seemingly broadly distributed data sets turn out to be duplicative or derivative versions of data that all stem from a single source of data may be unknowingly introduced, thereby creating functional bottlenecks that may impair the operation of the overall system.

It has been repeatedly said that the growing prevalence of the Internet has helped to “democratize” information. More specifically, any person with access to the Internet is able to put information out onto the Internet, and that information will become as easily accessible to anyone around the world as information also put out there by historians, scientists, government officials, news professionals, etc. Unfortunately, this same growing prevalence of the Internet has also given greater opportunities for those committing plagiarism to more easily obtain information from numerous sources, and to put out that same information under their own name, while giving no credit to those sources. Also, the vast number of scholastic, corporate, religious and/or governmental entities, as well as the vast number of individual persons, that now regularly put out information onto the Internet has created a situation in which those seeking information on almost any subject are often overwhelmed by the number of different apparent sources of information on that subject.

Still further, many individuals who put out pieces of information on the Internet are frequently not trained in, and/or are uninterested in, best practices in generating what they put out, including making use of original and/or contemporary pieces of information to the degree possible, and/or taking care to carefully delineate their presentations of fact from their opinions. As a result, the Internet has become a vast “free for all” environment in which there are many pieces of information that may not be properly attributed to sources, may present opinions as fact, and/or may be of highly questionable accuracy.

In entering into licensing agreements with various providers, curators of online library systems are often relying on those providers to employ some degree of curation, themselves, to separate out reliable information from questionable information, and to make original and/or contemporary pieces of information more readily available. In essence, curators of online library systems are seeking to make use of additional curation services performed by those providers such that the overall aggregate of the information that makes up such online library systems can be relied upon as meeting at least some minimum level of quality of accuracy and completeness. Many of such providers may, themselves, be publishers of new original pieces of information and/or new compilations of information assembled from other particular sources, and may have acquired a reputation for the accuracy and/or reliability of what they publish. Such reputations may be among the factors relied upon by curators of online library systems in selecting their providers. Again, to provide a breadth of sources, curators of online library systems may enter into licensing agreements with multiple competing providers of information. At the very least, they may seek to ensure that their users have access to an array of source that are not limited by the choices of sources made by a single provider.

Unfortunately for such curators, there is often no way to test or evaluate the quality and/or variety of information that they have arranged to be provided to their users from the providers through such licensing. Thus, such curators often have little ability to verify whether they are succeeding in their efforts to provide a sufficient variety of source of information to their users. As a result, such curators may be caused to remain oblivious to instances in which multiple providers that they have entered into licenses with are actually providing the very same information on at least a subset of subjects from the very same source. Where such a situation exists, aside from concerns that this may result in users having access to an all too limited selection of information concerning a particular subject, there is also the concern that such a single source of information may become the source of an information access bottleneck. More specifically, where each of multiple providers of information direct users to the same source of a particular piece of information, the storage, processing and/or network bandwidth capabilities of that one source may be insufficient to support accesses and queries made by so many users to that particular piece of information, thereby leading to instances where that particular piece of information may not be reliably accessible, and/or may become accessible only after a considerable period of delay, thereby impairing the overall functionality of the online library system.

In some situations, such of a situation of there being a single piece of information on a subject that becomes so widely sought after may be entirely unavoidable where the number of original sources of information on that particular subject is highly limited. By way of example, on such topics as the monitoring of geological activity along earthquake faults and/or in the vicinity of volcanoes, it may be that the US Geological Survey (USGS) of the US Government is the only source of various instrument measurements in the field. Thus, a report put out by the USGS on the subject of particular activity associated with a particular fault line or volcano may be the only source of that information, and therefore, may become the one piece of information that is included by each of multiple providers that a curator may license information from. Having knowledge of the fact there being such unavoidable bottlenecks may enable curators of online library systems to take action to improve reliability of access and/or reduce occurrences of delay in access. However, such curators are unable to take such action if they are not made aware of the existence of such situations.

Curators may also be caused to remain oblivious to instances in which one provider has made available a data set concerning a particular subject that was created as an aggregation of other data sets concerning different aspects of the subject that are individually available through one or more other providers. There may also be situations in which information on a particular subject from two different sources (and which may be available through two different providers) cover numerous different aspects of the particular subject, but may both have one particular aspect of the subject in which there is a high degree of overlap therebetween.

SUMMARY

This summary is not intended to identify only key or essential features of the described subject matter, nor is it intended to be used in isolation to determine the scope of the described subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

An apparatus includes a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations including receive, at a broker device and via a network, multiple normalized metadata portions based on a set of metadata portions originating from a set of vendor devices of a distributed online library system, wherein: the set of vendor devices stores a set of data sets of the distributed online library system; each normalized metadata portion of the multiple normalized metadata portions comprises multiple pieces of information descriptive of contents of a corresponding portion of data of a set of portions of data retrieved from the set of data sets; each portion of data in the set of portions of data is identified by a vendor device of the set of vendor devices as relevant to a subject of interest that is specified in a query originating from a client device of the distributed online library system; and the query comprises a request for a visualization of relationships among the portions of data in the set of portions of data. The processor is also caused to, for each pair of normalized metadata portions of the multiple metadata portions, compare the multiple pieces of information therebetween to identify at least one pair of identical portions of data within the set of portions of data; and in response to the identification of a pair of identical portions of data within the set of portions of data, analyze the pieces of information of the corresponding pair of normalized metadata portions to determine if there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices. The processor is further caused to, in response to a determination that there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices, perform operations including generate a visualization including: a first graphical element that represents a first portion of data of the pair of identical portions of data; a second graphical element that represents a second portion of data of the pair of identical portions of data; a third graphical element that represents the other identical portion of data stored within the other device; and a pair of graphical elements that each represent the dependency relationship between the other identical portion of data and each of the first portion of data and the second portion of data. The processor is still further caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

The multiple pieces of information of each normalized metadata portion comprises a data portion identifier of the corresponding portion of data that comprises at least one of: a file name of the corresponding portion of data stored as a file within a vendor device of the set of vendor devices; a title of the corresponding portion of data; an author of the corresponding portion of data; a publisher of the corresponding portion of data; an Internet Protocol (IP) address of the vendor device of the set of vendor devices within which the corresponding portion of data is stored; or a Universal Resource Locator (URL) of the vendor device of the set of vendor devices within which the corresponding portion of data is stored. To identify at least one pair of identical portions of data within the set of portions of data, the processor may be caused to compare the data portion identifiers of each pair of normalized metadata portions of the multiple normalized metadata portions.

At least a subset of the set of portions of data may include a multi-dimensional array data structure; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include multiple metrics that are each descriptive of contents of the corresponding portion of data. Each metric of the multiple metrics may include at least one of: a quantity of dimensions; a size of each dimension; a data type of stored data values; a bit width of stored data values; a specified data value employed as a null value; or a measure of sparseness of data values other than the null value. To identify at least one pair of identical portions of data within the set of portions of data, the processor may be caused to: identify a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and compare the multiple metrics of each pair of normalized metadata portions of the subset of normalized metadata portions.

In response to a determination that a pair of portions of data of the set of portions of data are not identical, the processor may be caused to perform operations including: compare indications of size of each portion of data of the pair of portions of data in the corresponding pair of normalized metadata portions to determine which portion of data is smaller; analyze the multiple pieces of information of the normalized metadata portion that corresponds to the smaller portion of data of the pair of portions of data to identify a type of data structure present within the smaller portion of data; analyze the multiple pieces of information of the normalized metadata portion that corresponds to the larger portion of data of the pair of portions of data to determine if a data structure of the same type is present within the larger portion of data of the pair of portions of data; in response to a determination that a data structure of the same type is present within the larger portion, compare pieces of information of the multiple pieces of information of each normalized metadata portion that specify multiple metrics of the data structures of the smaller portion of data and of the larger portion of data to determine whether the data structures of the smaller portion of data and the larger portion of data have identical metrics; and in response to a determination that the data structures of the smaller portion of data and the larger portion of data have identical metrics, compare at least a subset of data values of the data structure of the smaller portion of data to corresponding data values of the data structure of the larger portion of data to determine if the data structures of the smaller portion of data and the larger portion of data are identical and that there is an inclusion relationship in which the smaller portion of data is included in the larger portion of data. The processor may also be caused to, in response to a determination that there is an inclusion relationship between the smaller portion of data and the larger portion of data, perform operations including generate a visualization including: a first graphical element that represents the smaller portion of data; a second graphical element that represents the larger portion of data; and a third graphical element that represents the inclusion relationship in which the smaller portion of data is included in the larger portion of data. The processor may be further caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

Each portion of data in at least a subset of the set of portions of data may include a document; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include at least one document identifier of a document, and at least one citation to another document that is present within the document. Each document identifier of the at least one document identifier may include at least one of: a title; an author; an identifier of a publisher; or a date of publication. To identify a citation relationship between at least one pair of portions of data within the subset of portions of data in which a document of one portion of data includes a citation to a document of another portion of data, the processor may be caused to: identify a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and compare document identifiers to citations between each pair of normalized metadata portions of the subset of the multiple normalized metadata portions. In response to an identification of a citation relationship between a pair of portions of data within the subset of portions of data, the processor may be caused to perform operations including generate a visualization including: a first graphical element that represents a first portion of data of the pair of portions of data in which a document includes the citation; a second graphical element that represents a second portion of data of the pair of portions of data that includes the document that is cited by the citation; and a third graphical element that represents the citation relationship. The processor may also be caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

The processor may be caused to perform operations including: receive the query from the client device via the network; translate the query from a first communications protocol in which information in the query is expressed in a first manner, and to a second communications protocol in which the information in the query is expressed in a second manner in a translated query; and relay the translated query to the set of vendor devices via the network to enable the set of vendor devices to perform a search of metadata corresponding to multiple data sets to identify the set of portions of data as relevant to the subject of interest specified in the query and in the translated query.

Prior to translating the query, the processor may be caused to perform operations including: analyze access credentials included with the query to determine whether the client device or a user of the client device is authorized to enter the query into the distributed online library system; and in response to a determination that the entry of the query is not authorized, transmit an indication of lack of authorization to the client device and refrain from translating the query and relaying the translated query.

The processor may be caused to perform operations including: receive the set of metadata portions from the set of vendor devices via the network; and translate each metadata portion of the set of metadata portions from a first communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a first manner, and to a second communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a second manner in a corresponding normalized metadata portion of the multiple normalized metadata portions.

As part of the translation of each metadata portion to the corresponding normalized metadata portion, the processor may be caused to perform operations including analyze the multiple pieces of information of each metadata portion to determine whether there is a piece of information of a predetermined set of pieces of information to be included in the corresponding normalized metadata portion that is not present among the multiple pieces of information of the metadata portion, and in response to a determination that there is a piece of information of the predetermined set of pieces of information that is not present among the multiple pieces of information of a metadata portion, perform operations comprising: generate a further query to obtain the piece of information that is not present among the multiple pieces of information of the metadata portion from the corresponding portion of data; transmit the further query to the vendor device of the set of vendor devices from which the metadata portion was received; receive further data from the vendor device that comprises the piece of information that is not present among the multiple pieces of information of the metadata portion; and augment the corresponding normalized metadata portion with the piece of information.

The processor may be caused to perform operations including analyze the query to determine whether the query comprises a request for the provision of the set of portions of data that correspond to the multiple normalized metadata portions to be provide along with a visualization of relationships among the set of portions of data, and in response to a determination that the query does comprise a request for the provision of the set of portions of data, perform operations including: receive the set of portions of data from the set of vendor devices; and relay the set of portions of data to the client device along with the transmission of the visualization to the client device.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium includes instructions operable to cause a processor to perform operations including receive, at a broker device and via a network, multiple normalized metadata portions based on a set of metadata portions originating from a set of vendor devices of a distributed online library system, wherein: the set of vendor devices stores a set of data sets of the distributed online library system; each normalized metadata portion of the multiple normalized metadata portions comprises multiple pieces of information descriptive of contents of a corresponding portion of data of a set of portions of data retrieved from the set of data sets; each portion of data in the set of portions of data is identified by a vendor device of the set of vendor devices as relevant to a subject of interest that is specified in a query originating from a client device of the distributed online library system; and the query comprises a request for a visualization of relationships among the portions of data in the set of portions of data. The processor is also caused to, for each pair of normalized metadata portions of the multiple metadata portions, compare the multiple pieces of information therebetween to identify at least one pair of identical portions of data within the set of portions of data; and in response to the identification of a pair of identical portions of data within the set of portions of data, analyze the pieces of information of the corresponding pair of normalized metadata portions to determine if there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices. The processor is further caused to, in response to a determination that there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices, perform operations including generate a visualization including: a first graphical element that represents a first portion of data of the pair of identical portions of data; a second graphical element that represents a second portion of data of the pair of identical portions of data; a third graphical element that represents the other identical portion of data stored within the other device; and a pair of graphical elements that each represent the dependency relationship between the other identical portion of data and each of the first portion of data and the second portion of data. The processor is still further caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

The multiple pieces of information of each normalized metadata portion may include a data portion identifier of the corresponding portion of data that includes at least one of: a file name of the corresponding portion of data stored as a file within a vendor device of the set of vendor devices; a title of the corresponding portion of data; an author of the corresponding portion of data; a publisher of the corresponding portion of data; an Internet Protocol (IP) address of the vendor device of the set of vendor devices within which the corresponding portion of data is stored; or a Universal Resource Locator (URL) of the vendor device of the set of vendor devices within which the corresponding portion of data is stored. To identify at least one pair of identical portions of data within the set of portions of data, the processor may be caused to compare the data portion identifiers of each pair of normalized metadata portions of the multiple normalized metadata portions.

At least a subset of the set of portions of data may include a multi-dimensional array data structure; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include multiple metrics that are each descriptive of contents of the corresponding portion of data. Each metric of the multiple metrics may include at least one of: a quantity of dimensions; a size of each dimension; a data type of stored data values; a bit width of stored data values; a specified data value employed as a null value; or a measure of sparseness of data values other than the null value. To identify at least one pair of identical portions of data within the set of portions of data, the processor is caused to: identify a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and compare the multiple metrics of each pair of normalized metadata portions of the subset of normalized metadata portions.

In response to a determination that a pair of portions of data of the set of portions of data are not identical, the processor may be caused to perform operations including: compare indications of size of each portion of data of the pair of portions of data in the corresponding pair of normalized metadata portions to determine which portion of data is smaller; analyze the multiple pieces of information of the normalized metadata portion that corresponds to the smaller portion of data of the pair of portions of data to identify a type of data structure present within the smaller portion of data; analyze the multiple pieces of information of the normalized metadata portion that corresponds to the larger portion of data of the pair of portions of data to determine if a data structure of the same type is present within the larger portion of data of the pair of portions of data; in response to a determination that a data structure of the same type is present within the larger portion, compare pieces of information of the multiple pieces of information of each normalized metadata portion that specify multiple metrics of the data structures of the smaller portion of data and of the larger portion of data to determine whether the data structures of the smaller portion of data and the larger portion of data have identical metrics; and in response to a determination that the data structures of the smaller portion of data and the larger portion of data have identical metrics, compare at least a subset of data values of the data structure of the smaller portion of data to corresponding data values of the data structure of the larger portion of data to determine if the data structures of the smaller portion of data and the larger portion of data are identical and that there is an inclusion relationship in which the smaller portion of data is included in the larger portion of data. The processor may also be caused to, in response to a determination that there is an inclusion relationship between the smaller portion of data and the larger portion of data, perform operations including generate a visualization including: a first graphical element that represents the smaller portion of data; a second graphical element that represents the larger portion of data; and a third graphical element that represents the inclusion relationship in which the smaller portion of data is included in the larger portion of data. The processor may be further caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

Each portion of data in at least a subset of the set of portions of data may include a document; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include at least one document identifier of a document, and at least one citation to another document that is present within the document. Each document identifier of the at least one document identifier may include at least one of: a title; an author; an identifier of a publisher; or a date of publication. To identify a citation relationship between at least one pair of portions of data within the subset of portions of data in which a document of one portion of data includes a citation to a document of another portion of data, the processor may be caused to: identify a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and compare document identifiers to citations between each pair of normalized metadata portions of the subset of the multiple normalized metadata portions. In response to an identification of a citation relationship between a pair of portions of data within the subset of portions of data, the processor may be caused to perform operations including generate a visualization including: a first graphical element that represents a first portion of data of the pair of portions of data in which a document includes the citation; a second graphical element that represents a second portion of data of the pair of portions of data that includes the document that is cited by the citation; and a third graphical element that represents the citation relationship. The processor may also be caused to transmit the visualization to the client device to enable a visual presentation of the visualization.

The processor may be caused to perform operations including: receive the query from the client device via the network; translate the query from a first communications protocol in which information in the query is expressed in a first manner, and to a second communications protocol in which the information in the query is expressed in a second manner in a translated query; and relay the translated query to the set of vendor devices via the network to enable the set of vendor devices to perform a search of metadata corresponding to multiple data sets to identify the set of portions of data as relevant to the subject of interest specified in the query and in the translated query.

Prior to translating the query, the processor may be caused to perform operations including: analyze access credentials included with the query to determine whether the client device or a user of the client device is authorized to enter the query into the distributed online library system; and in response to a determination that the entry of the query is not authorized, transmit an indication of lack of authorization to the client device and refrain from translating the query and relaying the translated query.

The processor may be caused to perform operations including: receive the set of metadata portions from the set of vendor devices via the network; and translate each metadata portion of the set of metadata portions from a first communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a first manner, and to a second communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a second manner in a corresponding normalized metadata portion of the multiple normalized metadata portions.

As part of the translation of each metadata portion to the corresponding normalized metadata portion, the processor may be caused to perform operations including analyze the multiple pieces of information of each metadata portion to determine whether there is a piece of information of a predetermined set of pieces of information to be included in the corresponding normalized metadata portion that is not present among the multiple pieces of information of the metadata portion, and in response to a determination that there is a piece of information of the predetermined set of pieces of information that is not present among the multiple pieces of information of a metadata portion, perform operations including: generate a further query to obtain the piece of information that is not present among the multiple pieces of information of the metadata portion from the corresponding portion of data; transmit the further query to the vendor device of the set of vendor devices from which the metadata portion was received; receive further data from the vendor device that comprises the piece of information that is not present among the multiple pieces of information of the metadata portion; and augment the corresponding normalized metadata portion with the piece of information.

The processor may be caused to perform operations including analyze the query to determine whether the query comprises a request for the provision of the set of portions of data that correspond to the multiple normalized metadata portions to be provide along with a visualization of relationships among the set of portions of data, and in response to a determination that the query does comprise a request for the provision of the set of portions of data, perform operations including: receive the set of portions of data from the set of vendor devices; and relay the set of portions of data to the client device along with the transmission of the visualization to the client device.

A computer-implemented method includes receiving, by a processor at a broker device, and via a network, multiple normalized metadata portions based on a set of metadata portions originating from a set of vendor devices of a distributed online library system, wherein: the set of vendor devices stores a set of data sets of the distributed online library system; each normalized metadata portion of the multiple normalized metadata portions comprises multiple pieces of information descriptive of contents of a corresponding portion of data of a set of portions of data retrieved from the set of data sets; each portion of data in the set of portions of data is identified by a vendor device of the set of vendor devices as relevant to a subject of interest that is specified in a query originating from a client device of the distributed online library system; and the query comprises a request for a visualization of relationships among the portions of data in the set of portions of data. The method also includes: for each pair of normalized metadata portions of the multiple metadata portions, comparing, by the processor, the multiple pieces of information therebetween to identify at least one pair of identical portions of data within the set of portions of data; and in response to the identification of a pair of identical portions of data within the set of portions of data, analyzing, by the processor, the pieces of information of the corresponding pair of normalized metadata portions to determine if there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices. The method further includes, in response to a determination that there is a dependency relationship between each portion of data of the pair of identical portions of data and another identical portion of data stored within another device that is not among the set of vendor devices, performing, by the processor, operations including generating a visualization including: a first graphical element that represents a first portion of data of the pair of identical portions of data; a second graphical element that represents a second portion of data of the pair of identical portions of data; a third graphical element that represents the other identical portion of data stored within the other device; and a pair of graphical elements that each represent the dependency relationship between the other identical portion of data and each of the first portion of data and the second portion of data. The method still further includes transmitting, by the processor and via the network, the visualization to the client device to enable a visual presentation of the visualization.

The multiple pieces of information of each normalized metadata portion may include a data portion identifier of the corresponding portion of data that comprises at least one of: a file name of the corresponding portion of data stored as a file within a vendor device of the set of vendor devices; a title of the corresponding portion of data; an author of the corresponding portion of data; a publisher of the corresponding portion of data; an Internet Protocol (IP) address of the vendor device of the set of vendor devices within which the corresponding portion of data is stored; or a Universal Resource Locator (URL) of the vendor device of the set of vendor devices within which the corresponding portion of data is stored. To identify at least one pair of identical portions of data within the set of portions of data, the method may include comparing, by the processor, the data portion identifiers of each pair of normalized metadata portions of the multiple normalized metadata portions.

At least a subset of the set of portions of data may include a multi-dimensional array data structure; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include multiple metrics that are each descriptive of contents of the corresponding portion of data. Each metric of the multiple metrics may include at least one of: a quantity of dimensions; a size of each dimension; a data type of stored data values; a bit width of stored data values; a specified data value employed as a null value; or a measure of sparseness of data values other than the null value. To identify at least one pair of identical portions of data within the set of portions of data, the method may include: identifying, by the processor, a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and comparing, by the processor, the multiple metrics of each pair of normalized metadata portions of the subset of normalized metadata portions.

The computer-implemented method may include, in response to a determination that a pair of portions of data of the set of portions of data are not identical, performing operations including: comparing, by the processor, indications of size of each portion of data of the pair of portions of data in the corresponding pair of normalized metadata portions to determine which portion of data is smaller; analyzing, by the processor, the multiple pieces of information of the normalized metadata portion that corresponds to the smaller portion of data of the pair of portions of data to identify a type of data structure present within the smaller portion of data; analyzing, by the processor, the multiple pieces of information of the normalized metadata portion that corresponds to the larger portion of data of the pair of portions of data to determine if a data structure of the same type is present within the larger portion of data of the pair of portions of data; in response to a determination that a data structure of the same type is present within the larger portion, comparing, by the processor, pieces of information of the multiple pieces of information of each normalized metadata portion that specify multiple metrics of the data structures of the smaller portion of data and of the larger portion of data to determine whether the data structures of the smaller portion of data and the larger portion of data have identical metrics; and in response to a determination that the data structures of the smaller portion of data and the larger portion of data have identical metrics, comparing, by the processor, at least a subset of data values of the data structure of the smaller portion of data to corresponding data values of the data structure of the larger portion of data to determine if the data structures of the smaller portion of data and the larger portion of data are identical and that there is an inclusion relationship in which the smaller portion of data is included in the larger portion of data. The method may also include, in response to a determination that there is an inclusion relationship between the smaller portion of data and the larger portion of data, performing operations including generating, by the processor, a visualization including: a first graphical element that represents the smaller portion of data; a second graphical element that represents the larger portion of data; and a third graphical element that represents the inclusion relationship in which the smaller portion of data is included in the larger portion of data. The method may further include transmitting, by the processor and via the network, the visualization to the client device to enable a visual presentation of the visualization.

Each portion of data in at least a subset of the set of portions of data may include a document; and the multiple pieces of information of each normalized metadata portion that corresponds to a portion of data of the subset of portions of data may include at least one document identifier of a document, and at least one citation to another document that is present within the document. Each document identifier of the at least one document identifier may include at least one of: a title; an author; an identifier of a publisher; or a date of publication. To identify a citation relationship between at least one pair of portions of data within the subset of portions of data in which a document of one portion of data includes a citation to a document of another portion of data, the method may include: identifying a subset of the multiple normalized metadata portions that corresponds to the subset of data portions; and comparing document identifiers to citations between each pair of normalized metadata portions of the subset of the multiple normalized metadata portions. In response to an identification of a citation relationship between a pair of portions of data within the subset of portions of data, the method may include performing operations including generating, by the processor, a visualization including: a first graphical element that represents a first portion of data of the pair of portions of data in which a document includes the citation; a second graphical element that represents a second portion of data of the pair of portions of data that includes the document that is cited by the citation; and a third graphical element that represents the citation relationship. The method may also include transmitting, by the processor and via the network, the visualization to the client device to enable a visual presentation of the visualization.

The computer-implemented method may include: receiving, by the processor, the query from the client device via the network; translating, by the processor, the query from a first communications protocol in which information in the query is expressed in a first manner, and to a second communications protocol in which the information in the query is expressed in a second manner in a translated query; and relaying, by the processor, the translated query to the set of vendor devices via the network to enable the set of vendor devices to perform a search of metadata corresponding to multiple data sets to identify the set of portions of data as relevant to the subject of interest specified in the query and in the translated query.

The computer-implemented method may include, prior to translating the query, performing operations including: analyzing, by the processor, access credentials included with the query to determine whether the client device or a user of the client device is authorized to enter the query into the distributed online library system; and in response to a determination that the entry of the query is not authorized, transmitting, by the processor, an indication of lack of authorization to the client device and refrain from translating the query and relaying the translated query.

The computer-implemented method may include: receiving, by the processor, the set of metadata portions from the set of vendor devices via the network; and translating, by the processor, each metadata portion of the set of metadata portions from a first communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a first manner, and to a second communications protocol in which the multiple pieces of information about the corresponding portion of data is expressed in a second manner in a corresponding normalized metadata portion of the multiple normalized metadata portions.

As part of the translation of each metadata portion to the corresponding normalized metadata portion, the method may include performing operations including analyzing, by the processor, the multiple pieces of information of each metadata portion to determine whether there is a piece of information of a predetermined set of pieces of information to be included in the corresponding normalized metadata portion that is not present among the multiple pieces of information of the metadata portion. The method may also include, in response to a determination that there is a piece of information of the predetermined set of pieces of information that is not present among the multiple pieces of information of a metadata portion, performing operations including: generating, by the processor, a further query to obtain the piece of information that is not present among the multiple pieces of information of the metadata portion from the corresponding portion of data; transmitting, by the processor, the further query to the vendor device of the set of vendor devices from which the metadata portion was received; receiving, by the processor, further data from the vendor device that comprises the piece of information that is not present among the multiple pieces of information of the metadata portion; and augmenting, by the processor, the corresponding normalized metadata portion with the piece of information.

The computer-implemented method may include analyzing, by the processor, the query to determine whether the query comprises a request for the provision of the set of portions of data that correspond to the multiple normalized metadata portions to be provide along with a visualization of relationships among the set of portions of data. The method may also include, in response to a determination that the query does comprise a request for the provision of the set of portions of data, performing operations including: receiving, by the processor, the set of portions of data from the set of vendor devices; and relaying, by the processor, the set of portions of data to the client device along with the transmission of the visualization to the client device.

DETAILED DESCRIPTION

Various embodiments described herein are generally directed to the discovery and visualization of relationships among multiple portions of data (e.g., multiple data sets and/or portions thereof) concerning a selected subject that are made available within a distributed online library system. More precisely, in a distributed online library system, a broker device may serve as a gateway under the control of a curator of an online library to numerous sources of data that may be made accessible through one or more providers devices thereof under terms of agreement(s) with the curator. Each provider device may, in turn, serve as a gateway to numerous vendor devices, where each vendor device may directly store one or more data sets that each contain one or more portions of data on various subjects, and/or may store pointers and/or employ other mechanisms of redirection to one or more associated source devices that may store one or more of such data sets. A curator or other personnel may operate a client device to transmit a query, to the broker device, to provide the client device with indications of the storage locations of, and relationships among, the various portions of data available through the broker device that are in some way associated with a specified subject. The broker device may relay the query to each of the one or more provider devices in a broadcast thereto, and each provider device may, in turn, similarly relay the query to each of the one or more vendor devices to which each of the provider devices serves as a gateway. Each vendor device may respond by transmitting, to its associated provider device, one or more portions of metadata indicative of various details of the storage of portions of data by that vendor device and/or by one or more associated storage devices. In turn, each of the one or more provider devices may relay the received metadata portions to the broker device. The broker device may analyze the received metadata portions to identify relationships among the various data sets and/or portions thereof that correspond to the received metadata portions, and may generate a visualization of the identified relationships, which the broker device may then transmit to the requesting client device to be visually presented.

The data of each data set stored within a distributed online library system may be any of a variety of types of data (e.g., societal statistics data, business operations data, raw data from sensors of large scale experiments, financial data, medical treatment analysis data, data from geological or meteorogical instruments, streams of data collected from Internet-attached appliances, etc.). As the entities that may maintain a distributed online library system may be any of a variety of scholastic, industrial and/or governmental entities, any of a wide variety of uses may be made of such data sets by the personnel of such entities.

As will be familiar to those skilled in the art, it is a commonplace practice for a large data set to be stored within a storage system alongside metadata that provides a description of numerous aspects of the manner in which data is organized within the data set and/or characteristic of the data, itself, within the data set. By way of example, the metadata associated with a data set may describe the type of data structure employed to organize the data of the data set, including and not limited to, an array structure of a specified number of dimensions, or an indexed set of spreadsheets and/or documents, along with indications of a format employed by each. Also by way of example, the metadata associated with a data set may describe aspects of indexing system(s) used to access data values therein, including and not limited to, ranges of numerical values of the index for each dimension of a multi-dimensional array structure, or lists of text labels used to identify rows, columns, chapters, sections, figures, pictures. Further by way of example, the metadata associated with a data set may describe various characteristics of the data values, themselves, including and not limited to, indications of data types for data values in various portions of the data set (e.g., byte, word, double-word, quad-word, text characters, signed or unsigned integer values, floating point values, etc.), indications of one or more data values used as “null” data values, an indication of a degree of sparseness of non-null data values, or the size of each dimension in a multi-dimensional array. Still further by way of example, the metadata associated with a data set may describe various storage locations, including and not limited to, storage location(s) of the data set, of portions of data that are included in the data set, of data set(s) and/or other portions of data that were used in generating the data set, and/or of data set(s) and/or other portions of data that are cited and/or otherwise referred to.

As will also be familiar to those skilled in the art, there are numerous widely accepted and used standards for the storage of data that specify numerous parameters of the manner in which a data set may be stored, may be formatted for storage, may be indexed to make data values accessible from within storage, may be compressed for storage, may be distributed across multiple storage devices within storage, etc. In some embodiments, one or more of such parameters for the storage of a data set may be dictated, or at least influenced, by the selection of operating system and/or the file system used in storing a data set. Also, in some embodiments, one or more of such parameters for the storage of data may be dictated, or at least influenced by, the selection of database searching tools and/or data browsing applications used to retrieve and/or view information from a data set. Further, in some embodiments, the data type(s) of the data, itself, in a data set may dictate, or at least influence, one or more of such parameters for the storage of the data set. Still further, the fact of the need for the data in the data set to be encrypted and/or access thereto to be restricted may dictate and/or influence one or more of such parameters.

A curator of distributed online library system may select a subject to employ in evaluating the quality and/or breadth of the data currently available within the system, by operating a client device to generate a query to the system about what portions of data are present therein regarding the selected subject and the relationships among those portions of data. Such a query may or may not include a request to actually retrieve and provide those portions of data. In some embodiments, the distributed online library system may enforce some degree of limitation of access, such as a limitation of access to only individuals associated with the scholastic, industrial and/or governmental entity that maintains the system. Thus, the entry of any such query into the system may be required to be preceded with the entry of the credentials of the person who enters the query, although it may be that the particular client device they operate to enter the query may, itself, be provided with the necessary credentials to enable such access.

Upon entry of the query, the client device may transmit that query to a broker device of the distributed online library system. The broker device may serve as the collection point for all queries, and in so doing, may serve as the gateway to all of the devices of the system that are in some way responsible for storing the data of the system. As previously discussed, such a distributed online library system may rely on the provision of access to large quantities of data by one or more providers that may be caused to do so under licensing contracts and/or any of a variety of other arrangements. Indeed, in some embodiments, it may be that at least a subset of the data to which access is provided may be the property of the entity that maintains the system, such that the entity, itself, may serve as one of the providers.

Regardless of the identities of each of the one or more providers, each of the one or more providers may, at an earlier time, register a provider device with the broker device to establish an ongoing network connection therebetween. In so doing, a variety of parameters for that network connection may be configured to cause that provider device to employ a particular selected communications protocol for exchanging queries and responses to queries therebetween through a network connection. More specifically, the selection of such a communications protocol may entail the selection and use of a particular syntax and/or other particular parameters for the exchange, therebetween, of queries for, as well as portions of, metadata about portions of data within data sets. In some embodiments, it may be that the broker device employs a single selected communications protocol in exchanging such queries and information with any provider device such that all provider devices that are to communicate with the broker device must be configured to conform to the same set of communications requirements, including the particular selected communications protocol.

Alternatively or additionally, in some embodiments, one or more of the provider devices may be virtual machines (VMs) and/or may be embodied in the form of executable routines that are each to be executed within a VM. In such embodiments, such VMs may be instantiated and maintained within the broker device. It may be that the registration of a provider device is then carried out by the provision and/or installation of a routine that embodies the provider device by the provider to a curator of the distributed online library system for installation as or within one of the VMs of the broker device.

As previously alluded to, any of a variety of operating systems, file systems, database tools, etc. may be employed in performing the work of storing and providing access to data sets. As a result, each of the vendor devices may employ any of a wide variety of communications protocols for the exchange of queries and portions of metadata in response to queries. In some embodiments, each provider device may translate queries and portions of metadata between a communications protocol used in such exchanges with the broker device and a different communications protocol used in such exchanges with one or more vendor devices. It is envisioned that a provider will likely use the same communications protocol with all of its vendor devices such that their provider device will only need to translate between the communications protocol of the broker device and a single communications protocol for all of the provider's vendor devices.

However, embodiments are possible in which there may be more than one communications protocol employed by the vendor devices associated with a single provider. By way of example, it may be that the provider, itself, engages in licensing of data from one or more vendors, among which may be differences in the communications protocols used by their vendor devices. Thus, in some embodiments, the provider device associated with a particular provider may translate between the communications protocol of the broker device and multiple other communications protocols used by different ones of the vendor devices made available through that particular provider. Alternatively, a single provider may choose to support multiple different communications protocols among multiple vendor devices through the use of a different provider device for each such communications protocol.

Regardless of the exact manner in which each provider device may be implemented, regardless of the quantity of provider devices that may be employed by each provider, and regardless of the number of communications protocols that each provider device may translate between, upon receiving the query from the client device, the broker device may relay the query to each provider device of the one or more provider devices that have been registered with the broker device. In a manner that is analogous to the registration of provider devices with the broker device, it may be that each vendor device must similarly be registered with the particular provider device by which it is added to a distributed online library system. Thus, just as the broker device may serve as the gateway to all of the provider devices within a distributed online library system, each provider device may serve as a gateway that controls access to the multiple vendor devices that are registered with it.

In embodiments in which at least a subset of provider devices are operated by entities other than the entity that maintains the distributed online library system, it may be that one or more of the provider devices enforces restrictions in accesses to information that may be more stringent than the restrictions to distributed online library system that may be enforced by the broker device. By way of example, while the broker device may restrict access to the information of the distributed online library system to persons and/or devices that are associated with the entity that maintains the distributed online library system, a particular provider device may further restrict access to information to no more than a preselected maximum quantity of persons and/or devices at any one time. Alternatively or additionally, it may be that access to particular data sets and/or portions of data contained within particular data sets of a particular vendor device is time limited to be available for a predetermined period of time that may have been specifically agreed to in a licensing provision. As yet another alternative, it may be that, among the personnel of the entity that maintains the distributed online library system, only a particular group of persons are permitted to have access to one or more particular data sets that may be associated with a particular vendor device. Therefore, while the broker device may relay the query to all of the provider devices, one or more of the provider devices may refrain from relaying the query to one or more of the vendor devices that are registered with it. In each case where a provider device does relay the query, as previously discussed, the provider device may translate the query between the communications protocol by which it was received from the broker device and into the different communications protocol by which it is to be transmitted to one or more vendor devices.

Each of the vendor devices that does receive a translated form of the query may search the metadata associated with one or more data sets to identify any portions of data that may be relevant to the subject of the query. Each vendor device that successfully identifies such a portion of data may transmit a portion of metadata that corresponds to each such a portion of data back to the provider device. Each provider device that receives such portions of metadata may translate those portions of metadata from the communications protocol by which they were transmitted from one or more of the vendor devices, and into the communications protocol by which they are to be transmitted to the broker device.

Upon receipt, through one or more provider devices, of the portions of metadata provided by multiple vendor devices concerning multiple different portions of data, the broker device may perform various analyses and various comparisons among the different portions of metadata to identify relationships among their corresponding portions of data that are relevant to the subject of the query. Such analyses and/or comparisons may involve the text of various labels of data sets, portions of data sets, chapters and/or sections of documents, rows and/or columns of tables, etc. Alternatively or additionally, such analyses and/or comparisons may involve the size, organization, sparseness, data types and/or other characteristics of the portions of data that correspond to each of the portions of metadata.

As will be familiar to those skilled in the art, the choice of the operating system, the file system, the database searching tools and/or the browsing applications used in the storage and/or retrieval of data sets within each of the vendor devices may necessarily be correlated to the types of information that is included within associated metadata. Thus, the metadata maintained by different vendor devices that employ different selections of such components may contain different types of information. As will be apparent to those skilled in the art, such differences in metadata among different vendor devices may hamper the performance of comparisons made by the broker device among the portions of metadata it receives from different vendor devices. In some embodiments, the broker device may address such differences by transmitting further inquiries directed at one or more particular vendor devices (through appropriate one provider device(s)) for additional pieces of information that were not included in the portions of metadata provided therefrom, but which may be have been provided in portions of metadata from other vendor devices for other corresponding portions of data. Alternatively, in other embodiments, such further queries may originate from the one or more provider devices. The responses to such further queries may then be incorporated into the translated versions of the portions of metadata transmitted by such provider device(s) to the broker device.

Regardless of whether such further inquiries to round out the information provided in the portions of metadata are made, the broker device may generate one or more visualizations that include indications of the portions of data that have been identified by the vendor devices as relevant to the subject of the query, together with indications of relationships that have been identified thereamong. As will be explained in greater detail, various colors, patterns, symbols and/or other visual cues may be used to indicate differing types of relationships. The broker device may then transmit such visualization(s) to the requesting client device from which the original query was received by the broker device. Among the relationships that may be depicted in such visualizations may be relatively weak relationships in which multiple portions of data

Systems depicted in some of the figures may be provided in various configurations. In some embodiments, the systems may be configured as a distributed system where one or more components of the system are distributed across one or more networks in a cloud computing system and/or a fog computing system.

FIG. 1is a block diagram that provides an illustration of the hardware components of a data transmission network100, according to embodiments of the present technology. Data transmission network100is a specialized computer system that may be used for processing large amounts of data where a large number of computer processing cycles are required.

Data transmission network100may also include computing environment114. Computing environment114may be a specialized computer or other machine that processes the data received within the data transmission network100. Data transmission network100also includes one or more network devices102. Network devices102may include client devices that attempt to communicate with computing environment114. For example, network devices102may send data to the computing environment114to be processed, may send signals to the computing environment114to control different aspects of the computing environment or the data it is processing, among other reasons. Network devices102may interact with the computing environment114through a number of ways, such as, for example, over one or more networks108. As shown inFIG. 1, computing environment114may include one or more other systems. For example, computing environment114may include a database system118and/or a communications grid120.

In other embodiments, network devices may provide a large amount of data, either all at once or streaming over a period of time (e.g., using event stream processing (ESP), described further with respect toFIGS. 8-10), to the computing environment114via networks108. For example, network devices102may include network computers, sensors, databases, or other devices that may transmit or otherwise provide data to computing environment114. For example, network devices may include local area network devices, such as routers, hubs, switches, or other computer networking devices. These devices may provide a variety of stored or generated data, such as network data or data specific to the network devices themselves. Network devices may also include sensors that monitor their environment or other devices to collect data regarding that environment or those devices, and such network devices may provide data they collect over time. Network devices may also include devices within the internet of things, such as devices within a home automation network. Some of these devices may be referred to as edge devices, and may involve edge computing circuitry. Data may be transmitted by network devices directly to computing environment114or to network-attached data stores, such as network-attached data stores110for storage so that the data may be retrieved later by the computing environment114or other portions of data transmission network100.

Data transmission network100may also include one or more network-attached data stores110. Network-attached data stores110are used to store data to be processed by the computing environment114as well as any intermediate or final data generated by the computing system in non-volatile memory. However in certain embodiments, the configuration of the computing environment114allows its operations to be performed such that intermediate and final data results can be stored solely in volatile memory (e.g., RAM), without a requirement that intermediate or final data results be stored to non-volatile types of memory (e.g., disk). This can be useful in certain situations, such as when the computing environment114receives ad hoc queries from a user and when responses, which are generated by processing large amounts of data, need to be generated on-the-fly. In this non-limiting situation, the computing environment114may be configured to retain the processed information within memory so that responses can be generated for the user at different levels of detail as well as allow a user to interactively query against this information.

Network-attached data stores may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, network-attached data storage may include storage other than primary storage located within computing environment114that is directly accessible by processors located therein. Network-attached data storage may include secondary, tertiary or auxiliary storage, such as large hard drives, servers, virtual memory, among other types. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals. Examples of a non-transitory medium may include, for example, a magnetic disk or tape, optical storage media such as compact disk or digital versatile disk, flash memory, memory or memory devices. A computer-program product may include code and/or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others. Furthermore, the data stores may hold a variety of different types of data. For example, network-attached data stores110may hold unstructured (e.g., raw) data, such as manufacturing data (e.g., a database containing records identifying products being manufactured with parameter data for each product, such as colors and models) or product sales databases (e.g., a database containing individual data records identifying details of individual product sales).

The unstructured data may be presented to the computing environment114in different forms such as a flat file or a conglomerate of data records, and may have data values and accompanying time stamps. The computing environment114may be used to analyze the unstructured data in a variety of ways to determine the best way to structure (e.g., hierarchically) that data, such that the structured data is tailored to a type of further analysis that a user wishes to perform on the data. For example, after being processed, the unstructured time stamped data may be aggregated by time (e.g., into daily time period units) to generate time series data and/or structured hierarchically according to one or more dimensions (e.g., parameters, attributes, and/or variables). For example, data may be stored in a hierarchical data structure, such as a ROLAP OR MOLAP database, or may be stored in another tabular form, such as in a flat-hierarchy form.

Data transmission network100may also include one or more server farms106. Computing environment114may route select communications or data to the one or more sever farms106or one or more servers within the server farms. Server farms106can be configured to provide information in a predetermined manner. For example, server farms106may access data to transmit in response to a communication. Server farms106may be separately housed from each other device within data transmission network100, such as computing environment114, and/or may be part of a device or system.

Server farms106may host a variety of different types of data processing as part of data transmission network100. Server farms106may receive a variety of different data from network devices, from computing environment114, from cloud network116, or from other sources. The data may have been obtained or collected from one or more sensors, as inputs from a control database, or may have been received as inputs from an external system or device. Server farms106may assist in processing the data by turning raw data into processed data based on one or more rules implemented by the server farms. For example, sensor data may be analyzed to determine changes in an environment over time or in real-time.

Data transmission network100may also include one or more cloud networks116. Cloud network116may include a cloud infrastructure system that provides cloud services. In certain embodiments, services provided by the cloud network116may include a host of services that are made available to users of the cloud infrastructure system on demand Cloud network116is shown inFIG. 1as being connected to computing environment114(and therefore having computing environment114as its client or user), but cloud network116may be connected to or utilized by any of the devices inFIG. 1. Services provided by the cloud network can dynamically scale to meet the needs of its users. The cloud network116may comprise one or more computers, servers, and/or systems. In some embodiments, the computers, servers, and/or systems that make up the cloud network116are different from the user's own on-premises computers, servers, and/or systems. For example, the cloud network116may host an application, and a user may, via a communication network such as the Internet, on demand, order and use the application.

While each device, server and system inFIG. 1is shown as a single device, it will be appreciated that multiple devices may instead be used. For example, a set of network devices can be used to transmit various communications from a single user, or remote server140may include a server stack. As another example, data may be processed as part of computing environment114.

Each communication within data transmission network100(e.g., between client devices, between servers106and computing environment114or between a server and a device) may occur over one or more networks108. Networks108may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (LAN), a wide area network (WAN), or a wireless local area network (WLAN). A wireless network may include a wireless interface or combination of wireless interfaces. As an example, a network in the one or more networks108may include a short-range communication channel, such as a BLUETOOTH® communication channel or a BLUETOOTH® Low Energy communication channel. A wired network may include a wired interface. The wired and/or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the network114, as will be further described with respect toFIG. 2. The one or more networks108can be incorporated entirely within or can include an intranet, an extranet, or a combination thereof. In one embodiment, communications between two or more systems and/or devices can be achieved by a secure communications protocol, such as secure sockets layer (SSL) or transport layer security (TLS). In addition, data and/or transactional details may be encrypted.

Some aspects may utilize the Internet of Things (IoT), where things (e.g., machines, devices, phones, sensors) can be connected to networks and the data from these things can be collected and processed within the things and/or external to the things. For example, the IoT can include sensors in many different devices, and high value analytics can be applied to identify hidden relationships and drive increased efficiencies. This can apply to both big data analytics and real-time (e.g., ESP) analytics. This will be described further below with respect toFIG. 2.

As noted, computing environment114may include a communications grid120and a transmission network database system118. Communications grid120may be a grid-based computing system for processing large amounts of data. The transmission network database system118may be for managing, storing, and retrieving large amounts of data that are distributed to and stored in the one or more network-attached data stores110or other data stores that reside at different locations within the transmission network database system118. The compute nodes in the grid-based computing system120and the transmission network database system118may share the same processor hardware, such as processors that are located within computing environment114.

FIG. 2illustrates an example network including an example set of devices communicating with each other over an exchange system and via a network, according to embodiments of the present technology. As noted, each communication within data transmission network100may occur over one or more networks. System200includes a network device204configured to communicate with a variety of types of client devices, for example client devices230, over a variety of types of communication channels.

As shown inFIG. 2, network device204can transmit a communication over a network (e.g., a cellular network via a base station210). The communication can be routed to another network device, such as network devices205-209, via base station210. The communication can also be routed to computing environment214via base station210. For example, network device204may collect data either from its surrounding environment or from other network devices (such as network devices205-209) and transmit that data to computing environment214.

Although network devices204-209are shown inFIG. 2as a mobile phone, laptop computer, tablet computer, temperature sensor, motion sensor, and audio sensor respectively, the network devices may be or include sensors that are sensitive to detecting aspects of their environment. For example, the network devices may include sensors such as water sensors, power sensors, electrical current sensors, chemical sensors, optical sensors, pressure sensors, geographic or position sensors (e.g., GPS), velocity sensors, acceleration sensors, flow rate sensors, among others. Examples of characteristics that may be sensed include force, torque, load, strain, position, temperature, air pressure, fluid flow, chemical properties, resistance, electromagnetic fields, radiation, irradiance, proximity, acoustics, moisture, distance, speed, vibrations, acceleration, electrical potential, electrical current, among others. The sensors may be mounted to various components used as part of a variety of different types of systems (e.g., an oil drilling operation). The network devices may detect and record data related to the environment that it monitors, and transmit that data to computing environment214.

As noted, one type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment according to certain embodiments includes an oil drilling system. For example, the one or more drilling operation sensors may include surface sensors that measure a hook load, a fluid rate, a temperature and a density in and out of the wellbore, a standpipe pressure, a surface torque, a rotation speed of a drill pipe, a rate of penetration, a mechanical specific energy, etc. and downhole sensors that measure a rotation speed of a bit, fluid densities, downhole torque, downhole vibration (axial, tangential, lateral), a weight applied at a drill bit, an annular pressure, a differential pressure, an azimuth, an inclination, a dog leg severity, a measured depth, a vertical depth, a downhole temperature, etc. Besides the raw data collected directly by the sensors, other data may include parameters either developed by the sensors or assigned to the system by a client or other controlling device. For example, one or more drilling operation control parameters may control settings such as a mud motor speed to flow ratio, a bit diameter, a predicted formation top, seismic data, weather data, etc. Other data may be generated using physical models such as an earth model, a weather model, a seismic model, a bottom hole assembly model, a well plan model, an annular friction model, etc. In addition to sensor and control settings, predicted outputs, of for example, the rate of penetration, mechanical specific energy, hook load, flow in fluid rate, flow out fluid rate, pump pressure, surface torque, rotation speed of the drill pipe, annular pressure, annular friction pressure, annular temperature, equivalent circulating density, etc. may also be stored in the data warehouse.

In another example, another type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment according to certain embodiments includes a home automation or similar automated network in a different environment, such as an office space, school, public space, sports venue, or a variety of other locations. Network devices in such an automated network may include network devices that allow a user to access, control, and/or configure various home appliances located within the user's home (e.g., a television, radio, light, fan, humidifier, sensor, microwave, iron, and/or the like), or outside of the user's home (e.g., exterior motion sensors, exterior lighting, garage door openers, sprinkler systems, or the like). For example, network device102may include a home automation switch that may be coupled with a home appliance. In another embodiment, a network device can allow a user to access, control, and/or configure devices, such as office-related devices (e.g., copy machine, printer, or fax machine), audio and/or video related devices (e.g., a receiver, a speaker, a projector, a DVD player, or a television), media-playback devices (e.g., a compact disc player, a CD player, or the like), computing devices (e.g., a home computer, a laptop computer, a tablet, a personal digital assistant (PDA), a computing device, or a wearable device), lighting devices (e.g., a lamp or recessed lighting), devices associated with a security system, devices associated with an alarm system, devices that can be operated in an automobile (e.g., radio devices, navigation devices), and/or the like. Data may be collected from such various sensors in raw form, or data may be processed by the sensors to create parameters or other data either developed by the sensors based on the raw data or assigned to the system by a client or other controlling device.

In another example, another type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment according to certain embodiments includes a power or energy grid. A variety of different network devices may be included in an energy grid, such as various devices within one or more power plants, energy farms (e.g., wind farm, solar farm, among others) energy storage facilities, factories, homes and businesses of consumers, among others. One or more of such devices may include one or more sensors that detect energy gain or loss, electrical input or output or loss, and a variety of other efficiencies. These sensors may collect data to inform users of how the energy grid, and individual devices within the grid, may be functioning and how they may be made more efficient.

Network device sensors may also perform processing on data it collects before transmitting the data to the computing environment114, or before deciding whether to transmit data to the computing environment114. For example, network devices may determine whether data collected meets certain rules, for example by comparing data or values calculated from the data and comparing that data to one or more thresholds. The network device may use this data and/or comparisons to determine if the data should be transmitted to the computing environment214for further use or processing.

Computing environment214may include machines220and240. Although computing environment214is shown inFIG. 2as having two machines,220and240, computing environment214may have only one machine or may have more than two machines. The machines that make up computing environment214may include specialized computers, servers, or other machines that are configured to individually and/or collectively process large amounts of data. The computing environment214may also include storage devices that include one or more databases of structured data, such as data organized in one or more hierarchies, or unstructured data. The databases may communicate with the processing devices within computing environment214to distribute data to them. Since network devices may transmit data to computing environment214, that data may be received by the computing environment214and subsequently stored within those storage devices. Data used by computing environment214may also be stored in data stores235, which may also be a part of or connected to computing environment214.

Computing environment214can communicate with various devices via one or more routers225or other inter-network or intra-network connection components. For example, computing environment214may communicate with devices230via one or more routers225. Computing environment214may collect, analyze and/or store data from or pertaining to communications, client device operations, client rules, and/or user-associated actions stored at one or more data stores235. Such data may influence communication routing to the devices within computing environment214, how data is stored or processed within computing environment214, among other actions.

Notably, various other devices can further be used to influence communication routing and/or processing between devices within computing environment214and with devices outside of computing environment214. For example, as shown inFIG. 2, computing environment214may include a web server240. Thus, computing environment214can retrieve data of interest, such as client information (e.g., product information, client rules, etc.), technical product details, news, current or predicted weather, and so on.

In addition to computing environment214collecting data (e.g., as received from network devices, such as sensors, and client devices or other sources) to be processed as part of a big data analytics project, it may also receive data in real time as part of a streaming analytics environment. As noted, data may be collected using a variety of sources as communicated via different kinds of networks or locally. Such data may be received on a real-time streaming basis. For example, network devices may receive data periodically from network device sensors as the sensors continuously sense, monitor and track changes in their environments. Devices within computing environment214may also perform pre-analysis on data it receives to determine if the data received should be processed as part of an ongoing project. The data received and collected by computing environment214, no matter what the source or method or timing of receipt, may be processed over a period of time for a client to determine results data based on the client's needs and rules.

FIG. 3illustrates a representation of a conceptual model of a communications protocol system, according to embodiments of the present technology. More specifically,FIG. 3identifies operation of a computing environment in an Open Systems Interaction model that corresponds to various connection components. The model300shows, for example, how a computing environment, such as computing environment314(or computing environment214inFIG. 2) may communicate with other devices in its network, and control how communications between the computing environment and other devices are executed and under what conditions.

The model can include layers301-307. The layers are arranged in a stack. Each layer in the stack serves the layer one level higher than it (except for the application layer, which is the highest layer), and is served by the layer one level below it (except for the physical layer, which is the lowest layer). The physical layer is the lowest layer because it receives and transmits raw bites of data, and is the farthest layer from the user in a communications system. On the other hand, the application layer is the highest layer because it interacts directly with a software application.

As noted, the model includes a physical layer301. Physical layer301represents physical communication, and can define parameters of that physical communication. For example, such physical communication may come in the form of electrical, optical, or electromagnetic signals. Physical layer301also defines protocols that may control communications within a data transmission network.

Link layer302defines links and mechanisms used to transmit (i.e., move) data across a network. The link layer302manages node-to-node communications, such as within a grid computing environment. Link layer302can detect and correct errors (e.g., transmission errors in the physical layer301). Link layer302can also include a media access control (MAC) layer and logical link control (LLC) layer.

Network layer303defines the protocol for routing within a network. In other words, the network layer coordinates transferring data across nodes in a same network (e.g., such as a grid computing environment). Network layer303can also define the processes used to structure local addressing within the network.

Transport layer304can manage the transmission of data and the quality of the transmission and/or receipt of that data. Transport layer304can provide a protocol for transferring data, such as, for example, a Transmission Control Protocol (TCP). Transport layer304can assemble and disassemble data frames for transmission. The transport layer can also detect transmission errors occurring in the layers below it.

Session layer305can establish, maintain, and manage communication connections between devices on a network. In other words, the session layer controls the dialogues or nature of communications between network devices on the network. The session layer may also establish checkpointing, adjournment, termination, and restart procedures.

Presentation layer306can provide translation for communications between the application and network layers. In other words, this layer may encrypt, decrypt and/or format data based on data types and/or encodings known to be accepted by an application or network layer.

Application layer307interacts directly with software applications and end users, and manages communications between them. Application layer307can identify destinations, local resource states or availability and/or communication content or formatting using the applications.

Intra-network connection components321and322are shown to operate in lower levels, such as physical layer301and link layer302, respectively. For example, a hub can operate in the physical layer, a switch can operate in the link layer, and a router can operate in the network layer. Inter-network connection components323and328are shown to operate on higher levels, such as layers303-307. For example, routers can operate in the network layer and network devices can operate in the transport, session, presentation, and application layers.

As noted, a computing environment314can interact with and/or operate on, in various embodiments, one, more, all or any of the various layers. For example, computing environment314can interact with a hub (e.g., via the link layer) so as to adjust which devices the hub communicates with. The physical layer may be served by the link layer, so it may implement such data from the link layer. For example, the computing environment314may control which devices it will receive data from. For example, if the computing environment314knows that a certain network device has turned off, broken, or otherwise become unavailable or unreliable, the computing environment314may instruct the hub to prevent any data from being transmitted to the computing environment314from that network device. Such a process may be beneficial to avoid receiving data that is inaccurate or that has been influenced by an uncontrolled environment. As another example, computing environment314can communicate with a bridge, switch, router or gateway and influence which device within the system (e.g., system200) the component selects as a destination. In some embodiments, computing environment314can interact with various layers by exchanging communications with equipment operating on a particular layer by routing or modifying existing communications. In another embodiment, such as in a grid computing environment, a node may determine how data within the environment should be routed (e.g., which node should receive certain data) based on certain parameters or information provided by other layers within the model.

As noted, the computing environment314may be a part of a communications grid environment, the communications of which may be implemented as shown in the protocol ofFIG. 3. For example, referring back toFIG. 2, one or more of machines220and240may be part of a communications grid computing environment. A gridded computing environment may be employed in a distributed system with non-interactive workloads where data resides in memory on the machines, or compute nodes. In such an environment, analytic code, instead of a database management system, controls the processing performed by the nodes. Data is co-located by pre-distributing it to the grid nodes, and the analytic code on each node loads the local data into memory. Each node may be assigned a particular task such as a portion of a processing project, or to organize or control other nodes within the grid.

FIG. 4illustrates a communications grid computing system400including a variety of control and worker nodes, according to embodiments of the present technology.

Communications grid computing system400includes three control nodes and one or more worker nodes. Communications grid computing system400includes control nodes402,404, and406. The control nodes are communicatively connected via communication paths451,453, and455. Therefore, the control nodes may transmit information (e.g., related to the communications grid or notifications), to and receive information from each other. Although communications grid computing system400is shown inFIG. 4as including three control nodes, the communications grid may include more or less than three control nodes.

Communications grid computing system (or just “communications grid”)400also includes one or more worker nodes. Shown inFIG. 4are six worker nodes410-420. AlthoughFIG. 4shows six worker nodes, a communications grid according to embodiments of the present technology may include more or less than six worker nodes. The number of worker nodes included in a communications grid may be dependent upon how large the project or data set is being processed by the communications grid, the capacity of each worker node, the time designated for the communications grid to complete the project, among others. Each worker node within the communications grid400may be connected (wired or wirelessly, and directly or indirectly) to control nodes402-406. Therefore, each worker node may receive information from the control nodes (e.g., an instruction to perform work on a project) and may transmit information to the control nodes (e.g., a result from work performed on a project). Furthermore, worker nodes may communicate with each other (either directly or indirectly). For example, worker nodes may transmit data between each other related to a job being performed or an individual task within a job being performed by that worker node. However, in certain embodiments, worker nodes may not, for example, be connected (communicatively or otherwise) to certain other worker nodes. In an embodiment, worker nodes may only be able to communicate with the control node that controls it, and may not be able to communicate with other worker nodes in the communications grid, whether they are other worker nodes controlled by the control node that controls the worker node, or worker nodes that are controlled by other control nodes in the communications grid.

A control node may connect with an external device with which the control node may communicate (e.g., a grid user, such as a server or computer, may connect to a controller of the grid). For example, a server or computer may connect to control nodes and may transmit a project or job to the node. The project may include a data set. The data set may be of any size. Once the control node receives such a project including a large data set, the control node may distribute the data set or projects related to the data set to be performed by worker nodes. Alternatively, for a project including a large data set, the data set may be received or stored by a machine other than a control node (e.g., a HADOOP® standard-compliant data node employing the HADOOP® Distributed File System, or HDFS).

Control nodes may maintain knowledge of the status of the nodes in the grid (i.e., grid status information), accept work requests from clients, subdivide the work across worker nodes, coordinate the worker nodes, among other responsibilities. Worker nodes may accept work requests from a control node and provide the control node with results of the work performed by the worker node. A grid may be started from a single node (e.g., a machine, computer, server, etc.). This first node may be assigned or may start as the primary control node that will control any additional nodes that enter the grid.

When a project is submitted for execution (e.g., by a client or a controller of the grid) it may be assigned to a set of nodes. After the nodes are assigned to a project, a data structure (i.e., a communicator) may be created. The communicator may be used by the project for information to be shared between the project code running on each node. A communication handle may be created on each node. A handle, for example, is a reference to the communicator that is valid within a single process on a single node, and the handle may be used when requesting communications between nodes.

A control node, such as control node402, may be designated as the primary control node. A server, computer or other external device may connect to the primary control node. Once the control node receives a project, the primary control node may distribute portions of the project to its worker nodes for execution. For example, when a project is initiated on communications grid400, primary control node402controls the work to be performed for the project in order to complete the project as requested or instructed. The primary control node may distribute work to the worker nodes based on various factors, such as which subsets or portions of projects may be completed most efficiently and in the correct amount of time. For example, a worker node may perform analysis on a portion of data that is already local (e.g., stored on) the worker node. The primary control node also coordinates and processes the results of the work performed by each worker node after each worker node executes and completes its job. For example, the primary control node may receive a result from one or more worker nodes, and the control node may organize (e.g., collect and assemble) the results received and compile them to produce a complete result for the project received from the end user.

Any remaining control nodes, such as control nodes404and406, may be assigned as backup control nodes for the project. In an embodiment, backup control nodes may not control any portion of the project. Instead, backup control nodes may serve as a backup for the primary control node and take over as primary control node if the primary control node were to fail. If a communications grid were to include only a single control node, and the control node were to fail (e.g., the control node is shut off or breaks) then the communications grid as a whole may fail and any project or job being run on the communications grid may fail and may not complete. While the project may be run again, such a failure may cause a delay (severe delay in some cases, such as overnight delay) in completion of the project. Therefore, a grid with multiple control nodes, including a backup control node, may be beneficial.

To add another node or machine to the grid, the primary control node may open a pair of listening sockets, for example. A socket may be used to accept work requests from clients, and the second socket may be used to accept connections from other grid nodes. The primary control node may be provided with a list of other nodes (e.g., other machines, computers, servers) that will participate in the grid, and the role that each node will fill in the grid. Upon startup of the primary control node (e.g., the first node on the grid), the primary control node may use a network protocol to start the server process on every other node in the grid. Command line parameters, for example, may inform each node of one or more pieces of information, such as: the role that the node will have in the grid, the host name of the primary control node, the port number on which the primary control node is accepting connections from peer nodes, among others. The information may also be provided in a configuration file, transmitted over a secure shell tunnel, recovered from a configuration server, among others. While the other machines in the grid may not initially know about the configuration of the grid, that information may also be sent to each other node by the primary control node. Updates of the grid information may also be subsequently sent to those nodes.

For any control node other than the primary control node added to the grid, the control node may open three sockets. The first socket may accept work requests from clients, the second socket may accept connections from other grid members, and the third socket may connect (e.g., permanently) to the primary control node. When a control node (e.g., primary control node) receives a connection from another control node, it first checks to see if the peer node is in the list of configured nodes in the grid. If it is not on the list, the control node may clear the connection. If it is on the list, it may then attempt to authenticate the connection. If authentication is successful, the authenticating node may transmit information to its peer, such as the port number on which a node is listening for connections, the host name of the node, information about how to authenticate the node, among other information. When a node, such as the new control node, receives information about another active node, it will check to see if it already has a connection to that other node. If it does not have a connection to that node, it may then establish a connection to that control node.

Any worker node added to the grid may establish a connection to the primary control node and any other control nodes on the grid. After establishing the connection, it may authenticate itself to the grid (e.g., any control nodes, including both primary and backup, or a server or user controlling the grid). After successful authentication, the worker node may accept configuration information from the control node.

When a node joins a communications grid (e.g., when the node is powered on or connected to an existing node on the grid or both), the node is assigned (e.g., by an operating system of the grid) a universally unique identifier (UUID). This unique identifier may help other nodes and external entities (devices, users, etc.) to identify the node and distinguish it from other nodes. When a node is connected to the grid, the node may share its unique identifier with the other nodes in the grid. Since each node may share its unique identifier, each node may know the unique identifier of every other node on the grid. Unique identifiers may also designate a hierarchy of each of the nodes (e.g., backup control nodes) within the grid. For example, the unique identifiers of each of the backup control nodes may be stored in a list of backup control nodes to indicate an order in which the backup control nodes will take over for a failed primary control node to become a new primary control node. However, a hierarchy of nodes may also be determined using methods other than using the unique identifiers of the nodes. For example, the hierarchy may be predetermined, or may be assigned based on other predetermined factors.

The grid may add new machines at any time (e.g., initiated from any control node). Upon adding a new node to the grid, the control node may first add the new node to its table of grid nodes. The control node may also then notify every other control node about the new node. The nodes receiving the notification may acknowledge that they have updated their configuration information.

Primary control node402may, for example, transmit one or more communications to backup control nodes404and406(and, for example, to other control or worker nodes within the communications grid). Such communications may sent periodically, at fixed time intervals, between known fixed stages of the project's execution, among other protocols. The communications transmitted by primary control node402may be of varied types and may include a variety of types of information. For example, primary control node402may transmit snapshots (e.g., status information) of the communications grid so that backup control node404always has a recent snapshot of the communications grid. The snapshot or grid status may include, for example, the structure of the grid (including, for example, the worker nodes in the grid, unique identifiers of the nodes, or their relationships with the primary control node) and the status of a project (including, for example, the status of each worker node's portion of the project). The snapshot may also include analysis or results received from worker nodes in the communications grid. The backup control nodes may receive and store the backup data received from the primary control node. The backup control nodes may transmit a request for such a snapshot (or other information) from the primary control node, or the primary control node may send such information periodically to the backup control nodes.

As noted, the backup data may allow the backup control node to take over as primary control node if the primary control node fails without requiring the grid to start the project over from scratch. If the primary control node fails, the backup control node that will take over as primary control node may retrieve the most recent version of the snapshot received from the primary control node and use the snapshot to continue the project from the stage of the project indicated by the backup data. This may prevent failure of the project as a whole.

A backup control node may use various methods to determine that the primary control node has failed. In one example of such a method, the primary control node may transmit (e.g., periodically) a communication to the backup control node that indicates that the primary control node is working and has not failed, such as a heartbeat communication. The backup control node may determine that the primary control node has failed if the backup control node has not received a heartbeat communication for a certain predetermined period of time. Alternatively, a backup control node may also receive a communication from the primary control node itself (before it failed) or from a worker node that the primary control node has failed, for example because the primary control node has failed to communicate with the worker node.

Different methods may be performed to determine which backup control node of a set of backup control nodes (e.g., backup control nodes404and406) will take over for failed primary control node402and become the new primary control node. For example, the new primary control node may be chosen based on a ranking or “hierarchy” of backup control nodes based on their unique identifiers. In an alternative embodiment, a backup control node may be assigned to be the new primary control node by another device in the communications grid or from an external device (e.g., a system infrastructure or an end user, such as a server or computer, controlling the communications grid). In another alternative embodiment, the backup control node that takes over as the new primary control node may be designated based on bandwidth or other statistics about the communications grid.

A worker node within the communications grid may also fail. If a worker node fails, work being performed by the failed worker node may be redistributed amongst the operational worker nodes. In an alternative embodiment, the primary control node may transmit a communication to each of the operable worker nodes still on the communications grid that each of the worker nodes should purposefully fail also. After each of the worker nodes fail, they may each retrieve their most recent saved checkpoint of their status and re-start the project from that checkpoint to minimize lost progress on the project being executed.

FIG. 5illustrates a flow chart showing an example process500for adjusting a communications grid or a work project in a communications grid after a failure of a node, according to embodiments of the present technology. The process may include, for example, receiving grid status information including a project status of a portion of a project being executed by a node in the communications grid, as described in operation502. For example, a control node (e.g., a backup control node connected to a primary control node and a worker node on a communications grid) may receive grid status information, where the grid status information includes a project status of the primary control node or a project status of the worker node. The project status of the primary control node and the project status of the worker node may include a status of one or more portions of a project being executed by the primary and worker nodes in the communications grid. The process may also include storing the grid status information, as described in operation504. For example, a control node (e.g., a backup control node) may store the received grid status information locally within the control node. Alternatively, the grid status information may be sent to another device for storage where the control node may have access to the information.

The process may also include receiving a failure communication corresponding to a node in the communications grid in operation506. For example, a node may receive a failure communication including an indication that the primary control node has failed, prompting a backup control node to take over for the primary control node. In an alternative embodiment, a node may receive a failure that a worker node has failed, prompting a control node to reassign the work being performed by the worker node. The process may also include reassigning a node or a portion of the project being executed by the failed node, as described in operation508. For example, a control node may designate the backup control node as a new primary control node based on the failure communication upon receiving the failure communication. If the failed node is a worker node, a control node may identify a project status of the failed worker node using the snapshot of the communications grid, where the project status of the failed worker node includes a status of a portion of the project being executed by the failed worker node at the failure time.

The process may also include receiving updated grid status information based on the reassignment, as described in operation510, and transmitting a set of instructions based on the updated grid status information to one or more nodes in the communications grid, as described in operation512. The updated grid status information may include an updated project status of the primary control node or an updated project status of the worker node. The updated information may be transmitted to the other nodes in the grid to update their stale stored information.

FIG. 6illustrates a portion of a communications grid computing system600including a control node and a worker node, according to embodiments of the present technology. Communications grid600computing system includes one control node (control node602) and one worker node (worker node610) for purposes of illustration, but may include more worker and/or control nodes. The control node602is communicatively connected to worker node610via communication path650. Therefore, control node602may transmit information (e.g., related to the communications grid or notifications), to and receive information from worker node610via path650.

Similar to inFIG. 4, communications grid computing system (or just “communications grid”)600includes data processing nodes (control node602and worker node610). Nodes602and610comprise multi-core data processors. Each node602and610includes a grid-enabled software component (GESC)620that executes on the data processor associated with that node and interfaces with buffer memory622also associated with that node. Each node602and610includes a database management software (DBMS)628that executes on a database server (not shown) at control node602and on a database server (not shown) at worker node610.

Each node also includes a data store624. Data stores624, similar to network-attached data stores110inFIG. 1and data stores235inFIG. 2, are used to store data to be processed by the nodes in the computing environment. Data stores624may also store any intermediate or final data generated by the computing system after being processed, for example in non-volatile memory. However in certain embodiments, the configuration of the grid computing environment allows its operations to be performed such that intermediate and final data results can be stored solely in volatile memory (e.g., RAM), without a requirement that intermediate or final data results be stored to non-volatile types of memory. Storing such data in volatile memory may be useful in certain situations, such as when the grid receives queries (e.g., ad hoc) from a client and when responses, which are generated by processing large amounts of data, need to be generated quickly or on-the-fly. In such a situation, the grid may be configured to retain the data within memory so that responses can be generated at different levels of detail and so that a client may interactively query against this information.

Each node also includes a user-defined function (UDF)626. The UDF provides a mechanism for the DMBS628to transfer data to or receive data from the database stored in the data stores624that are managed by the DBMS. For example, UDF626can be invoked by the DBMS to provide data to the GESC for processing. The UDF626may establish a socket connection (not shown) with the GESC to transfer the data. Alternatively, the UDF626can transfer data to the GESC by writing data to shared memory accessible by both the UDF and the GESC.

The GESC620at the nodes602and620may be connected via a network, such as network108shown inFIG. 1. Therefore, nodes602and620can communicate with each other via the network using a predetermined communication protocol such as, for example, the Message Passing Interface (MPI). Each GESC620can engage in point-to-point communication with the GESC at another node or in collective communication with multiple GESCs via the network. The GESC620at each node may contain identical (or nearly identical) software instructions. Each node may be capable of operating as either a control node or a worker node. The GESC at the control node602can communicate, over a communication path652, with a client device630. More specifically, control node602may communicate with client application632hosted by the client device630to receive queries and to respond to those queries after processing large amounts of data.

DMBS628may control the creation, maintenance, and use of database or data structure (not shown) within a nodes602or610. The database may organize data stored in data stores624. The DMBS628at control node602may accept requests for data and transfer the appropriate data for the request. With such a process, collections of data may be distributed across multiple physical locations. In this example, each node602and610stores a portion of the total data managed by the management system in its associated data store624.

Furthermore, the DBMS may be responsible for protecting against data loss using replication techniques. Replication includes providing a backup copy of data stored on one node on one or more other nodes. Therefore, if one node fails, the data from the failed node can be recovered from a replicated copy residing at another node. However, as described herein with respect toFIG. 4, data or status information for each node in the communications grid may also be shared with each node on the grid.

FIG. 7illustrates a flow chart showing an example method700for executing a project within a grid computing system, according to embodiments of the present technology. As described with respect toFIG. 6, the GESC at the control node may transmit data with a client device (e.g., client device630) to receive queries for executing a project and to respond to those queries after large amounts of data have been processed. The query may be transmitted to the control node, where the query may include a request for executing a project, as described in operation702. The query can contain instructions on the type of data analysis to be performed in the project and whether the project should be executed using the grid-based computing environment, as shown in operation704.

To initiate the project, the control node may determine if the query requests use of the grid-based computing environment to execute the project. If the determination is no, then the control node initiates execution of the project in a solo environment (e.g., at the control node), as described in operation710. If the determination is yes, the control node may initiate execution of the project in the grid-based computing environment, as described in operation706. In such a situation, the request may include a requested configuration of the grid. For example, the request may include a number of control nodes and a number of worker nodes to be used in the grid when executing the project. After the project has been completed, the control node may transmit results of the analysis yielded by the grid, as described in operation708. Whether the project is executed in a solo or grid-based environment, the control node provides the results of the project, as described in operation712.

As noted with respect toFIG. 2, the computing environments described herein may collect data (e.g., as received from network devices, such as sensors, such as network devices204-209inFIG. 2, and client devices or other sources) to be processed as part of a data analytics project, and data may be received in real time as part of a streaming analytics environment (e.g., ESP). Data may be collected using a variety of sources as communicated via different kinds of networks or locally, such as on a real-time streaming basis. For example, network devices may receive data periodically from network device sensors as the sensors continuously sense, monitor and track changes in their environments. More specifically, an increasing number of distributed applications develop or produce continuously flowing data from distributed sources by applying queries to the data before distributing the data to geographically distributed recipients. An event stream processing engine (ESPE) may continuously apply the queries to the data as it is received and determines which entities should receive the data. Client or other devices may also subscribe to the ESPE or other devices processing ESP data so that they can receive data after processing, based on for example the entities determined by the processing engine. For example, client devices230inFIG. 2may subscribe to the ESPE in computing environment214. In another example, event subscription devices1024a-c, described further with respect toFIG. 10, may also subscribe to the ESPE. The ESPE may determine or define how input data or event streams from network devices or other publishers (e.g., network devices204-209inFIG. 2) are transformed into meaningful output data to be consumed by subscribers, such as for example client devices230inFIG. 2.

FIG. 8illustrates a block diagram including components of an Event Stream Processing Engine (ESPE), according to embodiments of the present technology. ESPE800may include one or more projects802. A project may be described as a second-level container in an engine model managed by ESPE800where a thread pool size for the project may be defined by a user. Each project of the one or more projects802may include one or more continuous queries804that contain data flows, which are data transformations of incoming event streams. The one or more continuous queries804may include one or more source windows806and one or more derived windows808.

The ESPE may receive streaming data over a period of time related to certain events, such as events or other data sensed by one or more network devices. The ESPE may perform operations associated with processing data created by the one or more devices. For example, the ESPE may receive data from the one or more network devices204-209shown inFIG. 2. As noted, the network devices may include sensors that sense different aspects of their environments, and may collect data over time based on those sensed observations. For example, the ESPE may be implemented within one or more of machines220and240shown inFIG. 2. The ESPE may be implemented within such a machine by an ESP application. An ESP application may embed an ESPE with its own dedicated thread pool or pools into its application space where the main application thread can do application-specific work and the ESPE processes event streams at least by creating an instance of a model into processing objects.

The engine container is the top-level container in a model that manages the resources of the one or more projects802. In an illustrative embodiment, for example, there may be only one ESPE800for each instance of the ESP application, and ESPE800may have a unique engine name. Additionally, the one or more projects802may each have unique project names, and each query may have a unique continuous query name and begin with a uniquely named source window of the one or more source windows806. ESPE800may or may not be persistent.

Continuous query modeling involves defining directed graphs of windows for event stream manipulation and transformation. A window in the context of event stream manipulation and transformation is a processing node in an event stream processing model. A window in a continuous query can perform aggregations, computations, pattern-matching, and other operations on data flowing through the window. A continuous query may be described as a directed graph of source, relational, pattern matching, and procedural windows. The one or more source windows806and the one or more derived windows808represent continuously executing queries that generate updates to a query result set as new event blocks stream through ESPE800. A directed graph, for example, is a set of nodes connected by edges, where the edges have a direction associated with them.

An event object may be described as a packet of data accessible as a collection of fields, with at least one of the fields defined as a key or unique identifier (ID). The event object may be created using a variety of formats including binary, alphanumeric, XML, etc. Each event object may include one or more fields designated as a primary identifier (ID) for the event so ESPE800can support operation codes (opcodes) for events including insert, update, upsert, and delete. Upsert opcodes update the event if the key field already exists; otherwise, the event is inserted. For illustration, an event object may be a packed binary representation of a set of field values and include both metadata and field data associated with an event. The metadata may include an opcode indicating if the event represents an insert, update, delete, or upsert, a set of flags indicating if the event is a normal, partial-update, or a retention generated event from retention policy management, and a set of microsecond timestamps that can be used for latency measurements.

An event block object may be described as a grouping or package of event objects. An event stream may be described as a flow of event block objects. A continuous query of the one or more continuous queries804transforms a source event stream made up of streaming event block objects published into ESPE800into one or more output event streams using the one or more source windows806and the one or more derived windows808. A continuous query can also be thought of as data flow modeling.

The one or more source windows806are at the top of the directed graph and have no windows feeding into them. Event streams are published into the one or more source windows806, and from there, the event streams may be directed to the next set of connected windows as defined by the directed graph. The one or more derived windows808are all instantiated windows that are not source windows and that have other windows streaming events into them. The one or more derived windows808may perform computations or transformations on the incoming event streams. The one or more derived windows808transform event streams based on the window type (that is operators such as join, filter, compute, aggregate, copy, pattern match, procedural, union, etc.) and window settings. As event streams are published into ESPE800, they are continuously queried, and the resulting sets of derived windows in these queries are continuously updated.

FIG. 9illustrates a flow chart showing an example process including operations performed by an event stream processing engine, according to some embodiments of the present technology. As noted, the ESPE800(or an associated ESP application) defines how input event streams are transformed into meaningful output event streams. More specifically, the ESP application may define how input event streams from publishers (e.g., network devices providing sensed data) are transformed into meaningful output event streams consumed by subscribers (e.g., a data analytics project being executed by a machine or set of machines).

Within the application, a user may interact with one or more user interface windows presented to the user in a display under control of the ESPE independently or through a browser application in an order selectable by the user. For example, a user may execute an ESP application, which causes presentation of a first user interface window, which may include a plurality of menus and selectors such as drop down menus, buttons, text boxes, hyperlinks, etc. associated with the ESP application as understood by a person of skill in the art. As further understood by a person of skill in the art, various operations may be performed in parallel, for example, using a plurality of threads.

At operation900, an ESP application may define and start an ESPE, thereby instantiating an ESPE at a device, such as machine220and/or240. In an operation902, the engine container is created. For illustration, ESPE800may be instantiated using a function call that specifies the engine container as a manager for the model.

In an operation904, the one or more continuous queries804are instantiated by ESPE800as a model. The one or more continuous queries804may be instantiated with a dedicated thread pool or pools that generate updates as new events stream through ESPE800. For illustration, the one or more continuous queries804may be created to model business processing logic within ESPE800, to predict events within ESPE800, to model a physical system within ESPE800, to predict the physical system state within ESPE800, etc. For example, as noted, ESPE800may be used to support sensor data monitoring and management (e.g., sensing may include force, torque, load, strain, position, temperature, air pressure, fluid flow, chemical properties, resistance, electromagnetic fields, radiation, irradiance, proximity, acoustics, moisture, distance, speed, vibrations, acceleration, electrical potential, or electrical current, etc.).

ESPE800may analyze and process events in motion or “event streams.” Instead of storing data and running queries against the stored data, ESPE800may store queries and stream data through them to allow continuous analysis of data as it is received. The one or more source windows806and the one or more derived windows808may be created based on the relational, pattern matching, and procedural algorithms that transform the input event streams into the output event streams to model, simulate, score, test, predict, etc. based on the continuous query model defined and application to the streamed data.

In an operation906, a publish/subscribe (pub/sub) capability is initialized for ESPE800. In an illustrative embodiment, a pub/sub capability is initialized for each project of the one or more projects802. To initialize and enable pub/sub capability for ESPE800, a port number may be provided. Pub/sub clients can use a host name of an ESP device running the ESPE and the port number to establish pub/sub connections to ESPE800.

FIG. 10illustrates an ESP system1000interfacing between publishing device1022and event subscribing devices1024a-c, according to embodiments of the present technology. ESP system1000may include ESP device or subsystem851, event publishing device1022, an event subscribing device A1024a, an event subscribing device B1024b, and an event subscribing device C1024c. Input event streams are output to ESP device851by publishing device1022. In alternative embodiments, the input event streams may be created by a plurality of publishing devices. The plurality of publishing devices further may publish event streams to other ESP devices. The one or more continuous queries instantiated by ESPE800may analyze and process the input event streams to form output event streams output to event subscribing device A1024a, event subscribing device B1024b, and event subscribing device C1024c. ESP system1000may include a greater or a fewer number of event subscribing devices of event subscribing devices.

Publish-subscribe is a message-oriented interaction paradigm based on indirect addressing. Processed data recipients specify their interest in receiving information from ESPE800by subscribing to specific classes of events, while information sources publish events to ESPE800without directly addressing the receiving parties. ESPE800coordinates the interactions and processes the data. In some cases, the data source receives confirmation that the published information has been received by a data recipient.

A publish/subscribe API may be described as a library that enables an event publisher, such as publishing device1022, to publish event streams into ESPE800or an event subscriber, such as event subscribing device A1024a, event subscribing device B1024b, and event subscribing device C1024c, to subscribe to event streams from ESPE800. For illustration, one or more publish/subscribe APIs may be defined. Using the publish/subscribe API, an event publishing application may publish event streams into a running event stream processor project source window of ESPE800, and the event subscription application may subscribe to an event stream processor project source window of ESPE800.

The publish/subscribe API provides cross-platform connectivity and endianness compatibility between ESP application and other networked applications, such as event publishing applications instantiated at publishing device1022, and event subscription applications instantiated at one or more of event subscribing device A1024a, event subscribing device B1024b, and event subscribing device C1024c.

Referring back toFIG. 9, operation906initializes the publish/subscribe capability of ESPE800. In an operation908, the one or more projects802are started. The one or more started projects may run in the background on an ESP device. In an operation910, an event block object is received from one or more computing device of the event publishing device1022.

ESP subsystem800may include a publishing client1002, ESPE800, a subscribing client A1004, a subscribing client B1006, and a subscribing client C1008. Publishing client1002may be started by an event publishing application executing at publishing device1022using the publish/subscribe API. Subscribing client A1004may be started by an event subscription application A, executing at event subscribing device A1024ausing the publish/subscribe API. Subscribing client B1006may be started by an event subscription application B executing at event subscribing device B1024busing the publish/subscribe API. Subscribing client C1008may be started by an event subscription application C executing at event subscribing device C1024cusing the publish/subscribe API.

An event block object containing one or more event objects is injected into a source window of the one or more source windows806from an instance of an event publishing application on event publishing device1022. The event block object may generated, for example, by the event publishing application and may be received by publishing client1002. A unique ID may be maintained as the event block object is passed between the one or more source windows806and/or the one or more derived windows808of ESPE800, and to subscribing client A1004, subscribing client B1006, and subscribing client C1008and to event subscription device A1024a, event subscription device B1024b, and event subscription device C1024c. Publishing client1002may further generate and include a unique embedded transaction ID in the event block object as the event block object is processed by a continuous query, as well as the unique ID that publishing device1022assigned to the event block object.

In an operation912, the event block object is processed through the one or more continuous queries804. In an operation914, the processed event block object is output to one or more computing devices of the event subscribing devices1024a-c. For example, subscribing client A1004, subscribing client B1006, and subscribing client C1008may send the received event block object to event subscription device A1024a, event subscription device B1024b, and event subscription device C1024c, respectively.

ESPE800maintains the event block containership aspect of the received event blocks from when the event block is published into a source window and works its way through the directed graph defined by the one or more continuous queries804with the various event translations before being output to subscribers. Subscribers can correlate a group of subscribed events back to a group of published events by comparing the unique ID of the event block object that a publisher, such as publishing device1022, attached to the event block object with the event block ID received by the subscriber.

In an operation916, a determination is made concerning whether or not processing is stopped. If processing is not stopped, processing continues in operation910to continue receiving the one or more event streams containing event block objects from the, for example, one or more network devices. If processing is stopped, processing continues in an operation918. In operation918, the started projects are stopped. In operation920, the ESPE is shutdown.

As noted, in some embodiments, big data is processed for an analytics project after the data is received and stored. In other embodiments, distributed applications process continuously flowing data in real-time from distributed sources by applying queries to the data before distributing the data to geographically distributed recipients. As noted, an event stream processing engine (ESPE) may continuously apply the queries to the data as it is received and determines which entities receive the processed data. This allows for large amounts of data being received and/or collected in a variety of environments to be processed and distributed in real time. For example, as shown with respect toFIG. 2, data may be collected from network devices that may include devices within the internet of things, such as devices within a home automation network. However, such data may be collected from a variety of different resources in a variety of different environments. In any such situation, embodiments of the present technology allow for real-time processing of such data.

Aspects of the current disclosure provide technical solutions to technical problems, such as computing problems that arise when an ESP device fails which results in a complete service interruption and potentially significant data loss. The data loss can be catastrophic when the streamed data is supporting mission critical operations such as those in support of an ongoing manufacturing or drilling operation. An embodiment of an ESP system achieves a rapid and seamless failover of ESPE running at the plurality of ESP devices without service interruption or data loss, thus significantly improving the reliability of an operational system that relies on the live or real-time processing of the data streams. The event publishing systems, the event subscribing systems, and each ESPE not executing at a failed ESP device are not aware of or effected by the failed ESP device. The ESP system may include thousands of event publishing systems and event subscribing systems. The ESP system keeps the failover logic and awareness within the boundaries of out-messaging network connector and out-messaging network device.

In one example embodiment, a system is provided to support a failover when event stream processing (ESP) event blocks. The system includes, but is not limited to, an out-messaging network device and a computing device. The computing device includes, but is not limited to, a processor and a computer-readable medium operably coupled to the processor. The processor is configured to execute an ESP engine (ESPE). The computer-readable medium has instructions stored thereon that, when executed by the processor, cause the computing device to support the failover. An event block object is received from the ESPE that includes a unique identifier. A first status of the computing device as active or standby is determined. When the first status is active, a second status of the computing device as newly active or not newly active is determined. Newly active is determined when the computing device is switched from a standby status to an active status. When the second status is newly active, a last published event block object identifier that uniquely identifies a last published event block object is determined. A next event block object is selected from a non-transitory computer-readable medium accessible by the computing device. The next event block object has an event block object identifier that is greater than the determined last published event block object identifier. The selected next event block object is published to an out-messaging network device. When the second status of the computing device is not newly active, the received event block object is published to the out-messaging network device. When the first status of the computing device is standby, the received event block object is stored in the non-transitory computer-readable medium.

FIG. 11is a flow chart of an example of a process for generating and using a machine-learning model according to some aspects. Machine learning is a branch of artificial intelligence that relates to mathematical models that can learn from, categorize, and make predictions about data. Such mathematical models, which can be referred to as machine-learning models, can classify input data among two or more classes; cluster input data among two or more groups; predict a result based on input data; identify patterns or trends in input data; identify a distribution of input data in a space; or any combination of these. Examples of machine-learning models can include (i) neural networks; (ii) decision trees, such as classification trees and regression trees; (iii) classifiers, such as Naïve bias classifiers, logistic regression classifiers, ridge regression classifiers, random forest classifiers, least absolute shrinkage and selector (LASSO) classifiers, and support vector machines; (iv) clusterers, such as k-means clusterers, mean-shift clusterers, and spectral clusterers; (v) factorizers, such as factorization machines, principal component analyzers and kernel principal component analyzers; and (vi) ensembles or other combinations of machine-learning models. In some examples, neural networks can include deep neural networks, feed-forward neural networks, recurrent neural networks, convolutional neural networks, radial basis function (RBF) neural networks, echo state neural networks, long short-term memory neural networks, bi-directional recurrent neural networks, gated neural networks, hierarchical recurrent neural networks, stochastic neural networks, modular neural networks, spiking neural networks, dynamic neural networks, cascading neural networks, neuro-fuzzy neural networks, or any combination of these.

Different machine-learning models may be used interchangeably to perform a task. Examples of tasks that can be performed at least partially using machine-learning models include various types of scoring; bioinformatics; cheminformatics; software engineering; fraud detection; customer segmentation; generating online recommendations; adaptive websites; determining customer lifetime value; search engines; placing advertisements in real time or near real time; classifying DNA sequences; affective computing; performing natural language processing and understanding; object recognition and computer vision; robotic locomotion; playing games; optimization and metaheuristics; detecting network intrusions; medical diagnosis and monitoring; or predicting when an asset, such as a machine, will need maintenance.

Any number and combination of tools can be used to create machine-learning models. Examples of tools for creating and managing machine-learning models can include SAS® Enterprise Miner, SAS® Rapid Predictive Modeler, and SAS® Model Manager, SAS Cloud Analytic Services (CAS)®, SAS Viya® of all which are by SAS Institute Inc. of Cary, N.C.

Machine-learning models can be constructed through an at least partially automated (e.g., with little or no human involvement) process called training. During training, input data can be iteratively supplied to a machine-learning model to enable the machine-learning model to identify patterns related to the input data or to identify relationships between the input data and output data. With training, the machine-learning model can be transformed from an untrained state to a trained state. Input data can be split into one or more training sets and one or more validation sets, and the training process may be repeated multiple times. The splitting may follow a k-fold cross-validation rule, a leave-one-out-rule, a leave-p-out rule, or a holdout rule. An overview of training and using a machine-learning model is described below with respect to the flow chart ofFIG. 11.

In block1104, training data is received. In some examples, the training data is received from a remote database or a local database, constructed from various subsets of data, or input by a user. The training data can be used in its raw form for training a machine-learning model or pre-processed into another form, which can then be used for training the machine-learning model. For example, the raw form of the training data can be smoothed, truncated, aggregated, clustered, or otherwise manipulated into another form, which can then be used for training the machine-learning model.

In block1106, a machine-learning model is trained using the training data. The machine-learning model can be trained in a supervised, unsupervised, or semi-supervised manner. In supervised training, each input in the training data is correlated to a desired output. This desired output may be a scalar, a vector, or a different type of data structure such as text or an image. This may enable the machine-learning model to learn a mapping between the inputs and desired outputs. In unsupervised training, the training data includes inputs, but not desired outputs, so that the machine-learning model has to find structure in the inputs on its own. In semi-supervised training, only some of the inputs in the training data are correlated to desired outputs.

In block1108, the machine-learning model is evaluated. For example, an evaluation dataset can be obtained, for example, via user input or from a database. The evaluation dataset can include inputs correlated to desired outputs. The inputs can be provided to the machine-learning model and the outputs from the machine-learning model can be compared to the desired outputs. If the outputs from the machine-learning model closely correspond with the desired outputs, the machine-learning model may have a high degree of accuracy. For example, if 90% or more of the outputs from the machine-learning model are the same as the desired outputs in the evaluation dataset, the machine-learning model may have a high degree of accuracy. Otherwise, the machine-learning model may have a low degree of accuracy. The 90% number is an example only. A realistic and desirable accuracy percentage is dependent on the problem and the data.

In some examples, if the machine-learning model has an inadequate degree of accuracy for a particular task, the process can return to block1106, where the machine-learning model can be further trained using additional training data or otherwise modified to improve accuracy. If the machine-learning model has an adequate degree of accuracy for the particular task, the process can continue to block1110.

In block1110, new data is received. In some examples, the new data is received from a remote database or a local database, constructed from various subsets of data, or input by a user. The new data may be unknown to the machine-learning model. For example, the machine-learning model may not have previously processed or analyzed the new data.

In block1112, the trained machine-learning model is used to analyze the new data and provide a result. For example, the new data can be provided as input to the trained machine-learning model. The trained machine-learning model can analyze the new data and provide a result that includes a classification of the new data into a particular class, a clustering of the new data into a particular group, a prediction based on the new data, or any combination of these.

In block1114, the result is post-processed. For example, the result can be added to, multiplied with, or otherwise combined with other data as part of a job. As another example, the result can be transformed from a first format, such as a time series format, into another format, such as a count series format. Any number and combination of operations can be performed on the result during post-processing.

A more specific example of a machine-learning model is the neural network1200shown inFIG. 12. The neural network1200is represented as multiple layers of interconnected neurons, such as neuron1208, that can exchange data between one another. The layers include an input layer1202for receiving input data, a hidden layer1204, and an output layer1206for providing a result. The hidden layer1204is referred to as hidden because it may not be directly observable or have its input directly accessible during the normal functioning of the neural network1200. Although the neural network1200is shown as having a specific number of layers and neurons for exemplary purposes, the neural network1200can have any number and combination of layers, and each layer can have any number and combination of neurons.

The neurons and connections between the neurons can have numeric weights, which can be tuned during training. For example, training data can be provided to the input layer1202of the neural network1200, and the neural network1200can use the training data to tune one or more numeric weights of the neural network1200. In some examples, the neural network1200can be trained using backpropagation. Backpropagation can include determining a gradient of a particular numeric weight based on a difference between an actual output of the neural network1200and a desired output of the neural network1200. Based on the gradient, one or more numeric weights of the neural network1200can be updated to reduce the difference, thereby increasing the accuracy of the neural network1200. This process can be repeated multiple times to train the neural network1200. For example, this process can be repeated hundreds or thousands of times to train the neural network1200.

In some examples, the neural network1200is a feed-forward neural network. In a feed-forward neural network, every neuron only propagates an output value to a subsequent layer of the neural network1200. For example, data may only move one direction (forward) from one neuron to the next neuron in a feed-forward neural network.

In other examples, the neural network1200is a recurrent neural network. A recurrent neural network can include one or more feedback loops, allowing data to propagate in both forward and backward through the neural network1200. This can allow for information to persist within the recurrent neural network. For example, a recurrent neural network can determine an output based at least partially on information that the recurrent neural network has seen before, giving the recurrent neural network the ability to use previous input to inform the output.

In some examples, the neural network1200operates by receiving a vector of numbers from one layer; transforming the vector of numbers into a new vector of numbers using a matrix of numeric weights, a nonlinearity, or both; and providing the new vector of numbers to a subsequent layer of the neural network1200. Each subsequent layer of the neural network1200can repeat this process until the neural network1200outputs a final result at the output layer1206. For example, the neural network1200can receive a vector of numbers as an input at the input layer1202. The neural network1200can multiply the vector of numbers by a matrix of numeric weights to determine a weighted vector. The matrix of numeric weights can be tuned during the training of the neural network1200. The neural network1200can transform the weighted vector using a nonlinearity, such as a sigmoid tangent or the hyperbolic tangent. In some examples, the nonlinearity can include a rectified linear unit, which can be expressed using the equation y=max(x, 0) where y is the output and x is an input value from the weighted vector. The transformed output can be supplied to a subsequent layer, such as the hidden layer1204, of the neural network1200. The subsequent layer of the neural network1200can receive the transformed output, multiply the transformed output by a matrix of numeric weights and a nonlinearity, and provide the result to yet another layer of the neural network1200. This process continues until the neural network1200outputs a final result at the output layer1206.

Other examples of the present disclosure may include any number and combination of machine-learning models having any number and combination of characteristics. The machine-learning model(s) can be trained in a supervised, semi-supervised, or unsupervised manner, or any combination of these. The machine-learning model(s) can be implemented using a single computing device or multiple computing devices, such as the communications grid computing system400discussed above.

Implementing some examples of the present disclosure at least in part by using machine-learning models can reduce the total number of processing iterations, time, memory, electrical power, or any combination of these consumed by a computing device when analyzing data. For example, a neural network may more readily identify patterns in data than other approaches. This may enable the neural network to analyze the data using fewer processing cycles and less memory than other approaches, while obtaining a similar or greater level of accuracy.

FIGS. 13A and 13Billustrates a block diagram of an example embodiment of a distributed online library system2000incorporating multiple vendor devices2200, one or more provider devices2400, a broker device2500and/or one or more client devices2800coupled by a network2999.FIGS. 14A and 14Billustrates a block diagram of an alternate example embodiment of the distributed online library system2000in which the broker device2500may perform the functions of the one or more provider devices2400. In both of the embodiments ofFIGS. 13A-Band14A-B, the distributed online library system2000provides personnel who operate the one or more client devices2800with access to data from data sets2130that may be stored by the multiple vendor devices2200. Additionally, in both of these embodiments, the distributed online library system2000also provides at least selected persons who operate the one or more client devices2800with visualizations of relationships that may exist among portions of data of the stored data sets2130.

In support of such operations, the devices2200,2400,2500and/or2800may exchange one or more queries (e.g., the depicted query data2732and/or translated query data2332), one or more portions of metadata (e.g., the depicted metadata portions2336and/or normalized metadata portions2436), and/or data representing one or more visualizations (e.g., the depicted visualization data2738). In various embodiments, the network2999may be a single network that may extend within a single building or other relatively limited area, a combination of connected networks that may extend a considerable distance, and/or may include the Internet. Thus, the network2999may be based on any of a variety (or combination) of communications technologies by which communications may be effected, including without limitation, wired technologies employing electrically and/or optically conductive cabling, and wireless technologies employing infrared, radio frequency (RF) or other forms of wireless transmission.FIG. 15illustrates a simplified block diagram of such operations and exchanges through the network2999to generate of such visualizations in response to queries originating at client device(s)2800that request such visualizations.

Turning toFIGS. 13A-B, as well as toFIG. 15, in various embodiments, each of the vendor devices2200may incorporate one or more of a processor2250, a storage2260, and a network interface2290to couple each of the vendor devices2200to the network2999. Within each of the vendor devices2200, the storage2260may store a control routine2240, and/or one or more data sets2130. Additionally, separate metadata2136may also be stored within the storage2260for each data set2130that is stored therein. Further, as depicted, in some embodiments, each data set2130stored therein may incorporate its corresponding metadata2136as a component of the data set2130. The control routine2240may incorporate a sequence of instructions operative on the processor2250of each of the vendor devices2200to implement logic to perform various functions, at least partially in parallel with the processors2250of others of the vendor devices2200.

Within each of the vendor devices2200, in executing the control routine2240, the processor2250may operate the network interface2290to await the receipt, from a provider device2400, of a query for information from a data set2130and/or from corresponding metadata2136. In response to the query, and based on the information that is requested therein, the processor2250may search one or more data sets2130and/or corresponding metadata2136to identify portions of data set(s)2130and/or portions of corresponding metadata2136that answer the query, and may then be caused to operate the network interface2290to transmit the identified portions of data set(s)2130and/or the identified portions of corresponding metadata2136back to the provider device2400from which the query was received.

It should be noted that, although the vendor devices2200are depicted as directly storing data set(s)2130and/or associated metadata2136within the storage2260, as will be explained in greater detail, there may be other embodiments in which one or more data sets2130and/or associated metadata2136may be stored within one or more separate storage devices with which one or more vendor devices2200may be in communication via the network2999. In such other embodiments, such vendor device(s)2200may store pointers and/or other information required for communications with such storage devices2100within the storage2260.

In various embodiments, each of the provider devices2400may incorporate one or more of a processor2450, a storage2460, and a network interface2490to couple each of the provider devices2400to the network2999. Within each of the one or more provider devices2400, the storage2460may store a control routine2440, rules data2435, the translated query data2332, the metadata portions2336and/or the normalized metadata portions2436. The control routine2440may incorporate a sequence of instructions operative on the processor2450of each of the provider devices2400to implement logic to perform various functions, at least partially in parallel with the processors2450of other provider devices2400in embodiments of the distributed online library system2000that include more than one of the provider devices2400.

In executing the control routine2440, the processor2450of each provider device2400may operate the network interface2490to await the receipt, from the broker device2500, of a query for information from data sets2130and/or corresponding metadata2136that are stored by and/or under the control of the multiple vendor devices2200. The processor2450may be caused to translate such a query from one communications protocol by which the query was received to another (which may be at least temporarily stored as the translated query data2332), and may then be caused to operate the network interface2490to relay the translated query onward to multiple vendor devices2200. The processor2450may then be caused to operate the network interface2490to await the receipt of portions of data sets2130and/or portions of corresponding metadata2136from a subset of the multiple vendor devices2200to which the processor2450relayed the translated query. Where portions of metadata are received, the processor2450may be caused to translate the received metadata portions2336from one communications protocol by which they were received to another (which may be at least temporarily stored as the normalized metadata portions2436), and may then be caused to relay the normalized metadata portions2436to the broker device2500.

In some embodiments, execution of the control routine2440by the processor2450may cause the processor2450to analyze access credentials of a client device2800and/or an operator of a client device2800from which the query originates to determine whether the query is authorized. As previously discussed, such an authorization check may be in addition to an authorization check that may be performed at the broker device2500, and may be based on different criteria, such as a limitation on the number of users that may be supported at a time, rather than on the identities of those users.

Also, as previously discussed, differences in what information is included in the metadata of one vendor device2200versus the metadata of another vendor device2200may result in the metadata portions2336that are provided by one vendor device2200lacking in one or more pieces of information concerning its corresponding data set2130that may be present in the metadata portions2336that are provided by another vendor device2200, and which may be deemed useful in identifying relationships between corresponding portions of data. Thus, in a subset of the provider devices2400in some embodiments, execution of the control routine2440by the processor2450may cause the processor2450to, in response to the receipt of the metadata portions2336, generate further queries for further information from the portions of data that correspond to each of the metadata portions2336, and to transmit such further queries back to the vendor device(s)2200from which the metadata portions2336are received. Upon subsequently receiving responses from those vendor devices to the further queries, the processor2450may be caused to augment the normalized metadata portions2436with further information based on those received responses to the further queries before transmitting the normalized metadata portions2436to the broker device2500.

This is one of the ways in which at least a subset of the metadata portions2336may be “normalized” to generate the corresponding normalized metadata portions2436. Stated differently, through such use of further queries, each of the normalized metadata portions2436of the set of normalized metadata portions2436that are ultimately provided to the broker device2500are caused to contain similar pieces of information. As will be explained in greater detail, the fact of each normalized metadata portion2436having similar pieces of information to the others enables various comparative analyses to be performed therebetween as part of identifying relationships among corresponding portions of data. As will also be explained, further normalization operations that may be performed in generating the normalized metadata portions2436from the metadata portions2336may include normalization of languages to a selected language or selected set of languages, vocabulary to selected vocabulary, formatting to a selected set of formats, data types to a selected set of data types, data widths to a selected set of data widths, etc. used for each of the pieces of information that are to be included within each normalized metadata portion2436to better enable those pieces of information to be used in such comparative analyses.

In various embodiments, the broker device2500may incorporate one or more of a processor2550, a storage2560, and a network interface2590to couple each of the broker device2500to the network2999. The storage2560may store a control routine2540, rules data2535, the normalized metadata portions2436, a visualization log2638, and/or the visualization data2738. The control routine2540may incorporate a sequence of instructions operative on the processor2550to implement logic to perform various functions.

In executing the control routine2540, the processor2550may operate the network interface2590to await the receipt, from one of the client devices2800, of a query for information from data sets2130and/or corresponding metadata2136that are stored by and/or under the control of the multiple vendor devices2200. The processor2550may be caused operate the network interface2590to relay the query onward to the one or more provider devices2400. The processor2550may then be caused to operate the network interface2590to await the receipt of portions of data sets2130and/or normalized data set portions2436from one or more provider devices2400. As will be explained in greater detail, the processor2550may be caused to perform various analyses with each of the normalized metadata portions2436to derive further characteristics of each portion of data of a data set2130that corresponds to one of the normalized metadata portions2436. The processor2550may be further caused to perform comparisons of various data characteristics of indicated by the normalized metadata2436and/or the derived further characteristics thereof to identify relationships that may be present among the corresponding portions of data. As will also be explained in greater detail, the processor2550may be caused to generate one or more visualizations of the identified relationships (which may be stored as the visualization data2738). Following the generation of such visualizations, the processor2550may operate the network interface2590to transmit those visualizations (e.g., transmit the visualization data2738) to the client device2800from which the query corresponding query was originally received.

In some embodiments, execution of the control routine2540by the processor2550may cause the processor2550to analyze access credentials of the client device2800from which the corresponding query was originally received to determine whether the query is authorized. As previously discussed, such an authorization may entail a check of whether the client device2800and/or a person operating the client device2800is associated with the entity that may maintain the distributed online library system2000.

Turning toFIGS. 14A-B, as well as toFIG. 15, an alternate embodiment is presented the processor2550of the broker device2500is caused by its execution of both the control routines2440and2540to perform substantially the same operations just described as being separately performed by the processors2450and2550of the embodiment ofFIGS. 13A-B. More specifically, each provider device2400that was implemented as a separate and distinct physical device in the embodiment ofFIGS. 13A-B, is instead, implemented with a VM2466instantiated within the broker device2500in the embodiment ofFIGS. 14A-B.

Thus, while each of such virtual versions of a provider device2400within a VM2466may still perform the same authorization checking operations, translation operations, and/or further query generation operations, the exchanges of the query data2732and normalized metadata portions2436with the broker device2500do not actually occur through the network2999. Instead, such network transmissions may be virtual in nature as such data is provided to and/or received from each of the VMs2466.

FIGS. 16A, 16B, 16C, 16D, 16E, 16F and 16G, together and in greater detail, illustrate an example of an embodiment of identifying relationships among portions of data through analyses and comparisons of corresponding portions of metadata, and generating visualizations of the identified relationships.FIGS. 16A and 16B, together, illustrate aspects of the reception, authorization checking, translation and broadcasting of a query that includes a request for a visualization of relationships among portions of data relevant to a specified subject.FIG. 16Cillustrates aspects of searching at least the metadata2136associated with various data sets2130to identify portions of data therein that are relevant to the subject of the query, and retrieving associated portions of metadata for analysis.FIG. 16Dillustrates aspects of generating further queries to obtain further pieces of information about at least some of the identified portions of data that are not already provided in the retrieved metadata portions2336.FIGS. 16E, 16F and 16G, together, illustrate aspects of the collection, translation and relaying of portions of metadata for analysis and comparison to identify relationships that are then depicted in a visualization2888for display.

Referring toFIGS. 16A-G, as recognizable to those skilled in the art, the control routines2240,2440,2540and2740, including the components of which each is composed, are selected to be operative on whatever type of processing component(s) that are selected to implement applicable ones of the processors2250,2450,2550and/or2750. In various embodiments, each of these routines may include one or more of an operating system, device drivers and/or application-level routines (e.g., so-called “software suites” provided on disc media, “applets” obtained from a remote server, etc.). Where an operating system is included, the operating system may be any of a variety of available operating systems appropriate for the processors2250,2450,2550and/or2750. Where one or more device drivers are included, those device drivers may provide support for any of a variety of other components, whether hardware or software components, of the vendor devices2200, the provider devices2400, the broker device2500and/or the client devices2800(or of VMs to implement any of these devices in virtual form).

Turning more specifically toFIG. 16A, as depicted, the control routine2840of an example embodiment of one of the client devices2800may include a user interface (UI) component2848that may be operable on the processor2850thereof to provide a UI for the entry of queries for processing by the distributed online library system2000. More specifically, the UI component2848may operate a display2880and/or an input device2820of the depicted client device2800to provide a text-based or graphical UI by which an operator thereof may manually enter aspects of a query. As has been discussed, the queries that may be entered may include queries to retrieve one or more portions of data from one or more of the data sets2130about a subject specified in the query, such as, for example, a query for data concerning the production of oranges within a particular specified region of the world throughout a particular specified period of time. As has been discussed, the distributed online library system2000may be maintained by a scholastic entity (e.g., a college or university), a commercial entity (e.g., any of a wide variety of business entities), and/or a governmental entity (e.g., a government department) for the purpose of enabling personnel associated with that entity to enter such queries to cause searches to be made of the data sets2130to obtain answers to such subject-related questions.

However, as has also been discussed, the queries that may be entered may include queries to ascertain aspects of the quality of the information provided by the distributed online library system2000in relation to a specified subject, such as, for example, a query for a visualization2888depicting the number of distinct portions of data that are available among the data sets2130concerning orange production and the nature of any relationships that may exist among those portions of data, including duplications of such portions of data, multiple dependencies on a single portion one of such portions of data, inclusions of one of such portions within another, and/or instances of overlap in information provided by such portions of data. More specifically, curator personnel of the entity that maintains the system2000may enter such queries as part of an effort to determine whether the distributed online library system2000has a sufficient breadth of information on the specified subject as to enable such best practices as cross-checking of data based on multiple sources, and/or consideration of a variety of opinions where there may be ongoing debate concerning some aspect of the specified subject. Again, among the situations that such curator personnel seek to detect by such analyses may also be situations in which bottlenecks may have become built into the distributed online library system2000in which what appears to be multiple sources of data are, in truth, multiple pathways to a single source of data such that there is a risk of saturation in accessing a single source of data, thereby leading to loss of reliability in accessing data, at all, and/or leading to lengthy delays in accessing data.

Focusing on such queries for visualizations2888of such relationships, the UI component2848may operate the display2880(or another output device, such as a speaker) to prompt an operator of the client device2800through manually entering the various pieces of information that may be needed to fully describe the parameters of such a query via the input device2820. By way of example, such a query may include indications of the subject of interest and/or any limitations on the types of relationships to be included in the visualization(s)2888to be generated. Alternatively or additionally, the parameters of such a query that may be so entered may include specifications limiting aspects of the search that is to be performed to identify portions of data that are relevant to the subject of interest (e.g., limiting the search to portions of data in which the subject of interest is found in row or column labels, or is found in documents of a particular format, etc.). Such information as may be so provided by the operator may be at least temporarily stored by the UI component2848as the query data2732. As depicted, in some embodiments, the UI component2848may additionally include an indication of the access credentials of the client device2800and/or the operator thereof within the query data2732. Where such access credentials are in some way associated with the specific operator, the UI component may prompt the operator to manually enter such credentials via the input device2820for inclusion in the query data2732. With the parameters of the query so gathered and stored, the UI component2848may transmit the query data2732to the broker device2500via the network2999.

As also depicted inFIG. 16A, the control routine2540of an example embodiment of the broker device2500may include a relay component2541that may be operable on the processor2550thereof to selectively relay the query (e.g., selectively relay the query data2732) to the provider device(s)2400based on whether or not the query is authorized. More specifically, the relay component2541may analyze the access credentials of the client device2800and/or the operator thereof to determine whether or not the query should be relayed to the provider devices2400as part of conveying the query to the vendor devices2200to be acted upon. In so doing, the relay component2541may compare the access credentials to indications stored in the depicted authorization data2531of what devices and/or persons are authorized to enter queries requesting details of relationships between portions of data. As has been discussed, the entry of queries may generally be limited to personnel and/or devices that are associated with the entity that maintains the distributed online library system2000. Alternatively or additionally, the entry of queries concerning relationships between portions of data may be limited to a selected subset of such personnel (e.g., those designated as curators of the system2000) and/or particular devices that may be under the control of such a subset of personnel.

If the relay component2541determines that the query is not authorized, then the relay component2541may transmit an indication of such lack of authorization back to the client device2800via the network2999, where the UI component2848may operate the display2880to provide a visual indication of the lack of authorization. However, presuming that the query (as received in the form of the query data2732) is authorized, the relay component2541may then relay the query via the network2999to each of the provider devices2400that may be present within the distributed online library system2000.

As has been previously discussed, in some embodiments, each of the provider devices2400to which the broker device2500may so relay the query may be required to have been registered with the broker device2500beforehand. Similarly, each vendor device2200to which a provider device may relay the query may also be required to have been registered with that provider device2400beforehand. Such a registration process may entail the provision and/or exchange of at least sufficient communications information as to enable at least one of the devices that engage each other in a registration procedure to successfully contact the other and exchange information with the other through the network2999(e.g., IP address, URL, login credentials, network encryption keys, communications protocol details, etc.).

Turning more specifically toFIG. 16B, as depicted, the control routine2440of an example embodiment of a provider device2400may include a query translation component2442that may be operable on the processor2450thereof to selectively translate and relay the query (e.g., selectively relay the query data2732) to the multiple vendor devices2200that have been registered with the provider device2400based on whether or not the query is authorized. More specifically, the query translation component2442may analyze the access credentials of the client device2800and/or the operator thereof to determine whether or not the query should be translated and relayed to the multiple vendor devices2200to be acted upon. In so doing, the relay component2541may compare the access credentials to indications stored in the depicted authorization data2431of what devices and/or persons are authorized to enter queries requesting details of relationships between portions of data. Alternatively or additionally, and as has been discussed, the query translation component2442may apply other criteria in determining whether the query is authorized, including and not limited to, whether a predetermined limit on a quantity of users and/or devices currently making queries of the distributed online library system2000has already been reached.

If the query translation component2442determines that the query is not authorized, then the query translation component2442may transmit an indication of such lack of authorization back to the broker device2500via the network2999, which may relay that indication back to the client device2800from which the query originated, where the UI component2848thereof may operate the display2880thereof to provide a visual indication of the lack of authorization. However, presuming that the query (as received in the form of the query data2732) is authorized, the query translation component2442may then translate the query between two different communications protocols.

More specifically, and as previously discussed, the communications protocol by which the query is communicated to the provider device2400by the broker device2500via the network2999may differ from the communications protocol by which the provider device2400may relay the query to the multiple vendor devices2200via the network2999. By way of example, the different communications protocols may employ different network protocols (e.g., different network ports, different packetizing, etc.). Also by way of example, the manner in which the parameters of the query are expressed within the query data2732may conform to the communications protocol used between the broker device2500and the provider device2400, but not between the provider device2400and the vendor devices2200that are registered with the provider device2400(e.g., different syntax, different formats, different query description languages, differences in sets of available logical operators, etc.).

Therefore, presuming that the query translation component2442determines that the query is authorized, the query translation component2442may generate the translated query data2332, in which the query is expressed in a different manner that may conform to the communications protocol used between the provider device2400and the vendor devices2200that are registered with the provider device2400. In so doing, the query translation component2442may retrieve indications of various rules for performing such a conversion of the query from the rules data2435, which may include rules specifying such aspects of conversion as protocol, syntax, formatting, equivalent logical operators and/or equivalent vocabulary. The query translation component2442may then relay the query (as the translated query data2332) via the network2999to each of the vendor devices2200that are registered with the provider device2400.

As previously discussed, the entity that maintains and operates the distributed online library system2000may engage one or more other entities (e.g., one or more “providers”) to provide at least some of the data that becomes part of the overall set of data that is included in the system2000. Thus, while the entity that maintains and operates the distributed online library system2000may directly exercise control over many of the devices2200,2400,2500and/or2800that make up the system2000, at least a subset of the provider devices2400may be under the control of one or more of such providers who agree under terms of a license or other form agreement to grant access, through a provider device2400, to data stored on one or more vendor devices2200for the benefit of operators of the client devices2800. It is from this difference in ownership and/or control over various ones of these devices that the need may arise for translations between communications protocols. More specifically, with so many different communications protocols having been developed over decades for the communication of data, as well as for the communication of metadata about data, considerable opportunity exists (indeed, a likelihood exists) that the communications protocol employed by a provider in the operation of their provider device2400in communications with vendor devices2200will be different from the communications protocol employed by the entity that maintains the system2000in the operation of their broker device2800in communications with the provider devices2400.

As also previously discussed, an entity that maintains and operates one of the provider devices2400may also operate at least a subset of the provider devices2200with which their provider device2400is in communication. However, as has also been previously discussed, it may be that an entity that maintains and operates one of the provider devices2400may, themselves, engage in licensing and/or other agreements with still other entities that maintain and operate at one or more of the vendor devices2200that are to store the data and/or pointers to data (e.g., one or more “vendors”). It may be that a provider configures a particular provider device2400to provide data for a particular distributed online library system2000, at least in part, by selecting particular vendor devices2200(and accordingly, selects one or more vendors) to be put coupled to that particular provider device2400based on what types data each of those vendor devices2200store (or at least store pointers to). Thus, it may be that neither the entity that operates the distributed online library system2000, nor the one or more entities that may operate the one or more provider devices2400thereof, are aware that a situation may have been built into the system2000in which what appeared to be unrelated pieces of data stored on separate vendor devices2000about a similar topic are all actually the very same piece of data stored on a single vendor device2000or a single other device, thereby creating the illusion of redundancy and/or variety in the storage of data where there is in truth a bottleneck.

Turning more specifically toFIG. 16C, as depicted, the control routine2240of an example embodiment of a vendor device2400may include a search component2242that may be operable on the processor2250thereof to perform a search of at least the metadata2136associated with each of one or more data sets2130that are stored by the vendor device2200to identify portions of data within the one or more data sets2130that meet the criteria specified in the translated query data2332, including being relevant to the subject of interest described therein. Additionally, in some embodiments, the search component2242may condition its performance on such a search on a determination of whether the query is authorized based on criteria similar to what has been previously discussed in regard to the relay component2541of the broker device2500and the query translation component2442of the provider device2400.

The data sets2130may vary greatly in size and complexity such that one data set2130may be a single relatively simple data structure such as a text document, while another data set2130may be include numerous data structures of widely varying types. However, it is envisioned that the vendor devices2200may each store one or more data sets2130that may each be of considerable size such that each data set2130may include numerous different data structures and/or data structures of considerable size and complexity. Thus, at least some data sets2130may be divisible into numerous portions that may each include a separate data structure of some size and/or of some complexity, including and not limited to, separate documents, separate spreadsheets, separate n-dimensional arrays, separate audio and/or visual recording, separate still images, etc.

As previously discussed, each data set2130may be accompanied by, and may even directly incorporate, a corresponding metadata2136that describes numerous aspects of its corresponding data set2130, along with numerous aspects of each of the data structures that may be present within its corresponding data set2130, and along with numerous aspects of the data that may be present within each of those data structures. More specifically, metadata2136associated with a particular data set2130may describe aspects of the data set2130, such as and not limited to, its file name, its location within a file system directory hierarchy, its location on a network (e.g., IP address, URL, etc.), its file type and/or the revision of the standard for that file type, the date/time on which it as created and/or was last modified, its current overall size, the identity of its creator and/or the last person/entity to modify it, any applicable copyright and/or applicable country's law, the date/time on which it was published, and/or a publication identifier that may assigned to it (e.g., an ISBN number). Alternatively or additionally, such metadata2136may describe aspects of data structure(s) that may be present within its corresponding data set2130, such as and not limited to, data structure type (e.g., n-dimensional array, text document, spreadsheet conforming to a particular standard, etc.), type of indexing scheme (e.g., rows and columns in a table, combination of chapters, sections and/or pages of a document, etc.), type of labels used in the indexing scheme (e.g., textual labels, a numeric index, etc.), type of ordering of the index labels (e.g., alphabetical, numeric, etc.), ordered listing of index labels used, current size of data structures and/or size limitations thereof, current number of dimensions of an n-dimensional data structure and/or limitations thereof, etc. Also alternatively or additionally, such metadata2136may describe aspects of the data within each data structure within its corresponding data set2130, such as and not limited to, data types (e.g., text characters, numerical values, logical bit-wise values, undefined n-byte sized fields, etc.), data widths (e.g., bit, byte, word, double-word, quad-word, n-bit floating point, signed or unsigned n-bit integer, etc.).

However, as has also been previously discussed, due to the wide variety of operating systems, file systems, database storage and retrieval tools, data set structure standards, data types predefined by each of numerous programming languages, etc., different metadata2136may include widely differing combinations of details concerning its corresponding data set2130. By way of example, the metadata2136of one data set2130may include a considerable number of details about data structures and/or data types within its data set2130, but may include minimal information concerning the file type of its data set2130or its location within a file system directory hierarchy. In contrast, the metadata2136of a different data set2130may include fuller details concerning the file type and/or other details of how and/or where its data set2130is stored, but may include minimal details about the indexing scheme used to organize data values within a multi-dimensional array within its data set2130.

Upon receiving the translated query data2332, the search component2242may perform searches of the metadata2136associated with each of multiple data sets2130to identify one or more portions of data within each such data set2130that are deemed relevant to the subject of interest specified in the translated query data2332and/or that meet other criteria that may also be specified therein. Upon identifying one or more of such portions of one or more data set2130, the search component2242may transmit portions of metadata2136(each in the form of one of the depicted data set portions2336) that correspond to such portions of data back to the provider device from which the translated query data2332was received via the network2999.

As those skilled in the art will readily recognize, depending on what the translated query data2332describes as the subject of interest and/or any limitations placed thereon, and depending on what information is included in the available metadata2136, it may or may not be possible for the search component2242to limit its search to the contents of the available metadata2136. By way of example, if the translated query data2332specifies that the search is to be limited to portions of data sets2130that include labels that include the subject of interest (e.g., production of oranges, social security numbers, street addresses, etc.), and if the available metadata2136includes all of the labels used throughout all of the corresponding data sets2130, then the search to identify such portions of data may be limited to the searching through the labels specified in the available metadata such that no search may be needed of any of the corresponding data sets2130. However, if by way of another example, the translated query data2332specifies that the search is to include any portion of a data set2130in which there is text of any kind that includes a mention of the subject of interest, then the search to identify such portions of data necessarily entails a search through at least all text that may be present within each data set2130for such a mention of the subject of interest. However, even in such an example where a search through the contents of data sets2130is still necessary, a search through the available metadata2136may still be performed as an initial step to at least rule out any data set2130that does not contain any text, at all.

Regardless of the extent and/or strategies employed in performing the searches, due to the above-described wide variation in the information that may be included in each metadata2136, the data set portions2336that are so transmitted to the depicted provider device2400may similarly vary widely in the information that each contains about its corresponding portion of a data set2130. As a result, one or more follow-up searches may be required to obtain more information about portions of data that have been identified as relevant to the subject of interest, as well as meeting any other criteria that may be specified in the translated query data2332.

It should be noted that, considerable efficiencies are realized by the performances of the searches within both metadata2136and corresponding data sets2130within the vendor devices2200, including the use of parallelism thereamong in the performances of these searches. Also, it may be that at least some of the data sets2130are updated on a frequent basis, thereby making the maintenance of copies of at least the metadata2136of each data set2130within the broker device2500impractical, as there would be considerable consumption of processing resources across multiple devices and considerable consumption of network communications bandwidth to recurringly update those copies.

Turning more specifically toFIG. 16D, as depicted, the control routine2440of the example embodiment of provider device2400introduced inFIG. 16Bmay also include a further query component2443that may be operable on the processor2450thereof to analyze each metadata portion2336to determine what pieces of information are not included therein such that a further query may be needed. More specifically, the further query component2443may retrieve indications, from the rules data2435, of what pieces of information are to be retrieved and included in portions of metadata that are to be relayed back to the broker device2500to better enable the identification of relationships. Thus, where one or more received metadata portions2336do not include one or more of such specified pieces of information for one or more corresponding identified portions of data, the further query component2443may generate the depicted further query data2333that specifies the parameters of a further search to be performed within the one or more identified portions of data to obtain those missing pieces of information.

By way of example, where the rules data specifies that the size of each dimension of a multi-dimensional array in an identified portion of data is among the pieces of information to be retrieved and included in metadata transmitted to the broker device2500, and yet, the available metadata does not include such information, then further query component2443may generate the further query data2333to specify the performance of a search of the contents of each such array that may be present in any of the identified portions of data to determine the size of each dimension thereof. The further query component2443may then transmit the further query data2333to applicable ones of the vendor devices2200(including the depicted example vendor device2200), and may await receipt of response(s) thereto.

FIG. 16Dalso depicts aspects of an alternate embodiment of the depicted vendor device2200in which at least a subset of the data sets2130may be stored within one or more separate and distinct source devices2100, instead of within the depicted vendor device2200. As also depicted, the metadata2136that corresponds to such separately stored data sets2130may alternatively or additionally be stored within the vendor device2200to improve the speed with which it is able to be accessed, and in recognition of the metadata2136usually being a fraction of the size of its corresponding data set2130. In such embodiments, such metadata2136that is stored so separately from the data set(s)2130that it corresponds to may be maintained in a distinct metadata log2236(seeFIG. 15), which may, in some embodiments, function effectively as a “cache” of such metadata2136. Alternatively or additionally, the vendor device2200may store one or more pointers and/or other information required to enable access by the vendor device2200to such separate storage device(s)2100. Such indirect storage of data set(s)2130within separate and distinct storage device(s)2100may be done as a way to expand the storage capacity of the vendor device2200. Alternatively or additionally, the entity that maintains one or more of the vendor devices2200(which may or may not be the same entity that maintains at least one of the provider devices2400) may engage in their own licensing of data from still other entities that may maintain such storage devices2100.

Turning more specifically toFIG. 16E, as depicted, the control routine2440of the example embodiment of provider device2400introduced inFIG. 16B, may further include a metadata translation component2446that may be operable on the processor2450thereof to translate and relay at least the received metadata portions2336to the broker device2500. More specifically, and in a manner similar to the earlier described translation of the query by the query translation component2442, the metadata translation component2446may translate each of the received metadata portions2336between two different communications protocols.

As previously discussed, the communications protocol by which portions of metadata are communicated to the provider device2400by each of the vendor devices2200via the network2999may differ from the communications protocol by which the provider device2400may relay portions of metadata to the broker device2500via the network2999. Again, and as previously discussed by way of example, the different communications protocols may employ different network protocols (e.g., different network ports, different packetizing, etc.). Also by way of example, the manner in which the pieces of information about a corresponding portion of data are expressed within each of the metadata portions2336may conform to the communications protocol used between the provider device2400and the vendor devices2200, but not between the provider device2400and the broker device2500(e.g., different syntax, different formats, different query description languages, differences in sets of available logical operators, etc.).

Therefore, for each metadata portion2336, the metadata translation component2446may generate a corresponding normalized metadata portion2436in which the pieces of information concerning the corresponding identified portion of data are expressed in a different manner that may conform to the communications protocol used between the provider device2400and the broker device2500. In so doing, the metadata translation component2446may retrieve indications of various rules for performing such a conversion of metadata information from the rules data2435, which may include rules specifying such aspects of conversion as protocol, syntax, formatting, and/or equivalent vocabulary. Additionally, where a metadata portion2336was determined by the further query component2443to be lacking in one or more pieces of information such that a further query was generated resulting in the receipt of corresponding further data2337, the metadata translation component2446may augment the corresponding normalized metadata portion2436to include the otherwise missing pieces of information that are supplied by that corresponding further data2337. The metadata translation component2446may then relay each normalized metadata portion2436to the broker device2500via the network2999.

Additionally, as part of generating each normalized metadata portion2436from its corresponding metadata portion2336and/or accompanying further data2337, the metadata translation component2446may also perform various forms of normalization of the manner in which pieces of information are expressed within each normalized metadata portion2436. By way of example, the formatting and/or other aspects of such pieces of information as times, dates, numerical values that include fractional amounts, addresses, spelling across variations and/or dialects of language (e.g., U.S. vs. British variants of English), etc. may be adjusted to conform to a single preselected set of normative formatting and/or other aspects.

Turning more specifically toFIG. 16F, as has been discussed above, for each portion of data of a data set2130that has been identified as relevant to the subject of the query, and that fits within one or more search restrictions that may have been specified by the query, corresponding metadata portions2336were retrieved and then converted into corresponding normalized metadata portions2436that have been provided to the broker device2500. Further, in response to situations in which different metadata portions2336include very different pieces of information concerning their respective portions of data, further queries to elicit further pieces of information for at least a subset of those portions of data may have been generated and used to enable some of the corresponding normalized metadata portions2436to be augmented with those pieces of information to reduce such variation in content among the normalized metadata portions2436. In this way, what pieces of information are include within each of the received normalized metadata portions2436is normalized to better enable those pieces of information to be compared as part of identifying relationships.

As depicted, the control routine2540of the example embodiment of broker device2500introduced inFIG. 16Amay also include a relationship identification component2546to identify relationships between portions of data of data sets2130based on corresponding ones of the normalized metadata portions2436. As also depicted, the control routine2540may further include a relationship visualization component2548to generate visualizations that depict such identified relationships.

In some embodiments, the relationship identification component2546may stage various comparisons of the pieces of information within the normalized metadata portions2436and/or pieces of information derived therefrom to start with relatively simple comparisons, before proceeding further to ever more complex ones. By way of example, as part of attempting to identify portions of data that are duplicates of each other and/or are copies of information from a source outside of the distributed online library system2000), the relationship identification component2546may begin with comparisons of identifying information associated with each portion of data, including and not limited to, file names, IP addresses, URLs, times/dates of publication, identities of publishers, identities of authors/creators, etc. As will be explained in greater detail through use of examples, it may be that the very same portion of data (e.g., the very same document, the very same spreadsheet, the very same hypercube, etc.) may have been made available through two or more different vendor devices2200(e.g., by each pointing to the same source device2100) such that this same very same portion of data is being caused to appear as if it were two or more different portions of data. Where such comparisons reveal highly similar, but not identical pieces of identifying information, various heuristic algorithms may be employed to determine whether the particular combination of differences and similarities is consistent with two or more portions of data being different versions of what is otherwise the same portion of data (e.g., different revisions of the same document).

A similar comparison of identifying information may be used to identify instances in which one portion of data contains a citation or other form of reference to one or more other portions of data. By way of example, a comparison may be made between the title, authorship, publishing information, etc., within each footnote or endnote indicated in the normalized metadata portion2436and respective ones of the titling and/or file names, author identifiers, publishing information, etc. indicated in each of the other normalized metadata portions2436to attempt to identify any citation matches.

It may be that attempts to identify duplicative portions of data may continue with comparisons between one or more metrics of each of the data portions. By way of example, portions of data that are identical to each other should have aspects of organization of their contents and/or indexing that are identical. Thus, for example, in comparisons among multi-dimensional arrays, comparisons of the number of dimensions, the size of each dimension, the degree of sparseness, the choice of a null data value, and/or any of a variety of others of such metrics may reveal that two or more of those multi-dimensional arrays have multiple identical ones of such metrics such that there is at least a high likelihood that they are duplicates of each other. In such situations in which there is such a basis to infer that two or more portions of data are identical, it may be deemed desirable for the relationship identification component2546to generate a further query to be transmitted to the corresponding vendor device(s)2200that requests that at least subparts of each of the suspected duplicative portions of data be provided to the broker device2500for direct comparison. In some of such situations, a form of random sampling of data values within each of the suspected duplicative portions may be used to select a subset of the data values to be retrieved and/or compared so as to avoid comparing all of the data values within each of the suspected duplicative portions. Thus, two portions of data may be deemed to be “identical” as a result of having identical content (at least to the degree that compared samples of content have been found to be identical) and having an identical organization of that content (e.g., organized into multi-dimensional arrays of identical number of dimensions and size of each dimension, or organized into documents of identical size and/or order of sentences, paragraphs, sections, pages, etc.).

A combination of various heuristic algorithms and/or metrics may be used to identify a situation in which a portion of data may be an aggregation, a compendium or other form of combination of multiple other portions of data. By way of example, a relatively large portion of data that is accessible from one vendor device2200may be a multi-chapter document that assembles a collection of tables, images, textual portions, etc., and one or more of each of those may be available elsewhere within the distributed online library system as separate and distinct portions of data. A heuristic analysis of the structure of the relatively large portion of data may reveal an organization of its contents that gives clues that at least some of its subparts may have been self contained portions of data that were imported into it. Such identification of such subparts may be employed as a trigger to compare aspects of each of those subparts to aspects of other portions of data to determine if there are any matches based on identifying information and/or various metrics thereof. Again, where a match is suspected, it may be deemed desirable to generate a further query to cause a direct comparison of at least a subset of the data values therebetween in an attempt to confirm the existence of a relationship in which one data portion is included within another.

In selecting the type of comparisons and/or other tests for relationships that are to be performed, and/or in determining what order in which to perform those comparisons and/or other tests, the relationship identification component2546may retrieve indications of a selection of comparisons and/or other tests to be performed and/or a specification of an order thereamong from the rules data2535. Additionally, the rules data2535may also specify one or more thresholds of similarity for various metric and/or statistics that may trigger the generation of further queries in which data values are accessed and compared to confirm that two or more portions of data are identical, etc.

Following the completion of comparisons and/or other tests to identify relationships among portions of data, the relationship visualization component may generate one or more visualizations2888to depict the identified relationships. Again, the query, as originally entered via the client device2800from which the query was received may specify that visualizations of only one or more specific types of relationships are requested. In addition to such limitations causing the relationship identification component2546to limit the comparisons and/or other tests that it may perform to those needed to identify only the specified types of relationships, such limitations may also cause the relationship visualization component to limit the visualization(s)2888to depicting only those same specified types of relationships.

Regardless of whether such limitations on what is to be included in the visualization(s)2888are specified in the original query, or not, the relationship visualization component2548may store a rendering of the resulting visualization(s)2888and/or a descriptive script thereof as the visualization data2738. The relationship visualization component may then transmit the visualization data2738to the client device from which the original query was received as an answer to that query. Additionally, copies of past visualizations may be stored for up to a predetermined period of retention time within a visualization log2638(seeFIG. 15), which may, in some embodiments, function effectively as a “cache” of recent visualizations2888that may be re-transmitted to a client device2800in response to a situation in which an identical query is received within a predetermined period of time that is deemed short enough for what is stored within the visualization log2638to not be deemed too “stale” to be valid. Alternatively or additionally, the visualization log2638may be employed to store previously generated visualizations2888for up to a predetermined period of retention time to enable a newer visualization2888generated in response to a query to be contrasted with an earlier visualization2888generated in response to an identical query to thereby provide a further visualization of the manner in which relationships among portions of data may have changed (e.g., perhaps as a result of a change having been made to what provider devices2400and/or vendor devices2200are included in the distributed online library system2000). Thus, for example, where a bottleneck may have been discovered through a visualization2888generated at an earlier time in which there were multiple pathways through different provider devices2400and/or different vendor devices2200to the very same device where a portion of data was stored, and later generated visualization2888may be contrasted with that earlier generated visualization2888as part of confirming that corrective action has been taken to remove that bottleneck.

Turning more specifically toFIG. 16G, as depicted, with the visualization data2738having been received at that client device2800, the UI component2848may then operate the display2880to visually present the visualization(s)2888.

FIG. 17illustrates an example embodiment of a citation relationship between two portions of data of two different data sets2130aand2130b, and a corresponding example embodiment of a visualization2888depicting this example citation relationship. More specifically, the depicted vendor device2200astores a data set2130aof which a portion of the data thereof is a document that includes a citation to another document that forms a portion of the data of another data set2130bthat is stored within another depicted vendor device2200b.

It may be that both documents were found to contain at least one mention or reference to a subject of interest specified in a query that, when acted upon with searches performed within each of the depicted vendor devices2200aand2200b, resulted in the identification of these two documents as portions of data deemed relevant to the subject of interest and meeting the criteria specified in the query for the search. With the corresponding portions of the metadata for each of these two documents having been provided to the depicted broker device2500, analyses performed at the broker device2500, including comparisons of the contents of the two corresponding portions of metadata, revealed indications in the metadata of one of the documents that it includes a citation to the other.

With this citation relationship having been identified, the depicted visualization2888is generated by the broker device to show this citation relationship. In the depicted visualization2888, each of the two documents, as stored as part of a separate data set2130within a separate vendor device2200, is depicted with a separate node. More specifically, the document that includes the citation is depicted with the designation “aaa” that denotes it being a portion of the data set2130athat is stored within the vendor device2200athat is registered with the depicted provider device2400a. Correspondingly, the document that is cited with that citation is depicted with the designation “bbb” that denotes it being a portion of the data set2130bthat is stored within the vendor device2200bthat is registered with the depicted provider device2400b.

As depicted, a dashed line that includes an arrow head at one end may be employed to visually indicate this citation relationship by pointing from the portion of data that includes the citation and toward the portion of data that is cited. It may be that different types of lines that differ in color, thickness, and/or selection of dotted and/or dashed pattern may be used, along with a selection of arrow head types and/or other graphical elements employed as line endings, are used to visually indicate any of a variety of relationships between stored portions of data that may be depicted as nodes.

It should be noted that this is a deliberately highly simplified example of an identified relationship between two portions of data, and a corresponding highly simplified example of a visualization2888of that relationship that is presented herein for purposes of illustration. It is envisioned that far more relationships of a far more interwoven nature would typically be found among what would typically be far more portions of data such that the resulting visualization2888would be far more complex than what is presented in this particular example. Accordingly, it is also envisioned that the designations given to each of the nodes that are meant to correspond with one of the depicted portions of data would be similarly more complex, and may include indications of physical locations and/or network locations of the various devices associated therewith.

FIG. 18illustrates an example embodiment of an inclusion relationship among three portions of data of three different data sets2130, and a corresponding example embodiment of a visualization2888depicting this example inclusion relationship. More specifically, the depicted vendor device2200astores a data set2130athat is fully included within another data set2130cthat is stored within another depicted vendor device2200b. Further, the depicted vendor device2200balso stores a data set2130bthat is also fully included within the data set2130c.

It may be that the use of heuristic algorithms to analyze at least a portion of the metadata corresponding to the data set2130crevealed that the data set2130cappeared to include more than one data structure that exhibited various clues of being a self contained data structure that was inserted into the data set2130c. This may have prompted an automated comparison of each of these seemingly self contained data structures to each of multiple other portions of data, including at least portions of each of the data sets2130aand2130b, which may have lead to the determination that two of these seemingly self contained data structures were identical to the data sets2130aand2130bsuch that the data set2130ccould be said to include each of the data sets2130aand2130b.

With this inclusion relationship having been identified, the depicted visualization2888is generated by the broker device to show this inclusion relationship. In the depicted visualization2888, each of the three data sets2130a,2130band2130c, as stored within one or the other of the vendor devices2200aor2200b, is depicted with a separate node. More specifically, the data set2130ais depicted with the designation “aaa” that denotes it being the data set2130athat is stored within the vendor device2200athat is registered with the depicted provider device2400a. Correspondingly, the data set2130bis depicted with the designation “bbb” that denotes it being the data set2130bthat is stored within the vendor device2200bthat is registered with the depicted provider device2400b. Also correspondingly, the data set2130cis depicted with the designation “bbc” that denotes it being the data set2130cthat is also stored within the vendor device2200bthat is registered with the depicted provider device2400b.

As depicted, two separate solid lines that each include an arrow head at one end may be employed to separately visually indicate each inclusion of one data set within another in this inclusion relationship by each pointing from the data set2130cand toward one of the data sets2130aand2130b. Again, it may be that different types of lines of different characteristics and/or having different graphical elements at one end, both ends or neither end, are used to visually indicate any of a variety of relationships between stored portions of data that may be depicted as nodes.

It should be noted that this is a deliberately highly simplified example of an identified relationship among three data sets, and a corresponding highly simplified example of a visualization2888of that relationship that is presented herein for purposes of illustration. Again, far more complex and interwoven relationships are envisioned as being more typical, and accordingly, far more complex depictions of relationships in visualizations are envisioned as being more typical. Thus, the relatively simplistic nature of the relationship and accompanying visualization thereof is presented herein for purposes of illustration and understanding, and should not be taken as limiting.

FIG. 19illustrates an example embodiment of a subject relationship between two different portions of data of two different data sets2130aand2130b, and a corresponding example embodiment of a visualization2888depicting this example similar subject relationship. More specifically, the depicted vendor device2200astores a data set2130aof which a portion of the data thereof is a table that includes a label indicating a similar subject to the subject indicated by a label of a row of data values of another data set2130bthat is stored within another depicted vendor device2200b.

It may be that both labels were found to indicate subjects of interest specified in a query that, when acted upon with searches performed within each of the depicted vendor devices2200aand2200b, resulted in the identification of the two portions of data associated with these two labels as portions of data deemed relevant to the subject of interest (e.g., a subject such as “orange production”) and meeting the criteria specified in the query for the search (e.g., a search of labels listed in corresponding portions of metadata). With the corresponding portions of the metadata for each of these two documents having been provided to the depicted broker device2500, analyses performed at the broker device2500, including comparisons of the contents of these labels as listed in their respective portions of metadata, revealed indications that the depicted table of data set2130aand the depicted row of data set2130bare both relevant to the subject of interest, and may therefore be related by subject to each other.

In some embodiments, this may be deemed too weak of a relationship between portions of data to be depicted in a visualization2888. However, in other embodiments, such a relationship may be deemed to be significant enough, and therefore, with this subject relationship having been identified, the depicted visualization2888is generated by the broker device to show this subject relationship. In the depicted visualization2888, each of the two portions of data, as stored as part of a separate data set2130within a separate vendor device2200, is depicted with a separate node. More specifically, the table of data set2130ais depicted with the designation “aaa” that denotes it being a portion of the data set2130athat is stored within the vendor device2200athat is registered with the depicted provider device2400a. Correspondingly, the row of data set2130bis depicted with the designation “bbb” that denotes it being a portion of the data set2130bthat is stored within the vendor device2200bthat is registered with the depicted provider device2400b.

As depicted, a relatively thin dotted line that includes no arrow head or other graphical element on either end may be employed to visually indicate this subject relationship. Again, it may be that different types of lines of different characteristics and/or having different graphical elements at one end, both ends or neither end, are used to visually indicate any of a variety of relationships between stored portions of data that may be depicted as nodes. Also, again, it should be noted that this is a deliberately highly simplified example of an identified relationship among two portions of data presented for purposes of illustration, and should not be taken as limiting.

FIG. 20illustrates an example embodiment of a dependency relationship among three data sets and a visualization thereof. More specifically, each of the depicted vendor devices2200a,2200band2200cstores a data set2130a,2130band2130c, respectively, where each of the data sets2130a,2130band2130cis duplicated from one of multiple versions of a data set2130xthat is stored by a separate source device2100x.

It may be that comparisons of identifying information within portions of metadata that correspond to each of these three data sets revealed that the data sets2130aand2130bare identical to each other. Further, it may be that the use of heuristic algorithms to analyze various metrics associated with each of these three data sets revealed that data set2130cis a slightly different version of the other two data sets2130aand2130b. Still further, it may be that analyses of the identifying information within the corresponding portions of metadata revealed that all three of these data sets emanate from the same depicted source device2100xsuch that, as a result, there is a dependency by all three of the vendor devices2200a,2220band2200con the source device2100xas the originator of all three of these data sets2130a,2130band2130c, respectively.

Such a dependency relationship may be of significance as a possible source of a bottleneck in the operation of the distributed online library system to which these vendor devices2200a,2200band2200cbelong if they are each configured to redirect accesses made to these data sets2130a,2130band2130c, respectively, to the single depicted source device2100x. In response to the discovery of such a potential bottleneck situation, a curator of such a system2000may choose to take steps to remove or otherwise address this dependency. Alternatively, the dependency relationship may be of a different nature in which each of the data sets2130a,2130band2130care actually stored within these vendor devices2200a,2200band2200c, respectively, but are each copied from the source device2100xsuch that there is still a dependency relationship of provenance to the source device2100xthat a curator of such a system2000may still deem important to be aware of in evaluating the quality of the data provided by the system2000.

With this dependency relationship having been identified, the depicted visualization2888is generated by the broker device to show this dependency relationship. In the depicted visualization2888, each of the three data sets2130a,2130band2130c, as associated with (but not necessarily stored within) one of the three vendor devices2200a,2200band2200c, respectively, is depicted with a separate node. More specifically, the data set2130ais depicted with the designation “aaa” that denotes it being the data set2130athat is stored within (or is at least associated with) the vendor device2200athat is registered with the depicted provider device2400a. Correspondingly, the data set2130bis depicted with the designation “bbb” that denotes it being the data set2130bthat is stored within (or is at least associated with) the vendor device2200bthat is registered with the depicted provider device2400b. Also correspondingly, the data set2130cis depicted with the designation “ccc” that denotes it being the data set2130cthat is stored within (or is at least associated with) the vendor device2200cthat is registered with the depicted provider device2400c. Still further, the source device2100xis also depicted with a separate node, which is depicted with the designation “x”.

As depicted, three separate solid lines that each include an arrow head at one end may be employed to separately visually indicate each dependency upon the source device2100xby each pointing thereto. Again, it may be that different types of lines of different characteristics and/or having different graphical elements at one end, both ends or neither end, are used to visually indicate any of a variety of relationships between stored portions of data that may be depicted as nodes. Also, again, it should be noted that this is a deliberately highly simplified example of an identified relationship among two portions of data presented for purposes of illustration, and should not be taken as limiting.

FIGS. 21A and 21B, together, illustrate an example embodiment of a logic flow3100. The logic flow3100may be representative of some or all of the operations executed by one or more embodiments described herein. More specifically, the logic flow3100may illustrate operations performed by the processor component2550in executing the control routine2540, and/or performed by other component(s) of the broker device2500.

At3110, a processor component of a broker device of a distributed online library system (e.g., the processor component2550of the broker device2500of the distributed online library system2000) may receive, via a network and from a client device (e.g., via the network2999, and from a client device2800), a query for information concerning relationships among portions of data that includes a specification of a subject of interest, and/or access credentials that are associated with the client device and/or a user thereof. As has been discussed, the distributed online library system may be maintained by a particular scholastic, commercial/industrial, or governmental entity that restricts its use to only the personnel of that entity and/or to the devices associated with those personnel or with that entity. As has also been discussed, such a query received from a client device may also include the specification of various limitations on any search that is to be performed in answer to the query, such as a specification that the subject of interest is to be searched for only within titles, headings, labels used in indexing data (e.g., row or column labels), a specification that only certain specified types of relationships among portions of data are to be searched for, etc.

At3112, the processor may check whether the client device and/or the user thereof is authorized to make the query. If not at3112, then at3114, the processor may transmit an indication of the rejection of the query back to the client device. As previously discussed, the UI by which the client device enabled the user to enter the query may then respond to the receipt, by the client device, of such an indication of rejection of the query by providing an indication to the user that the query is not authorized. However, if the query is authorized at3112, then at3120, the processor may relay the query to the one or more provider devices of the distributed online library system (e.g., the one or more provider devices2400) via the network, and may then await the receipt of responses therefrom.

At3130, as a response to having relayed the query to the one or more provider devices, the processor may receive multiple normalized metadata portions from the one or more provider devices (e.g., one or more normalized metadata portions2436). At3132, the processor may perform one or more analyses of the various pieces of information specified within each of the normalized metadata portions to derive one or more metrics that are not already directly specified therein (e.g., number and/or sizes of dimensions of multi-dimensional arrays, sparseness of data structures, etc.).

At3140, the processor may check whether the original query received from the client device specified that the visualization was to be limited to one or more specific relationship types. If not at3140, then at3142, the processor may perform various comparisons among pieces of information within the received normalized metadata portions to identity multiple types of relationships among the portions of data to which each of the normalized metadata portions correspond (e.g., either a portion of a data set2130, such as a single data structure or document within a data set2130, or a complete data set2130). At3150, the processor may then generate one or more visualizations (e.g., one or more of the visualizations2888) of the identified relationships. In some embodiments, a separate visualization may be generated for each different type of relationship. At3152, the processor may transmit the generated visualizations to the client device from which the original query was received to enable the display of the generated visualizations thereat. At3160, the processor may check whether the query included a request to provide the client device with the portions of data that were identified as relevant to the subject of interest and that met the criteria specified in the query for the search, which are the portions of data that correspond to the normalized metadata portions.

However, if at3140, the original query received from the client device did specify one or more particular relationships that any visualizations were to be limited to, then at3144, the processor may perform various comparisons among pieces of information within the received normalized metadata portions to identity relationships of only the one or more specified types that may be present among the portions of data to which each of the normalized metadata portions correspond. At3146, the processor may check whether any relationships of the one or more specified types were identified. If so at3146, then the processor may proceed with the generation of visualization(s) at3150. However, if not at3146, then at3160, the processor may proceed with checking whether the query included a request to provide the client device with the portions of data that were identified as relevant to the subject of interest and that met the criteria specified in the query for the search.

If, at3160, the query does specify the provision of the corresponding portions of data to the client device, then at3162, the processor may receive those corresponding portions of data from the same one or more provider devices that provided the normalized metadata portions. It should be noted that those corresponding portions of data may have already been received from those one or more provider devices together with the normalized metadata portions. At3164, the processor may relay those corresponding portions of data to the client device from which the original query was received, along with any visualizations that may also have been generated and transmitted to that client device.

FIGS. 22A and 22B, together, illustrate an example embodiment of a logic flow3200. The logic flow3200may be representative of some or all of the operations executed by one or more embodiments described herein. More specifically, the logic flow3200may illustrate operations performed by the processor component2450in executing the control routine2440, and/or performed by other component(s) of one of the provider devices2400.

At3210, a processor component of a provider device of a distributed online library system (e.g., the processor component2450of one of the provider devices2400of the distributed online library system2000) may receive, via a network and relayed by a broker device of the distributed online library system from a client device (e.g., via the network2999, and relayed by the broker device2500from a client device2800) a query for information concerning relationships among portions of data that includes a specification of a subject of interest, and/or access credentials that are associated with the client device and/or a user thereof. As has been discussed, the distributed online library system may be maintained by a particular scholastic, commercial/industrial, or governmental entity that restricts its use to only the personnel of that entity and/or to the devices associated with those personnel or with that entity. As has also been discussed, such a query may also include the specification of various limitations on any search that is to be performed in answer to the query, such as a specification that the subject of interest is to be searched for only within titles, headings, labels used in indexing data (e.g., row or column labels), a specification that only certain specified types of relationships among portions of data are to be searched for, etc.

At3212, the processor may check whether the client device and/or the user thereof is authorized to make the query. If not at3212, then at3214, the processor may transmit an indication of the rejection of the query back to the client device through the broker device. As previously discussed, the UI by which the client device enabled the user to enter the query may then respond to the receipt, by the client device, of such an indication of rejection of the query by providing an indication to the user that the query is not authorized. However, if the query is authorized at3212, then at3216, the processor may additionally check whether the query is authorized based on other restrictions, such as an upper limit on the number of queries that are permitted to be handled at one time, at least by the vendor devices (e.g., a subset of the vendor devices2200) that may be registered with the provider device. Again, if not at3216, then the processor may proceed with the transmission of the indication of rejection of the query at3214.

However, if the query is authorized at3216, then at3220, the processor may translate the query from the communications protocol under which the processor received the query from the broker device and into the communications protocol under which the processor is to relay the query to the vendor devices registered with the provider device, thereby generating a translated form of the query (e.g., the translated query data2332). As previously discussed, it may be that there are differences in the communications protocols by which the provider device exchanges queries and metadata portions with the broker device, and by which the provider device exchanges queries and metadata portions with the vendor devices that are registered with it. Among such differences may formatting, language, syntactic, and/or other differences in the manner in which pieces of information are expressed within the query and/or within portions of metadata. At3222, the processor may relay the resulting translated query to the vendor devices that are registered with the provider device, and may then await the receipt of responses therefrom.

At3230, as a response to having relayed the translated query to the vendor devices registered with the provider device, the processor may receive multiple metadata portions from one or more of those vendor devices (e.g., one or more metadata portions2336). At3232, the processor may translate each of the received metadata portions between the different communications protocols, thereby generating corresponding normalized metadata portions (e.g., the normalized metadata portions2436) that are to be relayed onward to the broker device.

At3240, as part of the translation of metadata portions, the processor may check whether any of the received metadata portions are missing one or more particular pieces of information, where there may be a list of pieces of information that are to be included in each of the normalized metadata portions that are to be relayed onward to the broker device. If no such pieces of information are missing from any of the received metadata portions at3240, then at3242, the processor may transmit the normalized metadata portions to the broker device for use in identifying relationships among the portions of data that correspond to the received metadata portions, and therefore also correspond to the normalized metadata portions.

However, if at3240, there are one or more received metadata portions that are missing one or more pieces of information that are meant to be included in the corresponding one or more normalized metadata portions, then at3250, the processor may generate and transmit one or more further queries to the one or more of the vendor devices from which at least the metadata portions that are missing such one or more pieces of information were received. At3252, the processor may receive the missing pieces of information from the one or more vendor devices, and at3254, the processor may augment the normalized metadata portions that correspond to the one or more metadata portions from which information was missing with the missing information to make those normalized metadata portions complete. At3242, the processor may then proceed with transmitting the normalized metadata portions to the broker device.

At3260, the processor may check whether the original query from the client device included a request to provide the client device with the portions of data that were identified as relevant to the subject of interest and that met the criteria specified in the query for the search. If, at3260, the query does specify the provision of the corresponding portions of data to the client device, then at3262, the processor may receive those corresponding portions of data from the same one or more vendor devices that provided the metadata portions. It should be noted that those corresponding portions of data may have already been received from those one or more vendor devices together with the metadata portions. At3264, the processor may relay those corresponding portions of data to the broker device to be further relayed to the client device from which the original query was received, along with any visualizations that may be generated and transmitted to that client device by the broker device.

In various embodiments, each of the processors2250,2450,2550and2850may include any of a wide variety of commercially available processors. Further, one or more of these processors may include multiple processors, a multi-threaded processor, a multi-core processor (whether the multiple cores coexist on the same or separate dies), and/or a multi-processor architecture of some other variety by which multiple physically separate processors are linked.

However, in a specific embodiment, the processor2550of each of the one or more broker devices2500may be selected to efficiently perform the analysis of multiple instances of queries at least partially in parallel. By way of example, the processor2550may incorporate a single-instruction multiple-data (SIMD) architecture, may incorporate multiple processing pipelines, and/or may incorporate the ability to support multiple simultaneous threads of execution per processing pipeline. Alternatively or additionally by way of example, the processor1550may incorporate multi-threaded capabilities and/or multiple processor cores to enable parallel performances of the tasks of more than job flow.

In various embodiments, each of the control routines2240,2440,2540and2840, including the components of which each is composed, may be selected to be operative on whatever type of processor or processors that are selected to implement applicable ones of the processors2250,2450,2550and/or2850within each one of the devices2200,2400,2500and/or2800, respectively. In various embodiments, each of these routines may include one or more of an operating system, device drivers and/or application-level routines (e.g., so-called “software suites” provided on disc media, “applets” obtained from a remote server, etc.). Where an operating system is included, the operating system may be any of a variety of available operating systems appropriate for the processors2250,2450,2550and/or2850. Where one or more device drivers are included, those device drivers may provide support for any of a variety of other components, whether hardware or software components, of the devices2200,2400,2500and/or2800.

In various embodiments, each of the storages2260,2460,2560and2860may be based on any of a wide variety of information storage technologies, including volatile technologies requiring the uninterrupted provision of electric power, and/or including technologies entailing the use of machine-readable storage media that may or may not be removable. Thus, each of these storages may include any of a wide variety of types (or combination of types) of storage device, including without limitation, read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDR-DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, polymer memory (e.g., ferroelectric polymer memory), ovonic memory, phase change or ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, magnetic or optical cards, one or more individual ferromagnetic disk drives, non-volatile storage class memory, or a plurality of storage devices organized into one or more arrays (e.g., multiple ferromagnetic disk drives organized into a Redundant Array of Independent Disks array, or RAID array). It should be noted that although each of these storages is depicted as a single block, one or more of these may include multiple storage devices that may be based on differing storage technologies. Thus, for example, one or more of each of these depicted storages may represent a combination of an optical drive or flash memory card reader by which programs and/or data may be stored and conveyed on some form of machine-readable storage media, a ferromagnetic disk drive to store programs and/or data locally for a relatively extended period, and one or more volatile solid state memory devices enabling relatively quick access to programs and/or data (e.g., SRAM or DRAM). It should also be noted that each of these storages may be made up of multiple storage components based on identical storage technology, but which may be maintained separately as a result of specialization in use (e.g., some DRAM devices employed as a main storage while other DRAM devices employed as a distinct frame buffer of a graphics controller).

However, in a specific embodiment, the storage2260in embodiments in which the one or more of the vendor devices2200store data sets2130may be implemented with a redundant array of independent discs (RAID) of a RAID level selected to provide fault tolerant storage.

In various embodiments, each of the input device(s)2810may each be any of a variety of types of input device that may each employ any of a wide variety of input detection and/or reception technologies. Examples of such input devices include, and are not limited to, microphones, remote controls, stylus pens, card readers, finger print readers, virtual reality interaction gloves, graphical input tablets, joysticks, keyboards, retina scanners, the touch input components of touch screens, trackballs, environmental sensors, and/or either cameras or camera arrays to monitor movement of persons to accept commands and/or data provided by those persons via gestures and/or facial expressions.

In various embodiments, each of the display(s)2880may each be any of a variety of types of display device that may each employ any of a wide variety of visual presentation technologies. Examples of such a display device includes, and is not limited to, a cathode-ray tube (CRT), an electroluminescent (EL) panel, a liquid crystal display (LCD), a gas plasma display, etc. In some embodiments, the displays2180and/or2880may each be a touchscreen display such that the input device(s)2810, respectively, may be incorporated therein as touch-sensitive components thereof.

In various embodiments, each of the network interfaces2290,2490,2590and2890may employ any of a wide variety of communications technologies enabling these devices to be coupled to other devices as has been described. Each of these interfaces includes circuitry providing at least some of the requisite functionality to enable such coupling. However, each of these interfaces may also be at least partially implemented with sequences of instructions executed by corresponding ones of the processors (e.g., to implement a protocol stack or other features). Where electrically and/or optically conductive cabling is employed, these interfaces may employ timings and/or protocols conforming to any of a variety of industry standards, including without limitation, RS-232C, RS-422, USB, Ethernet (IEEE-802.3) or IEEE-1394. Where the use of wireless transmissions is entailed, these interfaces may employ timings and/or protocols conforming to any of a variety of industry standards, including without limitation, IEEE 802.11a, 802.11ad, 802.11ah, 802.11ax, 802.11b, 802.11g, 802.16, 802.20 (commonly referred to as “Mobile Broadband Wireless Access”); Bluetooth; ZigBee; or a cellular radiotelephone service such as GSM with General Packet Radio Service (GSM/GPRS), CDMA/1×RTT, Enhanced Data Rates for Global Evolution (EDGE), Evolution Data Only/Optimized (EV-DO), Evolution For Data and Voice (EV-DV), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), 4G LTE, etc.

However, in a specific embodiment, one or more of the network interfaces2290,2490,2590and/or2890may be implemented with multiple copper-based or fiber-optic based network interface ports to provide redundant and/or parallel pathways in exchanging one or more of the portions of data of the data sets2130, one or more of the metadata portions2336, and/or one or more of the normalized metadata portions2436.

In various embodiments, the division of processing and/or storage resources among the federated devices1500, and/or the API architectures employed to support communications between the federated devices and other devices may be configured to and/or selected to conform to any of a variety of standards for distributed processing, including without limitation, IEEE P2413, AllJoyn, IoTivity, etc. By way of example, a subset of API and/or other architectural features of one or more of such standards may be employed to implement the relatively minimal degree of coordination described herein to provide greater efficiency in parallelizing processing of data, while minimizing exchanges of coordinating information that may lead to undesired instances of serialization among processes.

Some systems may use Hadoop®, an open-source framework for storing and analyzing big data in a distributed computing environment. Some systems may use cloud computing, which can enable ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Some grid systems may be implemented as a multi-node Hadoop® cluster, as understood by a person of skill in the art. Apache™ Hadoop® is an open-source software framework for distributed computing.