Patent ID: 12225098

DETAILED DESCRIPTION

Organizations large and small, government and non-government, may employ multiple information processing and communication systems. Such systems may facilitate personnel management, product and service production and provision, infrastructure, organizational communications both internal and external to the organization, and other logistics and readiness functions. Each of these systems may be developed and provided by sources or vendors. As a result, even if all systems are compatible in terms of overall information flow, data consumed and produced by the systems may not be compatible directly (i.e., without some form of conversion). For example, some organizational infrastructures may use an overall system in which efficient data transmission and consumption requires individual system components being able to process multiple data formats and data transmission protocols.

To facilitate and improve data interoperability, disclosed herein is a configurable agnostic data exchange (CADE) systems, and corresponding methods, that provides a means for exchanging data between systems and system components, thereby simplifying system operation, expansion, and efficiency. An example CADE system provides bi-directional data exchange between other systems, subsystems, components, and devices that may use differing data formats and differing data transmission protocols. The CADE system minimizes or eliminates compatibility issues by providing an innovative, interactive data format determination coupled with a well-defined, extensible and configurable schematic notation that decodes and translates input data streams. In an aspect, the CADE system provides configurable and efficient data exchange between heterogeneous systems, subsystems, components, and devices through the use of proxies that operate to define a common, intermediate (i.e., inter-proxy) form of data being consumed.

In an embodiment, the CADE system executes to perform data transmission and related operations. The first operation generates a configurable, extensible data protocol and corresponding data format and data format specification. To execute this first operation, the CADE system includes means to generate the configurable, extensible data protocol; the protocol allows data translation and data transmission between or among two or more external system components where the external system components may employ differing data formats and differing data transmission protocols. In an aspect, the CADE system employs automated and/or semi-automated means to generate the configurable, extensible data protocol, which represents a common, intermediate data format and corresponding data format specification. The second operation includes data translation and data transmission between and among the external system components. In this second operation, given input data streams with multiple data formats and data transmission protocols, the CADE system employs means to parse, extract, decode and translate data streams and then to format the translated data streams into the common, intermediate form. In an embodiment, the CADE system includes a configurable set of software and/or hardware proxies, each of which employs a set of rule-based means to parse, extract, and decode input data streams and to translate and format the decoded data streams to support multiple data formats and transmission protocols. In an aspect, these means provide automated identification of the format of a data stream and automated parsing, formatting, and further processing of the data stream. In another aspect, these means provide a (human) user an opportunity to determine the format of a data stream and to translate the data into any desired format. With either the automated or semi-automated processes, the proxies allow the CADE system to completely define the common, intermediate (inter-proxy) form for data received from external systems interfaced by the proxies. In an embodiment, each of two or more external devices of the external system has at least one associated proxy. In an aspect, the associated proxy may be dedicated to a specific external device. These operations are disclosed in more detail herein, including with respect toFIG.5.

Thus, the CADE system (1) executes automated and/or semi-automated tools to generate an inter-proxy data format, and (2) executes automated and/or semi-automated protocol parser/extractor/translator tools that employ a protocol specification and an extraction specification to automatically identify an input data format, including a data stream's schema, grammar, and transmission protocol and translate the input data stream into a data stream having an intermediate, inter-proxy data format. The protocol specification may include a schema and corresponding grammar. Generally speaking, a schema is a formal description of a data format; for example, a data format expressed in extensible markup language (XML). A grammar may consist of a rule set that may be employed to describe the schema's structure. The CADE system may use the automated and/or semi-automated tools to generate a schema and associated grammar that define the inter-proxy data format. Each proxy in the CADE system may generate, store and modify variants of the schema and its grammar so as to be able to translate between the data format(s) of its respective external device and the common, inter-proxy data format. By employing a schema and an associated grammar, the CADE system specifies the inter-proxy data format that is used for data transmission between the CADE system's proxies (see, for example,FIG.1A(1)). The CADE system's automated protocol parser/extractor/translator tools represent an improvement over manual or hand-coded protocol parsing, extracting, and translating mechanisms, which have at least two major weaknesses in comparison with the CADE system's automated protocol parsing, extracting, and translating processes. First, hand-coded protocol parsers are hard to reuse because they typically are tightly coupled to specific systems and deeply embedded into the system's working environments. Second, hand-coded protocol parsers tend to be error-prone and lack robustness. Writing an efficient and robust parser is a time-consuming and error-prone process and generally results in a limited-use protocol parser.

In another embodiment, the CADE system may provide an interactive, or semi-automated process for generating a schema that defines the inter-proxy data format. Since manual generation of a schema and grammar can require specialized training and can be tedious and error prone, this embodiment of the CADE system also is an improvement over hand-coded processes for generating schemas and grammars and may be utilized if automated processes are not available or desirable. To implement the semi-automated, interactive processes, the CADE system may include components that allow a human user to visualize the application of various syntaxes/formats, including partial specifications to derive the desired schema/grammar. This embodiment of the CADE system provides a “what-if” user interface that may be employed to create new inter-proxy schema/grammar specifications that in turn may be stored and retrieved as necessary.

In either the automated or semi-automated process noted above, the proxy architecture of the CADE system simplifies the specification and generation of the intermediate, common format used between proxies by eliminating the need to transform data to a preset schema; instead, the proxies allow a schema to be defined in in its most efficient form.FIGS.7A-7Dillustrate example data records, each of which employs a different format, but each of which contains exactly the same data. Data records710,720,730represent data records that may be received at a proxy110ifrom an associated external device10i. Data record740represents an inter-proxy data record formatted through operations of the CADE system according to the inter-proxy format.FIGS.7A-7Dare discussed in more detail herein.

In some implementations, the CADE system may consume and process data records of many different formats; however, these formats may share some similarities, such as one or more data frames containing one or more sub-frames. Both the frames and sub-frames may display similar format/extraction information. The CADE system may leverage these similarities to generate a short-hand notation containing the same information that ordinarily would be expressed in a grammar for the CADE system's common, internal (inter-proxy) format. Data verification rules also may be generated as part of the grammar specification. This grammar specification may allow the CADE system to execute in an automatic mode for some data processing and format identification, and a semi-automatic mode (i.e., with some human user feedback) for other data processing and format identification. In all cases (both the external device format and the common, inter-proxy format), the CADE system grammar specification may be expandable without requiring a redesign of the schema.

Thus, the CADE system may be used in many different network architectures. The CADE system may be implemented with different features, components, and capabilities. The CADE system components may be structured to support networks employing a client-server scheme, a peer-to-peer scheme, and/or a publish-subscribe scheme, or combinations of these and other schemes. The CADE system may operate in different use cases. The CADE system is described herein for a use case in which the schema/grammar and transmission protocols are known in advance and a use case in which one or more network devices employ an unknown, but knowable schema/grammar and transmission protocol.

FIGS.1A(1)-1A(3) illustrate example CADE system implementations in a computer network. AlthoughFIGS.1A(1)-1A(3) show a CADE system implemented in a computer network, the CADE system is not limited to such implementations and may be implemented in any system that uses digital data. InFIG.1A(1), CADE system100is shown implemented in network1A. Network1A includes multiple external devices10i, with one external device10iat each of multiple network nodes2. The external devices10imay be the identical, similar, or different network devices, computing devices, and/or communications devices. For example, the external devices10imay be computers, servers, routers, or switches. Each external device10imay send and receive data. For ease of description, and without limitation, the data transmitted generally will be referred to herein as a data stream. A data stream may be composed of data records, which in turn may be composed of data packets. A data packet may be composed of data frames, which in turn may be composed of sub-frames. The data streams, records, packets, frames, and sub-frames will follow a specific format or protocol and will be transmitted according to a specific data transmission protocol. The CADE system, as noted, generates a common, intermediate format that facilitates data transmission between and among external devices10iof network1A. The external devices10ithus represent any external interfaces (including clouds) with which the CADE system100may interface and interact. Each device10imay have associated with it, one or more proxies110i, which are components of the CADE system100. Other components of the CADE system100include a communications medium120and a user interface mechanism150(seeFIG.3A), which may be a component of computing platform105and/or a proxy110i. The communication medium120also may be a component of the network1A and is described in more detail later. The user interface mechanism150may be employed by aspects of the CADE system100, and also is described in more detail herein. AlthoughFIG.1A(1) shows the proxies110iseparated from their associated external devices,10i, in other networks or network configurations, the proxies110i, while still components of the CADE system100, may be located physically on or with the devices10i, on separate, individual computers, and/or on a single computing platform (such as computing platform105) as virtual machines or separate processes. Thus, the CADE system100may be viewed conceptually as a cloud that interfaces with multiple external devices10iand in an aspect streams data between any pair of external devices10i. In another aspect, some or all proxies communicate data through use of intermediary107(seeFIG.1A(3)). In an embodiment, data enters and exits the CADE system100in the native formats of the interfaced external devices10iwith the data transmitted between proxies110ibeing of the transformed, common, intermediate format (i.e., the inter-proxy format). As an example, and referring toFIGS.7A-7DandFIG.1A(3), proxy110A receives data record710from external device10A and transforms data record710into data record740having the inter-proxy format. Proxy710then transmits data record740to proxy110B and proxy110D. Proxy110B transforms data record740into data record720and provides data record720to external device10B. Proxy110D transforms data record740to data record730and provides data record730to external device10D. Note that one aspect of the data transformation and data transmission processes involves transforming non-data elements that may be appended to a data record.FIG.4A, for example, illustrates certain non-data elements that would be appended to a data record as part of the data transmission process. These non-data elements are identified and transformed along with the data elements illustrated in data records710-740. Thus, the proxies110iallow the CADE system100to employ an optimal definition of the common, intermediate format (i.e., between proxies110i) while ultimately being able to transmit all data consumed from the external devices10i. This structure of the CADE system100also allows the addition of proxies110iin an incremental fashion as the network1A adds new external devices10i. The CADE system100also allows multiple proxies110ito be employed for a single external device10ito increase data handling capacity, redundancy, and quality of service provisions.

FIGS.1B(1)-1B(4) illustrate example architectures for a proxy employed by the CADE system100. InFIG.1B(1), proxy110(1) is in the form of a standalone computing platform that may have a footprint similar to that of a standard credit card, and that may be plugged into a data port11of an external device10(1). InFIG.1B(2), proxy110(2) is implemented as a virtual machine on computing platform105, which in turn is a component of the CADE system100, and is coupled to external device10(2). InFIG.1B(3), proxy110(3) is implemented as a software plugin to standalone computer platform12, which in turn is coupled to external device10(3). InFIG.1B(4), proxy110(4) is implemented as a system on a chip (SoC) configuration in which a processor P1and other components C1and C2are installed on board B1that may be inserted into an external device10(4). Regardless of its physical form, a transmitting proxy110imay include the structure needed to decode data streams of uncertain format from an external device10iand translate the data streams into a common, inter-proxy format and a receiving proxy110imay include the structure to translate the inter-proxy data record into another format used by an external device10i.

FIGS.1C(1)-1C(3) illustrate an example schematic and an example physical implementation of the proxy110(1) shown inFIG.1B(1).FIG.1C(1) is a schematic illustrating example components of the proxy110(1). Proxy110(1) includes a central processor141, voltage regulator141A, system controller144, input/output (I/O) devices146, and memory components143. The memory components143include EBI143A connection to RAM, SRAM143B, Flash memory143C, and memory controller141D. Other memory devices may be used. In an aspect, the proxy110(1) may store limited data and instructions149on its corresponding external device10(1) or on non-transitory computer-readable data store145, as shown. These proxy components are connected by bus142. Of particular note, the proxy structure shown inFIG.1C(1) employs SRAM143B, which allows faster operations than would be possible with certain other memory types. Use of SRAM143B is made possible by the distributed nature of the proxy architecture shown, for example, inFIG.1A(1). That is, each proxy110iis subjected to a minimal processing load, which allows use of faster memory located closer to the central processor141than would be possible with a centralized proxy architecture. AlthoughFIG.1C(1) shows a schematic for a specific hardware implementation of proxies, other hardware implementations, as well as software implementations, may provide the same benefits.FIGS.1C(2) and1C(3) show an example physical implementation of the proxy110(1).

Depending on its configuration and control features, the CADE system100may operate according to different communications schemes, one of which is a client-server scheme and another of which is a publish-subscribe scheme. The CADE system100also may operate using other communications schemes or a combination of communications schemes. As described herein, the publish-subscribe scheme may increase processing speed and reduce memory use by the proxies110i. Referring toFIG.1A(2), in an example client-server scheme, each transmitting (or server) external device10Se and proxy110Se pair transmits data to a specific receiving (or client) proxy110Cl and external device10Cl pair (or pairs). In this client-server scheme, an external device/proxy pair may be a “server” or transmitter for some data transmissions and a “client” or receiver for other data transmissions. In this scheme, transmitting proxy T110Se may use a specific network address for each receiving proxy R110Cl. In such a traditional tightly coupled client-server scheme, the client external device10Cl cannot post data requests to the server external device10Se unless the server process is running, nor can the server external device10Se transmit data unless the client external device10Cl is running. As an alternative to such direct client-server addressing, a transmitting proxy T110Se may use indirect client-server processes such as broadcasting or multicasting data to some or all receiving proxies R110Cl.

FIG.1A(3) illustrates CADE system100′ in which an example publish-subscribe scheme may be implemented. In CADE system100′, proxies110A and110B are physically located with their external devices10A and10B, respectively, while proxy110D is a virtual machine resident on computing platform105and is accessed by external device10D. Proxy110G is shown unused by any external device10i. In the scheme ofFIG.1A(3), each proxy110imay be either a data publisher or a data subscriber, depending on the operational context of any given data transmission. The proxies110imay execute so that a published data description will match a subscriber's requested data description, thereby ensuring only desired data are transmitted between proxies110i. Proxy110A is in association and communication with external device10A. The external device10A employs a known schema/grammar; the proxy110A consumes data requested by other external devices10ifrom the external device10A when external device10A operates as a publisher and distributes the requested data to subscribers of the proxy110A (i.e. to other proxies110i). In an aspect of the publish-subscribe scheme, each proxy110icontrols “publication” and “subscription” using communications control mechanism117(seeFIG.2A). As a subscriber, proxy110B consumes data from the publishing proxy110A using the common, inter-proxy format and then reformats the consumed data according to the schema/grammar appropriate for its associated external device10B. The subscribing proxy110B then presents the data to its associated device10B in the appropriate format. In another aspect, the proxies110iinteract with the computing platform105, or other component of the CADE system100′, to publish data and to subscribe to data. For example, external device10A/proxy110A pair may publish to intermediary107and proxy110D may query intermediary107to obtain the data originally transmitted by external device10A. This aspect of the publish-subscribe scheme is disclosed further herein, including with respect toFIGS.3D(1) and3D(2).

In eitherFIG.1A(1) or1A(3), as the proxy110A (for example) consumes data from its associated external device10A, the data are reformatted and sent to other proxies110iin an efficient manner given the common, inter-proxy format (including transmission protocols) established among the proxies110i. If the communications medium120is a satellite communications medium, the proxy110A might first compress the data before transmission: if large data volumes are involved, the proxy110A may employ some form of caching and queueing. As shown inFIG.1A(1), if necessary or desired, an external device10i, such as external device10E may employ multiple proxies110E and110F to increase effective bandwidth for external device10E. In an aspect, as described in more detail herein, additional proxies110imay be provided for a specific external device10idynamically, for example, on an as needed or “on-demand” basis.

Thus, in an example, an external device10i, acting as either a client (in a client-server scheme) or a subscriber (in a publish-subscribe scheme), requests data in its appropriate schema/grammar, and according to a specific transmission protocol, and its associated proxy110itranslates the request into the common, inter-proxy format and transmission protocol and passes the translated request to a publisher/server proxy110i. A reverse process occurs at the publisher/server proxy110i/device10i. In this first use case, the CADE systems100and100′ employ proxies110ithat have identified and therefore “know” the schema/grammar and transmission protocol of their respective external devices10i, such that data translation in the proxy110imay be fully automated. Thus, the process of this example first use case works well because the schema/grammar of each device10i, and the appropriate transmission protocols are known to components of the CADE systems100and100′. However, the schema/grammar and/or transmission protocol need not be known in advance for the CADE systems100and100′ to perform their data translation functions.

An example second use case involves one or more external devices10ifor which the CADE system100is not provided with an advance identification of the appropriate device schema/grammar and/or communication protocol. To account for this unidentified schema/grammar situation, each proxy110imay be provided with multiple data schemas/grammars (and may be implemented on a platform that offers multiple physical interfaces) such that the CADE system100allows streaming data between any devices10i, including devices10iwith unidentified schemas/grammars. To select which schemas/grammars and protocols to employ for a given external device10i, the CADE system100may execute the user interface mechanism150(seeFIG.3A). The user interface mechanism150may execute in one or more of three identification modes, namely automatic, semi-automatic, and manual identification modes. As disclosed herein, including with respect toFIGS.2A-3E, some of these modes may involve a degree of human interaction and/or control.

FIG.2Ashows software components of an example proxy110. InFIG.2A, proxy110includes external device interface111, schema generation/selection application113, data transmission enhancement mechanism115, communications (e.g., publish-subscribe; client-server) control mechanism117, and common communications interface119. The external device interface111communicates directly with its proxy's associated external device10. The external device interface111may include one or more individual device interfaces, each interface with appropriate protocol support, depending how many different external device types the proxy110is intended to support. In an aspect, a proxy's external device interface111may be updated as a configuration of the networks1A or1B change to include additional external device types or because of updates to existing external devices10iwithin the networks1A or1B. The common communications interface119is selected based on a particular installation of the CADE system100. In an aspect, both the external device interface111and the common communications interface119offer at least a same level of security as the most secure external device10ibeing interfaced. The schema generation/selection application113, among other components, includes aspects of the user interface mechanism150(seeFIG.3A), and may be employed, in conjunction with a display device (not shown inFIG.2A) to provide a visual display, or user interface151, that enables human participation, as required, in automatic, semi-automatic, and manual discovery of data schemas and grammars, and communications protocols. The user interface151is disclosed in more detail herein, including with respect toFIGS.3A and8A-8C. The data transmission enhancement mechanism115may provide enhancements such as data compression and data caching. These and other aspects of the mechanism115are disclosed in more detail herein, including with respect toFIG.3C. The communications control mechanism117, among other functions, allows new external devices10ito connect, through an associated proxy110i, to other elements of the CADE system100. The communications control mechanism117is disclosed in more detail herein, including with respect toFIG.3D(1)-3E(2).

FIG.2Bis a conceptual illustration of a data transmission operation implemented by the example proxy110ofFIG.2A. InFIG.2B, data transmission operation200begins when transmitting external device10T provides,201, a data stream30to its associated proxy110T. The data stream30may be composed of data records, data packets, data frames, and data sub-frames, for example. In202, components of the proxy110T attempt to execute an automated process to transform the data stream30into a common, inter-proxy format. In203, the proxy110T transmits inter-proxy data stream31to a receiving proxy110R. In204, the receiving proxy110R transforms the inter-proxy data stream31into output data stream32having a format used by its associated external device10R. In205, the proxy110R transmits, and the external device10R receives, output data stream32.

FIG.3Aillustrates an example schema generation/selection application113of proxy110ofFIG.2A, and related user interface151, which is shown displayed on display device160. The schema generation/selection application113includes or accesses structural components that provide distinct but related functions. A first function is to process an incoming data stream30to identify the protocols used by data stream30, including the data schema and transmission protocols. In an aspect, the application113may include components that receive a specific identification, such as a network address, associated with the external device10iproviding the incoming data stream30. The application113then may use the network address to look up the data schema and transmission protocols. In another aspect, the source of the data stream30may not be explicitly identified. For this aspect, to satisfy the first function, components of the application113may execute to parse, extract, and identify elements of the data stream30. A second function involves generating an inter-proxy translator (if not already generated by or existing at proxy110i) that may be used to produce an inter-proxy data record based on the protocols of the incoming data stream30, and that subsequently may be used to translate between protocols of the incoming data stream30and the inter-proxy format. This second function is described in more detail herein. A third function involves, at the proxy level (i.e., at proxy110), applying the inter-proxy translator to generate intermediate, or inter-proxy data stream31having the inter-proxy format. The application113also provides for a reverse translation process in which the inter-proxy translator translates between the inter-proxy format and the protocols applicable to a related external device10i. InFIG.3A, the application113includes parser tool113A, extractor tool113B, and translator tool113C. These tools are described in more detail herein. The application113also includes user interface mechanism150, also described in more detail herein.

Referring toFIGS.1C(1) and3A, central processor141in cooperation with memory143and non-transitory, computer-readable data store145, which are connected by bus142, executes components of the application113. The data store145in turn includes library147. The library147stores two data record libraries147A and147B and instructions149.

Referring toFIGS.1C(1),3A, and3B, the central processor141may execute the application113, in part, to generate a common inter-proxy format from information contained in reference data record library147A. To complete this second function noted above, components of the application113include structural components derived from one or more specific grammars/data protocols. The central processor141begins the derivation process by comparing as many grammars as possible to find common, approximate, context free elements. Reference data record library147A may store multiple, existing formats and transmission protocols (reference data record formats A-N) that may be employed by external devices10i. Since the reference data record library147A serves as a basis for generating an intermediate format (schema/grammar and transmission protocol—that is, common, inter-proxy format provided in inter-proxy data format specification148′), the reference data record formats A-N may encompass a broad set of formats and protocols such that different characteristics may be included in the common, inter-proxy format, which then is stored in inter-proxy data record library147B. In an aspect, to generate the inter-proxy format, the approximate, context free elements from the reference data protocols A-N are combined to provide an extensible, context free grammar (ECFG). The ECFG is tested according to known data transmission protocols, and the ECFG is adjusted until a defined grammar is produced. The defined grammar provides a structure of a defined schema. The defined grammar provides a parsing specification and an extraction specification. The parser tool113A and the extractor tool113B use the two specifications to generate a parser and an extractor that may be applied to incoming data stream30to extract relevant data elements and non-data elements. A translator (e.g., one of translators148A,148B, and148C) generated by translator tool113C, is applied to the extracted elements to translate data records of the external device-provided data stream30into the common, inter-proxy format, which is stored in structural form in inter-proxy data record library147B as inter-proxy data format specification148′. Of note, a proxy110imay generate and store one or more translators, depending on the proxy's associated external device10i. As can be seen inFIG.3B, the inter-proxy data record library147B stores translators148A,148B, and148C, which are generated by each proxy110i.

Thus, the instructions149allow the central processor141to characterize the existing data record formats as part of an overall strategy for developing appropriate schemas for both “new, unknown” data inputs as well as a defined schema/grammar (the common, inter-proxy format).

In an embodiment, the application113takes advantage of the fact that computer networks and other data systems, when transmitting a data stream composed of data packets, data frames, and data sub-frames, for example, employ a method called frame synchronization to find valid data in a transmission that consists of data frames. The frame synchronization method may be based on identifying a synchronization word (syncword), sync character, sync sequence, or preamble to indicate the end of a sequence of header information and the start of data, and the data transmitting entity may insert a fixed, distinctive bit pattern (e.g., a sequence of bits—a syncword—that is distinguishable from data bits or data words) at the start of each data frame to mark the start of valid data. The receiving entity then searches for the fixed pattern in each data frame and achieves frame synchronization when a correlation between the input data and the fixed pattern is high. Components of the application113may use these and other bit patterns to identify the data and transmission protocols of incoming data stream30.

FIG.4Aillustrates a simplified representation of frame synchronization and data words in a data frame400composed of multiple sub-frames401. The words are shown, for simplicity purposes, as adjacent blocks; however, the actual data stream would, of course, be expressed in binary form (0s and 1s). Each sub-frame401includes a sub-frame identification (SFID) word402, frame synchronization (FS) word404, and data words WD1-WD8. Different formats and transmission protocols may use different frame synchronization techniques and formats. However, transmitting and receiving entities often know in advance what the format and transmission protocols are, simplifying the process of identifying the frame synchronization bits, for example. As a more specific example, in some data streams, the frame synchronization pattern is a known binary pattern that repeats at regular intervals within the data stream. A receiving frame synchronizer recognizes this pattern and aligns the data into sub-frames. The frame synchronization pattern may be followed by a counter (sub-frame ID) that indicates which sub-frame in the series is being transmitted. The receiving entity uses this information to assemble a data frame from the received sub-frames. The CADE system100may leverage frame synchronization bit patterns, as well as other bit patterns, in an unknown data stream to identify, automatically and without human intervention, and with a high degree of confidence, the format and transmission protocols of received data by iteratively looking for specific bit patterns. This bit pattern matching process employed in the CADE system100is not limited to frame synchronization bits, since data streams include other elements that may exhibit a specific pattern. In an aspect, the pattern matching may be based on a library of reference patterns, such as the reference data record library147A (seeFIGS.3A and3B). Finally, in an embodiment, the determination of a match condition does not depend on an exact match existing between the format of an incoming data record and an existing, known data record format. In this embodiment, a match condition signifies that, with a sufficient degree of confidence, a translator may be written automatically by operation of the CADE system100, where the translator translates the incoming data record to an inter-proxy data record having the inter-proxy format. Once the translator is written, its operation may be verified by translating the incoming data record to an inter-proxy data record. In another embodiment, a matching condition signifies that an existing translator may be used to translate the incoming data record into an inter-proxy data record. In addition, as disclosed herein, the CADE system100may operate in automatic, semi-automatic, and manual modes, and the specific data bit matching technique may differ depending on the mode.

Returning toFIG.3A, the parser tool113A identifies a protocol specification for the incoming data stream30. The protocol specification gives a grammar for parsing the data stream30into a number of elements. Extractor tool113B uses the protocol specification and corresponding grammar and an extraction specification to extract the parsed elements. The extraction specification indicates what elements need to be extracted. Having a separate extraction specification allows the protocol specification to be reused in different applications that need different elements to be extracted. Both specifications are generated using the extensible context free grammar (ECFG), which may augment rules for a baseline, or original-version ECFG as input data protocols change, or new data protocols are developed. The augmentations also may include actions that increase expressiveness of the ECFG, but still allow the ECFG to be automatically simplified and optimized.

As noted herein, data stream30may be composed of a number of discrete, related data elements and non-data elements. The data elements arrive at proxy110incrementally. There are two ways to process these elements: incrementally, which means processing each data element as it arrives, or buffering, which means temporarily storing a number of data elements until a certain amount of data stream30is collected. Incremental processing may be preferable because buffering may require large amounts of dynamically-allocated memory.

Referring toFIGS.1A(1),1C(1),3A,3B,4A, and7A-7D, components of the CADE system100operate to translate between data records710,720, and730ofFIGS.7A-7C, respectively, on the one hand, and data record740ofFIG.7Don the other hand. Data in the data records710-740may, for transmission, be encoded in data words WD1-WD8. In addition, frame synchronization words404and sub-frame identification words402may be appended to the data words WD1-WD8. As can be seen in the data records710-740, the data record740is formatted in a manner that can be translated into data records710-730by translators148A-148C, respectively.

Referring toFIG.3A, user interface mechanism150, which includes a user interface driver153, cooperates with other components of a platform on which the CADE system100is installed to provide a user interface151. The user interface151may be employed and understood by a human user; i.e., user104. In an embodiment, the user interface151may be displayed on display device160coupled to a computing platform on which proxies110iare instantiated as virtual machines. In embodiments in which a proxy110iis installed as a physical component on the external devices10i, the user interface151may be displayed on a display device native to the external device10i. In these and other embodiments, the user interface151may display how the external devices10iand the proxies110iinteract during transmission and reception of data. As noted herein, structural components of, or available to, the application113enable distinct but related functions. To achieve these functions, the application113and its components and accessible components execute in one or more operations. A first operation, as needed, provides automated or semi-automated identification of an unknown data stream format, and automated or semi-automated generation of a translator that formats data streams30received by proxy110from external device10into the common, inter-proxy format. Referring toFIG.7E, an embodiment of an automated translator generation process may begin when the application113executes to parse reference data records into the smallest possible units of a reference data record's elements. For example, a sub-frame may be parsed into sync words and data words. The application113saves information related to the thus parsed units in equivalency matrix750. The application113then processes the matrix750to find elements and units that perform the same functions across the reference data record formats. As can be seen inFIG.7E, the equivalency matrix750is composed of columns750C and rows750R. Each of the data records710-730is represented in a column750C. In addition, inter-proxy data element740also is represented. Each matrix row750R represents an element, sub-element, or smaller parsable unit such as shown in matrix row750R(1). Thus, the example matrix750ofFIG.7Eprovides a means for comparing possibly equivalent data record components to different record components to determine if it is possible to generate a common, agnostic data record format that may be used as the inter-proxy data record format. In an aspect, the matrix750may be displayed to the user104. In this aspect, the application113may suggest translation rules to the user104. In another aspect, the application113automatically may derive translation rules from the matrix. In yet another aspect, the translation rules may be displayed in the matrix750(not shown inFIG.7E). The translation rules, whether interactively derived by the user104or automatically derived by the application113, may be assembled by translator tool113C to generate a translator that operates between the inter-proxy data record and one of the reference data records. Of note, each reference data record may require a unique translator. The application113also performs a second operation, after the translator is generated and saved, to transform incoming data stream30into inter-proxy data stream31having the common, inter-proxy format. Thus, the application113performs two operations, either of which may be automated or semi-automated, to generate a translator (first operation) and then apply the translator (second operation). Aspects of the first and second operations also may be executed manually by a human user. Another example of possible translation follows: JavaScript Object Notation (JSON) is an open standard file format that uses human-readable text to transmit data objects consisting of attribute-value pairs and array data types and is a format used as a replacement for XML. JSON is a derivation of JavaScript. JSON filenames use the extension .json. The undefined type is not included in the JSON standard, and null may be used instead. In fact, the JSON standard specifies that a sparse array such as var v=[0]; v[3]=3; behaves in JavaScript as if it were var vx=[0, undefined, undefined, 3]; with the undefined entries being only implicit rather than explicit, translating to JSON as if it were var vx=[0, null, null, 3]; and with explicit null fillers for the undefined entries. The application113may use rules derived from these and other data record format differences to generate a translator.

FIG.5illustrates, pictorially, these first and second operations that may be executed through employment of the CADE system100. Block510illustrates an operation in which components of the CADE system100generate inter-proxy data format specification148′, and ultimately one or more translators148A-C. Block520illustrates automatic generation of the inter-proxy data format specification148′. In one aspect of block520, the schema generation/selection application113surveys a large number of existing format specifications to identify identical and or similar elements, code segments, or code modules, where identity and similarity are based on the function, instructions, and form of the data elements. For each identical or similar element, the application113saves the elements with a link to a generic instruction that represents approximately the similar element. The operation of block520continues until a full generic instruction set is assembled, if possible. The generic instruction set is then compiled to generate a generic grammar and corresponding schema. The generic grammar and corresponding schema then are tested against a number of existing formats to verify results identical to, or sufficiently similar to the existing formats are achieved with the generic instruction set. If satisfactory results cannot be achieved with the automatic operation, the semi-automatic operation of block530may be employed. The semi-automatic operation may employ the user interface151to suggest a number or list of generic code segments in terms of function and/or structure that may be similar to existing code segments, and user104selects from generic code segment list to assemble, or to complete assembly begun with the automatic operation of block520so that a common, inter-proxy data format specification148′ is produced, tested, and saved. The first operation then may provide a notification or prompt to the user104. In either block520or530, the universe of possible formats from which to choose may be limited to knowledge of formats used by the external devices10iin network1A.

In the second operation, in an aspect, proxy110receives multiple lines or frames of data and compares elements (e.g., words, sub-frames) of the lines or frames to known data and non-data format patterns stored in the reference data record library147A. The patterns may be stored as object code-level patterns and as source code-level patterns. If the proxy110is able to identify potential matches in a set of elements comprising a larger unit such as a sub-frame or a frame, the proxy110proceeds to examine the overall data pattern of the larger unit to confirm the overall pattern matches a stored data pattern. The proxy110repeats this process for a sufficient number of larger units, or frames. Note that this second operation does not require a complete examination of the data and the automatic mode of the second operation may proceed until a configurable confidence level is reached. If the second operation results in the possible identification of the format or schema of the data, the proxy110may present the identification to a user through a display on user interface151for user confirmation. In an aspect, proxy110may receive multiple lines or frames of data from a transmitting external device10and may compare the lines or frames of the received data to other lines or frames of data included in the reference data record library147A, but is not able to identify, with sufficient certainty, the format or schema of the data. In this aspect of the first operation, the proxy110may provide suggestions to the user104by way of the user interface151as to the meaning of one or more elements, sub-frames or frames of the data record. For example, the proxy110may identify a data frame as a suggested match to a specific format, schema, or grammar. The user interface151may, for example, show what the data looks like with various grammar choices selected. In a simple example, selecting ASCII will display each eight bits in the file as an ASCII character, which may tell the user104whether the data stream, in this example, is ASCII encoded. A visualization method may employ would color coding to aid the user104. For example, if a data element in the data stream30uses 16-bit words and the first word in the data stream is a frame sync word, the user interface151may show a color map of a large section of the data element where sync word matches are of a contrasting color to the rest of the data. With color mapping, sequential application of choices may tell the user104whether the operation converges to or diverges from a format match. Other displays of data items under various calculations/assumptions likewise may provide overall format clues to the user104.FIG.5, beginning in block540, illustrates the second operation, namely an inter-proxy format specification selection operation executed by the application113. In block550, the format of the incoming data stream30is known. In block552, the application113selects the appropriate translator148A-C and applies the selected translator to the incoming data stream30to generate an inter-proxy data stream31having the common, inter-proxy format. In block554, even though the format of the incoming data stream30is known, the human operator104may elect to invoke a semi-automatic, or interactive, translator selection process, possibly as a more reliable or accurate method for translator selection. Once the translator is selected, the application113applies the selected translator to reformat the incoming data stream30into the inter-proxy data stream31having the inter-proxy format.

In block560, the format of the incoming data stream30is unknown to the application113, but knowable. The operation of block560may be automatic (block562) or semi-automatic (block564).

Note that under limited circumstances, certain of the operations ofFIG.5may be performed by a suitably trained and experienced human operator104using hand coding. However, the hand coding operation would not parallel, or in most ways, follow the operations ofFIG.5. Furthermore, the computer operations include features and aspects not possible through hand-coding.

Thus, the application113includes the structural components to decode data streams of uncertain format and transform these data streams into the common, inter-proxy format and back to another format.

FIG.3Cillustrates the data transmission enhancement component115. Adding proxies on-the-fly, the data transmission component115may request addition of another proxy110iwhen data transmission slows below a threshold amount. Each proxy110imay cache data to be transmitted. Large data files may be compressed. Data to certain proxies110imay be placed in preference in queue. A transmitting and receiving proxy pair may use an acknowledgement/reply scheme. A transmitting proxy110imay establish a connection to a receiving proxy110iand transmit a data file. A sending proxy110imay send a data file a fixed number of times until receiving an acknowledgement from the addressee proxy, or until a threshold number of attempts is reached. Proxies may establish a serialization scheme where each transmission has a sequence number so that the addressee can see if any data files were missed. As part of the data translation to the inter-proxy format, the mechanism115may determine if any viruses or other problems are encoded into the data file being decoded.

InFIG.3C, data transmission enhancement component115includes bandwidth monitor115A, data compression module115B, cache/queue module115C, request/acknowledgement module115D, and security module115E. The bandwidth monitor115A monitors the data transmission rate at communications interface119and may execute one or more actions to increase data transmission rate and/or may execute a data transmission priority scheme to transmit certain data on a priority basis. One action that may be initiated is to add one or more additional proxies to provide, in essence, a parallel pathway for data flow from the proxy's associated external device10to one or more recipient external devices10i. The manner in which such “on-demand” proxies may be added may depend on the proxy architecture in use. For example, if all proxies110iare virtual machines, adding an on-demand proxy for a specific external device10iwould be a straightforward process. If proxy110is implemented as a device according toFIG.1B(1), an “on-demand” virtual machine proxy could be allocated to external device10, but the allocation would entail a more complex connection process. In an aspect, the “on-demand” proxy may be allocated to a specific external device10only as long as transmission demand requires its allocation.

The compression module115B may operate to compress an inter-proxy data stream31when the size of the data stream31exceeds a certain value and/or based on the nature of the communications medium120. In general, the external devices10iare employed at nodes of a private communications network (e.g., a private local area network (LAN) or a private wider area network (WAN)). The network may be wired or wireless; the network may be a satellite communications network. In a satellite communications network, data compression may improve data transmission.

The cache/queue module115C may include a cache memory, or may access a cache memory, and the module115C may buffer outgoing and incoming inter-proxy transmissions. In an aspect, outgoing inter-proxy transmissions may be queued in the cache according to some value such as a proxy or urgency value and a length of time in queue.

The request/acknowledgement module115D operates to provide notifications to the proxies110iwhen a receiving proxy110R sends a data request REQ to a transmitting proxy110T and corresponding acknowledgements ACK1and ACK2between the receiving proxy110R and the transmitting proxy110T.

Security module115E may operate to invoke various security measures for inter-proxy communications. For example, client-server communication schemes are particularly susceptible to distributed denial of service (DDoS) attacks; typically, the private LAN would require some connection to an outside network such as the Internet. One way such a DDoS attack may be implemented is by flooding a processor with data requests or other messaging. Since the example networks disclosed herein are private LANs or WANs, the risk of a DDoS attack may be minimal. Nonetheless, each proxy110i, through security module115E, may implement procedures to prevent a DDoS attack and, in the event of a DDoS attack, minimize its damage an expedite recovery. Such procedures are well known.

FIGS.3D(1)-3D(3) illustrate aspects of the communications control mechanism117. In an aspect, networks1A and1B may employ a publish-subscribe scheme for some or all external devices10i. In another aspect, the networks1A and1B may employ a peer-to-peer and/or a client-server scheme for some or all external devices10i.FIG.3D(1) illustrates components of the communication control mechanism117, including publish-subscribe data classes/data channels device117A, which in turn includes data content filter117A(1) and data topic filter117A(2). These components support a publish-subscribe scheme employing a data transmission pattern where data transmitting external devices10P (publishers) do not program the data transmission directly to specific receiving external devices10S (subscribers). Instead, the publishing external devices10P may categorize published data into classes without knowledge of which subscribing external devices10S may have an interest in the data. Similarly, subscribing external devices10S may express an interest in one or more data classes and only receive data transmissions for which the subscribing external device10S has expressed an interest, without knowledge of which publishing external device10P provided the data. In this aspect, the respective proxies110imirror the behaviors of their external devices10i. In this publish-subscribe scheme, an individual subscribing proxy110S may receive a subset of the total published data transmissions through a filtering process. One filtering process is topic-based. When topic-based filtering is employed, data transmissions may be published to “topics” or named logical topic channels. Subscribers (external devices10S and proxies110S) then receive all data transmissions published to the topics to which the subscribers subscribe, and all subscribers to a topic receive the same data transmissions. The publishing external device10P defines the classes of data transmissions to which subscribers can subscribe. Another filtering process is content-based, and data transmissions are delivered to a subscriber only if the attributes or content of those data transmissions match constraints defined by the subscriber. The subscriber classifies the data transmissions.

FIG.3D(2) illustrates another publish-subscribe intermediary117B that includes publish interface117B(1) and subscriber interface117B(2) by which publishing proxies110P post data transmissions to intermediary107(seeFIG.1A(3)), and by which subscribing proxies110S register with intermediary107, letting the intermediary107perform the filtering. The intermediary107employs a store and forward mechanism to route data transmissions from publishers to subscribers. In addition, the intermediary107may prioritize data transmissions in a queue before routing. Subscribers may register for specific data transmissions at build time, initialization time or runtime. In an aspect, subscribers may be added or removed at runtime.

With the publish-subscribe schemes ofFIGS.3D(1) and3D(2), each publisher and subscriber may share device meta-data in a multicast process. The publisher and the subscribers cache this information locally and route data transmissions based on the discovery. Publishers are loosely coupled to subscribers and need not know of their existence. With topic or content being the focus, publishers and subscribers operate without regard to network topology.

FIG.3D(3) illustrates a client-server component117C of communications control mechanism117. In a traditional client-server architecture, clients and servers exchange messages in a request-response messaging pattern. In such a request-response messaging pattern, a client sends a request message to a server, which receives and processes the request, and then returns a response. This messaging pattern allows two devices to engage in a two-way communication over a channel. This message pattern may be implemented in a synchronous fashion, which holds a connection open and waits until the response is delivered or a timeout period expires. However, the request-response scheme also may be implemented asynchronously, with a response being returned at some unknown, later time. Considering the network1A ofFIG.1A(1), for example, one or more of the external devices10imay operate as a server and other external devices10imay operate as clients in the traditional sense. To adequately process requests and responses between external device servers and clients, the proxies110imay include structural components to provide either synchronous or asynchronous data transmission through medium120.

InFIG.3D(3), client-server component117C includes address module117C(1) and request/acknowledge (REQ/ACK) module117C(2). The address module117C(1) may include network addresses for each external device10iand proxy110iwith which a specific proxy110may communicate. The address module117C(1) also indicates if the external device10iis a server or a client, and which servers have which clients. The request/acknowledge module117C(2) may operate to maintain a data transmission path between proxies110iwhen operating as either a server or as a client in a client-server scheme, including providing a timeout mechanism. Furthermore, server proxy110Se may receive requests from many distinct client proxies110Cl in a short period of time. To prevent overloading the server proxy110Se, the module117C(2) may provide a scheduling system to prioritize incoming requests from client proxies110Cl.

FIGS.3E(1) and3E(2) illustrate proxy discovery components of the communications control mechanism117. InFIG.3E(1) communications control mechanism117includes node discovery component117E, publish-subscribe node discovery component117F, client-server node discovery component117G, and proxy node database117H. The node discovery component117E implements a general process, modified as necessary by publish-subscribe node discovery component117F and/or client-server node discovery component117G, to determine when a new proxy enters the CADE system100, and to provide node information among the proxies110ifor each new proxy any information related to the proxies110ito the new proxy. For example, a new proxy, upon initial or subsequent connection and start-up may broadcast new proxy information. In a client-server scheme, the new proxy may receive a response from a proxy associated with a server external device. In a publish-subscribe scheme, the new proxy may receive information from one or all of the existing proxies110i. In either scheme, the proxy information for one or more proxies may be stored at each proxy110.FIG.3E(2) illustrates an example proxy node database117H. The database117H may record a proxy name, an address for the proxy tied to the corresponding external device. Whether the proxy is active (proxy G is not active, and is not associated with any external device (seeFIG.1A(3)). The database117H may include, where appropriate, an indication of the client (C) or server (S) status of the proxy.

FIG.4Billustrates pictorially an example process410for semi-automated suggestion of format based on format patterns stored in the reference data record library147A. As shown inFIG.4B, as well asFIGS.3A,4A and3B, in412, processor141loads and reads binary code (object code) corresponding to sub-frame401A into memory143(preferably SRAM143B). In414, format suggestion logic149A in the instructions149executes to compare the binary, or the source code conforming to the binary, of sub-frame401A to elements in reference data record library147A to find elements whose format or format pattern that match or are similar to the binary of sub-frame401A. In an aspect, the comparison includes pattern comparisons. In416, elements or words in the sub-frame401A that match a reference pattern may be arranged in descending order of likelihood of an actual match, and in418, the results may be presented in the user interface151along with an identification of the format and a confidence level in user interface151. Note that the user interface151displays confidence information as a percentage confidence level. Rather than providing a numerical level, the application113may express confidence information through a color coding scheme, such as shades of green used to indicate confidence greater than an adjustable minimum threshold value and shades of yellow for elements having confidence levels lower that the threshold.

FIG.4Cillustrates pictorially another example process420for semi-automated suggestion of format based on format patterns stored in the reference data record library147A. As shown inFIG.4C, along withFIGS.3A,4A, and3B, in422, user104selects de-compiled source code of sub-frame401A, and the source code is displayed in user interface151. In424, the user104selects reference data set A for comparison to sub-frame401A source code. In426, user104selects data elements of reference data set A and visually compares the selected data elements to sub-frame401A source code. The process420continues iteratively among the data elements and reference data record formats A-N until the user104identifies a matching reference data set, or failing to identify a reference data set, ends process420.

FIG.4Dillustrates pictorially, an example for semi-automatic, interactive processing and display of unknown data elements. InFIG.4D, process430begins in431when a proxy110receives data stream30from its associated external device10. The data stream30may include a number of data elements such as data packets, data frames, and data sub-frames such as the example ofFIG.4A. The proxy110, in this example, receives the data elements non-data elements without knowledge of the format of the element. The central processor141determines the format is unknown and posts a notice or prompt to user104as display in window151A. In433, central processor141displays in window151A, de-compiled source code for the data elements, on a rolling basis. In435, processor141compares source code of the data stream30to that of reference formats A-N and displays the closest matches in descending order in window151B. In437, central processor141displays two reference data record formats showing match values above threshold level151C (reference formats C and A are the two formats satisfying the threshold151C). In439, user104selects a matched reference data set (reference data set C is shown selected). Central processor141then displays parsed elements in window151F for comparison to the reference format C and the elements in data stream30, as shown in window151E. User104also applies highlight tool152to select and highlight a sub-set of the elements. In441, user104employs the highlight tool152to select different sub-sets of elements. The comparison may be done by user104by a manual-interactive operation supported by the central processor141or though operation of central processor141in a semi-automatic, interactive method. With the components of the CADE system100shown inFIG.4D, the user104may interactively try various alternatives to identify the unknown data record format. For example, the user104could iteratively start with a data word length and apply various offsets to see if they look correct. If it does, that portion of the specification can be saved. If that does not work, other word lengths and/or offsets can be tried. After a data word is identified and verified, a next step might, for example, be the determination of a sub-frame sync pattern. The CADE system100may, in cooperation with user104, may analyze a series of frames (based on the result of the first step) and try to identify recurring sub-frame sync patterns within each frame. The results may be displayed to the user104and the user104may be able to select a sub-frame sync pattern.

FIGS.6A-6Eare flow charts illustrating example operations of the CADE systems disclosed with respect toFIGS.1A(1)-3E(2). The example operations may result in displays of information to user104. Example displays are provided inFIGS.4B-4D and8A-8C. InFIG.6A, overall operation600begins in block601when a publishing proxy110ireceives an input data stream30from an external device10i, preprocesses the data stream30, and attempts to determine, in automatic mode, a format for a data record in the input data stream30. In an aspect, the determination of block601may include the central processor141consulting a lookup table of data record formats associated with the proxy's external (network) device10i. In block602, if the proxy110idetermines the format, operation600moves to block603. If the central processor141of proxy110iis not able to determine the format, operation600moves to block604.

In block603, the publishing proxy110itranslates the data record of input data stream30to a common, inter-proxy format, thereby generating an inter-proxy data record, and transmits the inter-proxy data record to a subscribing proxy110i. The operation600then moves to block608and ends.

In block604, the publishing proxy110iexecutes automatic, and if needed, semi-automatic format, discovery processes for the input data stream30. In block605, if either the automatic or semi-automatic process results in a determination of the input data stream format, the operation600moves to block606and the publishing proxy110isaves the information related to the determination, and optionally forwards (broadcasts) the information to all other proxies110iin the CADE system100. The operation600then moves to block608and ends. If neither the automatic or semi-automatic process results in a determination of the input data stream format, the publishing proxy110iprovides a system-wide notice to the CADE system100, and a notice to the external device10iand moves to operation610.

FIG.6Billustrates an example operation for generating an inter-proxy format specification. Generation of an inter-proxy format and specification may occur at initiation of the CADE system100, and occasionally thereafter such as when an existing reference data record format is revised or otherwise changed, or when the CADE system100encounters a new reference data record format. Furthermore, the CADE system100may employ more than one inter-proxy format. InFIG.6B, operation610begins in block611when a user104assembles a library of current data record formats and specifications and their associated source code, parsers and extractors, grammars and schemas, and data transmission protocols. In block612, the user104operates CADE system100to apply each parser and extractor to a data record format source code to confirm the source code parsing and extraction conform to the grammar and schema. The operation of block612may be displayed on user interface151. Once all data record formats are processed and saved, in block613, the application113may execute to automatically attempt to develop an inter-proxy format and specification, and associated grammar, schema, and transmission protocol. Consider the data records ofFIGS.7A-7C. The application113analyzes each of the illustrated data records710-730, and all other data records, which would be parsed and verified in block612to determine the grammar, schema, and transmission protocols, to determine if a common grammar, schema, and transmission protocol (e.g., according to740,FIG.7D) may be generated, and after generation, translated into each of the data record format specifications associated with data records710-730. In block614, if a common inter-proxy format specification can be generated automatically, the application113provides a prompt, block615, to save the common format specification, and in block616, the specification is saved in the inter-proxy data record library147B as the inter-proxy data format specification148′ for the common, inter-proxy format. In block614, if the common inter-proxy format specification cannot be generated automatically, operation610moves to block617, and the application113displays and saves any matching data records. The operation610then moves to block618and semi-automatic or manual, interactive operations are conducted. Following blocks616,617, or618the operation610may end, block619, and the user104attempts a manual, hand-coding operation.

FIG.6Cillustrates one alternate operation of block604in which the application113invokes an automatic discovery of a format of an input data record in input data stream30and generation of a translator to translate the input data record format into the common, inter-proxy format. In an aspect, the operation ofFIG.6Cmay be based on pattern matches at the object code level, but the comparison operations may be displayed at a higher level, such as at a source code or higher level. SeeFIGS.8A-8C, for example. InFIG.6C, operation604′ begins in block631, a proxy110ireceives one or more data elements from its associated external device10i. In block632, the application113initiates an iterative process to identify the format of the input data stream30. In an aspect, the identification process includes a method for automatically comparing structures of data elements contained in the data stream30to known data element structures as shown in reference data record formats A-N. In block633, the application113determines if a sufficient match has been made to confirm, with a given confidence level, the identity of the format of the incoming data stream30. In block633, if a sufficient match has been made, the operation604′ moves to block634; otherwise, the operation604′ moves to block636. In block634, the application113selects the translator corresponding to the identified format and translates the input data record according to the inter-proxy data format specification148′. In block635, the application113verifies the translation is correct. For example, the application113may translate the inter-proxy data record back into the format original of the input data record. In block636, the operation604′ provides a prompt or notification to user104that automatic identification and translation failed and suggest execution of a semi-automatic identification and translation process. Following either block635or636, operation604′ ends, block638.

FIG.6D, in conjunction withFIGS.8B and8C, illustrates another alternate operation of block604in which the application113invokes a semi-automatic process to translate an input data record into an inter-proxy data record having the common, inter-proxy format. In an aspect, the semi-automatic operation ofFIG.6Dmay employ source code pattern matches, as opposed to object code pattern matches, to determine if a match condition exists. Furthermore, the central processor141may make match suggestions that are displayed to, and either accepted or rejected by user104. InFIG.6D, operation604″ begins in block640when the application113provides a prompt through user interface151, or other form of notification, to user104. In block641, the user104initiates the semi-automatic process of operation604″ using controls provided through the user interface mechanism150. In block642, the user104selects one or all reference data record formats A-N to be used for comparison, and the central processor141loads the reference data record formats A-N into SRAM143B and provides an expandable listing of the reference data record formats A-N in navigation window823. In block643, the central processor141causes the incoming data record (i.e., the data record having an unknown format) to be displayed in window824as a sequence of the smallest parsable units of the incoming data record. InFIG.8B, eight such units are displayed; however, window824is scrollable, and the incoming data record may include more than eight parsable units. As a consequence of limited display real estate, the user104may scroll the window824to view and interact with other elements parsed and extracted from the input data record. In block644, the central processor141causes scrollable analysis window826to be displayed at a predetermined start position as an overlay encompassing a number of units or elements. In addition to being scrollable, the analysis window826may be expanded or contracted vertically or horizontally to expand or contract the focus of the semi-automated format identification process. For example, the analysis window826may be contracted vertically so that only one element is encompassed in the analysis window826. In block645, the user104provides a selection of a reference data record format to be displayed in window828. As can be seen inFIG.8B, user104selects reference data record format A, and in response, the central processor141expands the reference data record format A display in navigation window823to display elements 1-n of the reference data record format A and displays the reference data record format A in window828. If the number of elements exceed the display real estate of window828, the central processor141provides scroll bar829so that the user104may scroll the display of window828to view the entirety of elements of reference data record format A. In block646, the central processor141executes suggestion logic149A to provide an interactive suggestion as to a possible match between the element or elements encompassed by the analysis window826and the elements displayed in window828. In an aspect, in making this match suggestion, the central processor141compares the function and/or the structure of the encompassed elements and provides a suggestion as to a match by highlighting elements in window828that may match. In the example ofFIG.8B, elements2-4are highlighted by operation of the central processor141. In an aspect, if the matching elements of the reference data record format were not displayed in window828, the central processor141may scroll the display until the matching elements are displayed. In an aspect, the highlighting may include a color coding scheme where a specific color or color shade indicates a degree of matching and/or a confidence level in the match as to the highlighted elements. In an aspect, the central processor141may identify more than one grouping of elements from reference data record format A that possibly match the elements encompassed in analysis window826. In this aspect, in block647, the central processor141determines if more than one element grouping matches. If more than one element grouping matches, the operation604″ moves to block648and provides a pop-up interactive window827in the tool bar822identifying the location of the other matching element groupings. The operation604″ then returns to block646. In optional block646A, the user104may command the central processor141to display an expanded view of both the elements encompassed in the analysis window826and the highlighted elements shown in window828by clicking one the analysis window826. Should the user104click on window826, the central processor141provides overlay display826A, an example of which is shown inFIG.8C. The overlay display826A includes a close button826B that, when selected, causes the central processor141to close the overlay display826A. In block647, if there is not more than one matching element grouping, the operation604″ moves to block649and the central processor141determines if the element grouping encompassed by the window826is the last un-examined element or element grouping. If the encompassed element or element grouping is not the last un-examined element or element grouping, operation604″ moves to block651. In block651, the central processor141provides a scroll prompt to, and in return receives a scroll command, from user104and repositions analysis window826accordingly. The operation604″ then returns to block646, and the operations of block646to651repeat until in block649, the central processor141determines no un-examined elements or element groupings exist, at which point, the operation604″ moves to block653and the central processor141computes an overall, or total match for the incoming data record in comparison to the selected reference data record format (reference data record format A in the example ofFIG.6D). In block655, if a total match exists, the operation604″ moves to block657and the central processor141provides a match display, with a degree of confidence, in window827. In block655, if the central processor141determines no total match exists, the operation604″ moves to block658and the central processor141displays a no match notification in window827. The operation604″ then returns to block645. Note that the central processor141could determine a match condition based on the first iteration of block646and optional block646A. Following block657, the user104may elect to return to block645by selecting another reference data set for analysis and comparison. Otherwise, the user104may end the operation604″ ends, block659.

A further operation of the CADE systems provides for a manual interactive code comparison operation. This manual interactive code comparison operation differs from that of operation604″ in that the central processor141does not make code match suggestions. Instead, all code comparisons are effectuated by the user104manually scrolling contents of windows824and828(seeFIG.8B) and deciding if a match condition exists.

FIG.6Eillustrates the match operation of block655,FIG.6D, in detail. The operation604″ ofFIG.6Dinitially determines “local” matches between individual elements or between sub-sets of elements. Just one “local” match may be sufficient to determine a “total” match between the format of an incoming data record and a reference data record format (e.g., reference data record format A, as shown inFIG.8B). However, the central processor141may need to find more than one “local” match to determine if the formats match.FIG.6Eillustrates a total match determination in detail. The total match determination ofFIG.6Econsists of two possible determinations. A first match determination is based on a sufficient total format match such that an existing translator, such as translator148A, may be used to translate the format of the incoming data record into the inter-proxy format. That is, the translator148A, for example, will successfully translate the incoming data record into an inter-proxy data record that may be provided to any proxy110iin networks1A or1B. A second match determination is based on a sufficient total format match such that execution of components of the CADE system100, and specifically execution of the application113by central processor141, may generate and apply a new translator that will translate the incoming data record into an inter-proxy data record.

InFIG.6E, operation655abegins in block661when the central processor141receives information related to the total matched elements between the incoming data record and a chosen reference data record format A (seeFIG.8B). In block663, the central processor141determines if the match is sufficient within a given confidence level, and with a sufficient probability, to designate an existing translator (translator148A) as capable of translating the incoming data record into an inter-proxy data record. If in block663, the central processor141determines the match is sufficient, operation655amoves to block664and the central processor141designates and saves the translator designation. If in block663, the match is not sufficient, the operation655amoves to block665, and the central processor141determines if the reference data set tested in block663is the last available reference data set. If in block665, the reference data set tested in block663is not the last available reference data set, operation655amoves to block666, and the central processor141, or alternately user104, selects another reference data record format from the navigation window823, the operation655areturns to operation604″, block645, and the remaining processes of operation604″ are performed. If in block665, the tested reference data set is the last available reference data set, the operation655amoves to block667and the central processor141determines, based on the match information of blocks661and663, if the central processor141can write a new translator that will translate the incoming data record into an inter-proxy data record having the current inter-proxy format. If in block667the central processor141can write a new translator, the operation655amoves to block668and the central processor141writes the new translator. Following block668, operation655amoves to block669and the central processor141verifies the new translator by attempting to translate the incoming data record into an inter-proxy data record having the current inter-proxy format. In block671, if the translation is successful, the operation669moves to block673, and the central processor141saves the new translator in the inter-proxy data record library147B. If in block671the translation is not successful, operation655areturns to block611, operation610,FIG.6B. Following block673, operation655amoves to block675and ends.

FIG.8Aillustrates an example display, in user interface151, generated by the CADE system100. InFIG.8A, user interface151presents display810associated with a specific code comparison, notably Code Comparison A as listed in title bar811. The display810includes tool bar812, which may be employed to vary information shown in display810. In the tool bar812, selection of navigate provides navigator window813, which may be one of many available navigator windows, displaying reference data record formats A-F. As can be seen, reference data record format A is selected, expanding to display available elements. The available elements may be those parsed and extracted by a parser/extractor designed for reference data record format A. Finally, window814displays a data element (sub-frame401A) from data frame400, and window815displays element1from reference data record format A. A display such as display810may allow a semi-automatic, suggestion-based evaluation. However, the display810also may allow a semi-automatic comparison without a suggestion feature.

FIG.8Billustrates an example display, in user interface151, generated by the CADE system100. InFIG.8B, user interface151presents display820associated with a specific code comparison, notably Code Comparison A as listed in title bar821. The display820includes tool bar822, which may be employed to vary information shown in display820. In the tool bar822, selection of navigate provides navigator window823, which may be one of many available navigator windows, displaying reference data record formats A-F. As can be seen, reference data record format A is selected, expanding to display available data elements. The available data elements may be those parsed and extracted by a parser/extractor designed for reference data record format A. Window824includes scroll bar825. Window824displays data elements1-8in the order in which the data elements1-8were received in data stream30. Window828includes scroll bar829and within the visible portion of window828, displays the first eight data elements of reference data record format A. As can be seen in window824, the user104has invoked sliding window826, shown encompassing data elements1-3. The sliding window826may be shrunk or expanded vertically to encompass fewer or more data elements. Moreover, sliding window826may be positioned vertically in window824. These two features of sliding window826allow the user104to position the window826to encompass one or more data elements of interest. In an aspect, when the sliding window826is positioned, the application113operates to scroll the reference A elements in window828that most closely conform to the selected data elements in window824. The application113then may highlight the conforming data elements in window828. In another aspect, the application113does not scroll the data elements but does highlight the data elements so that the user104may see the highlights when the user104manually scrolls the window828. In yet another aspect, movement of window826, or changes in its size, merely allows the user104to concentrate on a sub-set of the data elements in window824. In this aspect, the user104scrolls the window828manually to look for conforming data elements.

FIG.8Cillustrates display830provided on user interface151. Display830includes pop-up overlay display826A that shows elements of the incoming data record, in more detail than is provided in window826, compared to elements of reference data set A, also shown in more detail. The overlay display826A appears when user104clicks on sliding window826.

Certain of the devices shown inFIGS.1A(1)-2A include a computing system. The computing system includes a processor (CPU) and a system bus that couples various system components including a system memory such as read only memory (ROM) and random access memory (RAM), to the processor. Other system memory may be available for use as well. The computing system may include more than one processor or a group or cluster of computing system networked together to provide greater processing capability. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. A basic input/output (BIOS) stored in the ROM or the like, may provide basic routines that help to transfer information between elements within the computing system, such as during start-up. The computing system further includes data stores, which maintain a database according to known database management systems. The data stores may be embodied in many forms, such as a hard disk drive, a magnetic disk drive, an optical disk drive, tape drive, or another type of computer readable media which can store data that are accessible by the processor, such as magnetic cassettes, flash memory cards, digital versatile disks, cartridges, random access memories (RAM) and, read only memory (ROM). The data stores may be connected to the system bus by a drive interface. The data stores provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computing system.

To enable human (and in some instances, machine) user interaction, the computing system may include an input device, such as a microphone for speech and audio, a touch sensitive screen for gesture or graphical input, keyboard, mouse, motion input, and so forth. An output device can include one or more of a number of output mechanisms. In some instances, multimodal systems enable a user to provide multiple types of input to communicate with the computing system. A communications interface generally enables the computing device system to communicate with one or more other computing devices using various communication and network protocols.

The preceding disclosure refers to a flowchart and accompanying description to illustrate the embodiments represented inFIGS.4B-4D,5, and6A-6E. The disclosed devices, components, and systems contemplate using or implementing any suitable technique for performing the steps illustrated. Thus,FIGS.4B-4D,5, and6A-6Eare for illustration purposes only and the described or similar steps may be performed at any appropriate time, including concurrently, individually, or in combination. In addition, many of the steps in the flow chart may take place simultaneously and/or in different orders than as shown and described. Moreover, the disclosed systems may use processes and methods with additional, fewer, and/or different steps.

Embodiments disclosed herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the herein disclosed structures and their equivalents. Some embodiments can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by one or more processors. A computer storage medium can be, or can be included in, a computer-readable storage device, a computer-readable storage substrate, or a random or serial access memory. The computer storage medium can also be, or can be included in, one or more separate physical components or media such as multiple CDs, disks, or other storage devices. The computer readable storage medium does not include a transitory signal.

The herein disclosed methods can be implemented as operations performed by a processor on data stored on one or more computer-readable storage devices or received from other sources.

A computer program (also known as a program, module, engine, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.