DYNAMIC SCHEMA MAPPING BETWEEN MICROSERVICES

Disclosed dynamic schema mapping systems and methods monitor network traffic between different microservices and train mapping models based on the monitored network traffic using unsupervised training. This training of the mapping models generates a probability distribution tensor that shows the probabilistic associations of different key-value pairs of the schemas of different microservices. The trained mapping models are used to map a schema from a source microservice to another schema at a destination microservice. Should the translated schema be incompatible with the destination microservice, a semi-supervised approach is taken to make the translated schema compatible. The trained models may be reinforced (e.g., the probability distribution tensor may be updated) as more network traffic is collected and analyzed. The dynamic mapping therefore allows a system to be schema-agnostic, and developers may be able to define application interfaces or interaction schemas without the necessity of accounting for compatibility constraint between the different schemas.

BACKGROUND

A backend of a system infrastructure typically deploys a plurality of microservices. Microservices provide different types of functionalities within the system infrastructure. For instance, a first microservice may generate an intake user interface to prompt user data entry, a second microservice may generate a presentation user interface to display the processed user data, a third microservice may handle the processing of the entered user data to generate the displayed user data, etc. Other microservices may store the data and allow access thereto. Microservices therefore generally allow segmented, granular level of functionalities within the larger system infrastructure. Microservices communicate through synchronous and asynchronous interfaces. For instance, a communication interface for microservices may include a message to a message queue and or a message reply to message queue.

Different microservices, however, use different schemas. Schemas, also referred to as metadata models, are generally the data format used by the microservices to store, transmit, and or receive data. The data format typically includes a cluster of fields, types of the fields (e.g., string, number, etc.), size of the fields, and the like. The schemas are generally different because the microservices may be developed by different software teams at different points in time to solve different sets of problems. For instance, each microservice may have a tailored schema based on the design requirements, resource constraints, developer preferences, etc. associated with that particular microservice. These tailored schemas may not match because, e.g., the data fields may have a different organization or different field names, among other things, thereby reducing the inter-compatibility between the microservices.

Conventional techniques of handling the difference between schemas are manual, cumbersome, and inefficient. For instance, application programming interface (API) standardization has been used to handle the differences. This standardization, however, includes a manual lookup of the microservices' schemas followed by a manual construction of the API calls between the microservices to account for the differences. In addition to the cumbersome nature of this manual process, the constructed API calls remain rigid and therefore inefficient—these calls will only address the manually discerned differences between the known microservices. There is no automatic generalization to incorporate additional and newer microservices.

As such, a significant improvement on the inter-compatibility between different microservices is therefore desired.

SUMMARY

Embodiments disclosed herein solve the aforementioned technical problems and may provide other technical solutions as well. The disclosed dynamic schema mapping systems and methods monitor network traffic between different microservices and train mapping models based on the monitored network traffic using unsupervised training. This training of the mapping models generates a probability distribution tensor to show probabilistic associations of different key-value pairs of the schemas of different microservices. The trained mapping models may be used to map (or translate) a schema from a source microservice to another schema at a destination microservice. Should the translated schema be incompatible with the destination microservice, a semi-supervised approach is taken to make the translated schema compatible. The trained models may be reinforced (e.g., the probability distribution tensor may be updated) as more network traffic is collected and analyzed. The dynamic mapping therefore allows a system to be schema-agnostic, and developers are able to define schemas without the necessity of accounting for compatibility constraints between the different schemas.

DETAILED DESCRIPTION OF SEVERAL EMBODIMENTS

Use of different schemas between different microservices provides a huge technical challenge in a system infrastructure. Because the microservices have to communicate with each other to collaboratively realize several functionalities provided by the system, the schemas have to be translated. The conventional solution has been to perform a manual translation, which is cumbersome, time consuming, and static. Furthermore, a microservice developer has an additional constraint when developing a microservice, he or she will have to be concerned about the compatibility of the microservice with other microservices within the system infrastructure.

Embodiments disclosed herein are directed to solving these technical challenges and also to providing a schema-agnostic development of microservices. The embodiments train—during runtime—mapping models from the network traffic data, where the mapping models probabilistically map the different schemas. For example, a field of one schema may be probabilistically mapped to another field of another schema, even when the names, sizes, and or other attributes of the two fields are different. Different training algorithms may be used to generate the models. An example training model comprises a naïve Bayes algorithm. The training models may generate a probability distribution tensor, which expresses the probabilistic relationship between different key-value pairs (e.g., generated by the naïve Bayes model) between the different schemas.

In one or more embodiments, the initial training is unsupervised. The models attempt to generate generalized classes from specific instances, e.g., using predictive probability such as naïve Bayes algorithms. For example, specific instances of “consumer price,” “customer price,” “market price,” “sale price,” etc. may be mapped into a generalized class of “retail price.” The key-value pairs generated by the generalized classification may therefore be, in the key (general class)-value (specific instance of the generalized class) format: retail price-consumer price, retail price-customer price, retail price-market price, retail price-sale price, etc. This generalization is used to map the specific instances. For example, consumer price in a first schema may be mapped to retail price; customer price in a second schema may also be mapped to the retail price; and because each of the instances are mapped to the same class, the instances can be mapped to each other, thereby matching the consumer price to customer price. The probabilistic relationships between the different key-value pairs generated at multiple points in the network are represented by the probability distribution tensor.

The initial training, however, may not provide a desired level of accuracy. A semi-supervised approach may then be used to increase accuracy. For example, one or more trained models (using the probability distribution tensor) will translate a first schema of a first microservice to a second schema of a second microservice. The second schema (translated from the first schema) is sent to the second microservice. If the second microservice generates an error message (e.g., indicating an incompatible schema), the second schema may be manually evaluated and or another machine learning model may be trained based on the error. When the error is corrected, the trained mapping models and the probability distribution tensor are updated to account for the error correction. As more network traffic data is collected and as more translations are performed, the trained mapping models are progressively reinforced. In some embodiments, the reinforcements are performed until a desired level of translation accuracy is reached.

FIG.1shows an example of a system100configured for dynamic schema mapping between microservices, based on the principles disclosed herein. It should be understood that the components of the system100shown inFIG.1and described herein are merely examples and systems with additional, alternative, or fewer number of components should be considered within the scope of this disclosure.

As shown, the system100comprises client devices150a,150band servers120,130interconnected through a network130. A first server120hosts a first microservice122and a first database124and a second server130hosts a second microservice132and a second database134. The client devices150a,150bhave user interfaces152a,152b,which may be used to communicate with the microservices122,132using the network140. For example, communication between the elements is facilitated by one or more application programming interfaces (APIs). APIs of system100may be proprietary and or may include such APIs as Amazon® Web Services (AWS) APIs or the like. The network140may be the Internet and or other public or private networks or combinations thereof. The network140therefore should be understood to include any type of circuit switching network, packet switching network, or a combination thereof. Non-limiting examples of the network140may include a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), and the like.

Client devices150include any device configured to present user interfaces (UIs)152and receive user inputs154. The UIs152are configured to display responses156to the user inputs154. The responses156include, for example, personalized answers, call queue confirmation, contact information of an appropriate subject matter expert, and or other outputs generated by the first server120. The UIs152also capture session data including UI screen identifiers (id), product id (e.g., product SKU), input text/product language, geography, platform type (e.g., online vs. mobile), and or other context features. Exemplary client devices150include a smartphone, personal computer, tablet, laptop computer, and or other device.

In some embodiments, the first microservice122and or second microservice132implements an information service, which is any network140accessible service that maintains financial data, medical data, personal identification data, and or other data types. For example, the information service may include QuickBooks® and its variants by Intuit® of Mountain View, California. The information service provides one or more features that use the structured form representations and structured metadata generated by the system100. It should however be understood that the two microservices122,132are just for illustration; and the system100may include a large number of microservices.

The microservices122,132may, however, use different schemas. For example, the first microservice122may have been developed to solve a particular type of problem using a particular type of data and the second microservice122may have developed to solve another type of problem with another type of data. The schemas therefore may have different fields, different field names for the same information, different lengths of the fields, etc. For example, the first UI152amay be associated with the first microservice122and the second UI152bmay be associated with the second microservice132. The data captured using the first UI152amay therefore be incompatible with the second microservice132and the data captured using the second UI152bmay be incompatible with the first microservice122. Therefore, in accordance with the disclosed principles, a translation between the schemas is supported by the system100.

First server120, second server130, first database124, second database134, and client devices150are each depicted as single devices for ease of illustration, but those of ordinary skill in the art will appreciate that first server120, second server130, first database124, second database134, and or client devices150may be embodied in different forms for different implementations. For example, any or each of first server120and second server130may include a plurality of servers or one or more of the first database124and second database134. Alternatively, the operations performed by any or each of first server120and second server130may be performed on fewer (e.g., one or two) servers. In another example, a plurality of client devices150may communicate with first server120and/or second server130. A single user may have multiple client devices150, and/or there may be multiple users each having their own client devices150.

FIG.2shows an example architecture200for dynamic schema mapping, based on the principles disclosed herein. The example architecture200may be implemented by any combination of the components of the system100shown inFIG.1. It should be understood that the architecture200and its constituent components are just for illustration and should not be considered limiting. Architectures with additional, alternative, and fewer number of components should also be considered within the scope of this disclosure. Within the architecture200, a dynamic schema agent202generates a mapping between a first microservice204aand a second microservice204bbased on collected and analyzed network traffic data.

In the illustrated embodiment, the first microservice204auses a first data schema and the second microservice204buses a second data schema. In an example embodiment in which the microservices204a,204bare for used recordkeeping, the first microservice204auses a first set of data fields and associated attributes (e.g., length of entry for a particular data field) and the second microservice204buses a second set of data fields and associated attributes. As a non-limiting example, Table 1 shows a subset of various data fields for the first microservice:

Table 2 shows a subset of data fields of the example second microservice:

TABLE 2Subset of data fields of the second microserviceProduct without stocklevelsPictureIdentifierSupplier CodePurchase costOn handStock Keeping UnitProductTypeCost of Goods a/cAvailable StockMinimum Order QuantityVariant DescriptionRetail PriceDefault Sales AccountAsset a/cCustoms DescriptionsThis item is taxable

In the illustrated embodiment, the microservices204a,204buse different data schemas with different fields to store, retrieve, and process the same type of data records. The dynamic schema agent202automatically maps the different schemas such that the operations between the microservices204a,204bare compatible.

The dynamic schema agent202performs the mapping based on the data—associated with communication and processing by the microservices204a,204b—from several different sources. For example, the dynamic schema agent202may interrogate (e.g., request a piece of data) each microservice204a,204b.In one or more embodiments, a shared application fabric plugin206is provided to the microservices204a,204b,where the plugin206listens to the communication between the microservices204a,204b.The dynamic schema agent202gathers data from an event database210and or historical database208(e.g., the historical database208may receive the event data in the event database210in batches such as daily batches, weekly batches, etc.). Additionally, the dynamic schema agent202may gather data from another database212, which should be understood to be any kind of database used by one or more of the microservices204a,204b.

FIG.3shows details of a portion of the example architecture200shown inFIG.2, based on the principles disclosed herein. In particular, the components of the dynamic schema agent202are shown. As shown, the dynamic schema agent202receives batch data302from the historical database208and listens to the real-time data304of the event database210. Based on training models using the batch data302and or the real-time data304, the dynamic schema agent202generates and or updates schema mappers308. In other words, the schema mappers308include mapping models and or a probability distribution tensor, which are used for mapping and translating between several different schemas.

In addition to the schema mappers308, the illustrated dynamic schema agent202includes an application programming interface (API)306to interact with other components within the architecture200, such as the historical database208and the event database210. In some embodiments, the API306may be a REST API. The illustrated dynamic schema agent202also includes a data layer310that interfaces the schema mappers208with a mapping results database310. For example, the mapping results database310includes a probability distribution tensor that is used by the schema mappers310. The data layers310allow the schema mappers308to access and or update the probability distribution tensor in the mapping results database310.

FIG.4shows a flow diagram of an example method400of generating a schema mapping between different microservices, based on the principles disclosed herein. The schema mapping may be organized into a library and or a database (e.g., mapping results database310), which may be accessed when different schemas have to be converted. It should be understood that method400shown inFIG.4and described herein is just an example, and methods with additional, alternative, and fewer number of steps should be considered within the scope of this disclosure. The steps of the method400may be performed by one or more components of the system100shown inFIG.1and or one or more components of the architecture200shown inFIGS.2-3.

The method400begins at step402where dynamic schema agents (an example dynamic schema agent202is shown inFIGS.2-3) are installed. The dynamic schema agents may be installed at different points in a network. For example, the dynamic schema agents may be installed as plugins to different network nodes, e.g., microservices and or communications link in between. The installation of the dynamic schema agents at different points in the network is an example—and any kind of software and or hardware deployment to monitor network traffic between different microservices to map the schemas should be considered within the scope of this disclosure.

At step404, the dynamic schema agents are used to monitor network traffic. The network traffic may include communications, e.g., API calls, data exchange, etc., between the different microservices. The monitoring includes tracking the different source and destination microservices, e.g., a source and destination interceptor within the dynamic schema agent extracts the information on the source microservice and the destination microservice to determine where the communication is coming from and where it is going to.

At step406, an unsupervised training technique is used to train mapping models. The mapping models include machine learning models (e.g., within the schema mappers308shown inFIG.3), and the training utilizes the monitored traffic to train the machine learning models. It should be understood that step406can occur at any point in time for generating and or retraining the mapping models. For example, step406may be used for initial training using a threshold amount of initially gathered data. The initial training may generate initial models, which may then be further trained and reinforced as more and more network traffic data is monitored and gathered. In other example, the step406is used for retraining and or further training of a model already trained (e.g., through a previous iteration of the method400). The retraining may be used, for example, when a desired threshold accuracy changes, i.e., a prediction with a higher amount of accuracy is desired, and the retraining with additional data improves the prediction accuracy.

The training at step406may involve any kind of machine learning and or statistical model. The training generally determines different patterns within the different schemas to map different fields between the schemas such that the schemas become translatable and compatible. For instance, a first cluster of fields with different numbers and attributes in a first schema is translated using one or more trained mapping models to a second cluster of fields with other different numbers and attributes.

In some embodiments, the mapping models are naïve Bayesian models. The general principle of naïve Bayesian models is to determine classes for different instances. For example, a “customer price” field in a first schema and a “consumer price” in a second schema could be classified into the same class “retail price.” This classification is then used to map the specific instances. For instance, using the “retail price” classification (or generalization), the “customer price” field may be mapped to the “consumer price” field. The classification may be represented as key (i.e., class)-value (i.e., specific instance) pair: the key-value pairs for this example are retail price-customer price and retail price-consumer price.

To train the naïve Bayesian models (and or any other types of models), information from the network traffic is extracted. The extraction may use any kind of extraction methodology. For example, the text identifying the data (e.g., data field, data value) may be extracted from the network traffic. As another example, if the data traffic includes graphics to be rendered in a user interface (UI), image processing may be used to identify the text and extract the data therefrom. Any kind of extraction technique that extracts the data from the network traffic should be considered within the scope of this disclosure.

As described above, the training may be unsupervised. The naïve Bayes training algorithm (and or any other type of training algorithm) may determine the pattern in the network traffic data. For example, the training algorithm determines classes based on the observed instances. Some non-limiting examples of the classes may include: size of data values, length of key-value pairs, number of children for a particular field, ratio of verbs to nouns, and the like. These generalized classes are used for mapping: a field belonging to a class from a first schema may map to another field belonging to the same class from the second schema.

The network traffic data is monitored at multiple locations for multiple microservices thereby generating multiple key value pairs. Therefore, a three-dimensional probability distribution tensor (seeFIG.5) may be generated based on the training.

At step408, schemas between the microservices may be translated using the mapping models (e.g., by using the probability distribution tensor shown inFIG.5) At step410, errors in the translation are corrected using supervised training. For example, a first schema is translated into a second schema using the probability distribution tensor500. The second schema is sent to a receiving microservice. If the second microservice generates an error upon receipt of the second schema, an error condition is flagged, requiring human intervention to correct the error condition.

At step412, the mapping models are reinforced using the error correction of step410and or the continuous collection of the network traffic data. For instance, there may be a desired level of accuracy for the mapping models and or the probability distribution tensor500. The models may be retrained and reinforced until the desired level of accuracy is reached as reflected in the probability distribution tensor.

In some embodiments, the error corrections (e.g., of the translations between the schemas) include manual involvement and or training of additional error correction machine learning models. For example, the error corrections may include a supervised training, where the labels for the errors are hand-crafted and the error correction machine learning models are trained using the hand-crafted labels. The mapping models may therefore be continuously trained and reinforcement as new network traffic data and or translation error are available.

FIG.5shows an example of a probability distribution tensor500generated during the execution of the method400, based on the principles disclosed herein. Alternative representations should also be considered within the scope of this disclosure.

As shown, the probability distribution tensor500represents n key-value pairs (K1. . . Kn) for m microservices (MS1. . . MSm) at i instances. The dimensions of the probability distribution tensor500are therefore n*m*i. In operation, the network traffic data from the m microservices is monitored at i locations and for each location, the key-value matching is performed (e.g., using naïve Bayes models). Therefore, for each key-value for each microservice, there may be i probabilities. To take a specific example from the illustrated probability distribution tensor500, the key-value pair K1for microservices MS1has i probabilities P11.

FIGS.6A-6Cshow example mappings between two microservices, based on the principles disclosed herein. For example,FIG.6Ashows a first schema602of a first microservice and a second schema604of a second microservice. In the first schema602, a “variant” field has two entries “sku” and “price.” In the second schema604, the same entries “sku” and “price” are for a “product” field. In the first schema the “variant” field relates the “product” field. Therefore, there are two possibilities for the dynamic schema mapping, e.g., as executed by one or more mapping models and or the probability distribution tensor described above. The first possibility may be one product with multiple variants and the second possibility may be multiple products with one variant each.

FIG.6Bshows two mappings based on the first possibility of one product with multiple variants. For example, mapping606shows a composite of two instances of the second schema604(bundle-1and bundle-2). In the mapping606, product-1and product-2are variants of product category_name1and product-3may be variant of product category_name2. Therefore, product-1from bundle-1and product-2from bundle-2may therefore be mapped to a same generalized class category_name1. Furthermore, product-3, which is common to both bundle-1and bundle-2, may be mapped to generalized class category_name2. Additional schemas may be added based on this mapping606.

As another example, mapping608shows a composite of two instances of the first schema602(composition-1and composition-2). In the mapping608, variant-1(in composition-1) and variant-2(in both of the composition-1and composition-2) may be mapped to the same product-1of product_type_1. Furthermore, variant-3(in composition-2) may be mapped onto product-2of product_type_2. Therefore, the product (generalized class)-variant (specific instance) mapping therefore satisfies the first possibility of the one product with multiple variants. Based on this mapping, additional schemas may be added to the mapping608.

FIG.6Cshows two mappings based on the second possibility with multiple products with one variant each. Mapping610shows a composite of two instances of the second schema604(bundle-1and bundle-2). The mapping610may be same as the corresponding mapping606each of the product-1, product-2, and product-3may be a product with a single variant. Mapping612, which shows a composite of two instances of the first schema602may however be different from the corresponding mapping608. In particular, although product-1and product-2may be a single product_type-1, product-1may have a single variant-1and product-2may have a single variant-2. Furthermore, product-3of product_type-2may also have a single variant-3. Each of the mappings610and612may then be used for adding additional schemas.

FIG.7shows a flow diagram of an example method700of translating schemas between different microservices, based on the principles disclosed herein. It should be understood that method700shown inFIG.7and described herein is just an example, and methods with additional, alternative, and fewer number of steps should be considered within the scope of this disclosure. The steps of the method700may be performed by one or more components of the system100shown inFIG.1and or one or more components of the architecture200shown inFIGS.2-3.

The method begins at step702, where a communication from the first microservice intended for a second microservice is received. The communication comprises data from the first microservice sent to the second microservice. At step704, it is determined whether the first microservice uses a first schema and the second microservice uses a second schema. The determination is based on the analysis of the communication packets, e.g., text detection, pixel detection to identify text, etc. At step706, data in the first schema is dynamically translated using one or more machine learning models to generate data for a second schema. The one or more machine learning models may include, e.g., a naïve Bayes model. At step708, a modified communication is generated by including the translated data in the second schema. At step710, the modified communication is transmitted to the second microservice.

Therefore, using the several embodiments disclosed herein, the schemas shown in Table 1 (a first microservice) and Table 2 (a second microservice) may be mapped as shown in Table 3:

FIG.8shows a block diagram of an example computing device800that implements various features and processes, based on the principles disclosed herein. For example, computing device800may function as first server120, second server130, client150a,client150b,or a portion or combination thereof in some embodiments. Additionally, the computing device800partially or wholly forms the architecture200and or wholly or partially hosts the dynamic schema agent202. The computing device800also performs one or more steps of the methods400and700. The computing device800is implemented on any electronic device that runs software applications derived from compiled instructions, including without limitation personal computers, servers, smart phones, media players, electronic tablets, game consoles, email devices, etc. In some implementations, the computing device800includes one or more processors802, one or more input devices804, one or more display devices806, one or more network interfaces808, and one or more computer-readable media812. Each of these components is be coupled by a bus810.

Display device806includes any display technology, including but not limited to display devices using Liquid Crystal Display (LCD) or Light Emitting Diode (LED) technology. Processor(s)802uses any processor technology, including but not limited to graphics processors and multi-core processors. Input device804includes any known input device technology, including but not limited to a keyboard (including a virtual keyboard), mouse, track ball, and touch-sensitive pad or display. Bus810includes any internal or external bus technology, including but not limited to ISA, EISA, PCI, PCI Express, USB, Serial ATA or FireWire. Computer-readable medium812includes any non-transitory computer readable medium that provides instructions to processor(s)802for execution, including without limitation, non-volatile storage media (e.g., optical disks, magnetic disks, flash drives, etc.), or volatile media (e.g., SDRAM, ROM, etc.).

Computer-readable medium812includes various instructions814for implementing an operating system (e.g., Mac OS®, Windows®, Linux). The operating system may be multi-user, multiprocessing, multitasking, multithreading, real-time, and the like. The operating system performs basic tasks, including but not limited to: recognizing input from input device804; sending output to display device806; keeping track of files and directories on computer-readable medium812; controlling peripheral devices (e.g., disk drives, printers, etc.) which can be controlled directly or through an I/O controller; and managing traffic on bus810. Network communications instructions816establish and maintain network connections (e.g., software for implementing communication protocols, such as TCP/IP, HTTP, Ethernet, telephony, etc.).

Dynamic schema mapping instructions818include instructions that implement the disclosed process for a mapping between different schemas, as described throughout this disclosure.

Application(s)820may comprise an application that uses or implements the processes described herein and/or other processes. The processes may also be implemented in the operating system.

To provide for interaction with a user, the features may be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer.