SERVERLESS DATA-REPRESENTATION-AS-A-SERVICE (DRAAS) TO ENABLE BUILDING GENERAL MULTI-MODAL INPUT DATA ML FLOWS

Techniques for enabling the building of general input data ML flows using a serverless data-representation-as-a-service (DRaaS) are provided. In one technique, in response to receiving a first data representation (DR) generation request from a first calling entity, first input data is retrieved based on the first DR generation request, a first set of DRs is generated (by a DR generator) based on the first input data, and the first set of DRs are made available to the first calling entity. In response to receiving a second DR generation request from a second calling entity that is different than the first calling entity, second input data is retrieved based on the second DR generation request, a second set of DRs is generated based on the second input data, and the second set of DRs are made available to the second calling entity.

BACKGROUND

When building a machine learning (ML) pipeline to solve, the available input data is often of different modality types. Input data modalities can be of different types, such as tabular, time series, text, image, video, documents, and audio. For example, in order to train a machine learning model to do product classification for an enterprise, the input data might consist of a product title and description in the text modality, images and/or video for the product, tabular attributes (e.g., Manufacturer Name, Brand Name, etc.), and documents, such as a Safety Data Sheet (needed for compliance), etc. How to build the most performant ML modeling pipeline for multi-modal data inputs is a hot research area in the data science community.

Currently, data scientists rely on significant manual coding and experimentation with different data representation techniques for each modality, trying early or late fusion ML models that combine those signals. This is a very time consuming process and often leads to models that are not the best performing.

Another large application of data representations is in single-modal input data setting where a downstream use case exists for object similarity, grouping, or ranking for the particular input data type. Performing a similarity image search given an input query image and identifying duplicate documents for a given seed document are two examples of a single-modal input data setting. If there was a system that generates data (e.g., vector) representations of the images or documents, then finding most similar images (or duplicate documents) is primarily a matter of applying a similarity function (e.g., cosine similarity) on top of those data representations.

DETAILED DESCRIPTION

General Overview

Currently, there are no tools, frameworks, libraries, or services that can provide automated data ingestion and “featurization/representation-as-a-service” for a given modality type that data scientists can readily leverage in their ML flows. In other words, there is no way to send a raw data object of any type to an online service and receive, in return, a meaningful vector representation of that raw data object. As more and more machine learning use cases require input data in multiple modalities, automated data ingestion and featurization is a critical gap in current open source tools and machine learning platforms.

A system and method for providing a cloud service that generates and provides data representations of input data objects are described. In one technique, a user initiates and sends a client data representation (DR) generation request (that is associated with one or more data objects) to the cloud service, the cloud service generates a data representation (e.g., a vector or an embedding) for each data object, and returns the generated data representation(s) to the user as a response to the DR generation request. In this way, data scientists do not need to spend time in feature engineering or developing and training DR generators (or the models upon which the DR generators are based). Also, embodiments effectively lower the skill level required to generate and maintain ML workflows that operate on data representations of data objects. Furthermore, the cloud service may offer better quality compared to other approaches.

Therefore, embodiments improve computer-related technology by providing a DRaaS (data representation-as-a-service) service for various input modalities, which service can be easily leveraged in ML workflows of various configurations. Thus, instead of providing Al Service APIs that take raw data input and return a prediction for a pre-defined use case, embodiments involve a serverless architecture where data representations are returned to the client. The client then has the flexibility to use these data representations in downstream ML flows and customize for specific use cases.

Embodiments described herein represent a significant departure from existing Al services paradigms, which provide predictions for a select number of pre-defined use cases, and do not allow for flexibility or customization of (1) what is being predicted or (2) tuning the featurization and an ensemble model process. For example, cloud providers, as part of their ML/AI offering, offer a collection of Al services that include pre-built, optimized ML models on a single input data modality and are focused on a narrow range of use cases for that input data modality, such as sentiment classification for input text, object detection for input images, and anomaly detection for input time series telemetry data. Having a DRaaS service provide data representations of customer reviews (input data modality of text), for example, allows data scientists to quickly customize for a fake review detection use case. In contrast, current language Al services as built can only be used for general use cases, such as sentiment analysis of the customer reviews, text classification, keyword detection, language detection, named entity recognition, and personal identifying information (PII) detection. Current language Al services are not built to answer a specific use case, such as product classification. Therefore, an ML practitioner needs to further develop on top of these outcomes to tailor and solve for a specific use case.

Many real world use cases typically need multiple input data modalities that need to be intelligently featurized and fused in downstream ML models in order to achieve high performance (seeFIG.1). This is different from a mixture of experts which are typically trained on each modality separately and have their predictions combined using some majority voting or other aggregation logic. Embodiments provide individual data modality data representations (not predictions) and, optionally, a joint representation across all modalities. In principle, such embodiments have the potential to yield superior performance over models that have only seen a subset of the predictive features and generate predictions only on those.

General ML Workflow

FIG.1is a block diagram that depicts an example of a general ML workflow100. Workflow100comprises multiple input data sources101-106, each corresponding to a different data modality or type. Input data sources101-106include tabular data, time series data, text data, image/video data, document data, and audio data. The first two modality types are structured data while the latter four modality types are unstructured.

Workflow100also comprises a block110that represents the majority of the work and effort, which is producing data transformations, performing feature engineering, and generating and training one or more ML models. Output from the ML models of block110are predictions120, which lead to decisions130, such as deduplication, grouping, recommending, etc.

Block110represents an opportunity for ML practitioners to leverage vector representations provided by a DRaaS service. With a DRaaS service, ML practitioners only need to work on generating and training one or more downstream ML models. The DRaaS service represents a general paradigm/pattern for building a custom ML pipeline, on any platform, and solves for any use case for any available data input modalities.

On any given input data modality, a DRaaS service is called, generates data representations (vector/matrix format) based on the input data that the DRaaS service ingests (seeFIG.2), and provides those data representations to the calling entity (or to a destination storage location specified in the call or by the caller). The DRaaS service may be called via an API, a software development kit (SDK), or an operator from a data science platform. The DRaaS service may be implemented in software, hardware, or a combination of software and hardware. The DRaaS service may be hosted on any cloud platform.

Each data representation of each data object of an input data modality type is represented as a numerical vector, which is available as is for further custom fusion algorithms to ingest in a general ML flow (seeFIG.3). Alternatively, numerical vectors can be used in single-modal data use cases (e.g., for similarity search/ranking/grouping), where such functions/models are to be applied (seeFIG.4).

FIG.2is a block diagram that depicts different input data modalities to a DRaaS service200, in an embodiment. In an embodiment, DRaaS service200includes a data representation (DR) generator for each of multiple input data modalities. In the depicted example, the input data modalities include text210, image/video220, document230, audio240, and time series or tabular250. The DRaaS service includes a language DR generator212, a vision DR generator222, a document Al DR generator232, a speech DR generator242, decision DR generator252. AlthoughFIG.2depicts a single DR generator for time series input data and tabular input data (which are two different types of input data modalities), there may be a separate DR generator for each. The output of the DR generators are data representations260.

WhileFIG.2depicts each generated/output data representation as a single dimension vector of values, one or more of the DR generators may output an N×M matrix of values. A DR generator takes in input (e.g., text string, a document, a two-dimensional image, time series data, an audio file) and may transform the input into a form that is acceptable to a model that the DR generator comprises. The model may be a trained neural network that takes transformed input and produces output (e.g., a vector or matrix of values) of a certain size.

The model of each DR generator may be trained on a large corpus of general data. For example, a model for vision DR generator222may be trained to recognize hundreds of different types of objects, from man-made objects to objects found in nature. As another example, a model for language DR generator212may be trained to recognize many types of sentiment reflected in textual data found in customer reviews. Alternatively, the models of some DR generators may be trained on vertical data sets. For example, there may be two DR services for documents: one that is specialized for clinical notes and another that is specialized for legal documents (e.g., contracts).

Data representations may be unsupervised or supervised depending on whether labels were available to DRaaS service200. For example, some DR generators (or models of DR generators) of DRaaS service200may have been trained using an unsupervised machine learning technique while some DR generators of DRaaS service200may have been trained using a supervised machine learning technique. Examples of supervised machine learning techniques include decision tree, logistic regression, linear regression, and support vector machines. Examples of unsupervised machine learning techniques include K-means clustering, principal component analysis, hierarchical clustering, and various deep learning autoencoder architectures.

In an embodiment, DRaaS service200includes multiple calling endpoints, each endpoint corresponding to a different input data modality type. For example, if calling entities desire to use a language DR generator of DRaaS service200, then the calling entities send DR generation requests to a first endpoint corresponding to the language DR generator. If calling entities desire to use a speech DR generator of DRaaS service200, then the calling entities send DR generation requests to a second endpoint corresponding to the speech DR generator.

General Paradigm

FIG.3is a block diagram that depicts a general paradigm300for solving any use case where multi-modal data inputs are present, in an embodiment. Paradigm300may represent a specific embodiment that includes an input data modality310, a DRaaS service320, other input data312, a feature engineering library322, a custom fusion ML model330, output predictions340, and decisions350. In general paradigm300, DRaaS service320generates and provides data representations, which enables downstream custom fusion models in multi-modal data scenarios. A “fusion” model is one that accepts data representations as input that are from different data sources.

While DRaaS service320has a line connecting DRaaS service320with custom fusion ML model330, this does not necessarily mean that DRaaS service320communicates with custom fusion ML model330. Instead, the output from DRaaS service320is input to custom fusion ML model330.

Because the model of DRaaS service320has not been trained in conjunction with the training of custom fusion ML model330, the accuracy of data representations that DRaaS service320generates might be relatively low. However, any such “lost ground” may be made up in the training of custom fusion ML model330based on those data representations.

Single Use Cases

FIG.4is a block diagram that depicts an example system400that provides individual data type representations that enables single-modal use cases, in an embodiment. System400includes input data sources402-410, DRaaS service420, and DR generators422-430that are part of (e.g., components or sub-services of) DRaaS service420. Each DR generator corresponds to a different input data modality, such as text and images. Specifically, text input402is processed by a language DR generator422, image/video input404is processed by a vision DR generator424, document input406is processed by document Al DR generator426, audio input408is processed by speech DR generator428, and time series/tabular input410is processed by decision DR generator430. Each generator generates a data representation, such as a vector of size N or a matrix of size N×M. Different generators may generate data representations of different sizes.

Such output data representations are then consumed, respectively, by one of trained similarity/clustering ML models442-450. Each of similarity/clustering ML models442-450may operate differently. For example, ML model442may be a text similarity ML model that, given two data representations as input, generates output that indicates whether the data objects (e.g., two text strings) represented by the two data representations are similar. As another example, ML model444may be a clustering ML model that takes many (e.g., a batch of) data representations as input and outputs one or more clusters (or groups) of data representations, each cluster representing a set of data representations (e.g., of images) that ML model444predicts are similar enough to each other that the set of data representations should be clustered or grouped together and treated as distinct from other clusters. As currently represented in FIG.4, similarity/clustering is done separately on each DR output. However, in principle, the similarity/cluster computation may be done across modalities if more than one modality is present in a request from an entity.

The output of each similarly ML model is an indication of whether two data objects are similar (whose data representations were input to the similarity ML model) and the output of each clustering ML model is a set of one or more clusters, each cluster including references to one or more data objects whose data representations were input to the clustering ML model.

Joint Multi-Modal Representation

In an embodiment, a DRaaS system generates multiple data representations of different modalities and aggregates those data representations into a single joint multi-modal representation.FIG.5is a block diagram that depicts an example system500for generating and providing a joint multi-modal representation, in an embodiment. System500includes input data sources502-510, DRaaS service520, DR generators522-530that are part of (e.g., components or sub-services of) DRaaS service520, and an aggregation model/operator540, which is also part of DRaaS service520. (Aggregation model/operator540may be similarity/clustering output.) Thus, system500is similar to system400except that multiple data representations that are generated by two or more of DR generators522-530are aggregated or combined in some way to generate a single data representation.

The type of aggregation might depend on one or more downstream task outcomes, such as product classification, user recommendation, or fraud detection. Then, a ML practitioner works backwards from those outcomes (labels) to tune the aggregation of the different modality types. “Tuning the aggregation” means improving the overall quality of the joint data representation via providing fine outcome labels and re-training the model.

If aggregation model/operator540is an operator, then example operations include concatenation, mean, median, min, max, percentile. For example, if two single modal data representations are of length N and M, respectively, and the operation is concatenation, then the length of a joint data representation is length N+M. As another example, if three single modal data representations are of length N and the operation is mean, then the length of a joint data representation is length N.

If aggregation model/operator540is a model, then the model may output a joint data representation that is larger, smaller, or the same size as individual input data representations. For example, the model may be a neural network with one or more hidden layers. The model has been trained based on outputs from data generators522-530.

The output of aggregation model/operator540may be used by a ML model that is downstream relative to DRaaS service520, that is developed by a ML practitioner (and customer of DRaaS service520), and that has been trained by outputs from aggregation model/operator540.

Calling a DRAAS Service

A computing (calling) entity (e.g., a process, a program, a cloud application, a computing device executing code) may call a DRaaS service in one of two main modes: a single input mode or a batch mode. In single input mode, the DRaaS service receives a single data object (e.g., a text string, a document, an image, an audio file, a video file) and generates a single data representation based thereon. A call to the DRaaS service may include the data object or a reference to a storage location where the data object is stored. The storage location may be remote or local relative to the DRaaS service. If remote, the storage location may be managed by a third-party storage service and the reference (or storage location identification data) may include credential data that allows the DRaaS service to access the storage location.

Once the DRaaS service generates a data representation, the DRaaS service may return the data representation to the computing entity that called the DRaaS service. Alternatively, the DRaaS service stores the data representation at a (local or remote) location associated with the calling computing entity, which location may be specified in the triggering call.

In batch mode, a DRaaS service receives a batch of multiple data objects in a single call or identifies multiple data objects in response to a single call (from a computing entity that requests data representations for data objects) and generates multiple data representations, one for each data object in the batch. A call to the DRaaS service may include a reference (or storage location identification data) to a first storage location where the batch of data objects are stored. The DRaaS service uses first storage location identification data (e.g., specified in the call) to retrieve the batch of data objects (e.g., one by one), generates a data representation for each data object in the batch, and stores the data representation at a second storage location using second storage location identification data (which may have also been specified in the call), which may be associated with the first storage location. The DRaaS service may inform the calling computing entity when all the requested data representations have been generated and are accessible to the calling computing entity, especially if the number of data objects in the batch are over a certain size or if the time to generate the data representations exceed a threshold amount of time.

For each data representation that a DRaaS service generates, the DRaaS service may also generate data representation (DR) identification data that uniquely identifies the data representation relative to other data representations that the DRaaS service generates, whether universally or for the calling computing entity (or organization with which the calling computing entity is associated). For example, each data object (for which the DRaaS service generates a data representation) has a name and the DRaaS service appends one or more characters to that name. As a specific example, the characters that the DRaaS service appends are “_DR.” Additionally or alternatively, the DRaaS service appends a timestamp of when (a) the corresponding data representation is generated or (b) the request to generate one or more data representations was received.

In an embodiment, the calling entity (i.e., that sends a DR generation request to a DRaaS service) is part of a data science (DS) platform that is hosted on the same cloud on which a DRaaS service is hosted. The DS platform allows ML practitioners to generate ML workflows. An ML workflow comprises one or more operands and operators, one of which corresponds to a DRaaS service. Thus, the DRaaS service may be invoked as if it is an operator. An operand of such an operator is a single data object (e.g., an image, a document, or a video file), a batch of data objects (e.g., multiple text strings or multiple images), or a reference to whether the data object (or batch) is located. Another operator in a ML workflow may be a ML model that takes output of one or more other operators as input. When a series of operators is executed, and one of those operators corresponds to the DRaaS service, then at least a portion of the output of the DRaaS service may be input into a downstream operator in the series, such as a similarity operator, a clustering operator, a classification operator, etc.

Customization of Embeddings

Because a user of a DRaaS service relies on the training of the data representation generator(s) of the DRaaS service to generate data representations based on input data objects, the training might not be optimal for the subject matter area of interest of the user. For example, an image DR generator of a DRaaS service may be trained to recognize mammals in digital images while a user of the DRaaS service needs to leverage the image DR generator when making decisions based on images of street signs. Therefore, the image DR generator might generate data representations that reflect the existence of mammals but not the existence of street signs.

In an embodiment, a DRaaS service (or a DR generator thereof) is customized for a specific user or organization. A DR generator may be customized by copying the model upon which the DR generator is based and updating or re-training the model using additional training data. For example, a user causes a customization request to be transmitted to a DRaaS service. The customization request may include training data or storage location identification data that identifies a storage location where the training data is stored. The training data comprises a set of training instances. Each training instance in the set includes a data object (e.g., text string, document, or image) and a label. If the DR generator is classifier, then the label is one of multiple values, each value corresponding to a different class. For example, if the DR generator is an image classifier, then the training instance comprises an image and a label that may indicate whether the image depicts a certain type of object.

The customization request may also indicate an input modality type (e.g., text, image, document, video, or audio) that corresponds to the DR generator. In this way, DRaaS service knows which DR generator (if there are multiple DR generators) to update or retrain. Thus, a customization request is fundamentally different than a DR generation request for one or more data representations.

In response to receiving a customization request, the DRaaS service accesses a set of training instances, creates a copy of the ML model corresponding to the indicated input modality type, and trains that copy based on the set of training instances, resulting in a trained copy. The DRaaS service stores an entity ID in association with the trained copy, the entity ID uniquely identifying the entity (e.g., user or organization) that requested the customization. Thereafter, the DRaaS service invokes the trained copy when it receives DR generation requests from that entity.

A DR generation request from an entity that is associated with a customized version of DR generator may include customization data that indicates whether the customized version of the DR generator should be leveraged or whether a non-customized version of the DR generator should be leveraged. For example, a DR generation request may include a value in a particular position of the DR generation request where the value indicates that a customized version of a DR generator is to be used. If an entity is associated with multiple customized versions of a DR generator, then the value may specify or indicate which customized version to use. If a DR generation request does not include a value in the particular position of the DR request, then a generic DR generator is used, even if the requesting entity is associated with one or more customized versions of the DR generator.

Example Process

FIG.6is a flow diagram that depicts an example process600for providing data representations as a service, in an embodiment. Process600is implemented as a cloud service to any type of third-party calling entity. The calling entity is operated or instructed by an end user, which may be a representative of an organization. The end user may have an account with the cloud service, which account may require credentials (e.g., username and password) to access.

At block610, a data representation (DR) generation request is received from a calling entity. The calling entity may be an entity executing on the same cloud platform as the cloud service. Alternatively, the calling entity may be executing on platform or device that is remote relative to the cloud service. For example, the DR generation request may be an HTTP request that is sent over one or more networks (including the Internet) to the cloud platform on which the cloud service is executing. The DR generation request may include input data that will be input to a DR generator of the cloud service or may include storage location identification data that identifies a storage location in which the input data is stored.

At block620, input data is retrieved based on the DR generation request. Block620may involve the cloud service retrieving the input data from the DR generation request. Alternatively, if the DR generation request instead includes storage location identification data, then the cloud service uses the storage location identification data to retrieve the input data, such as by sending a (e.g., HTTP) request (e.g., to a third-party storage service) or making an API call that includes the storage location identification data or a portion thereof.

At block630, a DR generator of the cloud service generates, based on the input data, a set of one or more data representations. If the input data is a single data object (e.g., an image file), then the DR generator generates a single data representation, such as a single dimension vector of values. If the input data comprises multiple data objects (e.g., multiple documents), then the DR generator generates a data representation for each of the multiple data objects.

Block630may first involve selecting a DR generator from among multiple DR generators, each DR generator corresponding to a different input modality type, such as text, document, image, video, and audio. Such selecting may involve determining an input modality type associated with the DR generation request and then matching that to one of the DR generators. Determining the input modality type associated with the DR generation request may involve identifying an input modality type indicator in the DR generation request or determining the input modality type of the input data.

At block640, the set of one or more data representations are made available to the calling entity. Block640may involve sending a response to the calling entity that includes the set of one or more data representations. Alternatively, the response may include storage location identification data that identifies where the set of one or more data representations is stored. In this scenario, the calling entity uses the storage location identification data to retrieve the set of one or more data representations. Alternatively, the cloud service stores the set of one or more data representations in association with an account that is associated with the calling entity.

If the DR generation request includes only a single data object (or a reference to a single data object) and the calling entity requires data representations for multiple data objects, then the calling entity may generate a different DR generation request for each of the data objects. Thus, process600may be repeated with respect to the same calling entity but a different DR generation request and a different data object (and, thus, a different generated data representation).

Also, process600may be repeated but for different calling entities. The different calling entities may leverage the same DR generator (especially if the cloud service only includes a single DR generator) or different DR generators. Some DR generation requests may be considered “batch” requests (meaning each request includes or refers to multiple input data objects), whereas other DR generation requests (e.g., from other calling entities) may be considered “single” requests (meaning each request includes or refers to a single input data object).

Hardware Overview

Computer system700further includes a read only memory (ROM)708or other static storage device coupled to bus702for storing static information and instructions for processor704. A storage device710, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus702for storing information and instructions.

Software Overview

FIG.8is a block diagram of a basic software system800that may be employed for controlling the operation of computer system700. Software system800and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

Software system800is provided for directing the operation of computer system700. Software system800, which may be stored in system memory (RAM)706and on fixed storage (e.g., hard disk or flash memory)710, includes a kernel or operating system (OS)810.

The OS810manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as802A,802B,802C . . .802N, may be “loaded” (e.g., transferred from fixed storage710into memory706) for execution by the system800. The applications or other software intended for use on computer system700may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, an app store, or other online service).

Software system800includes a graphical user interface (GUI)815, for receiving user commands and data in a graphical (e.g., “point-and-click” or “touch gesture”) fashion. These inputs, in turn, may be acted upon by the system800in accordance with instructions from operating system810and/or application(s)802. The GUI815also serves to display the results of operation from the OS810and application(s)802, whereupon the user may supply additional inputs or terminate the session (e.g., log off).

OS810can execute directly on the bare hardware820(e.g., processor(s)704) of computer system700. Alternatively, a hypervisor or virtual machine monitor (VMM)830may be interposed between the bare hardware820and the OS810. In this configuration, VMM830acts as a software “cushion” or virtualization layer between the OS810and the bare hardware820of the computer system700.

VMM830instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine comprises a “guest” operating system, such as OS810, and one or more applications, such as application(s)802, designed to execute on the guest operating system. The VMM830presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.

In some instances, the VMM830may allow a guest operating system to run as if it is running on the bare hardware820of computer system700directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware820directly may also execute on VMM830without modification or reconfiguration. In other words, VMM830may provide full hardware and CPU virtualization to a guest operating system in some instances.

In other instances, a guest operating system may be specially designed or configured to execute on VMM830for efficiency. In these instances, the guest operating system is “aware” that it executes on a virtual machine monitor. In other words, VMM830may provide para-virtualization to a guest operating system in some instances.

Cloud Computing