REDUCING INFERENCE TIME PROCESSING DEEP LEARNING MODELS

Various embodiments are provided herein for decreasing a central processing unit (CPU) inference time, thereby shortening processing time of a deep learning model for which run-time complexity is proportional to an output sequence length. A compacted sequence is developed that is used to train the deep learning model. An output of the trained model becomes the compacted sequence on which a post-processing method operates to create an uncompacted version of a predicted sequence.

BACKGROUND

In computer programming, structured query language (SQL) is a standardized programming language that is used to manage relational databases and perform various operations on the data in them. So-called “Text-2-SQL” refers to the tasks of transforming a question (in natural language format) into an executable SQL query. From the perspective of deep learning networks (DL), text-2-SQL is a translation process. As one skilled in the art will appreciate, the DL approach requires training data for creating a model. So-called “Spider Corpus,” which in one embodiment, may be located at https://yale-lily.github.io/spider, is a widely-used examples of a training data repository, including thousands of unique questions that are cross referenced to thousands of unique, complex SQL queries on hundreds of databases.

Foundation Models (e.g., transformed-based models like T5) have been shown to have top performance on several Natural Language Processing (NLP) tasks, including Text-2-SQL, and translate a natural language question to its corresponding SQL query.

Clients typically do not have access to graphics processing units (GPUs) needed to run foundations models quickly. Instead, clients usually seek models that can be run on a Central Processing Unit (CPU) with high accuracy and low execution.

Transformers-based models like T5, having an Encoder-Decoder architecture, usually run very slow in a CPU environment, compared to its typical runtime on a GPU. The overall inference time is dominated by the decoder and is directly proportional to the number of characters of the output SQL query. In some cases, the encoder runs only once, but the decoder runs as many times as characters in the output sequence require. Thus, in situations where CPU use results in longer inference, and thereby processing times for deep learning applications, a need exists to develop solutions to shorten processing times when performing these tasks.

SUMMARY

According to an embodiment of the present invention, a method for decreasing a central processing unit (CPU) inference time, thereby shortening processing time of a deep learning model for which run-time complexity is proportional to an output sequence length, is provided. A compacted sequence is developed that is used to train the deep learning model. An output of the trained model becomes the compacted sequence on which a post-processing method operates to create an uncompacted version of a predicted sequence.

An embodiment includes a computer usable program product. The computer usable program product includes a computer-readable storage device, and program instructions stored on the storage device. The program instructions are used to develop a compacted sequence that is used to train the deep learning model. An output of the trained model becomes the compacted sequence on which a post-processing method operates to create an uncompacted version of a predicted sequence.

An embodiment includes a computer system. The computer system includes a processor, a computer-readable memory, and a computer-readable storage device, and program instructions stored on the storage device for execution by the processor via the memory. When executing the program instructions, the processor develops a compacted sequence that is used to train the deep learning model. An output of the trained model becomes the compacted sequence on which a post-processing method operates to create an uncompacted version of a predicted sequence.

Optionally, in certain embodiments, a formal language is used which is constrained to a grammar mechanism, and the grammar mechanism is implemented to create the uncompressed version of the predicted sequence.

Optionally, in certain embodiments, the post-processing method is executed, and the uncompacted version of the predicted sequence is created.

Optionally, in certain embodiments, during a training phase: at least one of a column name, a value in question, and a query is replaced by a shorter one of a column name, a shorter value in question, and a shorter query, and during an inference phase, reversing the shorter one of the column name, the shorter value in question, and the shorter query with the at least one of the column name, the value in question, and the query.

Optionally, in certain embodiments, developing the compacted sequence used to train the deep learning model further includes transforming a SQL query to a compacted, pseudo-SQL sequence for training a Text-2-SQL DL model.

Optionally, in certain embodiments, the Text-2-SQL DL model is used on a short-length pseudo-SQL dataset.

Optionally, in certain embodiments, the length of an SQL query is shortened using an automatic Short-SQL-Transform process by removing a FROM clause.

DETAILED DESCRIPTION OF THE DRAWINGS

As previously mentioned, so-called “Text-2-SQL” refers to the tasks of transforming a question (in natural language format) into an executable SQL query. From the perspective of deep learning networks (DL), text-2-SQL is a translation process. As one skilled in the art will appreciate, the DL approach requires training data for creating a model. So-called “Spider Corpus,” which in one embodiment, may be located at https://yale-lily.github.io/spider, is a widely-used examples of a training data repository, including thousands of unique questions that are cross referenced to thousands of unique, complex SQL queries on hundreds of databases.

Foundation Models (e.g., transformed-based models like T5) have been shown to have top performance on several Natural Language Processing (NLP) tasks, including Text-2-SQL, and translate a natural language question to its corresponding SQL query.

Clients typically do not have access to graphics processing units (GPUs) needed to run foundations models quickly. Instead, clients usually seek models that can be run on a Central Processing Unit (CPU) with high accuracy and low execution.

Transformers-based models like T5, having an Encoder-Decoder architecture, usually run very slow in a CPU environment, compared to its typical runtime on a GPU. The overall inference time is dominated by the decoder and is directly proportional to the number of characters of the output SQL query. In one embodiment, the encoder runs only once, but the decoder runs as many times as characters in the output sequence require.

To address situations such as the one previously described where CPU use results in longer inference, and thereby processing times for deep learning applications, various aspects of the mechanisms of the illustrated embodiments are proposed to, among other aspects, transform an input sequence to a shorter sequence to speed up the CPU inference time of deep learning models for which run-time complexity is proportional to the output sequence length (e.g., transformed-base models). This shorter sequence, or “compacted sequence” is used to train a transformed-based model on a specific task, for example. The output of the trained model will, in one embodiment, thereby will be a compacted sequence on which a post-processing method operates to create a full version of the predicted sequence. For formal languages constrained to a grammatical aspects, the mechanisms of the illustrated embodiments may, in one example, utilize the grammatical aspects to re-create the full, original version of the predicted sequence.

In additional embodiments, various aspects of the present invention provide mechanisms to transform SQL queries to a shorter, “Pseudo-SQL” version for training, again for example, a Text-2-SQL DL model, and use fast post processing methods to compose the final SQL query, thus speeding up CPU inference time while maintaining the deep learning model's accuracy. In one example, a method is provided, as implemented through a computer code executing on a computer processing device, to minimize CPU interference time of transformers-based Text-2-SQL by training on short length Pseudo-SQL datasets.

In further embodiments, various mechanisms are provided which implement an automatic Short-SQL-Transform process, which, in one aspect, shortens the length of a conventional SQL query by removing the “FROM” clause. Column names, in one exemplary embodiment, are automatically replaced by shorter names during training. During inference, the shortened column names are reversed back to the original names/nomenclature. Values in question and query are automatically replaced by shortened values. During inference, short values are reversed back to original values. An automatic Full-SQL-Transform process recreates the full SQL from the pseudo-SQL predicted by the Text-2-SQL Deep Learning model.

It should be noted that one or more calculations may be performed using various mathematical operations or functions that may involve one or more mathematical operations (e.g., solving differential equations or partial differential equations analytically or computationally, using addition, subtraction, division, multiplication, standard deviations, means, averages, percentages, statistical modeling using statistical distributions, by finding minimums, maximums or similar thresholds for combined variables, etc.).

In general, as may be used herein, “optimize” may refer to and/or defined as “maximize,” “minimize,” “best,” or attain one or more specific targets, objectives, goals, or intentions. Optimize may also refer to maximizing a benefit to a user (e.g., maximize a trained machine learning scheduling agent benefit). Optimize may also refer to making the most effective or functional use of a situation, opportunity, or resource.

Additionally, optimizing need not refer to a best solution or result but may refer to a solution or result that “is good enough” for a particular application, for example. In some implementations, an objective is to suggest a “best” combination of operations, schedules, PE's, and/or machine learning models/machine learning pipelines, but there may be a variety of factors that may result in alternate suggestion of a combination of operations, schedules, PE's, and/or machine learning models/machine learning pipelines yielding better results. Herein, the term “optimize” may refer to such results based on minima (or maxima, depending on what parameters are considered in the optimization problem). In an additional aspect, the terms “optimize” and/or “optimizing” may refer to an operation performed in order to achieve an improved result such as reduced execution costs or increased resource utilization, whether or not the optimum result is actually achieved. Similarly, the term “optimize” may refer to a component for performing such an improvement operation, and the term “optimized” may be used to describe the result of such an improvement operation.

As previously mentioned, the illustrated embodiments of the present invention introduce, among other aspects, mechanisms for speeding up CPU inference time while maintaining a Deep Learning model's accuracy, for example, of a Text-2-SQL DL model, through the transformation of SQL queries. For example, an automatic Short-SQL Transform process reduces the length of an SQL query, by removing the FROM clause, and performing mapping to shorter table, column, value, and keyword mappings. Tables and columns are sorted lexicographically, and shorter names are used instead of original names (e.g. T1 in lieu of Table 1), before query execution column names are mapped back to original names.

In further embodiments, a trained Deep Learning (DL) model on Pseudo-SQL queries is used to predict Pseudo-SQL queries. A Full-SQL-Transform process introduces back from the FROM clause, along with other mappings, to an inferred Pseudo-SQL query generated by a DL model. Short-SQL-Transform processes, according to those as will be described herein in accordance with the mechanisms of the present invention, may reduce the original SQL queries used during training, by mapping keywords and values to shorter words (e.g., SELECT becomes S #, and “UNIT_4_BUSINESS_NY” becomes “V1”). The Pseudo-SQL Query Transformation Process improves the Inference time of the Text-2-SQL model as processed on a CPU, again as will be further described.

To briefly summarize a previously described embodiment thus far, a Text-2-SQL dataset is updated to shorten Pseudo-SQL queries using a Short-SQL-Transform protocol. A Deep Learning model is then trained (e.g., transformer-based model) on a particular Pseudo-SQL dataset (e.g., Spider after transformation to Pseudo-SQL). Finally, during an inference phase, the trained Deep Learning model is leveraged to generate the Pseudo-SQL, and pass the compressed queries to a Full-SQL-Transform process, to regenerate the uncompressed, final SQL.

To transform SQL queries into a shorter, compressed Pseudo-SQL Representation, in one embodiment, a Short-SQL-Transform process may be implemented. This transform process converts table names, column names, values and keywords, for example. As a next step, the “FROM” clause is removed. Finally, “NESTED” clauses are designated with { } brackets, versus ( ) parentheses. In one embodiment, as a result, parentheses are used to designate other components of SQL queries, but curly brackets are reserved for nested queries. Once the Pseudo-SQL is returned into postprocessing, the aforementioned curly brackets are then replaced with the original parentheses.

To regenerate the uncompressed, full SQL query, then the Full-SQL-Transform process may be leveraged to generate the final SQL by updating the inferred SQL from the model with the corresponding “FROM” clause and nested subqueries, if any, using a fast-post-processing transformation method.

Consider the following example of creating a compacted, Pseudo-SQL query from an original, uncompacted SQL query string. The uncompacted string reads as follows:

As one of ordinary skill in the art will appreciate, the above string contains a SELECT command making references to a Table 1, having a listing of employees, and a Table 2, having a listing of responsible managers. From the employees Table 1, the managers Table 2 is instructed to be collectively searched, with the nested precondition that the salary field in Table 1 be selected where the name field is equal to Richard, minus an offset value of 159.

As the complexity and thereby inference and processing time of an SQL query is directly related to the number of string characters presented in the query, the inference and processing time of the above example can consume a considerably larger amount of CPU time, versus a query with far fewer characters, again as one of ordinary skill in the art would appreciate.

Taking the example Pseudo-SQL transform process further, consider the following transformation step of the original SQL query:

Here, the fields T1.name, andT2.mgr_name are shown in a compacted form. T1.name becomes T1.C1 (ostensibly relating to column1) and T2.mgr_name becomes T1.C2 (ostensibly relating to column2). The “FROM” clause is removed as shown, and T1.salary becomes T1.C4. The parenthesis marking the beginning of a nested query is changed to a curly bracket, and the compacted schema continues with T1.C5 representative of T1.salary from employee T1, and continues further with where T1.C1=″RD″ (“Richard”). The curly bracket replacing the closed parenthesis is then shown.

Taking the example Pseudo-SQL transform process further, consider the following final transformation step of the original SQL query:

Here, the Select commands are replaced with the shortened, compacted S #, and the Where commands are replaced with the shortened, compacted W # as shown. To summarize, then, the SQL queries are parsed, the SQL is decomposed in structural parts (e.g., SELECT items), the SQL is compacted by an algorithm that selects what parts are excluded (e.g., FROM clause) or modified (e.g., table names), and finally, the compacting algorithm makes sure that no ambiguities or inconsistencies are introduced.

As indicated, the final, Pseudo-SQL string contains a considerably less number of string characters, leading to the shortened inference and time for processing the compacted, Pseudo-SQL through a CPU as previously described, and again as one of ordinary skill in the art will appreciate.

Turning now toFIG.2, consider the following block/flow diagram of an exemplary Full-SQL-Transform process200, according to one embodiment of the present invention. A Pseudo-SQL-DL output202is provided as exemplary string204as shown in compacted form. The output202is then provided to the Full-SQL-Transform206as indicated. As a first step208, the transform06regenerates the full table names208from the predicted output202. Here, T1.A regenerates manufacturer.name, and T2.B regenerates product.price.

As a following step210, the transform06regenerates the “FROM” clause as shown, which is then provided to post-processing step212, which reconstructs the entire, uncompacted SQL string as indicated. The uncompacted SQL string is then provided as the full SQL output214, which is represented as string216as shown. In the depicted embodiment, then, after obtaining the Pseudo-SQL or compacted SQL query from the Deep Learning model, the Full-SQL-Transform process takes into consideration the correct nesting structure (e.g., based on { . . . { . . . } . . . }). Each query then requires an understanding of T1 . . . . TN tables and columns. Finally, the “FROM” clause is recreated, with the appropriate aliases and correct join path based on the process' knowledge of the schema.

Turning now toFIG.3, an exemplary spreadsheet diagram300of a conventional processing methodology of a Text-2-SQL incorporating a Deep Learning model is shown. Column302references the particular database to be searched, column304is representative of the natural language presented (e.g., “who manufactured Phone1”) as shown, column306represents the corresponding, predicted SQL query, column308shows corresponding inference time(s) for generating the predicted SQL query in column306, the length of the corresponding string of the predicted SQL (in this case generally exceeding 100 characters in length) in column308, and the corresponding inference time per character in column312. Here, as indicated, each of the natural language questions results in a significantly long string of predicted SQL, taking a total average inference time314of 4.279 seconds to complete.

FIG.4, in comparison, following, depicts spreadsheet400showing the shortened, compacted, Pseudo-SQL schema according to one embodiment of the present invention, and the resulting inference times. Column402again refers to the database “WH”, and column404again refers to the natural language query presented. In the depicted example of spreadsheet400, the predicted SQL using the Pseudo SQL format as previously described results in column406of string lengths (represented by column410) of significantly less characters, with corresponding inference times (as shown in column408) of considerably less time, and inference time per character (as shown in column412) of considerably shorter duration as well. Accordingly, the average inference time for each of the depicted SQL queries results in 1.613 seconds to generate the Pseudo SQL, a more than two times reduction in processing time (and corresponding increase in processing speed).

Turning now toFIG.5, method500shows an exemplary training phase for a transformed-based Text-2-SQL Deep Learning model, according to the mechanisms of the present invention. A Text-2-SQL dataset502with example string510as shown (having 78 characters) is provided to a Short-SQL-Transform process504, which results in the depicted compacted string512(having 33 characters) of the Text-2-Pseudo-SQL-dataset506. The compacted, Pseudo-SQL dataset is then used to train the transformer-based Text-2-SQL Deep Learning model508as shown.

Here again, the Short-SQL-Transform results in a one-half reduction in input size (e.g., 78 characters to 33 characters) for queries with joins. References to T1, T2, etc., referring to names of tables in the schema in the order they are listed, may be retained as shown.

Turning now toFIG.6, following, an exemplary method600for performing an inference phase, is depicted, in accordance with the mechanisms of the present invention. A natural language input question602is shown as604as, “Who buys product A with a price higher than 10?” The natural language input is used to train a transformer-based Text-2-SQL Deep Learning model606, to provide Pseudo-SQL output608, shown as compacted string610(having 33 characters). The Pseudo-SQL output608is provided to Full-SQL-Transform612to regenerate full SQL output614, depicted as string616. Here again, the time to infer full SQL is greatly reduced, since the applicable decoder only needs to predict the indicated Pseudo-SQL instead of full SQL.

Turning now toFIG.7, a flow chart diagram of an exemplary compaction methodology700as part of a Short-SQL-Transform process, according to various aspects of the present invention, is depicted. Each SQL query702is then parsed704into subparts706. Each subpart706is provided as input (as processing step708) to a compacting algorithm that, in light of various conditions, key words, and other considerations (step710) returns a compacted form of that SQL in a compacting step712. In compacting step712, among other aspects, functionality such as transforming table names, column names, values to shorter values, and consistency checks may occur. Once the compacted subparts714are obtained, they are joined together (step716), provided to a rewriting operation that constructs the final string (step718), rejoined (if necessary) (step720), and provided as the completed, compacted SQL 722.

FIG.8, following, depicts an exemplary schema diagram800, in which various attributes are leveraged in accordance with aspects of the present invention, for example, in regenerating a “FROM” clause in the reconstruction of an uncompacted SQL string from a given Pseudo-SQL input. Schema800depicts customers802, manufacturers808which are connected to products804, vendors816which are connected with sales814and in turn, sales_details810, and stock806shown connected with shops812and then sales814. Each of the various relationships and aspects of the schema800may be leveraged in a future step as will be described in generating a particular “FROM” clause.

In conjunction withFIG.8,FIG.9, following, depicts an exemplary data model900, again in which aspects thereof are leveraged in accordance with various aspects of the present invention. In one embodiment, aspects of the schema diagram800, and the data model900, are utilized to generate and/or re-generate a “FROM” clause in a particular SQL query.

InFIG.9, data model900shows as a connected graph that has one (or more than one) undirected and unweighted links between any two nodes. In the depicted example, data model900contains customers node902, sales node904, vendors node906, shops node908that are connected via the depicted links (e.g., by SHOP_ID link connecting sales904to shops908. In addition, data model900shows sales details node918, refresh functionality910, stock node912, product node914, and manufacturers node916.

In one embodiment, data model900is constructed using schema metadata from schema diagram800as depicted, hence data model900becomes schema specific, and is built with schema metadata offline. In constructing data model900, each of the node attributes are constructed using the schema metadata, such as node name as previously described, list(s) of neighboring node name(s), and list(s) of link(s) therebetween.

An algorithm for a resulting join path may then be developed and implemented to find the shortest join path that covers a set of nodes (tables) shown in the Pseudo-SQL. Turning now toFIG.10, following, join path1000is shown following a determination of the shortest join path covering a particular set of nodes described in the Pseudo-SQL. Consider the following example input Pseudo-SQL string:

select CUSTOMERS.CUSTOMER_ID whereMANUFACTURERS.NAME = 'ABC'
As a result of the foregoing, the following nodes are input to a Full-SQL-Transform operation for generation of the “FROM” clause: [CUSTOMERS, MANUFACTURERS]

Turning again toFIG.10, it may be determined that the shortest join path1000lies between customers node1002, sales node1006, sales_details node1018, products node1014, and manufacturers node1016as shading is indicated. Once the shortest join path1000is determined, the output, incorporating the “FROM” clause may be obtained as follows:

from MANUFACTURERSinner join PRODUCTS onMANUFACTURERS. MANUFACTURER_ID=PRODUCTS.MANUFACTURER_IDinner join SALES_DETAILS onPRODUCTS.PRODUCT_ID=SALES_DETAILS.PRODUCT_IDinner join SALES SALES onSALES_DETAILS.SALES_ID=SALES.SALES_IDinner join CUSTOMERS onSALES.CUSTOMER_ID=CUSTOMERS.CUSTOMER_ID

Taking the example Full-SQL-transform process thus described further, in one embodiment, column names may be sorted for each table in the schema description. Short names to use during training (e.g., name < >C1). Later, during inference, the shortened names in the Pseudo-SQL string may then be replaced back to the original names in the scheme (e.g., C1< >name). In a following step, values in question may be detected and replaced by an abbreviation (e.g., “TYPE_I_CASE_B” by “TICB”). During training, these values are then replaced in both the natural language question input and the resulting SQL queries. Turing a later inference phase, the shortened value is provided to the scheme description. And finally, during post-processing, the replace shortened values are replaced to the original values in the database.

Turning now toFIG.11, following, an exemplary method1100for decreasing a central processing unit (CPU) inference time, thereby shortening processing time of a deep learning model for which run-time complexity is proportional to an output sequence length, is depicted in accordance with one aspect of the mechanisms of the present invention. Method1100begins (step1110) with the development of a compacted sequence used to train a deep learning model. The output thereof of a trained model thereby becomes the compacted sequence on which a post-processing method operates to create an uncompacted version of a predicted sequence (step1120). Method110then ends (step1130)