HYBRID NATURAL LANGUAGE GENERATION (NLG) TECHNIQUES USING A SYMBOLIC NLG ENGINE AND A LARGE LANGUAGE MODEL (LLM)

Techniques for hybrid natural language generation (NLG) using a symbolic NLG engine and a large language model (LLM) are disclosed. The techniques include: obtaining a first text segment configuration; obtaining values of at least some of a first set of data variables; generating a first natural language text segment for an electronic document using a first text segment configuration, the values, and the symbolic NLG engine; obtaining an initial text segment; obtaining a second text segment configuration specifying information to use for generating a second natural language text segment using the ML NLG engine from the initial text segment; generating a second natural language text segment for the electronic document by using the initial text segment, the second text segment configuration, and the ML NLG engine; and outputting the generated electronic document.

RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 to European Patent Application No. 24315102.4, entitled “HYBRID NATURAL LANGUAGE GENERATION (NLG) TECHNIQUES USING A SYMBOLIC NLG ENGINE AND A LARGE LANGUAGE MODEL” filed on Mar. 21, 2024, which is herein incorporated by reference in its entirety.

FIELD

The techniques described herein relate to the field of automatic generation of natural language text, and more particularly to hybrid natural language generation (NLG) techniques using a symbolic NLG engine and a large language model (LLM).

BACKGROUND

Natural language generation (NLG) is the generation of human-language text (i.e., text in a human language) based on information in non-linguistic form. Natural language generation techniques may be used to generate natural language text for different NLG tasks, for example, to generate a report for a business based on financial data about the business, to generate a textual description of a day of trading of a particular stock based on data indicating the price of the stock throughout the day, to generate a confirmation e-mail for an online purchase made via the Internet based on data describing the purchase, to generate real-time comments about a sporting event using data about the game, or to generate text for a chatbot interacting with a customer based on data about the customer.

SUMMARY

Some embodiments provide for a method for generating natural language text with a natural language generation (NLG) system using a plurality of semantic objects including a first semantic object, the NLG system communicatively coupled to at least one data store. The method comprises: using at least one computer hardware processor to perform: obtaining a first specification of the first semantic object, the first specification specifying a first set of one or more data variables of the first semantic object, first attributes of the first semantic object, a first vocabulary of the first semantic object, and a first document structure configuration of the first semantic object; obtaining, from the at least one data store, first data related to the first set of data variables of the first semantic object; determining values of at least some of the first set of data variables using the first data obtained from the at least one data store; generating the natural language text including a first natural language text segment, using the first specification of the first semantic object, the values of at least some of the first set of data variables, and the NLG system, at least in part by: generating a first intermediate representation of the first semantic object using the first document structure configuration, the values of the first set of data variables, and the first attributes of the first semantic object; generating a second intermediate representation of the first semantic object from the first intermediate representation using the first vocabulary of the first semantic object; and generating the first natural language text segment from the second intermediate representation of the first semantic object; and outputting the generated natural language text.

Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for generating natural language text with a natural language generation (NLG) system using a plurality of semantic objects including a first semantic object, the NLG system communicatively coupled to at least one data store, the method comprising: obtaining a first specification of the first semantic object, the first specification specifying a first set of one or more data variables of the first semantic object, first attributes of the first semantic object, a first vocabulary of the first semantic object, and a first document structure configuration of the first semantic object; obtaining, from the at least one data store, first data related to the first set of data variables of the first semantic object; determining values of at least some of the first set of data variables using the first data obtained from the at least one data store; generating natural language text including a first natural language text segment, using the first specification of the first semantic object, the values of at least some of the first set of data variables, and the NLG system, at least in part by: generating a first intermediate representation of the first semantic object using the first document structure configuration, the values of the first set of data variables, and the first attributes of the first semantic object; generating a second intermediate representation of the first semantic object from the first intermediate representation using the first vocabulary of the first semantic object; and generating the first natural language text segment from the second intermediate representation of the first semantic object; and outputting the generated natural language text.

Some embodiments provide for at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for generating natural language text with a natural language generation (NLG) system using a plurality of semantic objects including a first semantic object, the NLG system communicatively coupled to at least one data store, the method comprising: obtaining a first specification of the first semantic object, the first specification specifying a first set of one or more data variables of the first semantic object, first attributes of the first semantic object, a first vocabulary of the first semantic object, and a first document structure configuration of the first semantic object; obtaining, from the at least one data store, first data related to the first set of data variables of the first semantic object; determining values of at least some of the first set of data variables using the first data obtained from the at least one data store; generating natural language text including a first natural language text segment, using the first specification of the first semantic object, the values of at least some of the first set of data variables, and the NLG system, at least in part by: generating a first intermediate representation of the first semantic object using the first document structure configuration, the values of the first set of data variables, and the first attributes of the first semantic object; generating a second intermediate representation of the first semantic object from the first intermediate representation using the first vocabulary of the first semantic object; and generating the first natural language text segment from the second intermediate representation of the first semantic object; and outputting the generated natural language text.

In some embodiments, the plurality of semantic objects includes a second semantic object, and the method further comprises: obtaining a second specification of the second semantic object, the second specification specifying a second set of one or more data variables of the second semantic object, second attributes of the second semantic object, a second vocabulary of the second semantic object, and a second document structure configuration of the second semantic object; obtaining, from the at least one data store, second data related to the second set of data variables of the second semantic object; and determining values of at least some of the second set of data variables using the second data obtained from the at least one data store.

In some embodiments, generating the natural language text comprises generating a second natural language text segment using the second specification of the second semantic object, the values of at least some of the second set of data variables.

In some embodiments, generating the second natural language text segment comprises: generating a first intermediate representation of the second semantic object using the second document structure configuration, the values of the second set of data variables, and the second attributes of the second semantic object; generating a second intermediate representation of the second semantic object from the first intermediate representation of the second semantic object using the second vocabulary of the second semantic object; and generating the second natural language text segment from the second intermediate representation of the second semantic object.

In some embodiments, generating the natural language text comprises generating a single sentence using the first specification of the first semantic object and the second specification of the second semantic object.

In some embodiments, generating the natural language text comprises: generating a first intermediate representation of the second semantic object using the second document structure configuration, the values of the second set of data variables, and the second attributes of the second semantic object; composing the first intermediate representation of the first semantic object and the first intermediate representation of the second semantic object to obtain a composed intermediate representation; and using the composed intermediate representation to generate the single sentence.

In some embodiments, the first document structure configuration for the first semantic objects specifies multiple document structures, wherein the method further comprises selecting from among the multiple document structures to obtain a selected document structure, and wherein generating the first intermediate representation is performed using the selected document structure.

In some embodiments, the first specification of the first semantic object further specifies a first analysis configuration, and wherein determining the values of at least one of the first set of data variables comprises processing the data obtained from the at least one data store using the first analysis configuration.

In some embodiments, the first specification of the first semantic object further comprises a content selection configuration indicating a subset of the first attributes to use for generating the natural language text, and wherein generating the first intermediate representation of the first semantic object is performed using the content selection configuration.

In some embodiments, the first specification of the first semantic object further comprises a micro-planning configuration, and wherein the method further comprises: applying automatic aggregation to the first intermediate representation of the first semantic object using the micro-planning configuration.

In some embodiments, the first specification of the first semantic object further comprises a micro-planning configuration, and wherein the method further comprises: applying referent generation to the second intermediate representation of the first semantic object using the micro-planning configuration.

In some embodiments, the first specification of the first semantic object further comprises a surface transformation configuration, and wherein the method further comprises: applying one or more surface transformations to the second intermediate representation of the first semantic object using the surface transformation configuration.

In some embodiments, the at least one data store is external to the NLG system and the NLG system is communicatively coupled to the at least one data store using a communication network.

In some embodiments, outputting the natural language text comprises providing the natural language text to a publishing system external to the NLG system.

In some embodiments, outputting the natural language text comprises: generating an electronic document including the natural language text; and transmitting the electronic document over at least one communication network to a user. In some embodiments, the electronic document comprises a webpage.

Some embodiments provide for a method for generating natural language text with a natural language generation (NLG) system using a plurality of semantic objects including a first semantic object, the NLG system communicatively coupled to at least one data store. The method comprises: using at least one computer hardware processor to perform: obtaining a first specification of the first semantic object, the first specification specifying a first set of one or more data variables of the first semantic object, first attributes of the first semantic object, and a first vocabulary of the first semantic object; obtaining, from the at least one data store, first data related to the first set of data variables of the first semantic object; determining values of at least some of the first set of data variables using the first data obtained from the at least one data store; generating the natural language text including a first natural language text segment, using the first specification of the first semantic object, the values of at least some of the first set of data variables, and the NLG system, at least in part by: generating a first intermediate representation of the first semantic object using, the values of the first set of data variables, and the first attributes of the first semantic object; generating a second intermediate representation of the first semantic object from the first intermediate representation using the first vocabulary of the first semantic object; and generating the first natural language text segment from the second intermediate representation of the first semantic object; and outputting the generated natural language text.

Some embodiments provide for a method for generating an electronic document containing natural language text with a natural language generation (NLG) system using a plurality of text segment configurations including a first text segment configuration and a second text segment configuration, the NLG system communicatively coupled to at least one data store, the NLG system including a symbolic NLG engine and a machine learning (ML) NLG engine, the method comprising: using at least one computer hardware processor to perform: generating a first portion of the electronic document by using the first text segment configuration and the symbolic NLG engine to generate a first natural language text segment to include in the first portion of the document, wherein generating the first portion comprises: obtaining a first text segment configuration, the first text segment configuration specifying a first set of one or more data variables; obtaining values of at least some of the first set of data variables using first data obtained from the at least one data store; and generating the first natural language text segment using the first text segment configuration, the values of the at least some of the first set of one or more data variables, and the symbolic NLG engine; and generating a second portion of the electronic document by using the second text segment configuration and the ML NLG engine to generate a second natural language text segment to include in the second portion of the document, wherein generating the second portion comprises: obtaining an initial text segment; obtaining a second text segment configuration, the second text segment configuration specifying information to use for generating the second natural language text segment using the ML NLG engine from the initial text segment; and generating the second natural language text segment by using the initial text segment, the second text segment configuration, and the ML NLG engine; and outputting the generated electronic document.

In some embodiments, the ML NLG engine is configured to generate natural language text using a large language model (LLM).

In some embodiments, generating the second natural language text segment using the ML NLG engine further comprises: generating, using the second text segment configuration, a prompt from the initial text segment; providing the prompt as input to the LLM; and processing the prompt with the LLM to obtain the second natural language text segment.

In some embodiments, the information to use for generating the second natural language text segment specifies at least one transformation to be made to a grammatical aspect, content, tone, and/or style of the initial text segment.

In some embodiments, the initial text segment is the first natural language text segment output from the symbolic NLG engine.

In some embodiments, the first text segment configuration further specifies first attributes, a first vocabulary, and a first document structure configuration, and generating the first natural language text segment comprises: generating a first intermediate representation of the first text segment configuration using the first document structure configuration, the values of the first set of data variables, and the first attributes; generating a second intermediate representation of the first text segment configuration from the first intermediate representation of the first text segment configuration using the first vocabulary; and generating the first natural language text segment from the second intermediate representation of the first text segment configuration.

In some embodiments, the first document structure configuration specifies multiple document structures, wherein the method further comprises selecting from among the multiple document structures to obtain a selected document structure, and wherein generating the first intermediate representation is performed using the selected document structure.

In some embodiments, the first text segment configuration further specifies a first analysis configuration, and wherein determining the values of at least one of the first set of data variables comprises processing the data obtained from the at least one data store using the first analysis configuration.

In some embodiments, the first text segment configuration further comprises a content selection configuration indicating a subset of the first attributes to use for generating the natural language text, and wherein generating the first intermediate representation of the first text segment configuration is performed using the content selection configuration.

In some embodiments, the first text segment configuration further comprises a micro-planning configuration, and wherein the method further comprises: applying automatic aggregation to the first intermediate representation of the first text segment configuration using the micro-planning configuration.

In some embodiments, the first text segment configuration further comprises a micro-planning configuration, and wherein the method further comprises: applying referent generation to the second intermediate representation of the first text segment configuration using the micro-planning configuration.

In some embodiments, the at least one data store is external to the NLG system and the NLG system is communicatively coupled to the at least one data store using a communication network.

In some embodiments, outputting the electronic document comprises providing the natural language text to a publishing system external to the NLG system.

In some embodiments, the method further comprises transmitting the electronic document over at least one communication network to a user.

In some embodiments, wherein the plurality of text segment configurations includes a third text segment configuration, the method comprising: determining, using data in the third text segment configuration, whether to generate a third portion of the electronic document by using the ML NLG engine or the symbolic NLG engine; when it is determined to use the ML NLG engine to generate the third portion of electronic document, generating the third portion of the electronic document using the ML NLG engine; and when it is determined to use the symbolic NLG engine to generate the third portion of electronic document, generating the third portion of the electronic document using the symbolic NLG engine.

Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for generating an electronic document containing natural language text with a natural language generation (NLG) system using a plurality of text segment configurations including a first text segment configuration and a second text segment configuration, the NLG system communicatively coupled to at least one data store, the NLG system including a symbolic NLG engine and a machine learning (ML) NLG engine, the method comprising: using at least one computer hardware processor to perform: generating a first portion of the electronic document by using the first text segment configuration and the symbolic NLG engine to generate a first natural language text segment to include in the first portion of the document, wherein generating the first portion comprises: obtaining a first text segment configuration, the first text segment configuration specifying a first set of one or more data variables; obtaining values of at least some of the first set of data variables using first data obtained from the at least one data store; and generating the first natural language text segment using the first text segment configuration, the values of the at least some of the first set of one or more data variables, and the symbolic NLG engine; and generating a second portion of the electronic document by using the second text segment configuration and the ML NLG engine to generate a second natural language text segment to include in the second portion of the document, wherein generating the second portion comprises: obtaining an initial text segment; obtaining a second text segment configuration, the second text segment configuration specifying information to use for generating the second natural language text segment using the ML NLG engine from the initial text segment; and generating the second natural language text segment by using the initial text segment, the second text segment configuration, and the ML NLG engine; and outputting the generated electronic document.

Some embodiments provide for at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for generating an electronic document containing natural language text with a natural language generation (NLG) system using a plurality of text segment configurations including a first text segment configuration and a second text segment configuration, the NLG system communicatively coupled to at least one data store, the NLG system including a symbolic NLG engine and a machine learning (ML) NLG engine, the method comprising: using at least one computer hardware processor to perform: generating a first portion of the electronic document by using the first text segment configuration and the symbolic NLG engine to generate a first natural language text segment to include in the first portion of the document, wherein generating the first portion comprises: obtaining a first text segment configuration, the first text segment configuration specifying a first set of one or more data variables; obtaining values of at least some of the first set of data variables using first data obtained from the at least one data store; and generating the first natural language text segment using the first text segment configuration, the values of the at least some of the first set of one or more data variables, and the symbolic NLG engine; and generating a second portion of the electronic document by using the second text segment configuration and the ML NLG engine to generate a second natural language text segment to include in the second portion of the document, wherein generating the second portion comprises: obtaining an initial text segment; obtaining a second text segment configuration, the second text segment configuration specifying information to use for generating the second natural language text segment using the ML NLG engine from the initial text segment; and generating the second natural language text segment by using the initial text segment, the second text segment configuration, and the ML NLG engine; and outputting the generated electronic document.

Some embodiments provide for a method for generating an electronic document containing natural language text with a natural language generation (NLG) system using a plurality of text segment configurations including a first text segment configuration, the NLG system communicatively coupled to at least one data store, the NLG system including a machine learning (ML) NLG engine, and a symbolic NLG engine, the method comprising: using at least one computer hardware processor to perform: determining, using data in the first text segment configuration, whether to generate a first portion of the electronic document by using the ML NLG engine or the symbolic NLG engine; when it is determined to use the ML NLG engine to generate the first portion of electronic document, generating the first portion of the electronic document using the ML NLG engine; when it is determined to use the symbolic NLG engine to generate the first portion of electronic document, generating the first portion of the electronic document using the symbolic NLG engine; and outputting the electronic document.

In some embodiments, the data in the first text segment configuration comprises a parameter indicating that either the ML NLG engine or the symbolic NLG engine be used in the generation of the electronic document.

In some embodiments, the data in the first text segment configuration comprises information specifying a textual transformation, and determining whether to generate a first portion of the electronic document by using the ML NLG engine or the symbolic NLG engine further comprises: determining to use the ML NLG engine when the data in the first text segment includes information specifying a textual transformation.

In some embodiments, the data in the first text segment configuration comprises information specifying one or more values to be accessed from the at least one data store, and determining whether to generate a first portion of the electronic document by using the ML NLG engine or the symbolic NLG engine further comprises: determining to use the symbolic NLG engine when the data in the first text segment includes information specifying one or more values to be accessed from the at least one data store.

Some embodiments provide for a method for generating an electronic document containing natural language text with a natural language generation (NLG) system using a plurality of text segment configurations including a first text segment configuration and a second text segment configuration, the NLG system communicatively coupled to at least one data store, the NLG system including a symbolic NLG engine and a machine learning (ML) NLG engine, the method comprising: using at least one computer hardware processor to perform: displaying a graphical user interface (GUI) having: a text display portion configured to display textual content; and a natural language generation (NLG) portion, the NLG portion comprising a plurality of GUI elements that enable a user to specify one or more parameters for generating natural language text, the one or more parameters including an NLG engine parameter specifying whether to use the symbolic NLG engine or the ML NLG engine to generate text; receiving, via the NLG portion, values for the one or more parameters for generating natural language text; generating the natural language text using, the NLG system and the NLG engine parameter; and inserting the generated natural language text in the text display portion.

DETAILED DESCRIPTION

The inventors have recognized and appreciated that conventional NLG technology may be improved upon by improving techniques for using natural language generation (NLG) systems to perform different types of NLG tasks. Conventional techniques for applying an NLG system to perform a new NLG task involve using either: (1) very complex NLG systems requiring an impractical amount of time and effort to customize to a specific NLG task, or (2) highly customized NLG systems that are restricted to performing a specific NLG task (e.g., generating profit and loss reports for a business) and cannot be adapted to perform another NLG task (e.g., generating automated e-mails to customers of the same business).

To address these technical shortcomings of conventional NLG technology, the inventors have developed an NLG system that facilitates rapid development of customized and high-quality NLG software applications to perform different NLG tasks by generating at least some of the text using novel constructs, developed by the inventors, which are termed “semantic objects” herein. Sometimes, a “semantic object” may be referred to as an “intention”. The NLG system may be rapidly and inexpensively configured to perform thousands of different NLG tasks, something that was not previously possible.

Conventional techniques for applying NLG technology to generate natural language text for a specific task involve a series of operations including: (1) identifying a task to be performed by identifying text whose generation is to be automated (e.g., identifying that a particular type of report, e-mail, or other document is to be generated automatically); (2) identifying the source (e.g., a database) of non-linguistic data (e.g., numbers, names, customer data, business data, etc.) to be inserted into the natural language text as it is being generated; (3) creating a new NLG system, which involves extensive programming using an existing NLG framework (e.g., one or more NLG application programming interfaces (API), software packages, etc.); (4) coupling the NLG system to the source of non-linguistic data; and (5) coupling the NLG system to one or more systems to which the NLG system is to provide generated text. These operations are time consuming, require low-level programming, and are expensive.

Although available NLG software and APIs may help to reduce the time spent on some of the above individual operations, the overall process needs to be repeated for every new NLG task, making such conventional approaches impractical, and severely limiting applicability of NLG systems to real-world industrial tasks. Moreover, since available NLG software and APIs were designed for general applicability, they are complex and require extensive configuration (e.g., configuring document structure, data selection, vocabulary, sentence boundaries, formatting, referent selection, etc.), which has to be done manually by users. As a result, creating and/or configuring an NLG system to perform an NLG task is rendered more complex and increases development times.

Another conventional approach is to use NLG systems highly tailored to specific tasks (e.g., an NLG system customized for developing a particular type of report, an NLG system customized for generating a summary of a sporting event, etc.). This improves development times because it reduces the possible inputs to some standard format and allows for the automation of many decisions that would otherwise have to be configured manually by the user. The automation of the configuration may for example take the form of domain specific vocabulary, fixed document structures, standardized data connectors, predefined lexicographic rules, and the like. The inventors have appreciated, however, that restricting an NLG system to a specific domain of application is excessively limiting, given that new NLG frameworks would be required for each new domain. For example, an NLG system highly tailored to the task of generating e-mails for businesses in one industry (e.g., airline industry) would not be applicable to the task of generating e-mails for businesses in another industry (e.g., banking industry). Such NLG systems are simply not adaptable to other tasks.

Accordingly, the inventors have developed an NLG system that uses semantic objects to facilitate rapid development of customized and high-quality NLG software applications to perform different NLG tasks. The NLG system may be configured with one set of one or more semantic objects to perform one NLG task and, at another time, may be configured with another set of one or more semantic objects to perform a different NLG task. In this way, the previously time-consuming, programming-heavy, and expensive activity (e.g., computationally expensive activity) of developing an NLG software application for an NLG task may be reduced to specifying a set of semantic objects for the task, which is substantially simpler than configuring conventional complex NLG systems. The same underlying NLG system (e.g., the illustrative NLG system 101 of FIG. 1A) may be configured for different NLG tasks using different sets of semantic objects. Among various benefits, using semantic objects automates the configuration of various stages of the NLG process including lexicalization, aggregation, referential expression generation, and surface realization, whereas in conventional systems such stages have to be configured manually.

Advantageously, by specifying a set of semantic objects for a specific NLG task, an NLG system as described herein can execute with improved computational efficiency. For example, deployment of a conventional NLG system may require extensive manual configuration for the specific NLG task. Such manual configuration, although needed to be compatible with the specific NLG task, may be an undesirable configuration with respect to computational efficiency and computational cost expenditure. For example, extensive manual configuration can lead to a bloated, complex NLG system with suboptimal memory accesses and/or redundant processing operations that incur increased computational costs (e.g., processing power, memory, storage, network bandwidth) with respect to a non-configured instance of the same NLG system. By contrast, the same underlying NLG system as described herein may be used for different NLG tasks through the use of different sets of semantic objects. Accordingly, physical hardware resources (e.g., processing power, memory, storage) can execute an NLG system configured with a set of semantic objects as described herein with increased efficiency and/or throughput with respect to a bloated, complex, and heavily manually configured NLG system for the same NLG task.

Advantageously, using a set of semantic objects can improve how physical hardware resources execute. For example, an NLG system as described herein can access smaller data objects (e.g., semantic objects) to generate NLG outputs in a streamlined manner, which contrasts executing a bloated and/or complex NLG system that is manually configured for a specific NLG task. In such an example, users can focus on developing semantic objects instead of manually configuring the NLG system, which can lead to improved efficiency with respect to deploying NLG systems for a plurality of NLG tasks (e.g., 10 tasks, 100 tasks, 1000 tasks, etc.).

In some embodiments, a semantic object specifies configuration data used by an NLG system (e.g., NLG system 101) to generate a respective natural language segment (e.g., a phrase, a sentence, a set of multiple sentences). In some embodiments, a semantic object contains multiple types of configuration data used to configure different parts of the NLG system to generate the respective natural language segment. For example, an NLG system may include components to perform functions including, but not limited to, data analysis, document structuring, lexicalization, micro-planning, and surface realization. In turn, the semantic object may include configuration data for configuring the NLG system to perform such functions. In this example, the semantic object may include an analysis configuration having data for configuring the NLG system to perform data analysis, a document structure configuration and attributes (e.g., data attributes and lexical attributes) having data for configuring the NLG system to perform document structuring, a vocabulary having data for configuring the NLG system to perform lexicalization, a micro-planning configuration having data for configuring the NLG system perform micro-planning, and a surface realization configuration having data for configuring the NLG system to perform surface realization.

In some embodiments, a semantic object includes a plurality of attributes, each of which may be substituted by one or more words from an associated vocabulary and/or data from another system when the semantic object is rendered into natural language text by the NLG system. In this way, a semantic object may specify the manner in which linguistic data (e.g., words and phrases to be used for generating text) and non-linguistic data (e.g., business data having numeric values) may be combined in generated natural language text.

In some embodiments, a specification of a semantic object, containing data associated with (e.g., defining) the semantic object, may be stored in any suitable non-transitory computer readable storage medium including volatile and non-volatile memory. In some embodiments, the specification of the semantic object may be stored using one or more files. In some embodiments, the specification may be defined using any suitable mark-up language (e.g., XML, SGML, YAML, JSON, etc.), as aspects of the technology described herein are not limited in this respect. In some embodiments, the specification of a semantic object may be stored in volatile memory using on or multiple data structure(s). Examples of semantic objects are provided herein including with reference to the example semantic objects 150 of FIGS. 1C and 1D, the semantic object of FIGS. 3A-3F, and the semantic object 200 of FIGS. 2A-2C.

Some embodiments are directed to techniques for generating one or more documents(s) (e.g., electronic document(s)) with a natural language system (NLG) system configured with a plurality of semantic objects. The NLG system may be configured to use individual semantic objects, from among the plurality of semantic objects, to generate respective natural language text segments (e.g., a phrase, a sentence, multiple sentences). For example, an NLG system may be configured to generate a document having multiple natural text segments by generating the multiple text segments using respective semantic objects.

Another conventional approach to generating natural language text involves execution of a machine learning (ML) model, such as a large language model (LLM). LLMs are a type of machine learning model trained to conduct a probability distribution over words. Some LLMs are trained to generate an output including natural language text by predicting the next most appropriate word to fill in a blank space in a sentence, phrase, paragraph, etc. LLMs are used in a variety of NLG tasks, such as content generation, language transformation (e.g., changing a tense of natural language text in a sentence, a paragraph, etc.), question answering, and text summarization.

The inventors have recognized that LLMs are unfit for some NLG tasks, such as content generation for regulatory submissions (e.g., government regulatory submissions). An example of a regulatory submission is clinical documentation (e.g., a patient narrative corresponding to how a clinical participant tolerated a new pharmaceutical drug) for a pharmaceutical drug-approval process. In such an example, an extensive approval process must be completed to obtain government approval to sell a new prescription drug. The approval process typically includes the submission of discovery and/or concept submissions and research submissions (e.g., preclinical research, clinical research). These submissions are typically prepared by writers (e.g., medical writers) that are highly specialized such that they have subject matter expertise, statistical acumen, and elegant writing skills. The inventors have recognized that LLMs are unfit to generate such submissions because of the probabilistic nature of LLMs that can lead to unreliable and/or inaccurate submissions. Indeed, LLMs can “hallucinate” facts, generate output based on training data which may not be current and is typically dated, and are not able to access data relevant to the query/prompt from databases in order to incorporate such data in the generated output.

Accordingly, the inventors have developed an NLG system that leverages a hybrid approach that uses a symbolic NLG engine and/or an ML NLG engine to facilitate rapid development of customized and high-quality NLG software applications to perform different NLG tasks. The hybrid NLG system overcomes the technical limitations of symbolic NLG techniques, which are tied to humans creating rules and thereby limits scalability, and the technical limitations of LLMs, which can produce unreliable and/or inaccurate outputs, by combining symbolic NLG with ML-based NLG in a unique way to create a hybrid NLG system, as described herein.

In some embodiments, the hybrid NLG system generates an output, such as an electronic document, by generating one or more first electronic document portions using a symbolic NLG engine and one or more second electronic document portions using an ML NLG engine. In some embodiments, the hybrid NLG system generates an electronic document by using the symbolic NLG engine to generate electronic document portions and using the ML NLG engine to transform one(s) of the electronic document portions. Examples of transformations include changing the tone and/or style of natural language text of the electronic document portion(s), changing a tense or other grammatic aspect of the natural language text, and summarizing the natural language text. Additional examples of transformations are provided herein.

The inventors have recognized that the hybrid NLG system achieves several benefits, relative to conventional symbolic- or ML-only NLG systems, including data privacy, regulatory and/or industry compliance (e.g., GxP compliance), accuracy and reliability, and model output traceability. In some embodiments, the hybrid NLG system achieves data privacy by training the ML NLG engine on synthetic data in a fully secure private environment. Advantageously, access to the fully secure private environment can be limited to authorized users and/or within a computing environment that is not connected to external network(s).

In some embodiments, the hybrid NLG system achieves accuracy and reliability by generating natural language text using the symbolic NLG engine and using the ML NLG engine to perform transformations on the generated natural language text. For example, the technical limitation of LLMs potentially producing unreliable and/or inaccurate outputs is overcome by generating natural language text using the symbolic NLG engine. Furthering the example, the technical limitation of symbolic NLG techniques having limited scalability is overcome by performing transformations on the generated natural language text without the need for humans creating rules to perform such transformations.

In some embodiments, the hybrid NLG system achieves model output traceability by tracing outputs of the symbolic NLG engine to rule(s) implemented by the symbolic NLG engine. For example, the technical limitation of LLMs potentially “hallucinating”, such as producing unreliable and/or inaccurate outputs, is overcome by generating natural language text using the symbolic NLG engine and thereby enabling users to trace output(s) of the symbolic NLG engine to specific rule(s).

Some embodiments are directed to techniques for generating natural language text with a natural language generation (NLG) system using a plurality of semantic objects including a first semantic object, the NLG system communicatively coupled to at least one data store (e.g., a database external to the NLG system). The techniques include a method comprising: (1) obtaining a first specification of the first semantic object, the first specification specifying a first set of one or more data variables of the first semantic object, first attributes of the first semantic object, a first vocabulary of the first semantic object, and a first document structure configuration of the first semantic object; (2) obtaining, from the at least one data store, first data related to the first set of data variables of the first semantic object; (3) determining values of at least some of the first set of data variables using the first data obtained from the at least one data store; (4) generating the natural language text including first natural language text, using the first specification of the first semantic object, the values of at least some of the first set of data variables, and the NLG system, at least in part by: (4a) generating a first intermediate representation of the first semantic object using the first document structure configuration, the values of the first set of data variables, and the first attributes of the first semantic object; (4b) generating a second intermediate representation of the first semantic object from the first intermediate representation using the first vocabulary of the first semantic object; and (4c) generating the first natural language text from the second intermediate representation of the first semantic object; and (5) outputting the generated natural language text.

In some embodiments, the plurality of semantic objects includes a second semantic object, the technique further comprising: obtaining a second specification of the second semantic object, the second specification specifying a second set of one or more data variables of the second semantic object, second attributes of the second semantic object, a second vocabulary of the second semantic object, and a second document structure configuration of the second semantic object; obtaining, from the at least one data store, second data related to the second set of data variables of the second semantic object; and determining values of at least some of the second set of data variables using the second data obtained from the at least one data store, wherein generating the natural language text comprises generating second natural language text using the second specification of the second semantic object, the values of at least some of the second set of data variables.

In some embodiments, the first semantic object and the second semantic object may be used to generate two different text segments (e.g., two different phrases, sentences, etc.). For example, in some embodiments, generating the second natural language text comprises: generating a first intermediate representation of the second semantic object using the second document structure configuration, the values of the second set of data variables, and the second attributes of the second semantic object; generating a second intermediate representation of the second semantic object from the first intermediate representation of the second semantic object using the second vocabulary of the second semantic object; and generating the second natural language text (e.g., a second sentence different from the first sentence) from the second intermediate representation of the second semantic object.

In some embodiments, the first semantic object and the second semantic object may be used to generate a single sentence. To this end, the intermediate representations generated from the first and second semantic object may be composed to form a composed intermediate representation that may be used to generate a single sentence. Accordingly, in some embodiments, generating the natural language text comprises: generating a first intermediate representation of the second semantic object using the second document structure configuration, the values of the second set of data variables, and the second attributes of the second semantic object; composing the first intermediate representation of the first semantic object and the first intermediate representation of the second semantic object to obtain a composed intermediate representation; and using the composed intermediate representation to generate the single sentence.

In some embodiments, the first document structure configuration for the first semantic objects specifies multiple document structures (e.g., as shown in FIG. 3D where two document structures called “first variant” and “second variant” are illustrated), wherein the method further comprises selecting from among the multiple document structures to obtain a selected document structure, and wherein generating the first intermediate representation is performed using the selected document structure.

In some embodiments, the first specification of the first semantic object further specifies a first analysis configuration (e.g., as illustrated in FIG. 3B), and wherein determining the values of at least one of the first set of data variables comprises processing the data obtained from the at least one data store using the first analysis configuration (e.g., as illustrated in FIGS. 4B and 4C).

In some embodiments, the first specification of the first semantic object further comprises a content selection configuration indicating a subset of the first attributes to use for generating the natural language text, and wherein generating the first intermediate representation of the first semantic object is performed using the content selection configuration.

In some embodiments, the first specification of the first semantic object further comprises a micro-planning configuration (e.g., as illustrated in FIG. 3F), and wherein the method further comprises: applying automatic aggregation to the first intermediate representation of the first semantic object using the micro-planning configuration.

In some embodiments, the first specification of the first semantic object further comprises a micro-planning configuration, and the method further comprises: applying referent generation to the second intermediate representation of the first semantic object using the micro-planning configuration.

In some embodiments, the first specification of the first semantic object further comprises a surface transformation configuration, and the method further comprises: applying one or more surface transformations to the second intermediate representation of the first semantic object using the surface transformation configuration.

In some embodiments, outputting the natural language text generated by an NLG system comprises providing the natural language text to a publishing system external to the NLG system. The publishing system may be configured to generate an electronic document (e.g., a webpage, a PDF file, a text document, etc.) including the natural language text; and transmit the electronic document over at least one communication network to a user.

Some embodiments are directed to a method for generating an electronic document (e.g., the electronic document 822 of FIG. 8A) containing natural language text (e.g., the first and second natural language text segments 818 and 820 of FIG. 8A) with a natural language generation (NLG) system (e.g., the NLG system 101a of FIG. 8A) using a plurality of text segment configurations including a first text segment configuration (e.g., the symbolic NLG text segment configuration 804 of FIG. 8A) and a second text segment configuration (e.g., the ML NLG text segment configuration 802 of FIG. 8A), the NLG system communicatively coupled to at least one data store (e.g., the business data store(s) 112 of FIG. 8A), the NLG system including a symbolic NLG engine (e.g., the symbolic NLG engine 814 of FIG. 8A) and a machine learning (ML) NLG engine (e.g., the ML NLG engine 816 of FIG. 8A), the method comprising: using at least one computer hardware processor (e.g., the processor of FIG. 7) to perform: generating a first portion of the electronic document (e.g., the first portion of the electronic document of FIG. 8A) by using the first text segment configuration and the symbolic NLG engine to generate a first natural language text segment (e.g., the first NLT segment 818 of FIG. 8A) to include in the first portion of the document, wherein generating the first portion comprises: obtaining a first text segment configuration (e.g., the symbolic NLG text segment configuration 802 of FIG. 8A), the first text segment configuration specifying a first set of one or more data variables (e.g., one(s) of the data variables 154 of FIG. 8B); obtaining values of at least some of the first set of data variables using first data obtained from the at least one data store; and generating the first natural language text segment using the first text segment configuration, the values of the at least some of the first set of one or more data variables, and the symbolic NLG engine; and generating a second portion of the electronic document (e.g., the second portion of the electronic document of FIG. 8A) by using the second text segment configuration and the ML NLG engine to generate a second natural language text segment (e.g., the second NLT segment 820 of FIG. 8A) to include in the second portion of the document, wherein generating the second portion comprises: obtaining an initial text segment (e.g., the initial text segment of FIG. 8A); obtaining a second text segment configuration (e.g., the ML NLG text segment configuration of FIG. 8A), the second text segment configuration specifying information to use for generating the second natural language text segment using the ML NLG engine from the initial text segment; and generating the second natural language text segment by using the initial text segment, the second text segment configuration, and the ML NLG engine; and outputting the generated electronic document.

Some embodiments relate to a method for generating an electronic document containing natural language text with a natural language generation (NLG) system (e.g., the NLG system of FIGS. 8A and/or 8B) using a plurality of text segment configurations including a first text segment configuration (e.g., the engine selector text segment configuration of FIGS. 8A and/or 8B), the NLG system communicatively coupled to at least one data store (e.g., the business data store(s) 112 of FIGS. 8A and/or 8B), the NLG system including a machine learning (ML) NLG engine (e.g., the ML NLG engine of FIGS. 8A and/or 8B), and a symbolic NLG engine (e.g., the symbolic NLG engine of FIGS. 8A and/or 8B), the method comprising: using at least one computer hardware processor (e.g., the processor of FIG. 7) to perform: determining, using data in the first text segment configuration, whether to generate a first portion of the electronic document (e.g., the first portion of the electronic document of FIG. 8A) by using the ML NLG engine or the symbolic NLG engine; when it is determined to use the ML NLG engine to generate the first portion of electronic document, generating the first portion of the electronic document using the ML NLG engine; when it is determined to use the symbolic NLG engine to generate the first portion of electronic document, generating the first portion of the electronic document using the symbolic NLG engine; and outputting the electronic document.

Some embodiments relate to a method for generating an electronic document containing natural language text with a natural language generation (NLG) system (e.g., the NLG system of FIGS. 8A and/or 8B) using a plurality of text segment configurations including a first text segment configuration (e.g., the symbolic NLG text segment configuration of FIGS. 8A and/or 8B) and a second text segment configuration (e.g., the ML NLG text segment configuration of FIGS. 8A and/or 8B), the NLG system communicatively coupled to at least one data store (e.g., the business data store(s) 112 of FIGS. 8A and/or 8B), the NLG system including a symbolic NLG engine (e.g., the symbolic NLG engine of FIGS. 8A and/or 8B) and a machine learning (ML) NLG engine (e.g., the ML NLG engine of FIGS. 8A and/or 8B), the method comprising: using at least one computer hardware processor (e.g., the processor of FIG. 7) to perform: displaying a graphical user interface (GUI) (e.g., one(s) of the GUIs of FIGS. 9A-11I) having: a text display portion (e.g., the text display portion of FIG. 9H) configured to display textual content (e.g., the natural language text of FIG. 9H); and a natural language generation (NLG) portion (e.g., the NLG portion of FIG. 9H), the NLG portion comprising a plurality of GUI elements (e.g., the plurality of GUI elements of FIG. 9H) that enable a user (e.g., the administrator and/or the user of FIG. 8A) to specify one or more parameters for generating natural language text, the one or more parameters including an NLG engine parameter specifying whether to use the symbolic NLG engine or the ML NLG engine to generate text; receiving, via the NLG portion, values for the one or more parameters for generating natural language text; generating the natural language text (e.g., the natural language text of FIG. 9H) using, the NLG system (e.g., the NLG system of FIGS. 8A and/or 8B) and the NLG engine parameter; and inserting the generated natural language text in the text display portion.

Following below are more detailed descriptions of various concepts related to, and embodiments of techniques for natural language generation using semantic objects. Various aspects described herein may be implemented in any of numerous ways. Examples of specific implementations are provided herein for illustrative purposes only. In addition, the various aspects described in the embodiments below may be used alone or in any combination, and are not limited to the combinations explicitly described herein.

Illustrative NLG Systems for Generating Natural Language Text Using Semantic Objects

FIG. 1A is a diagram of an illustrative environment 100 in which some embodiments of the technology described herein may operate. The environment 100 includes an NLG system 101 that may be configured by administrator 110 to perform a natural language generation task. As part of the configuration, administrator 110 may configure NLG system 101 to communicate with business data store(s) 112 to obtain business data 113, which may include data to be incorporated into natural language text generated by NLG system 101. The administrator 110 may configure NLG system 101 to perform the NLG task by specifying one or more semantic object(s) 122 and/or one or more document macro structures 120 that the NLG system 101 will utilize to generate natural language text.

In some embodiments, the natural language text (NLT) generated by NLG system 101 may be output to one or multiple destinations (e.g., one or more other devices, users, etc.). For example, as shown in FIG. 1A, NLG system 101 may generate NLT 103 using one or more of the semantic objects 122 and at least some of the business data 113, and provide the NLT 103 to publishing system 104. The publishing system 104 may include the NLT 103 into content 105 (e.g., a webpage, an electronic document, a report, a form, and/or any other suitable type of document) and may provide the content 105 to a user 108, for example, by providing content 105 to a software application program 106 (e.g., an Internet browser) executing on the computing device 107 (e.g., smartphone, laptop, desktop, or any other suitable computing device) of user 108.

It should be appreciated that although, in some embodiments, publishing system 104 may be a web server, this is only an example and publishing system 104 may include any other type(s) of computing device(s). For example, in some embodiments, publishing system 104 may include one or more computing devices operated by a business and which use the NLT text to create documents, reports, e-mails, forms, and/or any other suitable types of document(s) used in connection with the business.

As described above, in some embodiments, the NLG system 101 may obtain, from business data store(s) 112, business data 113 at least some of which may be included in natural language text to be generated by the NLG system 101. In some embodiments, this may be done using database queries (e.g., SQL or MDX queries) or in any other suitable way. The business data 113 may include any suitable type(s) of data related to a business and may be numeric data (e.g., the revenues, profits, and/or losses of a business in a given period of time, etc.), alphabetic data (e.g., names of customers, employees, projects, etc.), or alphanumeric data. For example, the business data may include the values 70,000,000 and 50,000,000 and these data may be used by the NLG system to generate the text “In 2018, the revenue decreased from $70M to $50M”. The business data 113 may be in any suitable format, as aspects of the technology described herein are not limited in this respect. The business data store(s) 112 may include one or multiple storage devices storing data in one or more formats of any suitable type. For example, the storage device(s) part of data store(s) 112 may store data using one or more database tables, spreadsheet files, flat text files, and/or files in any other suitable format. In some embodiments, the data store(s) 112 may include one or more databases of any suitable type(s).

In some embodiments, NLG system 101 may be configured to perform one or multiple NLG tasks. In some embodiments, the NLG system 101 may be configured to perform an NLG task by an NLG application 102 configured to perform the NLG task. An NLG application 102 may be configured to perform one or multiple NLG tasks, as aspects of the technology described herein are not limited in this respect.

As shown in FIG. 1A, the NLG application 102 has multiple components including document macro structure(s) 120, semantic object(s) 122, a database interface module 124, NLG software tools 128, and a semantic object-NLG software tool integration layer 126, which is referred to herein as “integration layer 126” for clarity. It should be appreciated that these components are illustrative and that, in some embodiments, an NLG application 102 may include one or more components in addition to or instead of these components.

In some embodiments, NLG application 102 may generate natural language text by: (1) obtaining non-linguistic content from semantic object(s) 122 and business data 113 (obtained via the database interface module 124); (2) determining a macro structure for the natural language text using document macro structure(s) 120; (3) using the integration layer 126 to generate natural language text from the non-linguistic content using the macro structure and the NLG software tools 128.

In some embodiments, document macro structure(s) 120 may include information indicating the overall organization of a document to be generated. As described herein, in some embodiments, each of multiple semantic objects may be used to generate a respective natural language text segment (e.g., a phrase, a single sentence, multiple sentences). The document macro structure(s) 120 may include information indicating the order in which to organize natural language text generated using semantic objects. For example, semantic objects 122 may include ten semantic objects and document macro structure(s) 120 may indicate the order in which the ten natural language text segments (generated using the ten semantic objects) are to be organized within the document to be generated. Additionally, in some embodiments, the document macro structure(s) 120 may include information specifying a template for a report (e.g., sections, section headings, formatting, etc.).

Semantic object(s) 122 may include any suitable number of semantic objects, as aspects of the technology described herein are not limited in this respect. Examples of semantic objects are described herein. In some embodiments, the semantic object(s) 122 may be part of NLG application 102. In other embodiments, the semantic object(s) 122 may be stored in semantic object data store 140, which may be within NLG system 101 or external to NLG system 101. In some such embodiments, NLG application 102 may be configured to access information about semantic objects 122 using semantic object access module 123 that is configured to communicate with semantic object data store 140. Semantic object data store 140 may be any suitable type of data store and may include one or multiple storage devices storing data in one or more formats of any suitable type. For example, the storage device(s) part of data store 140 may store data using one or more database tables, spreadsheet files, flat text files, and/or files in any other suitable format. In some embodiments, the data store 140 may include one or more databases of any suitable type(s).

In some embodiments, database interface module 124 may be configured to access business data 113 from business data store(s) 112. This may be done in any suitable way. In some embodiments, the database interface module 124 may be configured to obtain data from (either pull data from or be provided data by) the business data store(s) 112. The data may be provided via a communication network (not shown), such as the Internet or any other suitable network, as aspects of the technology described herein are not limited in this respect.

In some embodiments, a user (e.g., administrator 110) may configure NLG application 102 to perform a particular NLG task by configuring the components 120, 122, and 124. For example, the user may provide the NLG application 102 with one or more document macro structure(s) 120 and semantic object(s) 122 (e.g., by providing one or more configuration files, for instance specified using a mark-up language, that specify the document macro structure(s) 120 and semantic object(s) 122). Additionally, when the NLG task involves generating NLT containing business data, the user may configure the NLG application 102 to obtain the business data needed from the business data store(s) 112. This may be done in any suitable way. For example, the user may configure the database interface module 124 to obtain specific data records and/or values from particular tables in a database part of business data store(s) 112 so that these data may be obtained and subsequently used together with the semantic objects to generate natural language text.

In some embodiments, the integration layer 126 may be configured to employ NLG software tools 128 to generate natural language text using data contained in the document macro structure(s) 120, semantic object(s) 122, and/or data obtained using the database interface module 124. The techniques for how the integration layer 126 achieves this result are described herein including with reference to FIGS. 4A, 6 and numerous other examples.

In some embodiments, the NLG software tools 128 may include one or more software libraries, one or more application programming interfaces, one or more software programs, and/or any other suitable software tools used to facilitate generation of natural language text from non-linguistic data. In some embodiments, natural language text may be generated from non-linguistic data (e.g., data contained in document macro structure(s) 120, semantic object(s) 122, and/or data obtained using database interface module 124) using multiple stages of processing including, by way of example and not limitation: (1) an analysis stage; (2) a document planning stage; (3) a micro-planning stage; and a (4) surface realization stage. Accordingly, in some embodiments, the NLG software tools 128 may include software for performing the processing for the processing of these stages. The NLG software tools 128 may include software to perform one specific type of stage and/or software to perform multiple stages.

For example, as shown in FIG. 1B, NLG software tools 128 may include a document planning module 132 for performing processing for the document planning stage, a micro-planning module 134 for performing processing for the micro-planning stage, and a surface realization module 136 for performing processing for the surface realization stage. Each of the modules may include processor-executable instructions that when executed perform the functionality associated with the stage.

In some embodiments, the document planning stage of natural language generation builds an intermediate representation of the document to be generated. In some embodiments, the intermediate representation may describe the levels of the document (e.g., titles, sections, chapters, paragraphs, etc.). Additionally, the intermediate representation may describe relationships among elements of the document such as causal relationships or other rhetorical constructs. In some embodiments, the intermediate representation may comprise multiple entities, the smallest ones of which correspond to natural language segments generated from semantic objects, as described herein. A natural language segment may be a phrase, a single sentence, multiple sentences. A natural language segment may include a few words (e.g., one, two, between 3 and 10, between 5 and 20) to a paragraph in length (e.g., between 20 and 200 words).

In some embodiments, the document planning stage of natural language generation, implemented by document planning module 132, may include: (1) a content determination stage during which an NLG system may obtain content to be expressed in natural language text; and (2) a document structuring stage for determining the rhetorical structure of the text to be generated during which the NLG system may generate a document plan indicating how the obtained content is to be organized for presentation in the natural language text to be generated. For example, to generate natural language text about the weather, information about the weather may be obtained in the content determination stage (e.g., information indicating the temperature and information indicating the likelihood of rain) and organized for presentation in the document structuring stage (e.g., by determining that information indicating the likelihood of rain should be presented before information indicating the temperature).

In some embodiments, the document planning stage may be implemented at least in part by using one or more semantic objects (e.g., using one or more semantic object(s) 122). In some embodiments, a semantic object includes a set of attributes and each of one or more of the attributes may be associated with a vocabulary for rendering that attribute into corresponding natural language text. During the document planning stage, the NLG system 101 may determine to generate a natural language segment (e.g., a sentence) using a particular semantic object. The particular semantic object may be used to determine the structure for the natural language segment—the segment may be structured to have content corresponding to attributes of the semantic object. The order of presentation of the content may be determined in the document planning stage and, for example, may be determined based on the document plan configuration (e.g., document plan configuration 158) part of the semantic object (e.g., semantic object 150).

For example, during the document planning stage, the NLG system 101 may determine to generate a natural language segment using semantic object 150 having attributes 152, data variables 154, vocabulary 156, and document plan configuration 158. The segment may be generated to have content corresponding to attributes 152 and data variables 154 of the semantic object. The vocabulary 156 may be used to render the attributes 152 into natural language text. The business data 113 may be used to specify values for the data variables 154 (optionally, with further analysis being performed on the values). The order in which the content corresponding to attributes 152 and data variables 154 is presented in the generated natural language segment may be specified by document plan configuration 158. Further examples of this are described herein including with reference to FIGS. 2A-2F, 3A-3F, and 4A-4J.

In some embodiments, the micro-planning stage of NLG, implemented by micro-planning module 134, may involve determining, based at least in part on the document plan, a syntactic structure for the text to be generated. In some embodiments, the micro-planning stage may involve building syntactic structure of the text from a document structure, which may involve choosing the vocabulary, the syntactic constructs, and the sentence boundaries. In some embodiments, the micro-planning stage may include an aggregation stage, a lexicalization stage, and a referring expression stage. The aggregation stage may involve determining boundaries between sentences. The lexical choice stage may involve choosing words to describe particular concepts to be expressed in the text to be generated (e.g., determining whether “warm” or “hot” should be used to describe a temperature of 80 degrees). In some embodiments, vocabularies for semantic objects may be used to perform the lexical choice stage for sentences generated based on semantic objects. For example, the lexical choice stage may be performed by substituting one or more of the attributes (e.g., attributes 152) specified by a semantic object (e.g., semantic object 150) with a corresponding vocabulary word (or words) from the vocabulary (e.g., vocabulary 156) for the attribute. In some embodiments, lexicalization may be performed using one or more grammars, for example, one or more abstract categorical grammars (ACGs), tree adjoining grammars (TAGs), context free grammars (CFGs), functional identification grammars (FIGs), any other suitable grammars, and/or in any other suitable way. The referring expression stage may involve selecting expressions, for use in the text to be generated, to refer to concepts that appear more than once in the text (e.g., selecting the pronoun “it” to refer to “the weather” in a portion of the text to be generated).

In some embodiments, the surface realization stage of NLG, implemented by surface realization module 136, may involve transforming the syntactic structure of the document to be generated into text and may include a linguistic realization stage and a structural realization stage. The linguistic realization stage may involve generating actual text according to rules of syntax, morphology, and orthography, and may include putting words in order (e.g., in a manner consistent with the order of attributes in a semantic object for sentences being generated based on a semantic object), conjugating verbs, ensuring adjective-noun agreement, etc. Additional low-level rules relating to number formats, date formats, and capitalization may be applied at this stage. During the structural realization stage, the text generated in the linguistic realization stage may be output in a desired format (e.g., a PDF file, a webpage, an XML file, etc.). The above-described tasks may be performed by an NLG system sequentially in stages or in any other suitable way.

In some embodiments, any one or more of the NLG software tools 128 may be developed from scratch, use already available software tools, or be a combination of one or more available software tools and software developed from scratch, as aspects of the technology described herein are not limited in this respect. There are numerous available software tools which may be leveraged, in some embodiments. For example, the NLG software tools 128 may include one or more available software tools to perform micro-planning. A non-limiting example of such a tool is the EasyText software for micro-planning, described in the article called “EasyText: an operational NLG system.” L. Danlos, F. Meunier, and V. Combet. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 139-144. Association for Computational Linguistics, 2011, which article is incorporated by reference herein in its entirety. EasyText software is also described in U.S. Pat. No. 9,135,244, titled “Method and apparatus for configurable micro-planning,” which is incorporated by reference in its entirety.

As another example, the NLG software tools 128 may include one or more available software libraries for lexicalization. For example, the NLG software libraries 128 may include one or more software tools to perform lexicalization using abstract categorical grammars (ACGs) such as the ACG Toolkit (see e.g., https://acg.loria.fr/#Software).

As another example, the NLG software tools 128 may include one or more software tools to perform surface realization. A non-limiting example of such a tool is the SimpleNLG software for surface realization, described the article called “Simplenlg: A realization engine for practical applications”. A. Gatt and E. Reiter. In Proceedings of the 12th European Workshop on Natural Language Generation, pp. 90-93. Association for Computational Linguistics, 2009, which article is incorporated by reference herein in its entirety.

As may be appreciated from the foregoing, the techniques developed by the inventors for generating natural language text using semantic objects may utilize any of numerous types of NLG software tools. In some embodiments, such NLG software tools may include constructs for representing semantic object components including data variables, attributes, and vocabularies. For example, an NLG software tool part of tools 128 may represent data variables using a relational database schema, an object oriented programming (OOP) class hierarchy, or any other suitable structured representation. Additionally or alternatively, less structured representations, such as attribute-value pairs and/or NoSQL database schemas may be utilized. As another example, an NLG software tool part of tools 128 may represent semantic object attributes as tree structures with typed nodes (e.g., semantic trees), directed acyclic graphs, or other types of graphs, for example, ontologies using the web ontology representation language (OWL). As another example, an NLG software tool part of tools 128 may represent semantic object vocabularies using attribute-value pairs. As yet another example, an NLG software tool part of tools 128 may use an underlying linguistic theory to represent syntactic, morphological and phonological information, which may impose a representation. Non-limiting examples of such linguistic theories are Tree Adjoining Grammars, Functional Unification Grammars, Meaning-Text theory, and Abstract Categorical Grammars.

In some embodiments, the NLG software tools 128 used by NLG system 101 may be configured to generate a natural language text segment, such as a single sentence, from multiple semantic objects by composing intermediate representations generated from the semantic objects. To facilitate this, the NLG software tools 128 used may use semantic and/or syntactic representations that are compositional. Such compositionality may be achieved using software tools that rely on linguistic theories for the syntax-semantic interface, examples of which are provided herein.

FIG. 1C is a diagram of an illustrative semantic object 150, in accordance with some embodiments of the technology described herein. As shown in FIG. 1C, semantic object 150 includes attributes 152, which includes data attributes 152a and lexical attributes 152b, data variables 154, variable metadata 155, vocabulary 156, document plan configuration 158, which includes content selection configuration 158a and document structure configuration 158b, analysis configuration 160, micro-planning configuration 162, and surface realization configuration. These components of semantic object 150 are illustrative and, in some embodiments, one or more of the components may be omitted and/or one or more other components may be part of semantic object 150 in addition to or instead of one or more of the components shown herein. For example, in some embodiments, one or more of content selection configuration 158a, analysis configuration 160, micro-planning configuration 162, surface realization configuration 164 and variable metadata 155 may not be included in a semantic object, which is why they are shown using dashed lines.

In some embodiments, the attributes 152 of semantic object 150 contain content to be rendered as a natural language text segment. Each of the attributes represents a respective piece of information that may be included in the natural language text segment generated from the semantic object. Each semantic object attribute may be thought as containing a piece of information that will occupy a text slot in the rendered natural language segment and, for that reason, a semantic object attribute may be termed as a “text slot” herein. Thus, the attributes of a semantic object may be rendered as portions of text in the segment generated from the semantic object using the techniques described herein. Examples of semantic object attributes are provided herein including with reference to FIGS. 2A and 3C.

In some embodiments, in addition to information specifying content, each of one or more of the attributes may include information indicating its semantic role (e.g., a part of speech or a relationship) in the natural language segment to be generated. For example, an attribute may include information indicating that it is a predicate, a noun phrase, a verb, a modifier (e.g., an adverb or adjective), another part of speech, and/or another type of role in a linguistic formalism. As another example, in some embodiments, one or more of the attributes may include information indicating its relationship with one or more other data attributes. For example, an attribute may indicate that it is to precede, or follow, or be rendered within a threshold number of words of another attribute. As another example, an attribute may indicate that it semantically related (e.g., that it is a modifier of, like an adjective) another attribute (e.g., a noun phrase).

A semantic object attribute may be a data attribute or a lexical attribute. Indeed, as shown in FIG. 1C, attributes 152 include data attributes 152a and lexical attributes 152b. Data attributes 152 are discussed first; a discussion of lexical attributes 152b follows. In some embodiments, a data attribute contains numeric information to be rendered as text. For example, a data attribute may contain numerical values representing quantities such as business data (e.g., profit, loss, revenue, percent increase of revenue, number of employees, etc.), dates (e.g., Feb. 21, 2032), time (e.g., 12:03:00), and any other type of value to be rendered as part of natural language text.

In some embodiments, content of one or more data attributes 152a may be obtained from business data obtained from a data store (e.g., business data store(s) 112) as described with reference to FIG. 1A. To this end, the semantic object may include one or multiple data variables 154 whose values may be assigned to the data values obtained from the data store. In turn, the values of a data variables 154 may be assigned directly to the values of data attributes 152a and/or the values may be used to perform various computations (e.g., mean, variance, clustering, classification, correlation, statistical calculation, etc.) and the results of these values may be assigned to the values of data attributes 152.

For example, the data variables of a semantic object for generating a natural language segment about any change in the profit of a company may include a variable indicating a starting value of (e.g., $50 million) representing a company's profit one year and another variable indicating an ending value (e.g., $75 million) representing the company's profit the following year. These values of these variables may be set from data obtained from one of the company's databases and used to calculate the percentage change (e.g., increase) in profit for the company (e.g., 50%). In turn, the semantic object may have a data attribute representing the percentage change in the profit of the company and its value will therefore be set to 50%. That value will be rendered as part of a natural language segment generated from the semantic object (e.g., “Company's profit increased by 50% from 2018 to 2019”).

In some embodiments, one or more of the data variables 154 may be associated with metadata. The metadata for data variables 154 may be part of variable metadata 155. The metadata for a data variable may include any suitable type of metadata including, but not limited to, metadata indicating origin of the data (e.g., when it was created, who created it, what system it was obtained from, when it was edited, etc.), subjective importance of the data (e.g., “critical”, “important”, “normal”), and/or any other suitable type of metadata. Such metadata may be used in analyses performed on the data. For example, data that is not sufficiently recent (e.g., not updated within a threshold amount of time) may be omitted from a calculation.

In some embodiments, the analysis configuration 160 specifies the analysis and/or calculations to be performed on the values of data variables. As described above, the results of such calculations may be assigned to data attributes 152a, which in turn may be rendered as natural language text. Non-limiting examples of analyses include statistical calculations (e.g., mean, media, variance, confidence intervals and/or any other suitable statistical calculation), time series calculations (e.g., finding maximum or minimum in a time series, finding outliers relative to a baseline, finding the level of volatility/consistency of variation of the data in a given time period, machine learning calculations (e.g., clustering, classification, prediction, etc.), finding the contribution of different variables in a formula (e.g., via partial derivatives), inference, and/or any other suitable calculation or calculation(s), as aspects of the technology described herein are not limited in this respect. Any suitable mathematical software library or libraries may be used by NLG system 101 to perform the calculations. In some embodiments, the analysis configuration 160 may be specified using a mark-up language such as the Predictive Model Markup Language (PMML) or any other suitable mark-up language.

Returning to lexical attributes 152, a lexical attribute represents, at an abstract level, a chunk of text to be rendered into a natural language segment. The abstract representation of text by a lexical attribute may be transformed into actual text by using vocabulary 156, which contains text strings (e.g., words) that may be used to realize the lexical attribute as natural language in the rendered segment.

For example, as shown in FIG. 2B, a lexical attribute 210c representing the concept of an entity (e.g., the thing that the natural language segment will be talking about) may be associated with the vocabulary 212c containing the text strings: “revenue” and “profit”, such that the generated segments describe something about the revenue or profit of a company. As another example, as shown in FIG. 2B, a lexical attribute 210d representing the concept of “variation” or change may be associated with the vocabulary 212d containing the text strings: “increase”, “decrease”, “a growth”, “a fall”, “a stagnation”, “to go down”, “a drop”, and “to see an increase”. As yet another example, as shown in FIG. 2B, a lexical attribute 210e representing the concept of “variation intensity” may be associated with the vocabulary 212e containing the text strings: “important”, “high”, “disappointing”, “excellent”, and “low”, which are adjectives that may be used to describe the variation associated with the entity. Such vocabulary strings, once substituted for the lexical attributes and put through the various stages of the NLG pipeline, may lead to the generation of sentences such as “In 2018, the revenue saw an important increase from a low of $1,000,000 to $2,000,000”, as shown in FIG. 2D.

FIG. 1D is a diagram of an example vocabulary 156 for the semantic object 150 of FIG. 1C. As shown in FIG. 1D, vocabulary 156 includes lexical attributes A, B, C, labeled as 160a, 160b, and 160c, respectively. Attribute 160a is associated with respective metadata 165a having text stings 162a and syntactic type information 164a. Attribute 160b is associated with respective metadata 165b having text stings 162b and syntactic type information 164b. Attribute 160c is associated with respective metadata 165c having text stings 162c and syntactic type information 164c. Examples of text strings are provided herein. The syntactic type information may be information indicating the part of speech for the text strings (e.g., noun, verb, adjective, adverb, etc.). Examples of syntactic type information are shown herein including in FIG. 2B.

A semantic object may include any suitable number of attributes, as aspects of the technology described herein are not limited in this respect. This applies to data attributes and lexical attributes. For example, although in the embodiment of FIG. 1D, the vocabulary 156 is shown as having entries for three lexical attributes, a semantic object may have any suitable number of lexical attributes and the vocabulary 156 may include entries for all such lexical attributes.

As shown in FIG. 1D, entries in vocabulary for a semantic object may comprise (identifier, metadata) pairs, each identifier identifying a respective lexical attribute and the metadata containing text strings and, optionally (as indicated by dashed lines in the figure), syntactic type information for the lexical attribute. While, in some embodiments, vocabulary entries may be stored as paired data, in other embodiments the vocabulary entries may be stored in any other suitable way (e.g., as records in a database, using a mark-up language) in any suitable format and/or using any suitable data structure(s), as aspects of the technology described herein are not limited in this respect.

In some embodiments, the attributes 152 of a semantic object 150 may be processed through a series of stages to generate a natural language text segments. For example, in order to obtain values for the data attributes 152a, data may be obtained from one or more external sources and processed using analysis configuration 160. Next, a first intermediate representation of the semantic object may be generated using the attributes 152 and document plan configuration 158 in the document planning stage. Next, a second intermediate representation of the semantic object may be generated using the first intermediate representation, the vocabulary 156 and micro-planning configuration 162 during the microplanning stage. Finally, the natural language segment may be obtained from the second intermediate representation using the surface realization configuration 164 during the surface realization stage.

As shown in FIG. 1C, in some embodiments, the document plan configuration includes content selection configuration 158a and document structure configuration 158b. The content selection configuration 158a may indicate which ones of the attributes 152 of the semantic object are to be used in generating a natural language text segment. The configuration 158a may indicate that some or all of attributes 152 may be used.

In some embodiments, the document structure configuration 158b may include information used by the NLG system to determine how the content of the semantic attributes 152 is to be organized into the natural language text segment generated from the semantic object 150. The document structure configuration 158b may include a list of one or multiple options describing different structures for the natural language text segment generated from the semantic object 150. For example, document structure configuration 158b may include information specifying an order in which some or all of the semantic object attributes 152 are to appear in the natural language text segment, when rendered. For example, document structure configuration 158b may include a data structure (e.g., a semantic tree) describing relationship among the attributes. As another example, document structure configuration 158b may include information specifying a rhetorical relationship (e.g., a cause and effect) relationship between semantic attributes of a single semantic object or between multiple semantic objects. The document structure configuration 158b may be specified using in any suitable way and in any suitable format, for example, by using a mark-up language, examples of which are provided herein. Examples of document structure configuration 158b are described herein including with reference to FIGS. 2A, 2E, 3D, 5A, and 5C.

In the illustrative embodiment of FIGS. 1A-1D, document macro structure(s) 120 and document plan configuration 158 are separate from one another. In this embodiment, the document macro structure(s) 120 specifies structure of the overall document to be created from segments obtained from respective semantic objects (e.g., titles, sections, chapters, paragraphs, order of segments generated from semantic objects), but does not specify the structure of the text within each segment. On the other hand, the document plan configuration 158 specifies, for a semantic object, the structure of the text segment generated from the semantic object. However, in other embodiments, the information relating to macro-structure (part of document macro structure(s) 120 in this embodiment) and information relating to micro-structure (part of document plan configuration 158) need not be separated and may be part of a single configuration or may be split up into multiple other configurations, in any suitable way, as aspects of the technology described herein are not limited in this respect.

In some embodiments, micro-planning configuration 162 may include information to be used during the micro-planning stage of generating natural language text from semantic object 150. In some embodiments, this information may include information for controlling aggregation, word order, generating referential expressions, any information used by the software tool(s) of NLG software tools 128 used to perform micro-planning, and/or any other suitable information used to control the manner in which micro-planning is performed.

In some embodiments, surface realization configuration 164 may include information to be used during the surface realization stage of generating natural language text from semantic object 150. In some embodiments, this information may include information for specifying formats of numbers, dates, capitalization, any information used by the software tool(s) of NLG software tools 128 used to perform surface realization, and/or any other suitable information used to control the manner in which surface realization is performed.

Semantic Object Example #1

FIG. 2A is a diagram of an illustrative semantic object 200 having attributes 202, data variables 204, a vocabulary 206, and a document structure configuration 208, in accordance with some embodiments of the technology described herein. The example semantic object 200 may be used to generate text describing the variation of a numerical value (e.g., revenue of a company, height of a person, temperature outside, score in a game, etc.) over time.

As shown in FIG. 2A, data variables 204 include a numerical value V1, a date D1 associated with value V1, a numerical value V2, and a date D2 associated with value V2, where the date D2 is later date than the date D1. The attributes 202 include data attributes 202a corresponding to the data variables. In particular, the data attributes 202a START VALUE, START DATE, END VALUE, and END VALUE may be assigned the values V1, D1, V2, and D2, respectively.

The attributes 202 also include lexical attributes 202b including the VARIATION attribute representing a change—the main predicate in this example, the ENTITY attribute representing the entity whose value is changing over time, the VARIATION INTENSITY, START INTENSITY and END INTENSITY modifiers, which may be rendered as adjectives modifying the VARIATION, the START VALUE, and the END VALUE.

As shown in FIG. 2A, document structure configuration 208 describes how the various attributes are related. In this example, the document structure configuration 208 is specified as a semantic tree specifying relationships among the attributes as follows:

ENTITY

In this representation, the convention is that children of a node are between parentheses to the right of the node. The semantic convention is that modifiers have their modifiee as child node, and that the predicate has its arguments as child nodes.

FIG. 2B is a diagram of an example vocabulary 206 that may be used when rendering the attributes 202 for the semantic object 200 of FIG. 2A, in accordance with some embodiments of the technology described herein. Entries in vocabulary 206 comprise (identifier, metadata) pairs including pairs: (210a, 212a), (210b, 212b), (210c, 212c), (210d, 212d), (210e, 212e). The metadata may include information describing: text options that may be used to render attributes as text, associated parts of speech, and rules for the text options that are available for rendering text (not shown).

For example, pair (210a, 212a) indicates that the attribute START VALUE is of type “Currency” and the pair (210b, 212b) indicates that the attribute START DATE is of type “Date Year”. These value types may be used by the NLG system 101 (e.g., using NLG software tools to perform micro-planning and/or surface realization) to appropriately render these values into text having the format appropriate for currencies and dates. As another example, pair (210c, 212c) indicates that the ENTITY attribute may be rendered using the text strings: “revenue” or “profit”. The metadata 212c further indicates that the syntactic type of “ENTITY” is noun. As another example, pair (210d, 212d) indicates that the main predicate VARIATION may be rendered using the text strings: “increase”, “decrease”, “stagnate”, “a growth”, “a fall”, “a stagnation”, “to go down”, “a drop”, “to see an increase”. The metadata 212d further indicates that the syntactic type of VARIATION is “verb”. As yet another example, pair (210e, 212e) indicates that the VARIATION INTENSITY attribute may be rendered using the text strings: “important”, “high”, “disappointing”, “excellent”, and “low.” The metadata 212e further indicates that the syntactic type of VARIATION INTENSITY is “adjective”. Although not shown, vocabulary 206 may include analogous entries for any other attributes part of attributes 202 (e.g., START VALUE INTENSITY, END VALUE INTENSITY, etc.).

In addition, although not shown in FIG. 2B, vocabulary 206 may include rules specifying which text strings are available for use in rendering the attributes with which they are associated. The rules may be used to constrain the available options depending on the values of the data variables 204. For example, if value V1 is lower than value V2, the text strings “increase”, “a growth”, “to see an increase” may be selected, but text strings such as “decrease”, “a fall” and “to see a decrease” will not be selected. Similarly, rules may be added to control the textual objects available for modifiers depending on the value of the data variables.

To illustrate the types of natural language text segments that may be generated using semantic object 200, let us suppose that the data variables 204 are set as shown in panel 220 of FIG. 2C. In particular, the values V1, D1, V2, and D2 are set to values “1,000,000”, “Jan. 1, 2018”, “2,000,000”, and “Jan. 1, 2019,” respectively. With these values and other configurations part of semantic object 200, including those described in FIG. 2A, the natural language segments shown in FIG. 2D may be generated. For example, the following sentence 232 may be generated: “In 2018, the revenue saw an important increase from a low of $1,000,000 to $2,000,000 may be generated.” Alternatively, the sentence 236 could be generated: “Importantly, the revenue increased in 2018 from a low of $1,000,000 to $2,000,000”.

As described herein, semantic objects may be easily customized by adding or removing configurations. For example, an analysis configuration may be added to semantic object 200 for the computation of a percentage of change in order to generate sentences like sentence of 234, “The revenue rose by 100% last year”. As another example, a micro-planning configuration may be added to control word order to select from among segments such as “In 2018, the revenue increased” and “The revenue increased in 2018”. As yet another example, a micro-planning configuration may be added to separate text into several sentences to produce a segment such as sentences 238, “In 2018, the revenue increased by 10%. It reached a historic high of $2,000,000 from $1,000,000 the previous year.” As yet another example, a surface realization configuration may be added to customize the data format, the currency format, or to present the currency in a currency other than dollars (e.g., Euros).

Combinations of Semantic Objects

As described herein, natural language segments generated from different semantic objects may be placed next to each other in the overall document. Such ordering constraints may be specified by a document plan (e.g., in document macro structure(s) 120). In this type of combination, data in one semantic object does not impact the natural language text segment generated using another semantic object. For example, two semantic objects may be used to generate two different sentences that, when placed next to each other, results in the following text:

However, in some embodiments, semantic objects may be combined in other ways, such that the combination has an impact on the generated text segment. To this end, in some embodiments, multiple semantic objects may be composed into an extended semantic object, which may be used to facilitate automated generation of sentence structures having greater complexity.

In some embodiments, multiple semantic objects may be combined by: (1) combining the data variables of the different semantic objects into a single set of data variables; (2) combining the attributes of the different semantic objects into a single set of attributes; and/or (3) combining the vocabularies of the different semantic objects into a single vocabulary.

In some embodiments, the document structure configurations of the semantic objects may be combined to generate a document structure configuration for the extended semantic object by adding a new attribute representing a rhetorical relationship between the segments represented by constituent semantic objects and using the new attribute when rendering the extended semantic object obtained through this composition into text. In this way, intermediate representations generated from individual semantic objects during the document planning stage of NLG may be composed to obtain a composed intermediate representation, which is then provided as input to subsequent NLG stages (e.g., lexicalization, aggregation, referential expression generation, and surface realization).

For example, in some embodiments, it is possible to combine two semantic objects with a neutral “join” relationship (e.g., by adding an attribute representing the “JOIN” relationship to the extended semantic object or in any other suitable way) to merge the previously separate sentences into a single one. In this way, sentences such as following sentences may be obtained:

As another example, in some embodiments, it is possible to combine two semantic objects using a causal relationship (e.g., by adding a new CAUSE attribute to the extended semantic object or in any other suitable way) to merge the previously separate sentences into a single one. In this way, sentences such as the following sentences may be obtained:

The last two sentences in the above example illustrate the challenge of combining sentences. In order to link sentences with a causal relationship such as “because of” or “resulting in”, the syntactic type of some of the elements in the sentences have to be modified. For example, the verbs “increased” and “dropped” may be changed to the noun phrases “an increase” and “a drop”, respectively. As another example, the adverbs “importantly” and “significantly” are transformed into the adjectives “important” an “significant”, respectively.

In some embodiments, these types of syntactic changes may be implemented using the NLG software tools 128 when they have the compositionality property. The compositionality property ensures that when semantic objects are combined together, the output text is syntactically correct. For example, NLG software tools 128 that use the tree adjoining grammars (TAG), allows for syntactic constraints to be attached to words and groups of words so that the NLG system can choose, reorder, and/or combine the words together while making sure that the generated text is syntactically valid. Other syntactic formalisms such as feature unification grammars (FUG), context free grammars (CFG), etc. may be used.

Composing semantic objects using a causal relationship may be applied in context of the example semantic object 200 described above with reference to FIGS. 2A-2D. This example shows how to combine two semantic objects describing the variation of a numerical value together by a causal relationship. The combined semantic object may be obtained by combining the data variables, attributes, and vocabularies of two constituent semantic objects and adding: (1) a new CAUSE attribute to the combined set of attributes; and (2) a corresponding entry into the combined vocabulary including text strings that may be used to render the CAUSE attribute into text such as, for example, the text strings “to be caused by”, “because of”, and “as a result of”. The document structure configuration may be updated as well resulting in the document structure configuration 240, shown in FIG. 2E, for the combined semantic object.

The resulting combined semantic object may be used to generate sentences such as sentence 242, shown in FIG. 2F: “As a result of the increase in costs in 2018, benefits declined during the same period.” In this example “costs” and “benefits” are given by vocabularies associated with the ENTITY attributes of each constituent semantic object. As described above, rendering of the combined semantic object into natural language text may rely on the compositionality of the document structure configurations (e.g., using semantic trees), which ensures that only combinations of vocabularies that lead to syntactically valid sentences are generated.

Semantic Object Example #2

Another example of a semantic object is described with reference to FIGS. 3A-3F, which show various example components of its specification. Like the semantic object 200 of Example #1, this semantic object also may be used to generate natural language segments describing the variation of an entity over a time period. Examples of natural language text segments that may be generated using the specification of the semantic object are described herein including with reference to FIGS. 4A-4J.

FIG. 3A illustrates the specification 300 of the data variables of the semantic object in this example. As shown in FIG. 3A, the semantic object includes four data variables: variables 302 and 303 representing two numeric values (“first_dp” and “second_dp”) and variables 304 and 305 representing two dates (“first_date” and “second_date”).

FIG. 3B illustrates the analysis configuration 310 of the semantic object in this example. The analysis configuration defines three analyses: 312, 314, and 316 that may be performed on the data variables shown in FIG. 3A. For example, analysis 312 may be used to determine the difference between the values of variables 302 and 303. As another example, analysis 314 may be used to determine the percent change between the values represented by variables 302 and 303. As another example, analysis 316 may be used to determine the duration of the time period between the dates represented by variables 304 and 305.

FIG. 3C illustrates the attributes 320 of the semantic object in this example. As shown in FIG. 3C, attributes 320 include four data attributes 321, 322, 323, and 324, and four lexical attributes 325, 326, 327, and 328. Data attributes 321 and 322 correspond to the values of data variables 302 and 303, respectively. Data attributes 323 and 324 correspond to the output of analyses 314 and 316, respectively. Attribute 325 represents the entity (e.g., “revenue” or “profit” or “sales”, etc.) whose variation the generated natural language text segment will be describing. The attribute 326 represents the type of variation (e.g., “increase”, “decrease”, etc.) that the generated segment will be describing. The attributes 327 and 328 represent modifiers (e.g., adjectives or adverbs) of the variation as a whole and at the end date, respectively.

FIG. 3D illustrates the document structure configuration 330 of the semantic object in this example. As shown in FIG. 3D, the document structure configuration 330 includes two document structure configurations 332 and 334, either one of which may be used to generate natural language segments from the semantic object. These document structure configurations may give rise to text segments having structure. For example, configuration 332 may give rise to a sentence structured as: “In YYYY, the revenue increased substantially from $X to a high of $Y.” On the other hand, configuration 334 may give rise to a sentence structured as: “In YYYY, the high revenue of $Y increased substantially from $X”.

FIG. 3E illustrates the vocabulary 340 of the semantic object in this example. As shown in FIG. 3E, the vocabulary 340 has entries for the various attributes shown in FIG. 3C. Entries 341, 342, 343, 344 correspond to the data attributes 321, 322, 323, and 324, respectively. Entries 345, 346, 347, and 348 correspond to the lexical attributes 325, 326, 327, and 328, respectively. Each of the entries includes information that may be used to render the corresponding attribute into text. For example, entries corresponding to data attributes include information for rendering numbers into text including information about the precision, scale, format, units, and/or currency for the numbers. As another example, entries corresponding to lexical attributes include text strings (e.g., for roots of the words to be used to render the attribute into text), part-of speech information (e.g., “Verb”, “Noun Phrase”, “Adverb”, “Adjective”) and other information.

FIG. 3F illustrates the micro-planning configuration 350 of the semantic object in this example. The micro-planning configuration 350 includes options to be used during the aggregation stage of natural language generation.

An NLG system (e.g., system 101) may be used to generate various types of sentences depending on the values of the data points. For example, the following sentences could be generated:

Depending on the context in which the semantic object used, relative to segments generated using other semantic object, the following variations are also possible:

Aspects of how to generate natural language text segments from the semantic object of FIGS. 3A-3F are described herein including below with reference to FIGS. 4A-4J.

Illustrative Methods

FIG. 4A is a flowchart of an illustrative process 400 for generating natural language text using one or more semantic objects, in accordance with some embodiments of the technology described herein. The process 400 may be performed by any suitable natural language generation system and, for example, may be performed by NLG system 101 described with reference to FIGS. 1A-1D. Aspects of process 400 are described herein with reference to the semantic object specification shown in FIGS. 3A-3F.

Process 400 begins at act 402, where the specification of a semantic object is obtained. The specification may be obtained from any suitable source (e.g., semantic object data store 140, a configuration file or files, etc.). The specification may be in any suitable format. For example, the specification may be specified using a mark-up language such as YAML or any other suitable mark-up language including any other mark-up language described herein. As one example, the specification of the semantic object shown in FIGS. 3A-3F may be accessed at act 402.

In some embodiments, the specification accessed at act 402 may be used to instantiate one or more variables and/or initialize one or more data structures in the memory of the system executing process 400 in preparation for the subsequent operations of natural language generation. For example, a data structure may be generated to represent one or more of the document plan configurations in the specification. As another example, variables corresponding to the data variables may be instantiated.

Next, at act 404, data related to the data variables of the semantic object may be accessed. These data may be accessed from any suitable source. In some embodiments, at least some of the data may be accessed from a system external the NLG system (e.g., business data store(s) 112 described with reference to FIG. 1A). In other embodiments, at least some of the data may be provided to the NLG system in a configuration file, a function call, or in any other suitable way. An example of data that may be accessed during act 404 is illustrated in FIG. 4B, which shows example values for the variables 302, 303, 304, and 305 shown in FIG. 3A.

Next, at act 406, one or more mathematical analyses on the data obtained at act 404 may be performed. In some embodiments, the computations performed at act 406 may be specified by the analysis configuration (e.g., 160) of the semantic object. For example, with reference to the example of FIGS. 3A-3F, the mathematical analyses specified in analysis configuration 310 may performed, at act 406, on the data obtained at act 404. The results of these calculation are shown in FIG. 4C and include values of the absolute difference between the starting and ending value (50), the percentage of increase that it represents (100%) and the period of analysis (the year 2018).

Next, at act 408, a content selection configuration may be used to perform content selection by selecting only a subset of the attributes of the semantic object to be used for generating the natural language text segment. In some embodiments, the content selection configuration may include Boolean variables indicating whether respective attributes are to be included. For example, in some embodiments, a content selection configuration may indicate that a modifier attributes is to be omitted so that instead of a generated segment stating “Revenue increased significantly in 2019 by $20M”, the generated segment states “Revenue increased in 2019 by $20M”, without the editorializing. A content selection configuration may indicate which attributes to include, which attributes to exclude, or both which to include and which to exclude. A content selection configuration may specify whether to include and/or exclude each of the attributes or may specify this for only a subset of the attributes. When a content selection configuration is omitted, all attributes may be used.

Next, process 400 proceeds to act 415 during which a natural language text segment is generated using the semantic object whose specification was accessed during act 402, the data obtained at act 404, the results of any calculations performed on these data at act 406, and the selection of any attributes of interest. Act 415 includes multiple stages: (1) generating a first intermediate representation of the semantic object at act 410; (2) applying automatic aggregation to the first intermediate representation at act 412; (3) generating a second intermediate representation of the semantic object from the first intermediate representation of the semantic object using the vocabulary of the semantic object at act 414; (4) applying referent generation technique(s) to the second intermediate representation of the semantic object at act 416; and (5) applying surface transformation(s) to the second intermediate representation of the semantic object to generate the natural language text segment at act 418. After the natural language text segment is generated at act 415, the natural language segment may be output at act 420 (e.g., for use by the NLG system to insert into a larger set of natural language text being generated, for example, a document; sent to a remote user, sent to a content management system, stored in non-transitory memory for subsequent access, etc.). Acts 410, 412, 414, 416, and 418 are further described next.

As described above, at act 410, the first intermediate representation of the semantic object is generated using its document structure configuration, values for data variables, and attributes (both data and lexical attributes). For example, as shown in FIG. 4D, document structure configuration 332 may be selected from among the options in configuration 330 shown in FIG. 3D. In turn, numeric values and/or lexical data may be substituted in for the attributes as shown in FIG. 4E.

As shown in FIG. 4E, the values of data variables are put in place of the data attributes (e.g., the value “2018” replaces the data attribute “period”, the value “50” replaces the data attribute “start_value”, the value “100” replaces the data attribute “end_value”). Data in vocabulary entries (shown in FIG. 3E) are substituted for the lexical attributes (e.g., “important” is substituted for the lexical attribute “variation intensity”, “increase” is substituted for the lexical attribute “variation_predicate”, “revenue” is substituted for the lexical attribute “entity”, and “high” is substituted for lexical attribute “end_value_intensity”. It should be noted that although the language being inserted for the lexical attributes does not necessarily represent the final text strings to be rendered in the output natural language text segment. Multiple processing operations may modify the language inserted at this stage (e.g., changing part of speech: from “important” to “importantly”, conjugation of verbs: from “increase” to “increased” or “increasing”, and various other changes).

As may be appreciated from the foregoing, in some embodiments, the document planning phase may be implemented using document structure templates (e.g., as shown in FIG. 4D), which may include one or more optional elements. FIG. 5A shows another example of a document structure configuration, in which variables are represented by a $ sign. From this representation, the first intermediate representation may be obtained through substitution, for example as shown in FIG. 5B.

In some embodiments, a document structure configuration may be specified using a schema, which may be a template augmented with regular or context free grammar mechanisms, possible with conditional statements as well. For example, the template of FIG. 5A may be augmented to obtain the schema shown in FIG. 5C. This schema includes the element “foreach: analysis.contributors”, which specifies that the “cause” block is to be repeated for each contributor to the consequence variation detected by the analysis.

Next, process 400 proceeds to act 412 where automatic aggregation is applied to the first intermediate representation. In some embodiments, automatic aggregation may be used to compose the first intermediate representation with an intermediate representation of another semantic object. For example, as shown in FIG. 4F, the semantic object attribute “entity” has been replaced with an attribute “join” itself references two indicators: “indicator” and “other_indicator”. This may result in more complex sentences such as “In 2019, both the profit and revenue increased by 50%”. Although, in some embodiments, such joining may be specified using rules in a document plan, in other embodiments, opportunities to join (or otherwise rhetorically connect) semantic objects may be discovered using pattern recognition techniques and/or rules during the aggregation stage.

For example, in some embodiments, the NLG system performing process 400 may define rules of the form “pattern→replacement”. The “pattern” is a semantic structure pattern and the “replacement” is another semantic structure pattern that should replace the left-hand side of the rule. In some such embodiments, the aggregation procedure may be performed using the following search procedure:

The order in which rules are considered may be specified in advance so that the search procedure terminates and is performed efficiently. In some embodiments, the aggregation rules may be built into the NLG system 101. Additionally or alternatively, the aggregation rules may be specified as part of one or more semantic objects, for example, in their microplanning configuration (e.g., micro-planning configurations 162 and 350).

Next, process 400 proceeds to act 414, where a second intermediate representation of semantic object is generated from the first intermediate representation using the vocabulary of semantic object. This processing corresponds to the lexicalization stage of natural language generation. In some embodiments, generating the second intermediate representation from the first intermediate representation comprises generating a structure (e.g., string) annotated with syntactic and/or morphological information. For example, as shown in FIGS. 4E and 4G, the first intermediate representation shown in FIG. 4E may be transformed to the second intermediate representation shown in FIG. 4E shown in FIG. 4G (“in 2019, <revenue> <to_increase> importantly from <50> to a high <100>”). The lexicalization stage may be performed using any of numerous types of lexicalization techniques. In some embodiments, lexicalization may be performed using one or more grammars, for example, one or more abstract categorical grammars (ACGs), tree adjoining grammars (TAGs), context free grammars (CFGs), functional identification grammars (FIGs), any other suitable grammars, and/or in any other suitable way.

As one non-limiting example, in some of the embodiments in which abstract categorical grammars are used to perform lexicalization, the second intermediate representation may be generated from the first intermediate representation using the following algorithm:

Any of numerous available NLG software and/or database tools may be used to perform one or more of the above-described operations. Although, in some embodiments, ACGs may be used (e.g., due to compositionality and computational efficiency afforded by this approach), other approaches (e.g., TAGs, CFGs, etc.) may be used in some embodiments, as aspects of the technology described herein are not limited in this respect.

In some embodiments, although the lexicalization stage may be configured to produce different outputs. In some such embodiments, one of the options may be selected using any suitable criteria such as, for example, minimizing redundancy and/or repetition with text already written).

Next, process 400 proceeds to act 416 where referent generation techniques are applied to the second intermediate representation of semantic object. In this act, long range dependencies in the text may be resolved in order to reference entities and use anaphoric expressions instead of referential expressions, where possible.

For example, in some embodiments, for each entity reference in the second intermediate representation the following acts may be performed:

Aspects of selecting referential and anaphoric expressions are further described in U.S. Pat. No. 9,582,501, titled “Techniques for Generation of Natural Language Text”, granted on Feb. 28, 2017, which is incorporated by reference herein in its entirety.

Continuing with the example, suppose that the semantic object of FIGS. 3A-3F is used to generate the second intermediate representation (b) that appears in the overall document being generated between two segments whose respective second intermediate representations (a) and (c) are shown below:

These second intermediate representations (also shown in FIG. 4H) may be processed, at act 416, to create the strings shown in FIG. 4I. As shown in FIG. 4I, an anaphoric expression (“it”) has been selected to refer to the “revenue”. On the other hand, a referential expression, rather than an anaphoric expression, is selected to describe US revenues to disambiguate this quantity from the overall revenue previously mentioned.

Next, process 400 proceeds to act 418, where one or multiple surface transformations are applied to the second intermediate representation of semantic object to generate natural language text. This involves the application of various formatting and/or morphological rules to produce the final sentences such as the sentences shown in FIG. 4J.

In some embodiments, surface transformations (e.g., number agreement, contractions, capitalization, formatting, etc.) may be implemented using regular expressions and/or context free grammars. For example, in some embodiments, the following procedure may be employed:

Of course, it should be appreciated that any other suitable type(s) of surface realization techniques may be used, as aspects of the technology described herein are not limited in this respect.

It should be appreciated that process 400 is illustrative and that there are variations thereof. For example, in some embodiments, one or more of the acts shown in dashed lines may be omitted. That is, in some embodiments, one or more of acts 406 (mathematical analysis), 408 (content selection), 412 (automatic aggregation), and 416 (referent generation) may be omitted. In some embodiments, the determination as to whether any one or more of these acts is to be omitted may be made automatically based on the specification of the semantic object. When the specification of a semantic object does not include a configuration for a particular act (e.g., a mathematical analysis configuration) that act may be omitted (e.g., no mathematical calculations on the received data are performed).

FIG. 6 is a flowchart of an illustrative process 600 for generating natural language text using one or multiple semantic objects, in accordance with some embodiments of the technology described herein. The process 600 may be performed by any suitable natural language generation system and, for example, may be performed by NLG system 101 described with reference to FIGS. 1A-1D.

Process 600 begins at act 602, where the specification of a semantic object is obtained. The specification may specify data variables(s), attributes, a vocabulary, and a document structure configuration for the semantic object. For example, the specification may specify data variables(s) 154, attributes 152, vocabulary 156, and document structure configuration 158b (or document plan configuration 158 also including content selection configuration 158). The specification may include one or more other configurations, for example, an analysis configuration (e.g., analysis configuration 160), a micro-planning configuration (e.g., micro-planning configuration 162), and/or surface realization configuration (e.g., surface realization configuration 164). The specification may be obtained from any suitable source (e.g., semantic object data store 140, a configuration file or files, etc.). The specification may be in any suitable format. For example, the specification may be specified using a mark-up language such as YAML or any other suitable mark-up language including any other mark-up language described herein.

Next, at act 604, data related to the data variables of the semantic object may be accessed. These data may be accessed from any suitable source. In some embodiments, at least some of the data may be accessed from a system external the NLG system (e.g., business data store(s) 112 described with reference to FIG. 1A). In other embodiments, at least some of the data may be provided to the NLG system in a configuration file, a function call, or in any other suitable way.

Next, at act 606, values for the data variable(s) of the semantic object are determined. In some embodiments, at least one or some of the values are the same as the values obtained at act 604, in which case at least one or some of the data attributes are simply take on the obtained values. Additionally or alternatively, one or more values may be derived (e.g., using an analysis configuration) from the data obtained at act 604, and at least one or some of the data variables take on values obtain as a result of such calculation(s).

Next, process 600 proceeds to act 610 during which a natural language text segment is generated using the semantic object whose specification was accessed during act 602, the data obtained at act 604, and the results of any calculations performed on these data at act 606. Act 610 includes multiple stages: (1) generating a first intermediate representation of the semantic object at act 612; (2) generating a second intermediate representation of the semantic object from the first intermediate representation of the semantic object using the vocabulary of the semantic object at act 614; and (3) generating a natural language text segment from the second intermediate representation of the semantic object. After the natural language text segment is generated at act 610, the natural language segment may be output at act 620 (e.g., for use by the NLG system to insert into a larger set of natural language text being generated, for example, a document; sent to a remote user, sent to a content management system, stored in non-transitory memory for subsequent access, etc.).

At act 610, the first intermediate representation of the semantic object is generated using its document structure configuration, values for data variables, and attributes (both data and lexical attributes). In particular, once a document structure configuration is identified (e.g., selected from multiple options) numeric values and/or lexical data may be substituted in for the attributes. Examples of this processing are described herein including with reference to FIGS. 4A and 5A-5C.

At act 612, a second intermediate representation of semantic object is generated from the first intermediate representation using the vocabulary of semantic object. This processing corresponds to the lexicalization stage of natural language generation. In some embodiments, generating the second intermediate representation from the first intermediate representation comprises generating a structure (e.g., string) annotated with syntactic and/or morphological information. The lexicalization stage may be performed using any of numerous types of lexicalization techniques. In some embodiments, lexicalization may be performed using one or more grammars, for example, one or more abstract categorical grammars (ACGs), tree adjoining grammars (TAGs), context free grammars (CFGs), functional identification grammars (FIGs), any other suitable grammars, and/or in any other suitable way. Examples of this processing are described herein, including with reference to FIGS. 4E and 4G.

At act 614, natural language text is generated from the second intermediate representation obtained at act 612. Various types of NLG processing may be performed at this stage including generation of referential and/or anaphoric expressions and the application of various surface realization transformations. These acts may be performed in accordance with configuration data in the semantic for those acts and/or using existing configurations of the NLG software tools being used to perform these acts. Examples of this processing are described herein including with reference to FIGS. 4H-4J.

It should be appreciated that process 600 is illustrative and that there are variations. For example, in some embodiments, process 600 may include one or more other processing stages in addition to or instead of the stages shown. For example, in some embodiments, an automatic aggregation stage may be included.

Additional Implementation Detail

An illustrative implementation of a computer system 700 that may be used in connection with any of the embodiments of the disclosure provided herein is shown in FIG. 7. The computer system 700 may include one or more processors 710 and one or more articles of manufacture that comprise non-transitory computer-readable storage media (e.g., memory 720 and one or more non-volatile storage media 730). The processor(s) 710 may control writing data to and reading data from the memory 720 and the non-volatile storage device 730 in any suitable manner, as the aspects of the technology described herein are not limited in this respect. To perform any of the functionality described herein, the processor(s) 710 may execute one or more processor-executable instructions stored in one or more non-transitory computer-readable storage media (e.g., the memory 720), which may serve as non-transitory computer-readable storage media storing processor-executable instructions for execution by the processor(s) 710.

Illustrative Hybrid Natural Language Generation (NLG) Systems for Generating Natural Language Text Using a Symbolic NLG Engine and a Large Language Model (LLM)

FIG. 8A is a diagram of another illustrative environment 800 in which some embodiments of the technology described herein may operate. The environment 800 includes an NLG system 101a that may be configured by administrator 110 of FIG. 1A to perform a natural language generation task. In some embodiments, the NLG system 101a corresponds to and/or implements at least one or more portions of the NLG system 101 of FIGS. 1A and/or 1B. As part of the configuration, administrator 110 may configure NLG system 101a to communicate with business data store(s) 112 of FIG. 1A to obtain business data 113, which may include data to be incorporated into natural language text generated by NLG system 101a. The administrator 110 may configure NLG system 101a to perform the NLG task by specifying one or more symbolic NLG text segment configurations 802 (identified by SYM NLG Text Segment Configuration in FIG. 8A), one or more machine learning NLG text segment configurations 804 (identified by ML NLG Text Segment Configuration in FIG. 8A), an engine selector text segment configuration 806, and/or one or more document macro structures 120 of FIG. 1A that the NLG system 101a will utilize to generate natural language text.

In some embodiments, the natural language text (NLT) generated by NLG system 101a may be output to one or multiple destinations (e.g., one or more other devices, users, etc.). For example, as shown in FIG. 8A, NLG system 101 may generate NLT 103a using one or more of the symbolic NLG text segment configurations 802, one or more of the NLG text segment configurations 804, and/or at least some of the business data 113, and provide the NLT 103a to publishing system 104 of FIG. 1A. The publishing system 104 may include the NLT 103a into content 105 (e.g., a webpage, an electronic document, a report, a form, and/or any other suitable type of document) and may provide the content 105 to a user 108, for example, by providing content 105 to software application program 106 (e.g., an Internet browser) executing on the computing device 107 (e.g., smartphone, laptop, desktop, or any other suitable computing device) of user 108. For example, the content 105, the application 106, the computing device 107, and/or the user 108 of FIG. 8A can correspond to and/or be implemented by the content 105, the application 106, the computing device 107, and/or the user 108 of FIGS. 1A and/or 1B.

As described above, in some embodiments, the NLG system 101 may obtain, from business data store(s) 112, business data 113 at least some of which may be included in natural language text to be generated by the NLG system 101a. For example, the business data 113 can correspond to and/or be implemented by the business data 113 of FIGS. 1A and/or 1B.

In some embodiments, NLG system 101a may be configured to perform one or multiple NLG tasks. In some embodiments, the NLG system 101a may be configured to perform an NLG task by an NLG application 102a configured to perform the NLG task. An NLG application 102a may be configured to perform one or multiple NLG tasks, as aspects of the technology described herein are not limited in this respect. In some embodiments, the NLG application 102a corresponds to and/or implements at least one or more portions of the NLG application 102a of FIGS. 1A and/or 1B.

As shown in FIG. 8A, the NLG application 102a has multiple components including document macro structure(s) 120, symbolic NLG text segment configuration 802, database interface module 124, initial text segment 808, ML NLG text segment configuration 804, engine selector text segment configuration 806, NLG software tools 128a, and text segment configuration-NLG software tools integration layer 126a, which is referred to herein as “text segment configuration integration layer 126a” for clarity. In the shown example, document macro structure(s) 120, symbolic NLG text segment configuration 802, and database interface module 124 form and/or are a part of symbolic NLG engine input 810. Further, in the shown example, initial text segment 808 and ML NLG text segment configuration 804 form and/or are a part of ML NLG engine input 812. It should be appreciated that these components are illustrative and that, in some embodiments, an NLG application 102 may include one or more components in addition to or instead of these components.

In the illustrated example of FIG. 8A, NLG application 102a generates and/or outputs NLT 103a by using text segment configurations, such as symbolic NLG text segment configuration 802 and ML NLG text segment configuration 804. In some embodiments, text segment configurations contain and/or specify the information needed by an NLG engine to generate and/or output NLT 103a. For example, symbolic NLG text segment configuration 802 can correspond to and/or be implemented by a specification of a semantic object, such as semantic object(s) 122 of FIG. 1A and/or semantic object 150 of FIG. 1B. In such an example, symbolic NLG text segment configuration 802 can include a specification of semantic object(s) 122 for use by symbolic NLG engine 814 of NLG software tools 128a to output a first natural language text (NLT) segment 818. In some embodiments, symbolic NLG engine 814 corresponds to and/or is implemented by document planning module 132, database interface module 124, surface realization module 136, and/or, more generally, NLG software tools 128, of FIG. 1B. By way of another example, ML NLG text segment configuration 804 can correspond to and/or be implemented by data and/or information used to work with an ML model, such as an LLM. In such an example, the data/information may be used to generate a prompt to an LLM. Furthering the example, ML NLG text segment configuration 804 can include data/information used to generate a prompt for use by ML NLG engine 816 of NLG software tools 128a to output a second NLT segment 820.

In the illustrated example, ML NLG engine 816 is an ML model. Examples of ML models include support vector machines (SVMs), clustering models, graphical models (e.g., hidden Markov models, Markov random fields, Bayesian networks), random forests, decision trees, gradient boosting, and neural networks (NNs). An example of an NN is a deep neural network. An example of a deep neural network is a large language model (LLM). An LLM uses a transformer, which is a particular neural network architecture designed to process and generate data in sequence. The transformer is a set of neural networks that may include a decoder or an encoder and decoder pair with self-attention capabilities. The encoder and decoder extract meaning from a sequence of text and understand the relationships between words and phrases in the sequence. Inputs to LLMs are multi-dimensional vectors (e.g., word embeddings) to represent words so that words with similar contextual meanings or other relationships are close to each other in the vector space. Using word embeddings as input, transformers can pre-process text, for example, as numerical representations through the encoder and understand the context of words and phrases with similar meanings as well as other relationships between words such as parts of speech. Encoders output a continuous representation (e.g., an embedding) of the input to the decoder. LLMs apply the contexts and word relationships through the decoder to produce a unique output, such as a sequence of natural language text. For example, a decoder can iteratively determine words in sequence that have the highest probability of appearing in the same sentence in accordance with the contexts and word relationships of the input sequence of text.

Examples of types of LLM models include language representation models (LRMs), zero-shot models, multimodal models, and fine-tuned or domain-specific models. Specific examples of LLM models include Pathways Language Model (PaLM), XLNet, and Large Language Model Meta AI (LLaMA).

LRMs are configured and/or designed to understand and generate human language, which may include natural language text. Specifically, LRMs are pre-trained on massive text corpora and can be fine-tuned for specific tasks like text classification and language generation. Examples of LRMs include Generative Pre-trained Transformers (GPTs) (e.g., GPT-3.5, GPT-4, each of which may implement a chatbot such as ChatGPT), Bidirectional Encoder Representations from Transformers (BERT) models, and Robustly Optimized BERT Approach (ROBERTA) models.

Zero-shot models are configured and/or designed to perform tasks without specific training data. Specifically, zero-shot models can generalize and output predictions or natural language text for tasks they have not seen before. An example of a zero-shot model is GPT-3.

Multimodal models are configured and/or designed to generate outputs based on inputs of one or more various types, such as text and image data. Specifically, multimodal models are designed to understand and generate content across different modalities. An example of a multimodal model is Contrastive Language-Image Pretraining (CLIP), which learns visual concepts from natural language descriptions.

Fine-tuned or domain-specific models are LLMs that have undergone additional training on domain-specific data to improve their performance in particular areas or with particular tasks like text classification and language generation for medical-related writing. For example, a GPT-3 model could be fine-tuned on clinical reports to create a domain-specific medical writing chatbot or assist with generating clinical reports.

Returning to the illustrated example of FIG. 8A, ML NLG engine 816 is at least one LLM. Alternatively, ML NLG engine 816 may be a different type of ML model. By way of example, ML NLG engine 816 can be implemented by one or more LLMs trained to understand and, in some instances, generate natural language text as second NLT segment 820. In the illustrated example, ML NLG engine 816 executes one or more LLMs locally on NLG system 101a. For example, ML NLG engine 816 can execute an instance of an LLM, using NLG system 101a, using ML NLG engine input 812 as input to the LLM to generate second NLT segment 820 as output from the LLM. Additionally or alternatively, ML NLG engine 816 may execute call(s) to a third-party (e.g., computer server(s), public and/or private cloud network(s)) that host LLM(s). In such an example, ML NLG engine 816 can make call(s) to an application programming interface (API) of the third-party by providing ML NLG engine input 812, or portion(s) thereof, to the third-party via the API. For example, portion(s) of ML NLG engine input 812 can be passed as parameters to the third-party via the API. Furthering the example, the third-party can execute LLM(s) hosted by the third-party using ML NLG engine input 812 as input to generate output. Examples of output from the third-party LLM(s) include natural language text, audio, image, and/or video. For example, output from the third-party LLM(s) can be second NLT segment 820 or natural language text data that may be used to generate and/or form second NLT segment 820. Responsive to generating the output, the third-party may cause transmission of the output to NLG system 101a via the API and over one or more networks (e.g., network 828).

In some embodiments, ML NLG engine 816 may generate new combinations of natural language text in the form of natural-sounding language. In some embodiments, ML NLG engine 816 may generate other types of output such as new audio, images, video, and/or any combination(s) thereof. In some embodiments, ML NLG engine 816 may be trained to generate text in response to an input textual prompt, which may be generated using ML NLG text segment configuration 804. For example, ML NLG engine 816 can be implemented as least in part by a chatbot and/or a co-pilot that is trained to generate natural language text responsive to text input, which may be a prompt based on ML NLG text segment configuration 804. In some embodiments, the prompt may be configured for use with an LLM while, in other embodiments, the prompt may be configured for a different type of ML model. In some embodiments, ML NLG engine 816 may be configured to generate an image based on an input textual prompt. In some embodiments, ML NLG engine 816 can be implemented at least in part by a text-to-image model that is trained to generate digital images in response to text input, which may be a prompt based on ML NLG text segment configuration 804.

In example operation, NLG application 102a and/or, more generally, NLG system 101a, generates an electronic document 822 containing natural language text 103a using a plurality of text segment configurations including symbolic NLG text segment configuration 802 and ML NLG text segment configuration 804. In example operation, NLG application 102a and/or, more generally, NLG system 101a generates a first portion 824 of the electronic document 822 by using symbolic NLG text segment configuration 802 and symbolic NLG engine 814 to generate first NLT segment 818 to include in the first portion 824 of the document 822. In example operation, generating the first portion 824 includes obtaining symbolic NLG text segment configuration 802 specifying a first set of one or more data variables (e.g., data variables 154 of FIGS. 1B and/or 8B); obtaining values of at least some of the first set of data variables using first data obtained from business data store(s) 112; and generating first NLT segment 818 using the symbolic NLG text segment configuration 802, the values of the at least some of the first set of one or more data variables, and symbolic NLG engine 814.

In example operation, NLG application 102a and/or, more generally, NLG system 101a generates a second portion 826 of the electronic document 822 by using ML NLG text segment configuration 804 and ML NLG engine 816 to generate second NLT segment 820 to include in the second portion 826 of the document 822. In example operation, generating the second portion 826 includes obtaining initial text segment 808; obtaining ML NLG text segment configuration 804 specifying information to use for generating second NLT segment 820 using ML NLG engine 816 from the initial text segment 808; and generating second NLT segment 820 by using the initial text segment 808, ML NLG text segment configuration 804, and ML NLG engine 816. In example operation, NLG application 102a and/or, more generally, NLG system 101a outputs the generated electronic document 822 to publishing system 104 for access by software application program 106 and/or computing device(s) communicatively coupled to network 828. The network 828 shown is a communication network such as the Internet or any other suitable network, as aspects of the technology described herein are not limited in this respect.

In some embodiments, the initial text segment 808 can be obtained from an existing document, file, etc. Additionally or alternatively, the initial text segment 808 may be obtained from the business data store(s) 112. In some embodiments, the initial text segment 808 can be cut and pasted into a graphical user interface (GUI), as described in connection with one(s) of FIGS. 9A-11G. In some embodiments, the initial text segment 808 can be symbolic generated text. For example, the initial text segment 808 can be output from symbolic NLG engine 814. In such an example, symbolic NLG engine 814 is shown in FIG. 8A to provide output to ML NLG engine 816 (represented by an arrow with a dotted line).

Advantageously, the hybrid NLG system illustrated in the example of FIG. 8A achieves accuracy and reliability by generating the first NLT segment 818 using the symbolic NLG engine 814 and generating the second NLT segment 828 using the ML NLG engine 816. For example, the technical limitation of LLMs potentially producing unreliable and/or inaccurate outputs is overcome by generating natural language text using the symbolic NLG engine 814. Furthering the example, the technical limitation of symbolic NLG techniques having limited scalability is overcome by performing transformations on the generated natural language text using the ML NLG engine 816 without the need for humans creating rules to perform such transformations.

In some embodiments, the hybrid NLG system illustrated in the example of FIG. 8A achieves model output traceability by tracing outputs of the symbolic NLG engine 814 to rule(s) implemented by the symbolic NLG engine 814. For example, the technical limitation of LLMs potentially “hallucinating”, such as producing unreliable and/or inaccurate outputs, is overcome by generating natural language text using the symbolic NLG engine 814 and thereby enabling users to trace output(s) of the symbolic NLG engine 814 to specific rule(s).

In some embodiments, NLG application 102a determines which of ML NLG engine 816 and/or symbolic NLG engine 814 to execute to output NLT 103a. For example, text segment configuration integration layer 126a can obtain engine selector text segment configuration 806. Engine selector text segment configuration 806 can include data and/or information specifying whether ML NLG engine 816 or symbolic NLG engine 814 be used in the generation of the electronic document 822. Examples of data and/or information include an indication of a type of text user 108 intends to create and a type of transformation to perform. By way of example, text segment configuration integration layer 126a can determine, using data in engine selector text segment configuration 806, whether to generate first portion 824 and/or second portion 826 by using ML NLG engine 816 and/or symbolic NLG engine 814. Furthering the example, when it is determined to use ML NLG engine 816 to generate first portion 824 of electronic document 822, NLG software tools 128a generates first portion 824 of electronic document 822 using ML NLG engine 816. For example, data in the symbolic NLG text segment configuration 802 can include a parameter indicating that either ML NLG engine 816 or symbolic NLG engine 814 be used in generation of the electronic document 822. In such an example, the parameter can indicate a type of text user 108 intends to create or a type of transformation to perform on text. Furthering the example, text segment configuration integration layer 126a can determine that there is an insufficient number of semantic objects 150, symbolic NLG text segment configuration 802, etc., available for use for the type of text user 108 intends to create. Accordingly, text segment configuration integration layer 126a can cause NLG software tools 128a to use ML NLG engine 816. In another example, text segment configuration integration layer 126a can determine to use ML NLG engine 816 to generate one(s) of portions 824, 826 of electronic document 822 when the parameter indicates a type of transformation is to be performed on text, such as a transformation to a grammatical aspect, content, tone, and/or style of the initial text. Non-limiting examples of grammatical aspects include changing the tense of text (e.g., from present tense to past or future tense, from past tense to present or future tense, from future tense to present or past tense, etc.), changing the voice (e.g., from first voice to second voice or third voice, active voice to passive voice or vice versa, etc.), adding or removing possessives, etc. Non-limiting examples of content transformations of text include generating a summary of the text. Non-limiting examples of transforming tone include transforming the text to have any one of the following example tones (or other tones): formal (e.g., as may be used in professional or academic context), informal (e.g., conversational), pessimistic, optimistic, persuasive, tense, curious, aggressive, uplifting, assertive, informative, entertaining, sarcastic, humorous, argumentative, and cooperative. Non-limiting examples of style include expository, narrative, descriptive, persuasive, creative, technical, academic, poetic, review/summary, objective, subjective, business, and journalistic.

Alternatively, when it is determined to use symbolic NLG engine 814 to generate first portion 824 of electronic document 822, NLG software tools 128a generates first portion 824 of electronic document 822 using symbolic NLG engine 814. For example, data in engine selector text segment configuration 806 can include data and/or information specifying one or more values to be accessed from the business data store(s) 112. In such an example, text segment configuration integration layer 126a can determine to use symbolic NLG engine 814 when the information and/or data in engine selector text segment configuration 806 includes information specifying one or more values to be accessed from the business data store(s) 112.

In some embodiments, a user (e.g., user 108, administrator 110) may configure NLG application 102a to perform a particular NLG task by configuring the components 120, 802, 124, 804, 806, 808. For example, the user may provide the NLG application 102a with one or more document macro structure(s) 120, symbolic NLG text segment configuration 802, ML NLG text segment configuration 804, engine selector text segment configuration 806, initial text segment 808 (e.g., by providing one or more configuration files, for instance specified using a mark-up language, that specify the document macro structure(s) 120, symbolic NLG text segment configuration 802, ML NLG text segment configuration 80, engine selector text segment configuration 806, and/or initial text segment 808). Additionally, when the NLG task involves generating NLT containing business data, the user may configure the NLG application 102a to obtain the business data needed from the business data store(s) 112 as described herein.

Advantageously, configuring NLG application 102a to perform a particular NLG task by configuring the components 120, 802, 124, 804, 806, 808 enables a user to develop the NLG system 101a for a plurality of NLG tasks (e.g., 10 tasks, 100 tasks, 1000 tasks, etc.). For example, the underlying NLG system 101a, such as the text segment configuration layer 126a and the NLG software tools 128a, can be used for a plurality of tasks through the use of a different component configuration, such as the symbolic NLG text segment configuration 802. In such an example, a user can quickly deploy the NLG system 101a for a specific NLG task through the use of a different symbolic NLG text segment configuration 802.

In some embodiments, the text segment configuration integration layer 126a may be configured to employ NLG software tools 128a to generate natural language text using data contained in the document macro structure(s) 120, symbolic NLG text segment configuration 802, ML NLG text segment configuration 804, engine selector text segment configuration 806, and/or data obtained using the database interface module 124. The techniques for how the text segment configuration integration layer 126a achieves this result are described herein including with reference to FIGS. 4A, 6 and numerous other examples.

In some embodiments, the NLG software tools 128a may include one or more software libraries, one or more application programming interfaces, one or more software programs, and/or any other suitable software tools used to facilitate generation of natural language text from non-linguistic data as described herein. For example, natural language text may be generated from non-linguistic data (e.g., data contained in document macro structure(s) 120, symbolic NLG text segment configuration 802, ML NLG text segment configuration 804, engine selector text segment configuration 806, and/or data obtained using database interface module 124) using multiple stages of processing including, by way of example and not limitation: (1) an analysis stage; (2) a document planning stage; (3) a micro-planning stage; and a (4) surface realization stage as described herein. Accordingly, in some embodiments, the NLG software tools 128a may include software for performing the processing for the processing of these stages. The NLG software tools 128a may include software to perform one specific type of stage and/or software to perform multiple stages.

For example, as shown in FIG. 8B, during the document planning stage, the NLG system 101a may determine to generate a natural language segment using symbolic NLG text segment configuration 802 having attributes 152, data variables 154, vocabulary 156, and document plan configuration 158 of FIG. 1B. Symbolic NLG text segment configuration 802 and ML NLG text segment configuration 804 of FIG. 8A are shown in FIG. 8B to be stored in text segment configuration data store 830. In some embodiments, text segment configuration data store 830 can correspond to and/or be implemented at least in part by semantic object data store 140 of FIG. 1B. The segment may be generated to have content corresponding to attributes 152 and data variables 154 of the symbolic NLG text segment configuration 802. The vocabulary 156 may be used to render the attributes 152 into natural language text. The business data 113 may be used to specify values for the data variables 154 (optionally, with further analysis being performed on the values). The order in which the content corresponding to attributes 152 and data variables 154 is presented in the generated natural language segment may be specified by document plan configuration 158. Further examples of this are described herein including with reference to FIGS. 2A-2F, 3A-3F, and 4A-4J.

Also shown in the example of FIG. 8B is ML NLG text segment configuration 804, which is stored in 830. ML NLG text segment configuration 804 shown in FIG. 8B includes prompt template 832 and transformation instructions 834. For example, prompt template 832 includes information that may be used as input by ML NLG engine 816 to generate second NLT segment 820. In such an example, prompt template 832 can represent an ML model prompt, which may include natural language text, image(s), audio, and/or video. In some embodiments, prompt template 832 represents a prompt or information that can be used to generate a prompt that is configured for input to an LLM or a different type of ML model. Transformation instructions 834 of FIG. 8B includes information that may be used as input by ML NLG engine 816 to execute transformation(s) on natural language text. For example, transformation instructions 834 can include information specifying at least one transformation to be made by ML NLG engine 816 to a grammatical aspect, tone, and/or style of initial text segment 808.

Further shown in the example of FIG. 8B is symbolic NLG text segment configuration access module 836 and ML NLG text segment access module 838. In the illustrated example, symbolic NLG text segment configuration access module 836 is configured to access and/or obtain symbolic NLG text segment configuration 802 from text segment configuration data store 830. In some embodiments, symbolic NLG text segment configuration access module 836 can access and/or obtain symbolic NLG text segment configuration 802 via one or more communication networks. In the illustrated example, ML NLG text segment access module 838 is configured to access and/or obtain ML NLG text segment configuration 804 from text segment configuration data store 830. In some embodiments, ML NLG text segment access module 838 can access and/or obtain ML NLG text segment configuration 804 via one or more communication networks.

FIGS. 9A-9J show example graphical user interfaces (GUIs) representing example workflows for generating and/or transforming natural language text. A first GUI 900 shown in FIG. 9A represents a software user interface that can receive user input to configure an intention. For example, an “intention” as shown in FIGS. 9A-9J represents an intention to describe what a user would like to convey in natural language text. In some embodiments, an “intention” as shown in FIGS. 9A-9J can correspond to and/or be implemented by symbolic NLG text segment configuration 802, ML NLG text segment configuration 804, and/or engine selector text segment configuration 806. For example, a user can select an intention corresponding to symbolic NLG text segment configuration 802 from a down-down menu.

As shown in a second GUI 910 in FIG. 9B, examples of intentions (e.g., symbolic NLG text segment configuration 802, ML NLG text segment configuration 804, engine selector text segment configuration 806) include value-based intentions, reference-based intentions, descriptive statistical analysis, and variation-based intentions.

In the illustrated example of FIG. 9C, a third GUI 920 is generated responsive to selecting a table intention, which represents that a user would like to convey information to be presented in a table format. By selecting a table intention, the third GUI 920 is generated to receive user input such as a dataset to be used to generate the table and a title for the table (i.e., table title).

In the illustrated example of FIG. 9D, a fourth GUI 930 is generated responsive to selecting a dataset for creating the table. By selecting a dataset, the fourth GUI 930 is generated to receive user input configuring table creation.

In the illustrated example of FIG. 9E, a fifth GUI 940 is generated responsive to receiving user input to create the table according to the provided configuration(s). The fifth GUI 940 includes a table 942, which is generated by symbolic NLG engine 814. For example, configuration(s) shown in FIGS. 9A, 9B, 9C, 9D, and/or 9E can correspond to at least portion(s) of symbolic NLG text segment configuration 802 that, when ingested by symbolic NLG engine 814, outputs table 942.

In the illustrated example of FIG. 9F, a sixth GUI 950 is generated responsive to receiving user input to generate natural language text to accompany and/or correspond to the table 942.

In the illustrated example of FIG. 9G, a seventh GUI 960 is generated responsive to selecting a dataset for use in generating the natural language text. By selecting a dataset, the seventh GUI 960 is generated to receive user input configuring natural language text creation.

In the illustrated example of FIG. 9H, an eighth GUI 970 is generated responsive to receiving user input to generate the natural language text according to the provided configuration(s). The generated natural language text is shown in FIG. 9H as natural language text 972. In the illustrated example, natural language text 972 is generated by symbolic NLG engine 814 using information in the selected dataset. Alternatively, natural language text 972 may be generated by ML NLG engine 816.

In the illustrated example of FIG. 9H, the eighth GUI 970 has a text display portion 974 configured to display textual content, which is shown in FIG. 9H as the natural language text 972. Also shown in the eighth GUI 970 is an NLG portion 976, the NLG portion comprising a plurality of GUI elements 978 that enable a user to specify one or more parameters for generating natural language text, the one or more parameters including an NLG engine parameter specifying whether to use the symbolic NLG engine or the ML NLG engine to generate text. In example operation, the NLG system 101a receives, via the NLG portion 976, values for the one or more parameters for generating natural language text; generating the natural language text using, the NLG system 101a and the NLG engine parameter; and inserting the generated natural language text 972 in the text display portion 974.

In the illustrated example of FIG. 9I, a ninth GUI 980 is generated responsive to receiving user input to perform one or more transformations on natural language text 972. Examples of transformations shown in FIG. 9I include summarizing, changing a tense, changing a tone (e.g., make the text fluid and concise, and remove redundancies), using active voice, creating a bulleted list, and using plain language. In some embodiments, one(s) of the transformations shown in FIG. 9I can correspond to prompt template 832 and/or transformation instructions 834 of FIG. 8B. For example, the summarizing transformation can be implemented at least in part by prompt template 832, which can include information to be used to generate a prompt of “Summarize the selected natural language text”. In another example, the summarizing transformation can be implemented at least in part by transformation instructions 834, which can include information to be used by ML NLG engine 816 to summarize natural language text 972.

In the illustrated example of FIG. 9J, a tenth GUI 990 is generated responsive to receiving user input to perform a summarize transformation on natural language text 972 to generate natural language text 992, which is a summary of natural language text 972. For example, natural language text 972 can be provided to ML NLG engine 816 as input to generate natural language text 992 as output in accordance with ML NLG text segment configuration 804.

FIGS. 10A-10H show example GUIs representing example workflows for transforming existing natural language text in a first illustrative environment. The first illustrative environment is a standalone software application. The standalone software application may be executed as a desktop software application or as executable in a web browser.

In the illustrated example of FIG. 10A, a first GUI 1000 is shown to receive user input to perform transformation(s) on existing natural language text such as by reusing existent content for natural language generation. For example, symbolic NLG engine 814 can generate table 1002 and a user may desire to generate natural language text to accompany and/or correspond to table 1002. In such an example, the user may desire to generate natural language text from existing sources.

In the illustrated example of FIG. 10B, a second GUI 1010 is shown to receive user input indicating a type of content to be reused, such as natural language text or a table, for natural language generation.

In the illustrated example of FIG. 10C, a third GUI 1020 is shown to receive user input indicating a source of a type of content selected for reuse. Examples of content sources include external documents or a current report model in progress.

In the illustrated example of FIG. 10D, a fourth GUI 1030 is shown to receive user input indicating which portion(s) (e.g., document section(s)) of a selected content source is to be used for natural language text generation.

In the illustrated example of FIG. 10E, a fifth GUI 1040 is shown to select text 1042 from selected portion(s) of a content source to be used for natural language text generation. In the illustrated example of FIG. 10F, a sixth GUI 1050 is shown responsive to selecting text from the selected portion(s) of a content source to be used for natural language text generation.

In the illustrated example of FIG. 10G, a seventh GUI 1060 is shown responsive to receiving user input to perform one or more transformations on the selected text 1042. In the shown example, user input is to provide a past tense transformation, such as by changing a tense of the selected text 1042 into past tense.

In the illustrated example of FIG. 10H, an eighth GUI 1070 is shown responsive to receiving user input to perform a past tense transformation on the selected text 1042 to generate natural language text 1072, which is a transformation of the selected text 1042. For example, the selected text 1042 can be provided to ML NLG engine 816 as input to generate natural language text 1072 as output in accordance with ML NLG text segment configuration 804.

FIGS. 11A-11G show example GUIs representing example workflows for transforming existing natural language text in a second illustrative environment. The second illustrative environment is a software application that may execute in association with another software application. For example, the second illustrative environment can be a plug-in to a word processing software application or a web browser.

In the illustrated example of FIG. 11A, a first GUI 1100 is shown and configured to receive user input to identify a source from which natural language text is to be generated. In FIG. 11B, a second GUI 1110 is shown responsive to a selection of an external document. The second GUI 1110 shows a portion of natural language text from the external document. In FIG. 11C, a third GUI 1120 is shown responsive to receiving user input to perform one or more transformations on the portion of natural language text from the external document. In the shown example of FIG. 11C, user input is to provide a past tense transformation, such as by changing a tense of the portion of natural language text from the external document into past tense.

In the illustrated example of FIG. 11D, a fourth GUI 1130 is shown representing a first software application (e.g., a word processing software application) and a fifth GUI 1140 is shown representing a second software application (e.g., a plug-in). The fifth GUI 1140 shows source content to be transformed into past tense. The fourth GUI 1130 shows the result of the transformation (e.g., a tense of the source content transformed into past tense). A sixth GUI 1150 is shown in FIG. 11E representing the result of the transformation shown in the fourth GUI 1130.

In the illustrated example of FIG. 11F, a seventh GUI 1160 is shown responsive to receiving user input to perform one or more transformations on the result of the transformation. In the seventh GUI 1160, user input is to provide a summarize transformation, such as by summarizing the natural language text selected in the fourth GUI 1130. An eighth GUI 1170 is shown in FIG. 11G responsive to user input directing natural language text to be generated by performing a summarizing transformation on the selected text of the fourth GUI 1130. The result of the summarizing transformation is shown in a ninth GUI 1180 in FIG. 11H. For example, the output in the ninth GUI 1180 represents a summary of the selected text of the fourth GUI 1130. In such an example, the selected text of the fourth GUI 1130 can be provided to the ML NLG engine 816 as input to generate the summary as output in accordance with the ML NLG text segment configuration 804. In example operation, the generated summary shown in the ninth GUI 1180 can be inserted into an existing document, as shown in a tenth GUI 1190 illustrated in FIG. 11I.

FIG. 12 is a flowchart of an illustrative process 1200 for generating natural language text using a hybrid NLG system, in accordance with some embodiments of the technology described herein. The process 1200 may be performed by any suitable natural language generation system and, for example, may be performed by NLG system 101a described with reference to FIGS. 8A and/or 8B.

The process 1200 begins at act 1202, at which NLG system 101a may obtain a first text segment configuration specifying a first set of data variable(s). For example, NLG application 102a can obtain symbolic NLG text segment configuration 802.

At act 1204, NLG system 101a may obtain values of at least some of the first set of data variable(s) using first data obtained from data store(s). For example, database interface module 124 can obtain values of at least some of the data variables 154 from the business data store(s) 112.

At act 1206, NLG system 101a may generate a first natural language text segment using the first text segment configuration, the values, and a symbolic NLG engine. For example, NLG application 102a can generate first NLT segment 818 using symbolic NLG text segment configuration 802, the values of the at least some of the data variables 154, and symbolic NLG engine 814.

At act 1208, NLG system 101a may generate a first portion of an electronic document by using the first natural language text segment. For example, NLG application 102a can generate first portion 824 of electronic document 822 by using first NLT segment 818.

At act 1210, NLG system 101a may obtain an initial text segment. For example, NLG application 102a can obtain initial text segment 808. In such an example, initial text segment 808 can be provided by a user (e.g., administrator 110, user 108) from an external document or a document in progress. In another example, initial text segment 808 can be output from symbolic NLG engine 814.

At act 1212, NLG system 101a may obtain a second text segment configuration specifying information to use for generating a second natural language text segment using an ML NLG engine from the initial text segment. For example, NLG application 102a can obtain ML NLG text segment configuration 804 specifying information to use for generating second NLT segment 820 using ML NLG engine 816 from initial text segment 808.

At act 1214, NLG system 101a may generate a second natural language text segment using the initial text segment, the second text segment configuration, and the ML NLG engine. For example, NLG application 102a generate second NLT segment 820 using initial text segment 808, ML NLG text segment configuration 804, and ML NLG engine 816. At act 1216, NLG system 101a may generate a second portion of the electronic document by using the second natural language text segment. For example, NLG application 102a can generate second portion 826 by using second NLT segment 820. At act 1218, NLG system 101a may output the generated electronic document. For example, NLG application 102a may output electronic document 822 to publishing system 104.

Further Implementation Detail

The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of processor-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as described herein. Additionally, in some embodiments, one or more computer programs that when executed perform methods of the disclosure provided herein need not reside on a single computer or processor, but may be distributed in a modular fashion among different computers or processors to implement various aspects of the disclosure provided herein.

Also, various inventive concepts may be embodied as one or more processes, of which examples have been provided including with reference to FIGS. 4A and 6. The acts performed as part of each process may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

All definitions, as defined and used herein, should be understood to control over dictionary definitions, and/or ordinary meanings of the defined terms.

Having described several embodiments of the techniques described herein in detail, various modifications, and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the disclosure. Accordingly, the foregoing description is by way of example only, and is not intended as limiting. The techniques are limited only as defined by the following claims and the equivalents thereto.