System and method of document generation

A document generation system to generate a document that forms the basis for a patent application to be submitted for examination by a patent searching authority. The document generation system may include: 1) a model generator trained on a plurality of references obtained from a reference database and 2) a document generator configured to generate paragraphs of a document based on a trained model.

FIELD OF INVENTION

The present application relates to a system and method for document generation, and more particularly toward generating a document based on a plurality of seed sentences.

BACKGROUND

The conventional manner in which a patent application is drafted most often involves obtaining an invention disclosure from an inventor and manually synthesizing this information into a specification, claims, and drawings. This process is laborious, often expensive, and can vary widely depending on the complexity of the disclosure materials.

Various efforts have been made to increase efficiency with respect to the drafting process. One common approach is to provide written forms for both a patent draftsman and the inventor. The written form or invention disclosure form provided to the inventor may be arranged to extract information that the inventor may not have considered, including abstractions of a specific invention to a more general description of the invention. Such a form may also provide a roadmap of sorts for the draftsman to arrange the specification and claims. Even with the help of a disclosure form, the conventional approach to drafting a patent application ultimately involves a significant amount of manual synthesize of the disclosure information by the patent draftsman and significant manual effort to author the specification and claims.

Another type of form often utilized in the drafting process is a patent template that includes several pre-written sections and headings to facilitate efficiency in drafting the application. However, similar to the shortcomings of the invention disclosure form, the actual drafting process involves significant manual effort to author the specification and claims.

SUMMARY OF THE DESCRIPTION

A system and method are provided for generating a document based on one or more seed documents, which may include text arranged according to a tree structure. A document generation system according to one embodiment may generate a document that forms the basis for a patent application to be submitted for examination by a patent searching authority. The document generation system may include: 1) a model generator for one or more models trained on a plurality of references obtained from a reference database and 2) a document generator configured to generate paragraphs of a document based on the one or more trained models.

In one embodiment, there may be a method of generating a document that forms a basis for a patent specification to be submitted for examination by a patent searching authority. The method may include providing a plurality of input statements each defining a statement group of one or more tokens, and vectorizing each of the one or more tokens to generate one or more token vectors such that each token is represented by a vector within a vector space. The method may also include generating a plurality of document tokens based on the one or more token vectors respectively representative of the one or more tokens, the plurality of document tokens forming the document to be submitted for examination by the patent searching authority.

In another embodiment, a document generation system may be provided for generating a document to be submitted for examination by a patent searching authority. The system may include a memory and a controller. The memory may store a vector space translator and one or more sequence generation models, where the vector space translator includes vector information pertaining to a vector space for tokens. The controller may be configured to receive a plurality of input statements, and programmed or configured to tokenize the plurality of input statements into one or more tokens based on content of the plurality of input statements. The controller may be configured or programmed to vectorize, based on the vector information from the vector space translator stored in the memory, each of the one or more tokens to yield a token vector within the vector space for each of the one or more tokens. The controller may be configured to arrange a sequence of vectors for each of the plurality of input statements based on the token vector for each of the one or more tokens, and to feed the sequence of vectors to the one or more sequence generation models to generate one or more output vectors. The one or more output vectors may be translated, based on the vector information, to one or more output tokens that together form the document to be submitted for examination.

In yet another embodiment, a system may be provided for generating a document to be submitted for examination by a patent searching authority, where the system includes a seed statement receiver, a vector translator, a content fragmenter, a sequence generator, and a document compiler. The seed statement receiver may be configured to receive an incoming statement with one or more tokens. The vector translator may be configured to vectorize the one or more tokens to provide one or more token vectors, and the content fragmenter may be configured to generate a plurality of seed fragments from the incoming statement. Each of the plurality of seed fragments may include at least one of the one or more token vectors.

The sequence generator may be configured to generate an output vector based on one or more input vectors. The document compiler may be configured to provide the at least one token vector from each of the plurality of seed fragments to the sequence generator and to aggregate the output vector from the sequence generator for each of the plurality of seed fragments to form an aggregate sequence, whereby the aggregate sequence forms the basis for the document to be examined.

These and other objectives, advantages, and features of the invention will be more fully understood and appreciated by reference to the description of the current embodiments and the drawings.

Before the embodiments of the invention are explained in detail, it is to be understood that the invention is not limited to the details of operation or to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings.

DESCRIPTION

A document generation system according to one embodiment is shown inFIG. 1and generally designated200. The document generation system200may be configured to generate a document that forms the basis for a patent application to be submitted for examination by a patent searching authority. The document generation system200in the illustrated embodiment is divided into two principle components: 1) a model generator220and 2) a document generator210. These two components may be implemented as separate systems—although it should be understood that these two components may be implanted on the same system, or one or more aspects of one component on one system may be utilized in the other component on another system.

In the illustrated embodiment, the model generator220includes a processor222, memory224, and input/output interface226. The processor222may be a central processing unit (CPU), such as an Intel Core i7 Processor, with a plurality of cores (physical or logical, or a combination thereof) configured to process a plurality of threads or processes simultaneously. Memory224may be shared on an integrated circuit associated with the processor222, or may be separate from the integrated circuit associated with the processor222and in communication with the processor222via the input/output interface226, or a combination thereof. Optionally, the model generator220may include a graphics processing unit (GPU)228with many more cores than the processor222(e.g., hundreds or thousands more) to enable parallel processing of training parameters for one or more models initialized and trained by the model generator220. In this way, the GPU228may simultaneously process significantly more threads or processes than the processor222in training the one or more models. Use of the GPU228in this manner may significantly decrease the amount of training time associated with generating the one or more models.

The model generator220includes any and all electrical circuitry and components to carry out the functions and algorithms described herein. Generally speaking, the model generator220may be a controller with one or more microcontrollers, microprocessors, and/or other programmable electronics that are programmed to carry out the functions described herein. The controller may additionally or alternatively include other electronic components that are programmed to carry out the functions described herein, or that support the microcontrollers, microprocessors, and/or other electronics. The other electronic components include, but are not limited to, one or more field programmable gate arrays, systems on a chip, volatile or nonvolatile memory, discrete circuitry, integrated circuits, application specific integrated circuits (ASICs) and/or other hardware, software, or firmware. Such components can be physically configured in any suitable manner, such as by mounting them to one or more circuit boards, or arranging them in other manners, whether combined into a single unit or distributed across multiple units. Such components may be physically distributed in different positions in the model generator220, or they may reside in a common location within the model generator220. When physically distributed, the components may communicate using any suitable serial or parallel communication protocol, such as, but not limited to, CAN, LIN, FireWire, I2C, RS-232, RS-485, and Universal Serial Bus (USB).

As described herein, the terms module, model, and generator designate parts of the controller. For instance, a model in one embodiment is described as having one or more core functions and one or more parameters that affect output of the one or more core functions. Aspects of the model may be stored in memory of the controller, and may also form part of the controller configuration such that the model is part of the controller that is configured to operate to receive and translate one or more inputs and to output one or more outputs. Likewise, a module or a generator are parts of the controller such that the controller is configured to receive an input described in conjunction with a module or generator and provide an output corresponding to an algorithm associated with the module or generator.

The document generator210may be configured similar to the model generator220, and may include a processor222, memory224, and an input/output interface226. The document generator210may optionally include a GPU228. The document generator210may obtain, from model storage250, one or more models to generate an output from an input sequence. The one or more models stored in the model storage250may be pre-trained by the model generator220such that the more intensive processing associated with training is not conducted by the document generator210. As a result, it is possible to configure the document generator210with less processing power than the model generator220. For instance, the document generator210may not include the GPU228whereas the model generator220may include the GPU228. It should be understood, however, that the document generator210may include and utilize the GPU228to enhance performance speed in generating a document based on the one or more models stored in model storage250.

With a model generator220configured separately from the document generator210, one or more models may be generated ahead of time with the model generator220, and then stored in the model storage250for later use with the document generator210. The degree of processing power utilized in training the one or more models is greater than in generating an output from the one or more models after training. As a result, the model generator220may be configured to handle more processor intensive calculations associated with training the one or more models, leaving the less intensive calculations for the document generator210. This way, the document generator210may more closely resemble a consumer level configuration without the enhanced processing capabilities utilized for the model generator220. It should be understood however there is no requirement that the document generator210be less capable than the model generator220. The document generator210may be substantially the same or more capable than the model generator220.

The model storage250of the document generation system200in the illustrated embodiment may enable storage of the one or more models output by the model generator220. Storage of a model may facilitate use of the one or more models at a later time without the need to re-initialize and re-train the model. In one embodiment, the model is based on a neural network having a model topology with a plurality of neural net nodes with weighted units. The weights of the weighted units may be initialized and trained in a variety of ways. As an example, the weights may be trained by initializing the weights, providing the model with an input and a desired output, comparing an output of the model to the desired output, and adjusting the weights to yield an output more closely resembling the desired output. Storage of the model in the model storage250may entail saving the model topology and the weights so that the trained model can be reproduced from the model storage250. Additionally, or alternatively, a compiled version of the model may be stored in the model storage250.

The model storage250may be non-volatile memory enabling transfer of the model storage250among different systems, such as from the model generator220to the document generator210. In this way, one or more operational aspects of the document generation system200may be separated in space and time. In one embodiment, the model storage250may be provided as part of a software installation that includes operational aspects of the document generator210without one or more operational aspects of the model generator220. It should be understood, however, that the document generation system200may be provided as a software installation with both the document generator210and the model generator220.

In the illustrated embodiment, the document generation system200may include a vector database230. The vector database230may be configured to translate a token to a vector in vector space. For example, a token such as “upper” may be translated to a200dimension vector. The dimensionality of the vectors output from the vector database230may vary from application to application, depending on the desired quality of the vector database230. The vector database230may be generated from a large corpus of tokens with each token being assigned a vector in vector space. Examples of vector space models or databases are based on the word2vec algorithm used to produce word embeddings and the GloVe algorithm to do the same. Additional example implementations of word2vec are described in further detail in Efficient Estimation of Word Representations in Vector Space, submitted to Cornell University Library on Sep. 7, 2013, by Mikolov et al.—the disclosure of which is incorporated by reference herein in its entirety. Additional example implementations of GloVe are described in further detail in GloVe: Global Vectors for Word Representation, by Pennington et el.—the disclosure of which is incorporated by reference herein in its entirety. The vector space model may be generated in an unsupervised manner (e.g., providing an input without a target output) so that a corpus of tokens, possibly significantly large, can be processed to form the vector space translations of tokens. Examples of significantly large corpuses include the entirety of available Wikipedia articles or all or a portion of references available from one or more patent authorities, such as the database of references maintained by the U.S. Patent and Trademark Office (USPTO) including issued patents and patent application publications. The models and the vector database230in one embodiment may be dynamic, so that as additional technological advancements are made and token associations are created, the vector database230and models can adapt accordingly.

An example of token vectorization can be seen in the illustrated embodiment ofFIG. 22. One aspect of building a vector space of tokens is that the tokens having similar meanings tend to group together within the vector space. For instance, as shown in the illustrated embodiment ofFIG. 22, the terms “fastener” and “screw” are proximal to each other in the example 2-D vector space, and not proximal to the terms “plurality” and “more”. The illustrated embodiment provides an example of 2-D space, but it should be understood that the dimensionality can be increased, such as to 100 dimensions or greater including greater than 1,000 dimensions.

The document generation system200may be communicatively coupled to a database of references or reference database240. The reference database240may be similar to the corpus from which the vector database230is based. For instance, the reference database240may include all or a portion of references available from one or more patent authorities, including the database of references available from the USPTO. Additional or alternative patent authorities may be sourced for the reference database240, such as the European Patent Office, the World Intellectual Property Office and the Japanese Patent Office. The reference database240is not limited to a single language, such as English. Likewise, the vector database230is not limited to a single language. The reference database240may be a source of information for each reference, including specification text and claim text. The specification text and claim text may be tagged or identified as such, respectively, for all or a majority of the references available from the reference database240. Additional information with respect to each reference may be provided by the reference database240, such as bibliographic information, status of an application associated with the reference, and class or technology classification information, or any combination thereof. The reference database240may be stored locally with respect to one or more aspects of the document generation system200—e.g., the reference database240may be stored in memory224of the model generator220.

In one embodiment, models may be trained for specific classes of references, such as a type of technology class associated with a plurality of references. This way, a document generator210may be provided for types of technology, such as biomedical or cloud data computing.

The claim text of each reference may define a plurality of statements arranged in a tree structure. For instance, the plurality of statements may include one or more head statements50(e.g., independent claims) and one or more child statements60(e.g., dependent claims) associated with or dependent on at least one of the one or more head statements50and the one or more child statements60. In this way, a child statement60may depend on another child statement60, which itself depends on a head statement50such that a child statement60may be considered a grandchild or further descendent of at least one of the one or more head statements50. This tree structure of statements results in a child statement60incorporating all of the content of its lineage to a head statement50. In some cases, a child statement60may depend on more than one statement, such as the one or more head statements50or the one or more child statements60, or a combination thereof. The preamble of a head statement50or a preamble fragment110may define the general subject matter of the head statement50as well as any child thereof. The preamble of a child statement60may identify the dependency of the child statement60, defining an association within the tree structure of the claim text.

In one embodiment of the present disclosure, as discussed in further detail herein, the claim text can be grouped or classified in a variety of ways. For instance, the text of a head statement50may include a plurality of noun phrases100or objects comprising one or more word tokens that together define an aspect or feature of the head statement50, such as a thing, quality or action. Not all tokens of the claim text may define a noun phrase—some tokens may function differently, such as by linking two noun phrases100or functioning as a verb phrase.

All or portions of the head statement50may be grouped into a plurality of fragments—e.g., with each fragment including one or more noun phrases100. A fragment may be defined according to one or more criteria, such as tokens or noun phrases100being between delimiters (e.g., “:”, “;”, newline markers, or a combination thereof), on the same line (e.g., between newline markers), at the beginning of a line, after the beginning of the line, in the preamble of the head statement50or in a “wherein” clause, or any combination thereof. As an example, a preamble fragment110may be identified as the preamble of the head statement50that precedes the token “comprising”.

The specification text of each reference may include fragments or subsections that repeat or describe aspects of the claim text in further detail, including aspects associated with a fragment of the claim text. As an example, a paragraph of the specification text may utilize all or a subset of the noun phrases100associated with a preamble fragment110of a head statement50. In one embodiment, such a paragraph of the specification may also lack all or a subset of the noun phrases100associated with other fragments of the head statement50.

In the illustrated embodiment ofFIG. 1, the model generator220may analyze the claim text and specification text for each reference selected from the reference database240. From this analysis, the model generator220may generate a plurality of token sequence pairs, each including an input (a fragment of claim text) and an output (a fragment of specification text). The model generator220, as discussed herein, may generate more than one type of model depending on the type of claim fragment being used as an input and the one or more criteria for identifying a specification fragment to associate with the claim fragment. For instance, one model type may be configured and generated for preambles, including the preamble fragment110of a head statement50and a fragment of specification text (e.g., a paragraph) that includes all noun phrases100of the preamble fragment110but no noun phrases of the head statement50outside the preamble fragment110. Another model type may be generated for a base fragment120corresponding to the base topic noun phrases114that start lines in the head statement50but do not form part of “wherein” clauses. In one embodiment, this base fragment120may be associated with a fragment of specification text that includes all of the noun phrases100in the base fragment120but none or few noun phrases100that occur within the tokens of the base fragment120but after the base topic noun phrase114.

Using the identified inputs/outputs from the analysis of references from the reference database240, the model generator220may train one or more models to take as an input a seed fragment with no corresponding specification text and to generate specification text from the seed fragment. The one or more models may be stored in the model storage250, which can be communicated to the document generator210for use in generating a document.

In one embodiment, a seed document212may be processed into one or more seed fragments similar to the fragments utilized in training the respective types of models. The document generator210may provide the one or more seed fragments as input to a respective model obtained from the model storage250. The output provided from the one or more models, based on the input, may be compiled to form a document that provides a basis for a patent application to be examined by a patent searching authority. It should be understood that while the present disclosure focuses on generating a document as the basis for a patent application, the present disclosure is not so limited. The document generator may be configured to generate a document of any type, including, for example, a news article, legal opinion, manual, and a software requirements document based on a list of software requirement statements.

II. Tree Structure

As discussed herein, the plurality of statements utilized as a seed document212or obtained from a reference of the reference database240may define a tree structure including one or more head statements50and one or more child statements60. An example of this tree structure is shown in the illustrated embodiments ofFIGS. 3-13. It should be understood that the present disclosure may be utilized in conjunction with other types of documents arranged in a different or similar manner. For instance, the references may not define a tree structure, or may define a different type of organization, such as paragraphs arranged with a topic paragraph and subtopic paragraphs relating to aspects of the topic paragraph.

In the illustrated embodiment, it should be appreciated that the noun phrases100and associated descriptions may also form a tree structure. For instance, a noun phrase100recited in a child statement60may correspond to the same noun phrase100recited in the head statement. If the noun phrase100in the child statement leads to a statement fragment102, that statement fragment102may be considered a leaf of the noun phrase100recited in the head statement50. This type of tree structure is governed at least in part by antecedent basis rules applied in forming the head statement50and child statements60.

In the illustrated embodiment ofFIG. 3, a seed document212is depicted with various markers identifying groups of tokens or fragments. The seed document212includes a slash marker “/” that identifies fragment boundaries104and box markers to identify noun phrases100. For purposes of disclosure the slash markers and the box markers are provided throughout portions of the illustrated embodiments without reference numbers. As can be seen, there are a variety of criteria that may be utilized in identifying a fragment boundary104. For instance, the beginning of a claim statement preceding a number may correspond to a fragment boundary104, the end of a paragraph or a period, comma, or semicolon followed by a newline may correspond to a fragment boundary104. Punctuation such as a comma or semicolon without being followed by a newline may also correspond to a fragment boundary104. Additionally or alternatively, a particular type of token, such as a “wherein” or “comprising” token, may correspond to a fragment boundary104. These various criteria are used in identifying the fragment boundaries104depicted in the illustrated embodiment ofFIG. 3. It should be understood that fewer or more criteria may be utilized in identifying the fragment boundaries104.

The fragment boundaries104may define groups of tokens collectively defining a statement fragment102. The tokens within each statement fragment102may include noun phrases100that function as a subject or object of the statement fragment102. The tokens may define other phrases or grammatical functions, such as a verb phrase. In the illustrated embodiment, each of the statement fragments102is further defined by one or more noun phrases100—although, in some cases, a statement fragment102may not include any noun phrases100. Further defined aspects of the noun phrases100may be determined based on position within the statement fragment102. For instance, a noun phrase100positioned at or near the start of the statement fragment102or the first noun phrase of the statement fragment102may be defined as a topic noun phrase, and a noun phrase following the topic noun phrase may be defined as a predicate noun phrase116.

In the illustrated embodiments ofFIGS. 3-13, the position or content, or both, of a statement fragment102relative to one or more other statement fragments102may be used as a basis for categorizing the statement fragment102. The category or type of statement fragment102may form the criteria for training a model type or providing the statement fragment102as an input to the model type to generate an output.

For example, the statement fragment102may be provided as an input to a model type trained against training data from the reference database240with statement fragments102similarly arranged within a head statement50or a child statement60and associated with aspects of a specification that include the same or similar noun phrases as the statement fragment102. For instance, a paragraph or portion of the specification text that utilizes one, several or all of the noun phrases100of the statement fragment102or is similar to the base fragment120according to a similarity metric (e.g., vector similarity with respect to tokens of the base fragment120and the portion of the specification text, or one or more fuzzy comparisons of one or more tokens in the base fragment120with the tokens of the portion being indicative that the content of the base fragment120is substantially included within the portion of the specification text). Similar techniques may be utilized in determining portions of the specification text to train for statement fragments102obtained from references of the reference database240.

In the illustrated embodiment ofFIG. 4, one of the statement fragments102is shown among a plurality of statement fragments102with the other statement fragments102greyed out. The statement fragment102that is visible or active is part of a head statement50, and forms the first statement fragment defined by the start of the head statement50and the token “comprising”. This statement fragment102may be the preamble fragment110or head topic fragment.

The preamble fragment110as discussed herein may include a plurality of noun phrases100. The first noun phrase of the plurality may be identified as the head topic noun phrase112, which can provide context for all or substantially all of the tree that is defined by the head statement50and the child statements60of the head statement50. The preamble fragment110may also include one or more predicate noun phrases116subsequent to the head topic noun phrase112. The preamble fragment110may form the basis for training and generating output from a first type of model. For instance, the aspects of the preamble fragment110may provide a main topic of the document to be generated that can be provided to a model trained to generate one or more tokens that expand on the main topic.

B) Base Fragment

Turning to the illustrated embodiment ofFIGS. 6 and 7, the statement fragment102shown active or without being greyed out is positioned with a fragment boundary104following punctuation, a newline, or the end of a paragraph, or a combination thereof. This type of statement fragment102may be categorized according to one or more of these or similar criteria, such as following a newline and including a base topic noun phrase114at the beginning of the statement fragment102. Additionally, or alternatively, the criteria for identifying the type of statement fragment102depicted active inFIGS. 6 and 7may be presence of a noun phrase100at the head of the statement fragment102that has not been used in a preceding statement fragment102, or not used in a preceding statement fragment except the preamble fragment110. The statement fragment102categorized in this manner is described herein as a base fragment120.

The base fragment120may begin with a base topic noun phrase114, and include zero or more predicate noun phrases116after the base topic noun phrase114. For instance, in the illustrated embodiment ofFIG. 6, the base fragment120includes only the base topic noun phrase “a touch screen matrix display” without one or more predicate noun phrases116. The collection including the base topic noun phrase114and the zero or more predicate noun phrases116may provide context for aspects of the document to be generated that emphasize the base topic noun phrase114and possibly its relation to the zero or more predicate noun phrases116.

Several of the statement fragments102described herein are defined by fragment boundaries104. However, it should be understood that a statement fragment102is not so limited. The statement fragment102may be formed by a collection of one or more tokens of different statement fragments102defined by fragment boundaries104. An example of such a statement fragment102is depicted in the illustrated embodiment ofFIG. 5, where tokens of the statement fragment102are not greyed out.

In the illustrated embodiment ofFIG. 5, the statement fragment102is defined by a plurality of noun phrases100, including the head topic noun phrase112. As described herein, there may be a first noun phrase of a statement fragment102following a fragment boundary104, which may follow a newline, punctuation and a newline, or the end of a paragraph. This type of noun phrase100is shown in the illustrated embodiment ofFIG. 5and generally described as a base topic noun phrase, designated as114. The statement fragment102of the illustrated embodiment ofFIG. 5includes such a base topic noun phrase114. The collection of tokens that define the type of statement fragment102in the illustrated embodiment ofFIG. 5may include other types of noun phrases100or groups of one or more tokens, such as a verb phrase or one or more predicate noun phrases116.

The statement fragment102in the illustrated embodiment ofFIG. 5may form the basis for training and generating output from a second type of model as described herein. In other words, the second type of model may be trained based on a statement fragment102that is formed from all or parts of two or more other statement fragments102, and may be configured to generate output of one or more tokens based on similar input obtained from the seed document212.

In the illustrated embodiments ofFIGS. 8-11, a type of statement fragment102described as a feature fragment130is shown active or without being greyed out. The criteria that primarily identifies the feature fragment130from other types of statement fragments102is the noun phrase100positioned at the head of the feature fragment130(identified as a feature topic noun phrase132) corresponding to a noun phrase100recited in another statement fragment102.

The statement fragments102shown active in the illustrated embodiment ofFIGS. 8-11may be identified as feature fragments130primarily because they may further expand on the principle concept of the feature topic noun phrase132. This may be used as a basis for training a model type based on a plurality of feature fragments130and portions of specification text that are respectively similar to the plurality of feature fragments130according to a similarity metric. As an example, a portion of specification text may be identified based on presence of the same or similar noun phrases100as a feature fragment130. Additionally, or alternatively, absence of any noun phrases100in the portion of specification text that are not present in the feature fragment130may identify the portion of specification text as a candidate for training. In yet another similarity metric, the portion of specification text may be identified based on the absence of any noun phrases100other than the feature topic noun phrase132that are present in the statement fragment102from which the feature fragment130is associated by virtue of the feature topic noun phrase132.

Examples of a feature topic noun phrase132are outlined in the illustrated embodiments ofFIGS. 8-11. In the illustrated embodiment ofFIG. 8, the feature fragment130includes a feature topic noun phrase132that is the same as the base topic noun phrase114of the base fragment120preceding the feature fragment130.

In another example, depicted in the illustrated embodiment ofFIG. 10, the feature fragment130shown active without being greyed out includes a feature topic noun phrase132(“the third image”) that corresponds to a predicate noun phrase116of a statement fragment102preceding the feature fragment130. Likewise, the feature fragment130shown active in the illustrated embodiment ofFIG. 11includes a feature topic noun phrase132(“the information button”) that corresponds to a predicate noun phrase116of a statement fragment102prior to the feature fragment130.

The feature fragment130may be identified by presence of the feature topic noun phrase132, or additional or alternative criteria such as the statement fragment102being positioned with a fragment boundary104that may not follow a newline or follows a newline but includes a token “wherein” or other identifying token.

The feature topic noun phrase132of the feature fragment130may not be the same as the base topic noun phrase114of the base fragment120preceding the feature fragment130. For instance, the feature topic noun phrase132may correspond to any type of noun phrase100, such as a predicate noun phrase116, of a statement fragment102that precedes the feature fragment130. In this way, a feature fragment130may be based on another feature fragment130and so on.

The illustrated embodiments ofFIGS. 12 and 13provide examples of feature fragments130, shown active, in a child statement60and with a feature topic noun phrase132that expands on a noun phrase100recited in a statement fragment102that precedes the feature fragment130. In both cases depicted in the illustrated embodiments, the statement fragment102that precedes the feature fragments130shown active are themselves feature fragments130, reciting respectively “the more information” and “the information button” corresponding to the feature topic noun phrases132.

III. Model Input and Model Output (Paragraph Generation)

Several methods of training a model and generating a plurality of tokens are depicted in the illustrated embodiments ofFIGS. 14-18. The document compilation process according to one embodiment of the present disclosure may involve providing a statement fragment102as an input to a model type (e.g., Model Type M1) to generate one or more tokens as an output. The output may correspond to a paragraph of text incorporated into the document being generated. In this way, the model type may correspond to a paragraph type or fragment type.

The illustrated embodiment ofFIG. 14depicts a method1400for training and generating output from a Model Type M1. As discussed above, the statement fragment102provided to the Model Type M1 according to one embodiment is the preamble fragment110to generate a paragraph of one or more tokens relating to the content of the preamble fragment110.

The method1400may include obtaining one or more claim statements from a document, such as a reference from the reference database240or the seed document212, with the claim statements including one or more head statements50and zero or more child statements60. Step1402. The claim statements may be separated into individual statements corresponding to either a head statement50or child statement60. Each individual statement may be tokenized and parsed so that noun phrases100within the individual statement can be identified. Each individual statement may be fragmentized to yield one or more fragments according to criteria discussed herein, including fragmentized to yield the preamble fragment110. Steps1404,1406.

Tokenization and parsing in accordance with one embodiment may involve identifying words of a statement or fragment as tokens, assigning a part of speech to each token (also described as parts of speech (POS) tagging), and identifying associations between tokens to identity phrase groups within the statement or fragment. For instance, the statement “the quick brown fox jumps over the lazy dog” may be tokenized and parsed as follows:

The parts of speech symbols identified in the example above include DT, IN, JJ, NN, and VB and correspond respectively to a singular determiner, a preposition, an adjective, a singular noun, and a verb in base form. These symbols correspond to a subset of the part-of-speech tags used in the Brown Corpus or (Brown University Standard Corpus of Present-Day American English), which is incorporated herein by reference in its entirety. It should be understood that there may be many additional types of parts of speech utilized in tagging the tokens, and that the present disclosure is not limited to the POS tags identified in the example above. Further, it should be understood that an alternative tagging scheme other than the one utilized in the Brown Corpus may be utilized for POS tagging.

In the example above, the statement is parsed into groups of tokens (based at least in part on the POS tagging of the tokens) identified by the symbol NP corresponding to a noun phrase group. Additional or alternative groupings may be identified, including verb phrases and prepositional phrases. This type of classification of groups of adjacent tokens may be identified as chunking (e.g., noun phrase chunks). However, the present disclosure is not so limited—dependency parsing may also be utilized so that associations between non-adjacent tokens may be identified, such as the direct object and the noun object relative to a verb or prepositional object.

The POS tagging and parsing of tokens in a statement may be achieved in a variety of ways, depending on the application. One example tagger and parser is the SyntaxNet parser available from Google®, named colloquially as Parsey McParseface.

In the illustrated embodiment, the head statement50may be POS tagged, parsed, and fragmentized according the criteria set forth herein to yield a preamble fragment110that is POS tagged and parsed. Step1408. For instance, tokens (POS tagged and parsed) leading up to the token “comprising” may be identified as part of the preamble fragment110.

At this stage, the method1400may bifurcate between training the Model Type M1 or providing the preamble fragment110as an input to the Model Type M1 (already trained) to generate one or more tokens corresponding to a head paragraph to be included in the document compilation. The phantom lines to Step1420illustrate this possibility between training the Model Type M1 and use of an already trained version of the Model Type M1, which may be obtained from model storage250.

For purposes of disclosure, the method1400will now be described in connection with training the Model Type M1—but it should be understood that, after the preamble fragment110has been obtained, the method1400may proceed to use of an already trained version of the Model Type M1 at Step1420. In training the Model Type M1, a target output may be obtained for comparison against the preamble fragment110. The target output may be based on text (e.g., specification text) associated with the head statement50in a reference document obtained from the reference database240. This text may be tokenized, POS tagged, and parsed (e.g., parsed into noun phrases100), and fragmentized in a manner similar to the head statement50described with respect to Steps1402,1404and1406. Fragmentation, such as breaking the text into multiple statements, may be conducted prior to, after, or as an intermediate step to tokenizing, POS tagging, and parsing, and may include defining groups of statements or sentences associated with each paragraph of the specification text. Step1412.

The method1400according to the illustrated embodiment may identify portions of the text associated with the head statement50, depicted in the illustrated embodiment as specification text. Step1414. The portions of text may be identified based on a variety of criteria, as described herein, including portions of text that include noun phrases100that are similar to or the same as noun phrases100in the preamble fragment110. In one embodiment, it is not necessary for the identified text of the specification to include all of the noun phrases100of the preamble fragment110. Rather, use of noun phrases100that are similar is an indication of similarity between the preamble fragment110and the identified text. In one embodiment, the criteria for the portion of specification text may include having no noun phrases100present in other fragments of the head statement50and any child statement60related to the preamble fragment110.

Additionally or alternatively, a portion of the specification text may be identified by a similarity metric that is based on comparing distinct aspects of the specification text and claim statements. For instance, if the terms “widget” and “bar” are somewhat infrequent in the specification text and claim statements but the preamble fragment110and a portion of the specification text both utilize these terms, that portion of specification text may be identified for use as a target output of the Model Type M1.

It is noted that the identified portion of the specification may include more than one statement or sentence, preferably but not limited to a paragraph of text bounded by newlines in the specification text.

The preamble fragment110may be associated with the identified portion of specification text as an input/output pair. This process may be conducted numerous times, including associating a preamble fragment110with a portion of specification text from the same document from which the preamble fragment110is extracted. This way, a large set of input/output pairs may be collected and used for training the Model Type M1 to yield an output based on an input corresponding to a preamble fragment110. Steps1416and1418. The trained model may be stored in memory224or model storage250for use at a later stage.

With a trained version of the Model Type M1 stored in memory224or model storage250, the method1400may sidestep the training process, including steps1410,1412,1414,1416and1418, and proceed to generate output based on a preamble fragment110. Step1420. In this case, the preamble fragment110may be obtained from a seed document212, and provided to the Model Type M1 in order to generate a paragraph of text or one or more tokens.

In the illustrated embodiment ofFIG. 15, a method1500is provided for training and generating output from a Model Type M2. As discussed above, the statement fragment102provided to the Model Type M2 according to one embodiment is an aggregate fragment to generate a paragraph of one or more tokens, Paragraph Type P2, relating to the content of the base components of the head statement50, and optionally any base components of any child statements60dependent on the head statement50according to the tree structure outlined herein. The base components, identified as a base topic noun phrase114from one or more base fragments120, may relate to the basic structure of the content to which the claim statements are directed. The aggregate fragment may be defined according to these base components or base topic noun phrases114, and used as a basis for training the Model Type M2 to generate one or more tokens as an output that may relate to the relationship of the base components.

The method1500may include obtaining one or more claim statements, including one or more head statements50and zero or more child statements60, and separating, tokenizing, and fragmentizing the one or more claim statements, similar to steps1402and1404described herein with respect to the method1400. The method1500may also involve POS tagging and parsing the one or more claim statements to identify noun phrases100therein, similar to step1406of the method1400.

The method1500according to the illustrated embodiment may generate an aggregate fragment based on one or more base topic noun phrases114identified in one or more respective base fragments120of the one or more claim statements. Step1508. Similar to the method1400, the method1500may branch at this stage between 1) training the Model Type M2 to take as an input the aggregate fragment and to output a Paragraph Type P2, including a plurality of tokens pertaining to the content of the aggregate fragment, or 2) applying the aggregate fragment as the input to generate the output. This optional branch is shown in phantom lines inFIG. 15.

For purposes of disclosure, the method1500is described next in connection with training the Model Type M2, but it should be understood that, as discussed herein, the process may proceed to generating a Paragraph Type P2 from the aggregate fragment. Step1520.

In the illustrated embodiment, based on the aggregate fragment and its base topic noun phrases114, a portion of the specification text may be identified that is similar to the content of the aggregate fragment. Steps1510,1514. For instance, the identified portion may correspond to a paragraph of the specification text that recites noun phrases100that are the same or substantially similar to the base topic noun phrases114of the aggregate fragment. There are a variety of additional or alternative criteria described herein that may be utilized in identifying a portion of the specification text that is similar to the aggregate fragment, including a similarity metric that includes identifying a paragraph of the specification text that is more similar to the statement fragment102(e.g., aggregate fragment) than any other paragraphs of the specification text.

The specification text may be tokenized, POS tagged, parsed, and fragmentized in a manner similar to the process outlined with respect to Steps1410and1412described in connection with the illustrated embodiment ofFIG. 14. This analysis of the specification text may facilitate identifying a portion of the specification text that is similar to the aggregate fragment, and to compile a training input/output pair corresponding to the aggregate fragment and the identified portion of specification text. Similar to the method1400, multiple input/output pairs, thousands or hundreds of thousands or greater, may be generated for training based on references obtained from the reference database240. Step1416. The Model Type M2 may be trained, as described herein, based on the input/output training pairs, and stored in memory224or model storage250for use at a later stage to generate a Paragraph Type P2 based on an aggregate fragment provided as an input. Steps1518,1520.

Turning to the illustrated embodiments ofFIGS. 16 and 17, methods1600and1700are shown for training and generating output from a Model Type M3 or a Model Type M4 to respectively generate a Paragraph Type P3 or Paragraph Type P4 based on a base fragment120or a feature fragment130. The methods are similar in many respects to the methods1400and1500with several exceptions. For instance, the methods1600and1700may include obtaining one or more claim statements, including one or more head statements50and zero or more child statements60, and separating, tokenizing, and fragmentizing the one or more claim statements, similar to steps1402and1404described herein with respect to the method1400. The methods1600and1700may also involve POS tagging and parsing the one or more claim statements to identify noun phrases100therein, similar to step1406of the method1400.

The methods1600and1700according to the illustrated embodiment may obtain a base fragment120or a feature fragment130from the one or more claim statements, and identify a portion of specification text (e.g., a paragraph) that is similar to the base fragment120or the feature fragment130. Steps1610,1614,1710,1714. Similarity may be based on criteria described herein, including, for example, recitation of the same or substantially similar noun phrases100, such as a recitation of the base topic noun phrase114or the feature topic noun phrase132and any predicate noun phrases116. The specification text may be processed in a manner similar to that described herein in connection with the illustrated embodiments ofFIGS. 14 and 15to facilitate identifying the portion of the specification text that is similar to the base fragment120or the feature fragment130.

In the method1600, the base fragment120may be associated with a similar portion of specification text, such as a paragraph of the specification text, to form an input/output pair for training the Model Type M3. Step1616. This process may be conducted for many base fragments120from many references to obtain numerous input/output pairs for training. Likewise, the method1700may including forming input/output pairs for training the Model Type M4 based on the feature fragment130and a similar portion of the specification text. Step1716. With the input/output training pairs, preferably much greater than a thousand of them, the system according to one embodiment may train the respective Model Type M3 or Model Type M4, and save the trained model in memory224or model storage250. Steps1618,1718. The trained models may be provided to generate, respectively, a Paragraph Type P3 or Paragraph Type P4 based on an input corresponding to the base fragment120or the feature fragment130. Steps1620,1720.

The illustrated embodiment ofFIG. 18depicts yet another method1800for training and generating output from a Model Type M5 that corresponds to content related to a noun phrase100, such as a predicate noun phrase116, that is also not identified in the claim statements as a head topic noun phrase112, a base topic noun phrase114, or a feature topic noun phrase132. As an example, the noun phrase “algorithm” in the illustrated embodiment ofFIG. 3satisfies this criteria and may be provided to the Model Type M5 as an input. Step1806. A noun phrase100that satisfies this criteria may be used as a basis for identifying a portion of the specification text so that the Model Type M5 may be trained to generate a plurality of tokens that relate to the noun phrase100. Step1814. The criteria for identifying the portion of specification text may be similar to the techniques described herein with respect to the illustrated embodiments ofFIGS. 14-17, such as presence of the noun phrase100but no other noun phrases100of the head statement50or any of its child statements60.

With numerous training pairs corresponding to the noun phrase100identified for the Model Type M5 as an input and a similar portion of specification text as an output, the Model Type M5 may be trained as discussed herein. Steps1816,1818. The Model Type M5 may be stored in memory224or model storage250for use with respect to a noun phrase100that is potentially previously unseen in order to generate a plurality of tokens related to the noun phrase100. Step1820.

IV. Model Generation and Training

An example model is shown according to one embodiment of the present disclosure inFIG. 19, and generally designated300. The example model is depicted as a sequence-to-sequence type of model300utilizing one or more layers of a recurrent neural network (RNN), such as a long short-term memory (LSTM) network. It should be understood the present disclosure is not limited to use of RNN layers or LSTM layers, and that any type of model300and any number of layers may be utilized.

The model300may take as an input a plurality of tokens that are vectorized based on the vector database230. This way, the input to the model300may be a sequence of vectors302labeled V1, V2, V3 and V4, each of the vectors302having a dimension X (e.g., 128). The first layer of the model300is an LSTM encoder310which may encode the input sequence V1, V2, V3, V4 in a forward review manner, optionally backward review, to generate a context vector312provided to the LSTM decoder320. Each step of the LSTM encoder310may be identified as a state having a dimensionality Y (possibly the same or different from the vector302dimensionality) that is fed as an input to the next block of the LSTM encoder310. Each block of the LSTM encoder310may include many weights corresponding to the dimensionality of the LSTM encoder310that may be trained as discussed herein.

The LSTM decoder320may accept the context vector312output from the LSTM encoder310, and generate an output306that may be provided to a neural network330or layer that translates the output306from the LSTM decoder320to the vector space utilized by the vector database230. The output306of the first state or block of the LSTM decoder320may be provided to the next state or block. Optionally, each of the outputs306from the LSTM decoder320may be provided to the neural network330, to yield a sequence of prediction vectors304representative of a likely token.

In the illustrated embodiment, the output306from each state or block may be provided to the next block or state. Put differently, the output306from each block in the LSTM decoder320may be provided to the next block in the sequence to generate a sequent token vector (or a vector fed to the neural network330to yield a token vector). Although the output306of each block in the LSTM decoder320is shown being provided to the next block in the sequence, the output of the neural network330in vector space X may be provided as an input to the next block in the sequence. The sequence generated from the LSTM decoder320may be variable in length, terminating on output of one or more end of line tokens. It should be noted that the end of line token may be provided at the end of each training output so that the model300is trained to generate the end of line token when the output sequence is considered complete.

In an alternative embodiment, the context vector312provided by the LSTM encoder310may be provided to the LSTM decoder320, which may generate a vector output314that is provided to the neural network330. This alternative is depicted with phantom lines inFIG. 19. The neural network330may translate this vector output314to vector space or a word embedding corresponding to a token or word. The output from the neural network330or the word embedding may be appended to the input sequence V1, V2, V3, V4. This modified input sequence may be provided to the LSTM encoder310and LSTM decoder320to generate the next word embedding in the output sequence. The input sequence may be a sliding window of the last Z number of vectors, such as the last 25 vectors (zero padded if less than 25 vectors exist) corresponding to the word embeddings of the input sequence appended with vectors generated through the LSTM encoder310, LSTM decoder320, and neural network330. Although the model is described in connection with an LSTM encoder310and LSTM decoder320, any type of encoder or decoder, or both, may be utilized. Further, the neural network330is optional. Additionally, or alternatively, no decoder and no neural network may be utilized so that the context vector from the encoder may be directly mapped to vector space of the vector database230.

InFIG. 20, a method of training the model300in accordance with one embodiment is shown and designated2000. The method2000may include obtaining a plurality of training pairs of input sequences and outputs sequences, with the input sequence being tokens (T1 . . . Tm) and the target output sequence being tokens (OT1 . . . OTk). Step2010. The length of the input sequence may differ from the length of the output sequence, and the output sequence and/or input sequence may vary from training pair to training pair. The number of training pairs may be several thousand or more, and possibly several hundreds of thousands. Indeed, with the number of references available from the USPTO database being in the millions, it is possible to generate several million or more training pairs.

Each sequence of tokens may be vectorized into vector space based on the vector database230. Step2012. In the illustrated embodiment, each token is vectored to a dimension Y so that T1→VT_1, 1 . . . Y. The input sequence of tokens I1 after being vectored is identified as VI1, and the output sequence of tokens O1 after being vectored is identified as VO1.

In the illustrated embodiment, the method2000includes training a model to generate a single output that is appended to an input sequence for generating the next output. This may be described as a sliding window as discussed in connection with the illustrated embodiment ofFIG. 19. To yield a training set for such a configuration, the vector input sequence VI and vector output sequence VO are concatenated to form sequence MI. A sliding window of size b (e.g., 25) is applied across the sequence MI for each position a (e.g., the step size may be 1) with the input sequence corresponding to the subset MI[a−b . . . a] of the sequence MI[1 . . . m+k] where 1<a<m+k. The subset may be zero padded in cases where a<b or b>m+k. The output associated with each sequence subset of MI may be MI[a+1] or the next token vector in the sequence. Step2014.

The model may be initialized with weights for each node of the model, such as each node of the LSTM encoder310, the LSTM decoder320, and the neural network330. Step2016. The input sequences may be provided to the model to generate an output, and based on the error between the output and the target output, the weights may be adjusted. Step2018. In one embodiment, input sequences may be provided to the model several or more times (epochs) such that the weights are adjusted a number of times to reduce the error between the model output and the target output. This way, the model can be adjusted to more closely approximate the correct output or target output for a given input sequence. In one embodiment, gradient descent may be utilized to reduce the error by changing each weight in proportion to the derivative of the error with respect to the weight being changed.

After the model has been trained such that an acceptable degree of error has been achieved, the model may be stored in memory224or model storage250. Step2020.

The document generator210in accordance with one embodiment may generate a document based on a seed document212and output from one or more models obtained from the model storage250. In the illustrated embodiment ofFIGS. 2 and 21, the one or more models include the five types of models described herein and associated with various aspects of a seed document212, which is similar in some respects to the reference documents obtained from the reference database240queried for training the one or more models. For instance, the seed document212may include a plurality of statements or claim text arranged in a tree structure, including one or more head statements50and one or more child statements60. However, the seed document212provided to the document generator210may not include specification text or a substantial amount of specification text in contrast to the reference documents obtained from the reference database240. The document generator210in the illustrated embodiment is configured to generate specification text based on the plurality of statements or claim text. The document generator210may be configured for other type of documents, including a software requirements document based on a hierarchical list of software requirements.

As described herein, there are five types of inputs provided respectively to the five types of models. It should be understood that the five types of inputs are provided as example ways in which aspects of the one or more head statements50and one or more child statements60may be categorized into fragments or groups of tokens according to one or more criteria. The same or similar criteria may be used in generating one or more fragments from the references of the reference database240to train a respective model type. There may be more or fewer models utilized to compile a document, and one or more models configured differently.

The preamble fragment110of the references from the reference database240may be extracted from a head statement50and utilized as a basis for training the Model Type M1 to generate the Paragraph Type P1 as an output. The preamble fragment110may be identified from the seed document212and input to the Model Type M1 to generate a plurality of tokens as output in a similar manner. This output or Paragraph Type P1 may be compiled with output from one or more additional model types to generate a document.

According to one embodiment, depicted in the illustrated embodiment ofFIG. 2, the document generator210may utilize 5 types of models to compile a document based on a plurality of statements that define a tree structure of one or more head statements50and one or more child statements60. For purposes of disclosure, the model types are associated with paragraph types that are labeled Paragraph Type P1, Paragraph Type P2, Paragraph Type P3, Paragraph Type P4, and Paragraph Type P5 in the illustrated embodiment. Each of the paragraph types may be associated with a type of fragment extracted from the seed document212or the plurality of statements according to one or more criteria, as discussed herein.

The statement fragments102may be selectively input to one or more models to generate a paragraph output tree structure, such as the one outlined in the illustrated embodiment, that define organization of the output from the one or more models. It should be understood that that output of the one or more models may be organized differently or according to a different paragraph organization or arrangement.

The preamble fragment110may be associated with a topic set of tokens defined as the Paragraph Type P1 and generated from the Model Type M1. This set of tokens may form a paragraph of the document to be generated that provides an overview or context for the paragraphs to follow that are based on statement fragments102of the head statement50or child statement60that provides the preamble fragment110.

The aggregate fragment discussed herein may be formed of one or more noun phrases100of a plurality of statement fragments102, such as the base topic noun phrases114of a plurality of base fragments120. This way, the aggregate fragment may correspond to basic topics or elements of the head statement50, or head statement50and one or more child statements60. Providing this aggregate fragment as an input to a Model Type M2 may enable generation of a plurality of tokens defined as the Paragraph Type P2 and relating to the plurality of noun phrases100provided in the aggregate fragment, and which may be expanded upon through generation of further tokens (e.g., Model Types M3, M4, M5) based on the statement fragments102from which the noun phrases100have been extracted for inclusion in the aggregate fragment.

The tokens output from the Model Type M2 and based on input of the aggregate fragment may be appended as another paragraph to the Paragraph Type P1 generated from the Model Type M1.

The base fragment120extracted from the head statement50or child statement60may form the basis for providing input to a Model Type M3 trained to generate a plurality of tokens that expand on the concepts outlined in the base fragment120, including, for example, the base topic noun phrase114and zero or more predicate noun phrases116that follow the base topic noun phrase114. Other aspects or other tokens, such as verb phrases or linking verbs, of the base fragment120may be utilized by the Model Type M3 to generate the plurality of tokens defined as the Paragraph Type P3. The output from the Model Type M3 may be appended as another paragraph to the document being compiled, including the paragraphs from the Model Type M1 and Model Type M2.

At this stage, as depicted in the illustrated embodiment ofFIG. 2, the process of generating additional paragraphs for compiling the document to be generated may be iterative based on whether the base fragment120includes one or more predicate noun phrases116, and whether each predicate noun phrase116is further identified as a feature topic noun phrase132in a feature fragment130. For instance, as depicted in the illustrated embodiment, a base fragment120that is the first among several base fragments120is associated with the base topic noun phrase114designated base topic NP1 and includes multiple predicate noun phrases116designated predicate topics NP1 . . . M.

The first predicate topic NP1 is identified as a feature topic noun phrase132in a feature fragment130, and provided as an input to the Model Type M4 to generate a plurality of tokens to form the Paragraph Type P4 with text relevant to the predicate topic NP1 and its relation to other noun phrases100in the feature fragment130. This process may be iterated for each predicate noun phrase116(NP1 . . . NPM) that forms a feature topic noun phrase132of a feature fragment130. For each predicate noun phrase116that does not correspond to a feature topic noun phrase132, the predicate noun phrase116may be provided as an input for the Model Type M5 to generate a Paragraph Type P5 with tokens relevant to the predicate noun phrase116.

The same process as outlined for the first predicate topic NP1 of the base fragment120may be conducted for each of the following predicate topics NP2 . . . M. For each iteration, a paragraph of text may be generated from a type of model, and appended to paragraphs previously generated. The document may be compiled piece by piece in this way from paragraphs output from one of the models and based on statement fragments102of various classifications.

In some cases, a base fragment120may include a base topic noun phrase114with no other predicate noun phrases116. In this case, the base fragment120may be provided as an input to the Model Type M3 or a Model Type M5 to generate a plurality of tokens describing aspects included in the base fragment120, including the base topic noun phrase114. The iterative process identified above for one or more predicate noun phrases116may not be necessary for the base fragment120in this case because there are no predicate noun phrases116. This type of base fragment120is shown inFIG. 2in phantom lines with no predicate topics.

There also are instances in which there is no corresponding feature fragment130for a predicate noun phrase116of a statement fragment102. As discussed above, this predicate noun phrase116may be provided as an input to the Model Type M5 to generate a Paragraph Type P5. Optionally, the predicate noun phrase116may be provided by itself to the Model Type M3 or Model Type M4 to generate a plurality of tokens relating to the predicate noun phrase116. These tokens may form a paragraph of text included in the document being compiled.

In the illustrated embodiment, each of the base fragments120and feature fragments130may be iterated through, and outputs relating to each fragment are organized as paragraphs appended to each other for each iteration. It should be understood that headings corresponding to one or more noun phrases100may be inserted into the document before associated paragraphs are positioned to provide additional context to the paragraphs and structure of the compiled document.

In the illustrated embodiment ofFIG. 21, a method2100of compiling a document based on claim text with one or more head statements50and zero or more child statements60arranged in a tree structure is shown. Alternatively, the document may be compiled from a different type of text other than the claim text that is arranged in a different or similar type of tree structure, such as topic sentences and sub-topic sentences.

The method2100may involve obtaining the claim text, and one or more of tokenizing, POS tagging, parsing, and fragmentizing the claim text. Steps2102,2104. The noun phrases100of one or more statement fragments102, categorized according to one embodiment of the present disclosure, may be identified. Step2106.

The preamble fragment110of the claim text may be provided to the Model Type M1 to generate a Paragraph Type P1, and the aggregate fragment of the claim text may be provided to the Model Type M2 to generate a Paragraph Type P2. Steps2108,2110. These two paragraphs may provide general context to the subject matter of the claim text and outline the main components of the text, which may correspond to the base topic noun phrases114.

The iterative process of generating paragraphs that expand on this subject matter according to the arrangement and association of the noun phrases100may be performed. The method2100may include providing a base fragment120as an input to the Model Type M3 to generate a Paragraph Type P3 related to the subject matter of the base fragment120. Step2112. For each predicate noun phrase116of the base fragment120, a Paragraph Type P4 or a Paragraph Type P5 may be generated depending on whether the predicate noun phrase116corresponds to a feature topic noun phrase132of a feature fragment130. Steps2114,2116.

If a predicate noun phrase116of the base fragment120corresponds to a feature fragment130, each of the predicate noun phrases116of the feature fragment130may be processed to generate a Paragraph Type P4 or a Paragraph Type P5. As each paragraph type is generated it may be appended to the prior generated paragraph type so that the document is generated after each of the noun phrases100and associated statement fragment102have been provided as input to a model type to output a paragraph type. Step2118. The document generated according to one embodiment herein may incorporate one or more pre-defined paragraphs or form paragraphs based on a template. Because statement fragments102input to a model may be separated from other statement fragments, providing individual paragraphs as output, it should be understood that the arrangement of paragraphs is not limited to the construction shown in the illustrated embodiment and that the paragraphs may be arranged differently depending on the application.