SYSTEMS AND METHODS FOR SEMANTIC SEARCH SCOPING

A computer-implemented method for searching electronic documents is provided. The method executing a non-semantic search, such as lexical search, to identify documents from a document corpus that meet the search criteria of the non-semantic search. A subsequent semantic search can be scoped based on the results of the non-semantic search. The method can thus include executing a semantic search scoped to the documents identified in the non-semantic search result to generate a semantic search result that identifies content that is semantically relevant to a natural language query. Thus, the semantically relevant content can have both non-semantic (e.g., lexical) and semantic relevance.

COPYRIGHT NOTICE

TECHNICAL FIELD

This disclosure relates generally to searching electronic documents. Even more particularly, embodiments of the present application relate to enhancing searching of large bodies of potentially complex documents by scoping semantic searches using non-semantic searches.

BACKGROUND

In the modern world, many documents that are being created, utilized, and maintained are in electronic format. Several situations commonly arise that require an analysis or identification of certain relevant electronic documents from a relatively large pool of available electronic documents. These situations are generally referred to as information retrieval or search problems. These types of search problems crop up in a wide variety of contexts. For example, in litigation, an attorneys may have to search through a large volume of documents provided by their client and received during discovery to find information needed to

To illustrate in more detail, parties to litigation typically must share relevant evidence with opposing counsel through the discovery process. In many cases, each party makes a reasonable search of their records based on some set of terms or keywords and produces the results of the search to the other party. Discovery thus typically involves the gathering of potentially relevant materials, much of it digital, and then reviewing such materials to determine what to be shared with opposite parties. Additionally, during the litigation, the lawyers must continually review those documents produced both by their own client and by the opposing party to locate documents relevant to the case at hand. Litigation thus represents a microcosm of a more general problem raised by the high volume of electronic documents present in a variety of contexts. Namely, how a large volume of electronic documents can be understood, reviewed, or searched in order that documents relevant to a particular topic or user's interest may be located.

Document analysis systems help resolve these problems. A document analysis system is a computer-implemented system that allows users to search, analyze, review, or navigate information in a corpus to locate electronically stored information of interest. Document analysis systems are often tailored to specific contexts, such as electronic discovery, academic research, etc. E-discovery systems, for example, include tools to allow attorneys to search documents for review, exhaustively tag the documents, and use the tags to determine whether and how to produce documents, thus assisting in review for production. An attorney may also use a document analysis system during investigation, where the attorney determines the facts of a case and finds evidence for or against those facts.

Document analysis systems typically support lexical searching of documents. In a common scenario, a user of a document analysis system submits a query to the document analysis system to search a corpus of documents and the search engine of the document analysis system selects a set of results from the corpus based on the terms of the search query. The terms of search queries usually specify words, terms, phrases, logical relationships, metadata fields to be searched, synonyms, stemming variations, etc. The search engine performs a lexical search on metadata fields and, in some systems, document content for literal matches of words, terms, phrases or variants to identify documents and returns the documents that meet the logical constraints specified in the search.

With lexical search, the meaning or intent of the query can be lost. This can result in the search engine missing documents that meet the intent behind the query but do not match the words in the query. On the other hand, a lexical search may also return many documents that match the search terms but are not relevant to the intent behind the query.

Some document management systems also support other types of searches, such as semantic search, which attempts to understand the intent and contextual meaning behind a search query to provide more relevant results. Semantic searching is more time-consuming and computationally expensive than lexical searching and can produce less accurate results than lexical searching when the semantic search query does not contain enough context.

In document analysis systems that support multiple types of searches, each search type (e.g., lexical, semantic, etc.) is treated as a different problem domain and considered independently. If, for example, a user wishes to run a lexical search and a semantic search, they use a lexical search tool for the lexical search and a semantic search tool for the semantic search. If a user is interested in documents that match different types of searches, the user must compare the search results from the different types of searches. For example, if the user is interested in documents that both have specific lexical characteristics and semantic meaning, it is left to the user to determine which documents are of interest by reviewing both lexical search results and the independent semantic search results. Document analysis tools do not provide a way to seamlessly change between search types.

What is desired, therefore, are improved systems and methods for searching large bodies of potentially complex documents.

SUMMARY

Embodiments of the present disclosure provide systems and methods for scoping semantic searches to enhance the accuracy of semantic searches and, in some embodiments, generative artificial intelligence.

According to one aspect of the present disclosure, a computer-implemented method for searching electronic documents is provided. The method may include receiving a non-semantic search query from a user to search a document corpus and executing a non-semantic search according to the non-semantic search query to generate a first search result that identifies first documents from the document corpus. The non-semantic search, according to one embodiment, is a lexical search. The method may further include receiving a natural language query from the user and servicing the natural language query to generate a response to the user. Servicing the natural language query may include executing a semantic search scoped to the first documents—that is the documents identified in the results of the non-semantic search—to generate a semantic search result that identifies semantically relevant content that is semantically relevant to the natural language query. According to one embodiment, the natural language query is a query to an AI-search assistant.

Some embodiments include providing the first search result to the user in a graphical user interface and receiving, via user interaction with the graphical user interface, an indication to scope the semantic search to the first documents. According to another embodiment, the semantic search is automatically scoped to the first documents.

According to one embodiment, servicing the natural language query comprises generating an input to a large language model where the input comprises the natural language query and the semantically relevant content to cause the large language model to generate text to respond to the natural language query based on the semantically relevant content. One embodiment further comprises receiving generative text generated by the large language model in response to the input and providing the generative text to the user in response to the natural language query. The input to the large language model, according to one embodiment, includes the natural language query as a prompt and the semantically relevant content as a context for responding to the prompt.

In some embodiments, the semantically relevant content comprises text chunks associated with the documents identified in the non-semantic search results. In another example embodiment, the semantically related content comprises semantically related documents from the documents identified in the non-semantic search result.

According to another aspect of the present disclosure, a non-transitory, computer-readable medium for searching electronic documents is provided. The non-transitory, computer-readable medium embodies code that includes instructions executable for receiving a non-semantic search query from a user to search a document corpus and executing a non-semantic search according to the non-semantic search query to generate a first search result that identifies first documents from the document corpus. The non-semantic search, according to one embodiment, is a lexical search. The code may further include instructions executable for receiving a natural language query from the user and servicing the natural language query to generate a response to the user. Servicing the natural language query may include executing a semantic search scoped to the first documents—that is the documents identified in the results of the non-semantic search—to generate a semantic search result that identifies semantically relevant content that is semantically relevant to the natural language query. According to one embodiment, the natural language query is a query to an AI-search assistant.

Some embodiments include instructions executable for providing the first search result to the user in a graphical user interface and receiving, via user interaction with the graphical user interface, an indication to scope the semantic search to the first documents. According to another embodiment, the semantic search is automatically scoped to the first documents.

According to one embodiment, servicing the natural language query comprises generating an input to a large language model where the input comprises the natural language query and the semantically relevant content to cause the large language model to generate text to respond to the natural language query based on the semantically relevant content. One embodiment further comprises executable instructions for receiving generative text generated by the large language model in response to the input and providing the generative text to the user in response to the natural language query. The input to the large language model, according to one embodiment, includes the natural language query as a prompt and the semantically relevant content as a context for responding to the prompt. In some embodiments, the semantically relevant content comprises text chunks associated with the documents identified in the non-semantic search results. In another example embodiment, the semantically related content comprises semantically related documents from the documents identified in the non-semantic search result.

Another aspect of the present disclosure provides a computer system providing enhanced search. The computer system may include storage, processor, and memory. The storage may include a plurality of snippets. Each of the plurality of snippets may comprise snippet text extracted from a document in a document corpus and a reference to the document from which the snippet text of that snippet was extracted. The storage may further store an embedding store comprising a vector index of the plurality of snippets.

The memory stores a non-semantic search engine and a semantic search engine. The non-semantic search engine may be executable to search the document corpus. According to one embodiment, the non-semantic search engine is a lexical search engine. The semantic search engine may be executable to perform semantic searching of the document corpus using the vector index. The memory may further comprise instructions executable to scope semantic searches by the semantic search engine to documents identified in search results from the non-semantic search engine.

According to one embodiment, the memory further stores instructions executable to receive a non-semantic search result from the non-semantic search engine where the non-semantic search results comprise document identifiers for first documents from the document corpus. The memory may further store instructions executable to store the document identifiers from the non-semantic search results as query parameters for a subsequent search. The memory may further store instructions executable to receive a natural language query after the non-semantic search result. The natural language query includes a query string from a user. The memory may further store instructions executable to generate a request to the semantic search engine that includes the query string and the document identifiers that were stored as query parameters.

In some embodiments, the semantic search engine is executable to receive the query string and the document identifiers execute a corresponding semantic search scoped to the first documents to generate a semantic search result that identifies semantically relevant content that is semantically relevant to the query string.

According to one embodiment, the memory further stores instructions executable to generate an input to a large language model, the input to the large language model comprising the query string and the semantically relevant content from the semantic search result, receive generative text generated by the large language model to respond to the input string based on the semantically relevant content and display the generative text to the user.

DETAILED DESCRIPTION

The disclosure and various features and advantageous details thereof are explained more fully with reference to the exemplary, and therefore non-limiting, embodiments illustrated in the accompanying drawings and detailed in the following description. It should be understood, however, that the detailed description and specific examples, while indicating the preferred embodiments, are given by way of illustration only and not by way of limitation. Descriptions of known programming techniques, computer software, hardware, operating platforms, and protocols may be omitted so as not to unnecessarily obscure the disclosure in detail. Various substitutions, modifications, additions and/or rearrangements within the spirit and/or scope of the underlying inventive concept will become apparent to those skilled in the art from this disclosure.

Embodiments of the present disclosure provide systems and methods for enhancing semantic search and, in some embodiments, artificial intelligence (AI)-based text generation by scoping servicing of natural language queries based on the results of non-semantic searches. More particularly, embodiments may, for example, scope semantic searches based on the results of lexical searches.

According to one embodiment, a document analysis system can provide an interface that allows a user to run multiple types of searches (e.g., non-semantic search, semantic search, or other types of searches) or a system to implement a workflow that runs multiple types of searches. The search results from a search are used to scope a next search (or other subsequent search), either automatically or based on user selection. For example, the search results of a non-semantic search can be used to scope a semantic search to documents returned by the non-semantic search. Embodiments of the present disclosure allow a user or system to switch between search types while maintaining context from a prior search, even if the prior search was of a different type. Embodiments can increase the relevancy of a search by limiting the search to a narrower context provided by the results of a prior search, potentially of a different search type.

Generative AI is becoming an increasingly important tool for analyzing documents. Embodiments of the present disclosure can be employed to scope the content used by generative AI when responding to a query to focus the generative AI on documents that have specific lexical or semantic characteristics, thus increasing the accuracy of the responses produced by the generative AI.

FIG.1is a diagrammatic representation of one embodiment of a document analysis system100. Document analysis system100executes on a processor101, which, in some embodiments, comprises a plurality of processors of a plurality of computers that execute code to provide a document analysis system. The code is embodied, in some embodiments, on a non-transitory, computer readable medium.

Document analysis system100supports various types of search technologies for searching a large corpus of documents (document corpus102). In some embodiments, document corpus102includes documents stored on a variety of storage technologies, including across heterogeneous storage technologies (e.g., stored on remote or local databases, remote or local file systems, cloud stores or other storage technologies). Documents in document corpus102are assigned unique document identifiers (document IDs) in document analysis system100.

In the embodiment illustrated, document analysis system100includes a user interface104through which a user105can interact with the system, a semantic search engine106, a semantic embedding engine108, and a second search engine110. According to one embodiment, second search engine110performs a different type of search from semantic search engine106. For example, in one embodiment, second search engine110is a lexical search engine that performs a lexical search of documents in corpus102.

User interface104allows user105to submit search queries according to the supported search types. Document analysis system100can service new queries in the context of prior searches. For example, document analysis system100may constrain a lexical search to the results of the semantic search. Thus, iterative searches may be performed in the context of prior searches.

Document analysis system100includes a text store112, a snippet store114, and an embedding store116for storing data related to searching document corpus102. Each of text store112, snippet store114, and embedding store116comprises a file system, a database, or other storage technologies or combinations thereof. While illustrated separately, two or more of text store112, snippet store114, or embedding store116may represent portions of the same data store.

The text of searchable documents from document corpus102is stored in document analysis system100as index text118for the documents. According to one embodiment, the index text118for a document comprises a character array of characters from the document—for example, as a single dimensional character array in one embodiment.

To support semantic search, the documents from document corpus102may be semantically embedded as document text vectors that represent the document text for semantic searching. More particularly, the documents from document corpus102may be broken down into more manageable chunks of text, referred to herein as original text chunks, and the original text chunks semantically embedded as document text vectors. As discussed below, the process of semantically embedding an original text chunk may involve normalizing the original text chunk and semantically embedding the normalized text chunk as the document text vector representing the text chunk.

According to one embodiment, each of the original text chunks associated with a document is a sequence of characters within the index text118of a document. In an even more particular embodiment, an original text chunk is a single dimension aware sequence of characters within the index text118of a document. According to one embodiment, an original text chunk has a start offset and end offset within the character array of the index text118of a document. In some embodiments, the original text chunks follow a set of delineation rules, such as, but not limited to junk text is excluded, punctuation is preserved, capitalization is preserved. The amount of text in an original text chunk will depend on the chunking rules applied. According to one embodiment, the documents from document corpus102are chunked into defined grammatical units, such as sentences.

The original text chunks are stored as snippets120in snippet store114. According to one embodiment, a snippet120comprises an original text chunk (snippet text), a document ID of a document from document corpus102, and an offset in a document coordinate system indicating the location of the snippet text in the document (that is, the location of the snippet text in the document having the document ID with which the snippet is associated). A snippet may thus be an original text chunk in the context of a document. Each snippet120can be assigned a unique id according to one embodiment.

The snippet texts—that is, the original text chunks—are semantically embedded as document text vectors. Various text embedding models known or developed in the art can be used to semantically embed a text chunk as a document text vector for semantic search. According to one embodiment, a multi-qa-mpnet-base-dot-v1 model is used to generate the text embeddings.

As will be appreciated, semantic embedding may involve text normalization, such as, but not limited to removing white space, removing email line breaks, etc. Thus, according to one embodiment, the embedding process may normalize the original text chunks from snippets120as normalized text chunks for embedding as the document text vectors. In some cases, multiple original text chunks from the same document or across documents normalize to the same normalized text chunk (e.g., the same normalized sentence) and thus the same semantic embedding (the same document text vector).

Embedding store116comprises a vector index122of snippets that associates the semantically embedded text chunks—that is, the document text vectors—with snippets. In a more particular embodiment, index122maps the document text vectors to normalized text chunks from which the document text vectors were generated and the snippets120from snippet store114that map to the normalized text chunks. Because the snippet text in multiple snippets may map to the same normalized text and hence semantic vector, multiple snippets120may map to the same normalized text chunk and document text vector in index122.

In some embodiments, one or more of the components of document analysis system100may be provided by a different system or a third party. Further, document analysis system100may include additional or alternative services. For example, document analysis system100may include various services to orchestrate searches and reviews of results. Thus, for example, a request illustrated as flowing from one component to another inFIG.2may be processed and conditioned by one or more intermediate services between the components.

Turning toFIG.2, one embodiment of a flow in document analysis system100is illustrated. UI104receives a non-semantic search query200for second search engine110. In an embodiment in which second search engine110is a lexical search engine, query200includes lexical search criteria, such as words, phrases, logical operators, or other search criteria supported by the lexical search engine. User interface104sends a search request202to search engine110with the search criteria. Search engine110executes the search and returns search results204(e.g., a lexical search result) that includes a list of hits. According to one embodiment, each hit is a reference to a document from corpus102that meets the search criteria. For each hit, search results204may include, for example, the document ID of the document. In some embodiments, search results204include, for each document referenced, one or more of the document name, representative content from the document, or metadata associated with the document. UI104displays search results204to the user (output206). UI304tracks one or more prior search results as potential scopes for servicing subsequent queries.

User105may wish to run a further search, such as a semantic search. In one embodiment, UI104provides the user the option to scope the next search to the documents returned from a previous search, such as search results204. Thus, the user may select to scope the next search to the documents returned in search results204(or other prior search result). Based on user interaction with UI104, UI104can thus receive input208indicating that the next search is to be scoped to documents returned from the previous search. In some embodiments, user305may select a subset of documents from search result406to which a subsequent search should be scoped. UI104, at parameter update210, updates query parameters with constraints to limit servicing of the next query to the documents identified in a previous search result (e.g., search results204). For example, UI104may store the document IDs of the documents identified in search results406or selected subset of documents as query parameters.

In another embodiment, document analysis system100is configured to automatically update the query parameters so that the next search in a defined logical flow, such as a search session, conversation, or a dialog, is to be done in the scope of the previously returned documents without explicit user selection or in the absence of user opt-out. In other words, parameter update210may occur automatically without user input.

As discussed, user interface104, according to some embodiments, supports multiple search technologies. For example, user interface104may also allow user105to submit a semantic search query to search corpus102. Thus, user interface104can receive a semantic search query212(e.g., a natural language search query) that includes an input string for semantic search. UI sends a search request214to semantic search engine106. If the search is to be scoped to documents returned in the previous search, the semantic search request214also includes query parameters that constrain the search scope based on prior search results. For example, semantic search request214may include the document IDs from prior search results204(e.g., as stored at parameter update210).

Semantic search engine106sends a request216to semantic embedding engine108to embed the user input string. Semantic embedding engine108semantically embeds the input string as a semantic vector (a query vector). According to one embodiment, semantic embedding engine108embeds the input string in the same way the original text chunks were embedded, including normalizing the input string for embedding in the same manner as the original chunks were normalized for embedding. Semantic embedding engine108returns a response218to semantic search engine106that includes the query vector (that is, the semantically embedded (normalized) input string).

Semantic search engine106performs a semantic search of index122using the query vector to identify semantically relevant content that is responsive to semantic search request214and generates semantic search results222based on the responsive semantically relevant content or references to the responsive semantically relevant content. According to one embodiment, semantic search engine106supports the approximate matching of text chunks that enables the semantics to be found with related meaning.

Various methods of identifying semantically relevant content may be employed. According to one embodiment, semantic search engine determines the similarity between the query vector (e.g., the semantically embedded input string) and a document text vector (a semantically embedded text chunk) and by computing a similarity score—for example, a cosine similarity—between the embeddings. Semantic search engine106can thus determine that a document text vector is a semantically relevant document text vector based on the similarity score determined for the document text vector.

In one embodiment, semantic search engine106identifies the normalized text chunks and snippets mapped to the semantically relevant document vectors in index122as semantically relevant to search query200. Further, in some embodiments, semantic search engine106identifies the documents identified in semantically relevant snippets as documents that are semantically relevant to search query200.

If semantic search request214includes search constraints, semantic search engine106applies a search scope filter220to filter the identified semantically relevant content to only include, as responsive semantically relevant content, the scoped semantically relevant content that is within the search scope specified by the search constraints. In one embodiment, for example, semantic search engine106may identify as scoped semantically relevant snippets the semantically relevant snippets that contain a document ID that matches a document ID provided as a constraint in search request214. Further, semantic search engine106may identify as scoped semantically relevant normalized text chunks the semantically relevant normalized text chunks that are mapped to responsive semantically relevant snippets in index122. Similarly, semantic search engine106may identify as scoped semantically relevant documents the semantically relevant documents that have a document ID that matches a document ID provided as a constraint in search request214.

In another embodiment, semantic search engine106is configured to limit the search of documents to documents identified in search request214. For example, semantic search engine106may filter entries in index122to only those entries corresponding to documents identified as a constraint in search request214and then perform a search to identify semantically relevant content. Thus, the semantically relevant content determined in this manner can be considered scoped semantically relevant content.

According to one embodiment, semantic search results222includes citations for responsive semantically relevant snippets. In an even more particular embodiment, semantic search results222includes citations for semantically relevant snippets scoped to documents identified in search request214. According to one embodiment, the citation for a snippet includes snippet information for the snippet, the snippet information for a snippet includes a text chunk (e.g., one or more of the normalized text chunk mapped to the snippet or the original text chunk from the snippet) and a reference to one or more of the snippet (e.g., the snippet ID) or the document identified in the snippet (e.g., the document ID from snippet). Other examples of information that may be included in the snippet information of a citation include but are not limited to the snippet offset, document metadata (e.g., Author or other metadata), of the document identified by the snippet, a citation for a relevance score for the normalized text chunk mapped to the snippet, or relevance score for the original text chunk from the snippet. In some embodiments, citations include entire snippets. A citation for a semantically relevant snippet may include, in some embodiments, snippet information for snippets before or after a semantically relevant snippet.

User interface104displays the results (e.g., the citations to the user) (output224). The user may be given the option to perform another semantic search or another type of search (e.g., a lexical search). According to one embodiment, the user is given an option to search within the scope of prior search results, such as search results222. Based on user interaction with UI104, UI104can thus receive an input226indicating that the next search is to be scoped to documents returned from a previous search—for example, the documents identified in the citations of semantic search results222or identified in the snippets referenced by the citations of semantic search result222. In some embodiments, user105may select a subset of the documents. UI104, at parameter update228, updates query parameters with constraints to limit servicing of the next query to the documents based on the previous search result (e.g., search results222). For example, UI104may store the document IDs of the documents identified in the citations of semantic search results222or identified in the snippets referenced by the citations of semantic search result222, or selected subset thereof.

In another embodiment, document analysis system100is configured to automatically update the query parameters so that the next search in a defined logical flow, such as a search session, conversation, or a dialog, is to be done in the scope of the previously returned documents without explicit user selection or in the absence of user opt-out. In other words, parameter update228may occur automatically without user input.

The user enters another search query230, for example, a search query for second search engine110. In an embodiment in which second search engine110is a lexical search engine, query230includes lexical search criteria, such as words, phrases, logical operators, or other search criteria supported by the lexical search engine. UI104formulates a request232to second search engine110that includes the user's query parameters with constraints to limit the search to the documents identified from semantic search results222(e.g., the document IDs included query parameter update228).

Search engine110executes the search and returns search results234(e.g., a lexical search result) that includes a list of hits. The results are displayed to user105(output236). According to one embodiment, each hit is a reference to a document from corpus102that meets the search criteria of search request232. For each hit, search results234may include, for example, the document ID of the document. In some embodiments, search results234include, for each document referenced, one or more of the document name, representative content from the document, or metadata associated with the document. The hits, however, are limited to the documents having document IDs included in the constraints of search request232.

Thus, user105may perform multiple searches in the context of a defined logical flow, such as a search session, conversation, or a dialog. According to one embodiment, if user105performs a first search and then, in the context of the defined logical flow, submits another search query, document analysis system100performs the new search within the scope of the documents returned by the prior search, even if the prior search was a different type of search. Thus UI104(or another component of document analysis system100) can scope a new search to the documents referenced in prior search results, either automatically or based on user selection. The user can thus seamlessly switch between semantic search and other types of searches in a defined logical flow.

In the embodiment ofFIG.2, semantic search engine106filters semantically relevant content to identify and return citations for scoped semantically relevant content. In other embodiments, the filtering of semantically relevant content to a scope can be applied at another component or service. For example, semantic search results222, in one embodiment, may include citations for snippets from documents not identified in search results204or not included in a subset of the documents selected by user105. UI104can apply a filter to limit the results to content from documents having document IDs included in search result204(e.g., to the documents having the document IDs stored at parameter update210).

FIG.3is a diagrammatic representation of one embodiment of a document analysis system300. Document analysis system300executes on a processor301, which, in some embodiments, comprises a plurality of processors of a plurality of computers that execute code to provide a document analysis system. The code is embodied, in some embodiments, on a non-transitory, computer readable medium.

Document analysis system300supports various types of search technologies for searching a large corpus of documents302. Document corpus302can comprise documents stored on a variety of storage technologies, including across heterogeneous storage technologies (e.g., stored on remote or local databases, remote or local file systems, cloud stores or other storage technologies). As will be appreciated, to support semantic search, documents may be embedded as vectors. Even more particularly, documents may be broken down into more manageable chunks and each chunk embedded as a vector.

In the embodiment illustrated, document analysis system300includes a user interface304through which a user305can interact with the system, a dialog orchestration engine306, a review service308, a semantic search engine310, a semantic embedding engine312, a dialog engine314, an LLM316, and a document search engine318. According to one embodiment, document search engine318is a lexical search engine that searches on metadata fields and, in some systems, document content for literal matches of words, terms, phrases or variants to identify documents and returns the documents that meet the logical constraints specified in the search.

Document analysis system300includes a text store320, a snippet store322, and an embedding store324for storing data related to searching document corpus302. Each of text store320, snippet store322, and embedding store324comprises a file system, a database, or other storage technologies or combinations thereof. While illustrated separately, two or more of text store320, snippet store322, or embedding store324may represent portions of the same data store.

In some embodiments, one or more of semantic search engine310, semantic embedding engine312, or LLM316are provided by a third party. Further, document analysis system300may include additional or alternative services. Thus, for example, a request illustrated as flowing from one component to another inFIG.4A,FIG.4B,FIG.4CorFIG.4Dmay be processed and conditioned by one or more intermediate services between the components.

The text of searchable documents from document corpus302is stored in document analysis system300as index text328for the documents. According to one embodiment, the index text328for a document comprises a character array of characters from the document—for example, as a single dimensional character array in one embodiment.

To support semantic search, the documents from document corpus302may be semantically embedded as document text vectors that represent the document text for semantic searching. More particularly, the documents from document corpus302may be broken down into more manageable chunks of text, referred to herein as original text chunks, and the original text chunks semantically embedded as document text vectors. As discussed below, the process of semantically embedding an original text chunk may involve normalizing the original text chunk and semantically embedding the normalized text chunk as the document text vector representing the text chunk.

According to one embodiment, each of the original text chunks associated with a document is a sequence of characters within the index text328of a document. In an even more particular embodiment, an original text chunk is a single dimension aware sequence of characters within the index text328of a document. According to one embodiment, an original text chunk has a start offset and end offset within the character array of the index text328of a document. In some embodiments, the original text chunks follow a set of delineation rules, such as, but not limited to junk text is excluded, punctuation is preserved, capitalization is preserved. The amount of text in an original text chunk will depend on the chunking rules applied. According to one embodiment, the documents from document corpus302are chunked into defined grammatical units, such as sentences.

The original text chunks are stored as snippets330in snippet store322. According to one embodiment, a snippet330comprises an original text chunk (snippet text), a document ID of a document from document corpus302, and an offset in a document coordinate system indicating the location of the snippet text in the document (that is, the location of the snippet text in the document having the document ID with which the snippet is associated). A snippet may thus be an original text chunk in the context of a document. Each snippet330can be assigned a unique id according to one embodiment.

The snippet texts—that is the original text chunks—from snippets330are embedded as document text vectors for semantic search. Various text embedding models known or developed in the art can be used to embed snippets text as semantic vectors for semantic search. According to one embodiment, a multi-qa-mpnet-base-dot-v1 model is used to generate document text vectors. As discussed above, original text chunks may be normalized for embedding as document text vectors.

Embedding store324comprises a vector index332of snippets that associates the semantically embedded text chunks—that is, the document text vectors—with snippets. In a more particular embodiment, index332maps the document text vectors to normalized text chunks from which the document text vectors were generated and the snippets330from snippet store322that map to the normalized text chunks. Because the snippet text in multiple snippets may map to the same normalized text and hence semantic vector, multiple snippets330may map to the same normalized text chunk and document text vector in index332.

User interface304allows user305to submit queries for searching and generative AI. Document analysis system300can service queries in the context of prior search results. For example, document analysis system300may constrain a semantic search to the results of a lexical search or vice versa. Thus, iterative searches may be performed in the context of prior searches.

Document analysis system300can apply retrieval augmented generation (RAG), which includes a retrieval stage, an augmentation stage, and a generation stage. In the retrieval stage, document analysis system300retrieves text of interest. In the augmentation stage, document analysis system300provides the text of interest to LLM316as, for example, context for a prompt to help LLM316generate a more accurate response to the prompt than LLM316would generate in absence of the context. In the generation stage, LLM316generates text to respond to the prompt. For example, LLM316may answer a question, summarize text, or otherwise generate text to respond to a prompt.

The text of interest for RAG may include, in some embodiments, snippets, normalized text chunks, original text chunks, or documents that are semantically relevant to the query. Accordingly, in the retrieval stage, semantic search engine310performs a semantic search of embedding store324to identify the text of interest such as semantically relevant snippets, semantically relevant normalized text chunks, semantically relevant original text chunks, or semantically relevant documents. In some embodiments, semantic search engine310retrieves and generates a semantic search result that includes the semantically relevant text of interest, which is passed by dialog engine314to LLM316as, for example, context for responding to the query. In other embodiments, semantic search engine310returns a semantic search result that references the semantically relevant snippets, semantically relevant normalized text chunks, semantically relevant original text chunks, or semantically relevant documents, and dialog orchestration engine306or dialog engine314retrieves the text of interest, which dialog engine314provides to LLM316as context.

As discussed, a semantic search may occur in the context of a prior non-semantic search. Thus, the semantic search to identify text of interest for RAG may be scoped to results from a prior non-semantic search.

In some embodiments, searches occur in the context of a dialog managed by dialog orchestration engine306. A dialog is a conversational context that is maintained to allow document analysis system300to accumulate state to disambiguate language that is syntactically ambiguous (e.g., what was “her” name) based on well-supported disambiguation elements such as named entities that are correctly implied by the context and not by the syntax. A dialog can comprise one or a sequence of dialog inputs and, in some embodiments, remembers outputs associated with the inputs. Thus, various inputs and outputs in a dialog are associated by a dialog ID. In other embodiments, there is no association between dialog inputs and outputs.

Turning toFIG.4A, one embodiment of a flow in document analysis system300is illustrated. User interface304allows user305to submit search queries to search corpus302. In some embodiments, user interface304supports different search technologies, including, for example, lexical searching and semantic searching, different semantic search engines or the like. User interface304also supports submitting queries to LLM316, a generative AI-model.

Turning briefly toFIG.5A, one example embodiment of a UI page550provided by one embodiment of UI304is illustrated. UI page550includes a list of documents552in a document corpus (e.g., document corpus302), a search bar554for entering lexical searches, a query bar556for querying an AI assistant (e.g., LLM316), and an AI assistant response area558. The document corpus includes 447,798 documents.

Returning toFIG.4A, UI304receives a non-semantic search query400for document search engine318. In an embodiment in which document search engine318is a lexical search engine, query400includes lexical search criteria, such as words, phrases, logical operators, or other search criteria supported by the lexical search engine.

UI304sends a dialog request402to dialog orchestration engine306that includes the user input search criteria. According to one embodiment, dialog orchestration engine306begins a dialog. Thus, in some embodiments, subsequent communications between components may include a dialog identifier for the dialog. As discussed, a dialog is a conversational context that is maintained to allow document analysis system300to accumulate state to disambiguate language that is syntactically ambiguous.

Dialog orchestration engine306sends a search request404to document search engine318to find documents. Document search engine318executes the search and returns search results406(e.g., a lexical search result) that includes a list of hits. According to one embodiment, each hit is a reference to a document from corpus302that meets the search criteria. For each hit, search results406may include, for example, the document ID of the document. In some embodiments, search results406include, for each document referenced, one or more of the document name, representative content from the document, or metadata associated with the document.

Dialog orchestration engine306returns a response408to UI304responsive to request402. Response408includes the search results406returned by document search engine318. UI304displays the search results to the user (output410). In one embodiment, UI304tracks one or more prior search results in a dialog as potential scopes for servicing subsequent queries.

Turning toFIG.5B, here the user has selected to perform a lexical search on specific tags available in the system as shown by search bar554which is populated with tags (“Enron-Oil, Gas, or Energy”). In this example, document search engine318returned 8,788 hits. The user is given the option to scope the context servicing subsequent queries to the results of the lexical search at control560(e.g., “Set current search as scope (8,788 documents).”

Returning toFIG.4A, user305may wish to submit a new query (query416), such as a semantic search query, a question to LLM316or another type of query. In one embodiment, UI304provides the user the option to scope servicing of a new query to the documents returned from a previous search. For example, the user may select to scope servicing of the next query to the documents referenced returned in the search results406included in response408(or other prior search results). In some embodiments, user305can select a subset of documents from search results406to which a subsequent search should be scoped. Based on user interaction with UI304, UI304can thus receive an input412indicating that the next query is to be scoped to documents returned in a previous search. UI304, at parameter update414, updates query parameters with constraints to limit servicing of the next query to the documents identified in a previous search result (e.g., search results406included in response408). For example, UI304updates the query parameters to include the document IDs returned from the prior search (e.g., all the document IDs returned in search results406included in response408or the document IDs for a selected subset of the documents). For example, if the user selects control560ofFIG.5B, document analysis system updates the query parameters for searching to include the document IDs of the 8,788 documents returned in response to the lexical search.

In another embodiment, document analysis system300is configured to automatically update the query parameters so that the next search in a defined logical flow, such as a search session, conversation, or a dialog, is to be done in the scope of the previously returned documents without explicit user selection or in the absence of user opt-out. In other words, parameter update414may occur automatically without user input.

As discussed, user interface304, according to some embodiments, supports multiple search technologies. For example, user interface304may also allow user305to submit a semantic search query or other type of query. Thus, user305, according to one embodiment, can submit a natural language query416that includes an input string, such as “what happened in california?”.

UI304sends a request418to dialog orchestration engine306that includes the user's input string. If servicing of the query is to be scoped to documents returned in a previous search, request418can include query parameters to constrain the scope of servicing the request. According to one embodiment, request418includes the document IDs of documents determined from a previous search result, such as the document IDs from search results406included in response408(e.g., as stored at parameter update414).

Continuing withFIG.4A, dialog orchestration engine306sends a semantic search request420to semantic search engine310to find citations. Semantic search request420includes the user input string (e.g., “what happened in california?”). In one embodiment, semantic search request420also includes the document IDs from request418(e.g., the document IDs set at parameter update414).

Semantic search engine310sends a request422to semantic embedding engine312to embed the user input string. Semantic embedding engine312semantically embeds the input string (e.g., “what happened in california”) as a semantic vector (a query vector). According to one embodiment, semantic embedding engine312embeds the input string in the same way the original text chunks were embedded, including normalizing the input string for embedding in the same manner as the original chunks were normalized for embedding. Semantic embedding engine312returns a response424to semantic search engine310that includes the query vector (that is, the semantically embedded (normalized) input string).

Semantic search engine310searches embedding data store324using the embedded query string to identify semantically relevant content responsive to semantic search request420. For example, semantic search engine310performs a semantic search of index332using the query vector to identify semantically relevant content that is responsive to semantic search request420and generates semantic search results426based on the responsive semantically relevant content or references to the responsive semantically relevant content. According to one embodiment, semantic search engine310supports the approximate matching of text chunks that enables the semantics to be found with related meaning. Non-limiting examples of semantically searching for relevant content are discussed above with respect to semantic search engine106.

If semantic search request420includes search constraints, semantic search engine310applies a search scope filter425to filter the identified semantically relevant content to only include, as responsive semantically relevant content, the scoped semantically relevant content that is within the search scope specified by the search constraints. In one embodiment, for example, semantic search engine310may identify as scoped semantically relevant snippets the semantically relevant snippets that contain a document ID that matches a document ID provided as a constraint in search request420. Further, semantic search engine310may identify as scoped semantically relevant normalized text chunks the semantically relevant normalized text chunks that are mapped to responsive semantically relevant snippets in index332. Similarly, semantic search engine310may identify as scoped semantically relevant documents the semantically relevant documents that have a document ID that matches a document ID provided as a constraint in search request420.

In another embodiment, semantic search engine310is configured to limit the search of documents to documents identified in search request420. For example, semantic search engine106may filter entries in index332to only those entries corresponding to documents identified as a constraint in search request420and then perform a search to identify semantically relevant content. Thus, the semantically relevant content determined in this manner can be considered scoped semantically relevant content.

According to one embodiment, semantic search results426includes citations for responsive semantically relevant snippets. In an even more particular embodiment, semantic search results426includes citations for semantically relevant snippets scoped to documents identified in search request420. According to one embodiment, the citation for a snippet includes snippet information for the snippet where the snippet information for a snippet includes a text chunk (e.g., one or more of the normalized text chunk mapped to the snippet or the original text chunk from the snippet) and a reference to one or more of the snippet (e.g., the snippet ID) or the document identified in the snippet (e.g., the document ID from snippet). Other examples of information that may be included in the snippet information of a citation include but are not limited to the snippet offset, document metadata (e.g., Author or other metadata) of the document identified by the snippet, a citation for a relevance score for the normalized text chunk mapped to the snippet, or relevance score for the original text chunk from the snippet. In some embodiments, the citation for a snippet includes the entire snippet. A citation for a semantically relevant snippet may include, in some embodiments, snippet information for snippets before or after a semantically relevant snippet.

Dialog orchestration engine306generates a request428to dialog engine314to process query416using LLM316. Request428includes the search query string (e.g., “what happened in california?”) and additional context information for servicing the query. The additional context information includes or references content of interest to be used as context for servicing query416. The context information includes, for example, semantically relevant snippets, semantically relevant normalized text chunks, semantically relevant original text chunks, or semantically relevant documents referenced or included in semantic search result426.

In one embodiment, request428includes citations from semantic search results426as context information. As discussed above, a citation may include a semantically relevant snippet, semantically relevant original text chunk from a snippet, or a semantically relevant text chunk. Thus, request428may include text of interest. In another embodiment, dialog orchestration engine306retrieves the documents identified in the semantically relevant snippets included or referenced in semantic search results426and provides the documents as context information with request428.

Based on request428, dialog engine314generates a prompt and context (indicated at430) for LLM316to process the input string entered by the user and inputs the prompt and context (input434) to LLM316. According to one embodiment, the prompt includes the input string from the user (e.g., “what happened in california?” from query416) and the context includes text of interest over which to process the prompt. As nonlimiting examples, dialog engine314may be configured to provide citations for semantically relevant snippets, semantically relevant snippets, original text chunks from semantically relevant snippets, semantically relevant normalized text chunks or documents referenced in semantically relevant snippets. If the text of interest is not included with request428, dialog engine314may retrieve the text interest based, for example, on the citations included with request428.

Dialog engine314thus formulates a prompt and context to prompt LLM316to answer the question from query416using the citations, snippets, original text chunks, normalized text chunks or documents that semantically relevant to that are semantically relevant to query416as context. Some example prompts and associated context structures are provided in the attached code appendix.

LLM316processes input434to generate text and returns a response436that includes generative text generated responsive to input434. For example, LLM316answers the question “what happened in california?” using the snippet text from or documents identified in the semantically relevant snippets included or referenced in semantic search result426to answer the question “what happened in california?” As will be appreciated, LLM316may identify the snippets or documents on which it based its answer.

Dialog engine314returns a response438to dialog orchestration engine306that includes the text generated by LLM316and the list of citations, snippets or documents used to generate the answer. Dialog orchestration engine306generates a response440to UI304that includes the semantic search results426returned by semantic search engine310and the text generated by LLM316. In one embodiment, response440includes the generative text and the citations, snippets, or documents on which LLM316based its response.

In the foregoing example of processing query416to generate response440, semantic search engine310implemented filtering to filter the semantically relevant content based on a search scope (e.g., document IDs) provided in search request420. In other embodiments, the filtering of semantically relevant content to a scope can be applied by other components or services. In one embodiment, dialog orchestration engine306may apply filters prior to forwarding citations to dialog engine314or retrieving text of interest for forwarding to dialog engine314to filter the citations from semantic search results426to include only those citations that include or reference semantically relevant snippets that contain a document ID that matches a document ID provided as a constraint in request418or that reference semantically relevant documents that have a document ID that matches a document ID provided as a constraint in request418. In another embodiment, dialog engine314may apply filters prior to retrieving text of interest for inclusion as context to LLM316or generating the context for LLM316to scope the text of interest based on document IDs provided in request428.

User interface304displays the results to the user (output442).FIG.5C, for example, illustrates one embodiment of interface page550updated to display generative text562generated by the AI assistant (LLM316) in response to the question “what happened in california?” using the scope set inFIG.5B.

The user can continue to ask questions and receive generated text answers. According to one embodiment, the user may interact with user interface304to generate inputs that cause interface304to update. In one embodiment, user interface304receives an input444that results in the user interface304updating the displayed information (update446). For example, the user can click on the citation expand button and UI304displays the list of snippets returned in the last answer (that is, the snippets returned in response440with respect to the last question asked in the dialog). In some embodiments, UI304displays the snippets in relevance order.

FIG.5Dillustrates AI assistant response area558in which the user has selected to expand the citations to show citations list564. Here the citations are those referenced in the generative AI text.

Further, in one embodiment, the user can select citations for review. Thus, input448may indicate a selection of citations for review. UI304updates a set of review query parameters to include the document ids from selected citations (parameter update450). In some embodiments, parameter update450may also include adding the snippet text from the selected citations to the query parameters. User305commits the search (input452). UI304generates a request454to review service308that includes the snippet text and document ids from the selected citation(s). For example, if the user selects the first snippet in citation list564ofFIG.5D, UI304can formulate a query with the text of the first snippet and the document ID 441842.

In addition to, or in the alternative to, allowing user305to select documents to review using citations, UI304can provide a search tool to allow the user to search documents using other criteria. Thus, in one embodiment, user305may provide an input467that indicates a new search query. In such an embodiment, user interface304at parameter update450, updates the query parameters to include the document IDs from the citations selected in input448, or automatically selected by UI304, to limit the query to those documents. In some embodiments, UI304updates the search input to include the document ids from all the citations returned in the update446. In some embodiments, UI304updates the search input to include the snippet text from the citations. Thus, in one embodiment, when the user commits the search (input452), UI304generates request454to review service308to include the search criteria from input467, the document IDs and snippet text from parameter update450.

Review service308parses the request454from UI304and generates a query (represented at456ofFIG.4A). According to one embodiment, review service308generates an elastic search query. Review service308sends the search query458to document search engine318and document search engine318services the query to generate a response460. In some embodiments, review service308enriches the results (represented at462) and returns a response464to UI304that includes search results. UI304displays an updated result list to user305(output466).

FIG.4Billustrates another example flow of document analysis system300. In this example, document analysis system services query400and query416as discussed above with respect toFIG.4Aand user interface304displays the results (e.g., the response text and citations to the user) (output442). According to one embodiment, the user is given an option to search within the scope of prior search results, such as search results included in response440. Based on user interaction with UI304, UI304can thus receive an input472indicating that the next search is to be scoped to documents returned from a previous search—for example, the documents identified in the citations included in response440. In some embodiments, user305may select a subset of the documents. UI304, at parameter update474, updates query parameters with constraints to limit servicing of the next query to the documents based on the previous search result (e.g., response440). For example, UI304may store the document IDs of the documents identified in the citations or snippets referenced by the citations included in response440, or selected subset thereof.

In another embodiment, document analysis system300is configured to automatically update the query parameters so that the next search in a defined logical flow, such as a search session, conversation, or a dialog, is to be done in the scope of the previously returned documents without explicit user selection or in the absence of user opt-out. In other words, parameter update474may occur automatically without user input.

The user enters another search query473, for example, a lexical search query to review service308. Query473thus includes lexical search criteria, such as words, phrases, logical operators, or other search criteria supported by the lexical search engine. UI304formulates a search request475to review service308. The search initiated based on query473, is scoped to the documents identified in the snippets returned or referenced in search results included in response440. For example, UI304updates the search input to limit the search input to only the documents having document IDs included parameter update474. In some embodiments, parameter update474occurs without user input472and the document IDs are added to the search input in the background and transparently to user305.

Review service308parses the request475from UI304and generates a query (represented at476ofFIG.4A). According to one embodiment, review service308generates an elastic search query. Review service308sends the search query478to document search engine318and document search engine318services the query to generate a response480. In some embodiments, review service308enriches the results (represented at482) and returns a response484to UI304that includes search results. UI304displays an updated result list486to user305.

InFIG.4C, semantic search engine310services semantic search request422as described in conjunction withFIG.4A. In the embodiment ofFIG.4C, however, dialog orchestration engine306returns the semantic search results426in response500to UI304without generative text generated by LLM and UI displays the result list502to user305. The user may then take various actions with respect to the result list, such as selecting citations from search results426for further searching (e.g., to cause UI304to generate a search request to review service308that includes document IDs from selected citations and snippet text from the selected citations), initiating a lexical search that is scoped to document IDs included in the citations from semantic search results426, or initiating other types of searches.

InFIG.4D, orchestration engine306receives request418from UI304that includes document IDs (e.g., as set in parameter update414). Dialog orchestration engine306sends a request510to dialog engine314that includes the input string from query416(e.g., “what happened in california?” and context information. In one embodiment, dialog orchestration engine306retrieves the documents identified by the document IDs included in request418and sends the documents to dialog engine314as context information with request510. In another embodiment, dialog orchestration engine306includes the document IDs in the context information but does not retrieve the documents.

Dialog engine314receives request510and generates a prompt and context (represented at512). The context includes the documents included or identified in the context information of request510. If the context information of request510does not include the documents, dialog engine314retrieves the documents identified by the document IDs in the context information. Thus, dialog engine314can generate an input to LLM316that includes the prompt and context information. LLM316generates text responsive to input514to produce generative text516. Dialog engine314returns a response518to dialog orchestration service306that includes the generative text. Dialog orchestration service306generates a response520to UI304that includes the generative text and UI304displays the generative text to user305(output522).

Thus, in the embodiment ofFIG.4D, the results of the non-semantic search initiated by query400are used to scope the generative AI.

While embodiments have been discussed above with respect to setting the scope for semantic search or AI-text generation using a non-semantic search, scope for a semantic search or AI-text generation may be set in a number of ways. For example, the user may select a folder that includes documents and use the contents of the folder as the scope.FIG.5EandFIG.5F, for example, illustrates an example in which a user has selected a folder566that includes 9,839 documents and has set the folder as the scope (indicated at scope568). Thus, when the user enters the natural language query, “who is sara shackleton” the scope will be the 9,838 documents. That is, the document ids of the 9,839 documents will be included as constraints for the request to semantic search engine310in searching for semantically relevant citations to provide LLM316to answer the question “who is sara shackleton.” As other examples, filters may be used.

FIG.6is a diagrammatic representation of one embodiment of a computing environment600that includes a document analysis computer system602connected to a client system604via network606.

Document analysis computer system602includes a processor610and memory620. Depending on the exact configuration and type of mobile device, memory620(storing, among other things, executable instructions) may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. Further, Document analysis computer system602may also include storage devices612, such as, but not limited to, solid state storage. Storage devices may provide storage for one or more of a document corpus, document index data, snippets, or a vector index. Similarly, document analysis computer system602may also have input device(s) and output device (I/O devices614) such as keyboard, mouse, pen, voice input, touch screen, speakers. Document analysis computer system602further includes communications interfaces616, such as a cellular interface, a Wi-Fi interface, or other interfaces.

Document analysis computer system602includes at least some form of non-transitory computer-readable media. The non-transitory computer-readable readable media can be any available media that can be accessed by processor610or other devices comprising the operating environment. By way of example, non-transitory computer-readable media may comprise computer storage media such as volatile memory, nonvolatile memory, removable storage, or non-removable storage for storage of information such as computer readable-instructions, data structures, program modules or other data. Computer storage media includes, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information.

As stated above, several program modules and data files may be stored in system memory620. While executing on processor610, program modules (e.g., applications, Input/Output (I/O) management, and other utilities) may perform processes including, but not limited to, one or more of the stages of the operational methods described with respect to document analysis100or document analysis system300. In one embodiment, system memory620stores an operating system and a document analysis application622. Document analysis application622is executable by processor610to provide a document analysis system that supports multiple types of searches of a document corpus628and can scope searches or LLM queries based on the results of a semantic search.

System memory620may include other program modules such as program modules to provide analytics or other services. Furthermore, the program modules may be distributed across computer systems in some embodiments.

Client system604includes a processor630and memory638. Depending on the exact configuration and type of computer system, memory638(storing, among other things, executable instructions) may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. Further, client system604may also include storage devices632. Similarly, client system604may also have input device(s) and output device (I/O devices634) such as keyboard, mouse, pen, voice input, touch screen, speakers. Client system604further includes communications interfaces636, such as a cellular interface, a Wi-Fi interface, or other interfaces.

Client system604includes at least some form of non-transitory computer-readable media. The non-transitory computer-readable readable media can be any available media that can be accessed by processor630or other devices comprising the operating environment. By way of example, non-transitory computer-readable media may comprise computer storage media such as volatile memory, nonvolatile memory, removable storage, or non-removable storage for storage of information such as computer readable-instructions, data structures, program modules or other data. Computer storage media includes, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information.

Several program modules and data files may be stored in system memory638. While executing on processor630, program modules (e.g., applications, Input/Output (I/O) management, and other utilities) may perform processes to enable a user to interact with a document analysis system (e.g., as provided by document analysis application622). In one embodiment, system memory638stores an operating system and a client application640. Client application, according to one embodiment, is a desktop application for interacting with document analysis application622. In one embodiment, client application640is a web browser. System memory638may include other program modules such as program modules to provide analytics or other services. Furthermore, the program modules may be distributed across computer systems in some embodiments.

The different aspects described herein may be employed using software, hardware, or a combination of software and hardware to implement and perform the systems and methods disclosed herein. Although specific devices have been recited throughout the disclosure as performing specific functions, one of skill in the art will appreciate that these devices are provided for illustrative purposes, and other devices may be employed to perform the functionality disclosed herein without departing from the scope of the disclosure.

Portions of the methods described herein may be implemented in suitable software code that may reside within RAM, ROM, a hard drive, or other non-transitory storage medium. Alternatively, the instructions may be stored as software code elements on a data storage array, magnetic tape, floppy diskette, optical storage device, or other appropriate data processing system readable medium or storage device.

Although the invention has been described with respect to specific embodiments thereof, these embodiments are merely illustrative, and not restrictive of the invention. The description herein of illustrated embodiments of the invention, including the description in the Abstract and Summary, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. Rather, the description is intended to describe illustrative embodiments, features and functions in order to provide a person of ordinary skill in the art context to understand the invention without limiting the invention to any particularly described embodiment, feature or function, including any such embodiment feature or function described in the Abstract or Summary. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the invention in light of the foregoing description of illustrated embodiments of the invention and are to be included within the spirit and scope of the invention. Thus, while the invention has been described herein with reference to particular embodiments thereof, a latitude of modification, various changes and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of embodiments of the invention will be employed without a corresponding use of other features without departing from the scope and spirit of the invention as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the invention.

Embodiments discussed herein can be implemented in a computer communicatively coupled to a network (for example, the Internet), another computer, or in a standalone computer. As is known to those skilled in the art, a suitable computer can include a CPU, at least one read-only memory (“ROM”), at least one random access memory (“RAM”), at least one hard drive (“HD”), and one or more input/output (“I/O”) device(s). The I/O devices can include a keyboard, monitor, printer, electronic pointing device (for example, mouse, trackball, stylus, touch pad, etc.), or the like.

ROM, RAM, and HD are computer memories for storing computer-executable instructions executable by the CPU or capable of being compiled or interpreted to be executable by the CPU. Suitable computer-executable instructions may reside on a computer readable medium (e.g., ROM, RAM, and/or HD), hardware circuitry or the like, or any combination thereof. Within this disclosure, the term “computer readable medium” is not limited to ROM, RAM, and HD and can include any type of data storage medium that can be read by a processor. For example, a computer-readable medium may refer to a data cartridge, a data backup magnetic tape, a floppy diskette, a flash memory drive, an optical data storage drive, a CD-ROM, ROM, RAM, HD, or the like. The processes described herein may be implemented in suitable computer-executable instructions that may reside on a computer readable medium (for example, a disk, CD-ROM, a memory, etc.). Alternatively, the computer-executable instructions may be stored as software code components on a direct access storage device array, magnetic tape, floppy diskette, optical storage device, or other appropriate computer-readable medium or storage device.

Although the foregoing specification describes specific embodiments, numerous changes in the details of the embodiments disclosed herein and additional embodiments will be apparent to, and may be made by, persons of ordinary skill in the art having reference to this disclosure. In this context, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of this disclosure.