Grouping documents based on document concepts

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving at least one electronic document, and identifying one or more words, phrases, or patterns used within the electronic document and that are based on a lexicon. Mapping, using a concept library, the one or more words, phrases, or patterns to a concept intended to be conveyed by the one or more words, phrases, or patterns according to the lexicon. Generating concept data based on the mapping, and storing the concept data in association with data identifying the electronic document.

BACKGROUND

Various individuals may convey similar abstract concepts using different words and phrases based on each individual's pre-dispositions and manners of thinking about the concepts. Categorization, searching, and comparison of documents based on abstract concepts becomes difficult and imprecise using traditional methods due to the differing ways that various authors of documents think about and describe similar topics.

SUMMARY

This specification relates to categorizing electronic textual documents based on concepts extracted from the documents.

Implementations of the present disclosure generally extract concepts conveyed using various different lexicons from electronic textual documents. More particularly, implementations of the present disclosure are directed to identifying in documents words, phrases, or patterns of words and phrases, specific to a variety of differing lexicons, that are intended to convey similar concepts; and to map the identified words, phrases, and patterns of words and phrases to the intended concepts using a set of concept libraries. In some implementations, concept data identifying the concepts conveyed within each document is then generated and stored in association with the data identifying the documents with which the concept data is associated. In some implementations, the concept data may include a measure of the strength or weight of individual concepts in relation to other concepts.

In general, innovative aspects of the subject matter described in this specification can be embodied in methods that include actions of receiving at least one electronic document, and identifying one or more words, phrases, or patterns used within the electronic document and that are based on a lexicon. Mapping, using a concept library, the one or more words, phrases, or patterns to a concept intended to be conveyed by the one or more words, phrases, or patterns according to the lexicon. Generating concept data based on the mapping, and storing the concept data in association with data identifying the electronic document. Other implementations of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

These and other implementations can each optionally include one or more of the following features. In some aspects, the concept library may include at least one lexicon having a plurality of context entries, where each context entry has a set of one or more words, phrases, or patterns and is mapped to at least one concept intended to be conveyed by the set of one or more words, phrases, or patterns of the respective context entry. The concept library may be one of a product library, a product feature library, a line of business library, or a life events library. The lexicon maybe one of a customer lexicon, a customer service representative lexicon, a legal staff lexicon, a marketing staff lexicon, or a technical staff lexicon.

The concept data may include binary data associated with each concept represented in the concept library, where the binary data indicates that a respective concept is either present or not present in the at least one electronic document. The concept data may include a concept score for each concept represented in the concept library, where each concept score indicates a frequency with each respective concept is conveyed in the at least one electronic document.

The methods may further include the actions of determining a set of concepts that are conveyed in both a first and a second electronic document based on comparing concept data associated with the first electronic document to concept data associated with the second electronic document, and providing, for display to the user, the set of concepts that are conveyed in both a first and a second electronic document in response to receiving a user request to compare the first and the second electronic document.

The methods may further include the actions of determining a set of concepts that are conveyed in a first electronic document but that are not conveyed in a second electronic document based on comparing concept data associated with the first electronic document to concept data associated with the second electronic document, and providing, for display to the user, the set of concepts that are conveyed in a first electronic document but that are not conveyed in a second electronic document in response to receiving a user request to compare the first and the second electronic document.

Particular implementations of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Implementations may enable efficient and consistent categorization of large quantities of electronic textual documents across various lexicons. Implementations may permit comparisons of electronic textual documents at an abstracted conceptual level notwithstanding variations in words and phrases used to convey similar concepts within the documents. Implementations may allow rapid searches for documents relating to specific concepts among large quantities of documents notwithstanding variations in words and phrases used to convey similar concepts within the documents.

DETAILED DESCRIPTION

Implementations of the present disclosure generally extract concepts conveyed using various different lexicons from electronic textual documents. More particularly, implementations of the present disclosure are directed to identifying in documents words, phrases, or patterns of words and phrases, specific to a variety of differing lexicons, that are intended to convey similar concepts; and to map the identified words, phrases, and patterns of words and phrases to the intended concepts using a set of concept libraries. In some implementations, concept data identifying the concepts conveyed within each document is then generated and stored in association with the data identifying the documents with which the concept data is associated. In some implementations, the concept data may include a measure of the strength or weight of individual concepts in relation to other concepts.

FIG. 1depicts an example system100that can execute implementations of the present disclosure. The example system100includes computing system102in communication with user computing device103and concept libraries104through a network105. The computing system102can include a computing device102aand computer-readable memory provided as a persistent storage device102b, and can represent various forms of server systems including, but not limited to, a web server, an application server, a proxy server, a network server, or a server farm. Computing devices can be any type of user computing device including, but are not limited to, one or more desktop computers, laptop computers, notebook computers, tablet computers, and other appropriate devices. Network105can include a large network or combination of networks, such as a local area network (LAN), wide area network (WAN), the Internet, a cellular network, a satellite network, one or more wireless access points, or a combination thereof connecting any number of mobile clients, fixed clients, and servers.

As described in more detail below in reference toFIG. 2, each concept library104includes a group of related concepts and contextual lexicon entries related to each concept. A concept library104maps one or more different user lexicons into a common set of concepts defined within the library104. Concept libraries104may define concept groups including, but not limited to, products, product features, lines of business, business organizational units, communication channels (e.g., telephone, internet, kiosk, etc.), computer systems/devices, application software, life events, or other appropriate concepts of interest.

In some examples, one or more of the concept libraries104may be stored on the persistent storage device102bof computing system102. In some examples, the concept libraries104may be stored in one or more databases on other persistent storage devices accessible to computing system102through network105. In some implementations, the concept libraries104may be stored in one or more cloud databases on other persistent storage devices accessible to computing system102through network105.

The concept libraries104may be used by computing system102to automatically and consistently categorize documents106based on concepts conveyed within the documents notwithstanding specific user lexicons used within each document to describe the concepts. Documents106may include any type of electronic text document, including but not limited to, word processor files, portable document files, e-mails, chat conversations, speech to text transcriptions, or website or user application text entry forms, for example.

Computing system102may use one or more libraries104to identify various concepts defined within a library104based on specific words, phrases, and/or patterns of words and phrases contained within a document106that map to the defined concepts. The computing system102then may generate concept data108(described in more detail below with reference toFIG. 3) that identifies one or more concepts conveyed within the document106or within each document in a set of documents. The concept data108may be used to categorize, compare, or search the various documents106based on one or more standard sets of concepts as defined by concept libraries104.

For example, a concept library104may define a group of products and map those products to words, phrases, or patterns of words and phrases used by customers, customer service representatives (CSR), legal staff, and engineers to describe each of the defined products. A user may send a group of documents106to computing system102to extract concept data108related to each of the documents106(e.g., from computing device103through network105). Moreover, individual documents within the group of documents106may include text generated by customers, CSRs, legal staff, and engineers, each of which use a different lexicon. Computing system102can use the example product concept library104to identify various products described within each document notwithstanding differing lexicons that may be used in individual documents. Then, computing system102can generate concept data108for each document that identifies the products described in the individual documents regardless of the lexicon used to describe the products. The computing system120may then use the concept data108to categorize the documents by product, to compare documents across differing lexicons, or to perform user searches of the documents using a standardized concept searching index. In some implementations, the concept data108may allow for more efficient use of computing resources when system102performs functions such as document categorization, comparison, and searching. In some examples, the concept data108may be stored in one or more databases on other persistent storage devices accessible to computing system102through network105. In some implementations, the concept data108may be stored in one or more cloud databases on other persistent storage devices accessible to computing system102through network105.

FIG. 2depicts a graphical representation of an example concept library200. As noted above, concept libraries200may be used (e.g., by computing system102) to map concepts or groups of concepts to combinations of words, phrases and/or patterns of words and phrases used within different lexicons to describe the similar or the same concepts. For example, a library200may be generated to map words, phrases, or patterns of words and phrases used by customers (e.g., a lexicon) to describe various products offered by a business to the specific products (e.g., the concepts). Further, the various words, phrases, or patterns of words and phrases used by members of a legal department to describe the same set of products may be different from those used by the customers. Thus, the library200also may map words, phrases, or patterns of words and phrases used by legal staff (e.g., another lexicon) to describe the same products to the specific products. In some examples, a library200may be generated to map multiple different lexicons to any particular group of concepts including, but not limited to, products, product features, lines of business, business organizational units, communication channels, computer systems/devices, application software, life events, or other appropriate concepts.

More specifically, a concept library200includes one or more lexicons202, each of which includes multiple context entries204. Each context entry204may include a set of one or more words (W), phrases (PH), patterns (PS), or any combination of the three that may be used to describe a particular concept206in the related lexicon202(e.g., a customer lexicon). In other words, the lexicons202map various sets of words, phrases, and patterns used by speakers of the particular lexicon to concepts206that those words phrases, or patterns are intended, by users of the lexicon, to convey. Furthermore, each library may map the various lexicons202to a specific group of concepts206(e.g., products, product features, lines of business, etc.).

For example, lexicons202may be defined to map the various words, phrases, and patterns that are used by different groups of people (e.g., groups of people in different departments within an organization such as a business) to convey similar concepts206(e.g., products offered by the business). Lexicons202may include, but are not limited to, a customer lexicon (Lcust), a customer service representative (CSR) lexicon (LCSR), an engineering department lexicon (Leng), and a legal department (Llegal).

For example,FIG. 2depicts an exemplary concept library200for a line of products (LLprod), such as, for example, financial products. Products (concepts206) defined in the product library (LLprod) may include, for example, consumer loan, credit card, auto insurance policy, and home insurance policy. Taking the credit card as an example, when discussing a credit card, customers may tend to refer a business's credit card products as a “charge card,” “credit card,” “creditcard,” or “payment card,” for example. Referring to the same credit card product, customer service representatives may refer to the credit card products as “credit card,” “mastercard,” “master card,” “visa,” “amex,” or “American Express.” Referring again to the same credit card products, other groups (e.g., legal staff, engineers, and marketing staff) may use yet other words, phrases, or patterns to describe the products. Therefore, the product library (LLprod)1includes each set of words, phrases, and patterns in corresponding context entries204(e.g., context entry (W, PH, Ps)1for each respective lexicon202(e.g., Lcust−LN), where the first context entry (W, PH, Ps)1for each lexicon describes a first concept206(e.g., credit cards or product1), and the nth context entry (W, PH, Ps)1for each lexicon describes an nth concept206(e.g., productn).

In some implementations, a concept library200may include only one lexicon202. In such an implementation the various words, phrases, and patterns used by each different lexicon to describe a particular concept may be grouped together in a single context entry204and mapped to the particular concept. In other words, the context entries204for each different lexicon may be combined into a single entry as opposed to being divided into separate contextual entries, one for each different lexicon, as described above.

In some implementations, libraries200may be updated as the words, phrases, and patterns used within one or more lexicons to describe particular concepts change. For example, one or more computing systems may monitor changes in each lexicon and update corresponding libraries as appropriate. In some implementations, a library200may be updated to modify, add, or remove various concepts defined within the library. For example, as products change, new products are offered, and old products are discontinued.

In some implementation, one or more lexicons may be unique to a particular organization, for example, a business. The lexicon may represent words or phrases used internally by the business to describe products, product features, lines of business, and other business affairs in a manner that maintains confidentiality of the businesses internal processes.

FIG. 3depicts graphical representations300,320,350of example concept data (e.g., concept data108). Concept data108may be generated by a computing system, such as computing system102, to represent concepts conveyed within electronic documents using various different lexicons to describe the concepts.

Graphical representation300depicts concept data (cells308) extracted from multiple documents (D1-DM) according to one implementation, where the rows302represent individual documents (D1-DM), and the columns304represent individual concepts (P1-PN). In graphical representation300, the concept data308is represented as binary data, where a “1” indicates that a particular concept is conveyed within a particular document, and a “0” indicates that a particular concept is not conveyed within a particular document. For example, as depicted in graphic representation300, concepts P2, P3, P5, and PNare conveyed within document D1as indicated by the “1's” in the appropriate columns304of row302a.

As described above, the concepts may represent various products (P1-PN) offered by a business, therefore, based on the concept data extracted for document D1, the document may be categorized as describing products P2, P3, P5, and PN, irrespective of the lexicon used to describe products P2, P3, P5, and PNin document D1.

In addition, documents D1and D2may be compared based on their respective concept data, for example, to determine which concepts described in document D2are also described in document D1. Based on comparing rows302aand302bit can quickly be determined that document D1describes all of the products (e.g., P3and PN) described in document D2, irrespective of the lexicons used in each document to describe the products.

Graphical representation320depicts concept data (cells328) extracted from multiple documents (D1-DM) according to another implementation. Similar to graphical representation300, in graphical representation320the rows322represent individual documents (D1-DM), and the columns324represent individual concepts (P1-PN). However, in graphical representation320, the concept data328is depicted by scores, where a magnitude of a score represents the strength with which or frequency of which a concept is conveyed in a given document. For example, as depicted in graphic representation320, products P2, P3, P5, and PNare each described within document D1as indicated by the non-zero scores in the appropriate columns324of row322a. However, based on the score magnitudes product P3, with a score of “4,” is described more frequently than product P2, with a score of “1.”

Graphical representation350depicts concept data (cells358) extracted from multiple documents (D1-DM) according to yet another implementation. Similar to graphical representation300, in graphical representation350the rows352represent individual documents (D1-DM), and the concept data358is represented as binary data. However, graphical representation350illustrates a more detailed set of concept data, where the columns354represent individual concepts (e.g., products P1-PN) nested with sub-concepts associated with each concept (e.g., product features PiF1-PiFn). For example, product P1may be a credit card associated with product features (P1F1-P1Fn), such as fees, interest rate, credit limit, etc. Graphical representation350may, for example, represent context data obtained based on using multiple related libraries (e.g., libraries104,200) to extract context data from one or more documents.

FIG. 4depicts an example process400that can be executed in accordance with implementations of the present disclosure. In some examples, the example process400can be provided as one or more computer-executable programs executed using one or more computing devices. In some examples, the process400is executed to extract concepts conveyed within one or more electronic textual documents using one or more different lexicons.

At least one electronic document is received by a computing device (410). Electronic documents may include any type of electronic text document, including but not limited to, word processor files, portable document files, e-mails, chat conversations, speech to text transcriptions, or website or user application text entry forms, for example.

One or more words, phrases, or patterns that are used within the electronic document and that are based on a lexicon are identified within the electronic document (420). For example, a particular word, phrase, or pattern of words may be used within lexicon A to describe concept X, while a different word, phrase or pattern of words may be used within lexicon B to describe the same concept X. For example, the lexicon may be one of a customer lexicon, a customer service representative lexicon, a legal staff lexicon, a marketing staff lexicon, or a technical staff lexicon. In some examples, known sets of one or more words, phrases, or patterns corresponding to one or more known lexicons may be identified within the electronic document.

The one or more words, phrases, or patterns are mapped using a concept library to a concept intended to be conveyed by the one or more words, phrases, or patterns in the lexicon (430). For example, the concept library may include at least one lexicon having a plurality of context entries. Each context entry of the concept library may include a set of one or more words, phrases, or patterns that are mapped to at least one concept which is intended to be conveyed, according to the respective lexicon, by the set of one or more words, phrases, or patterns that are included in the respective context entry. In some examples, the concept library may be one of a product library, a product feature library, a line of business library, or a life events library. In other words, the concept library may map words, phrases, or patterns of words using in one or more lexicons to concept sets that include, but are not limited to, products, product features, lines of business, or life events.

Concept data is generated for the document based on the mapping (440), and the concept data is stored in association with data identifying the electronic document (450). The concept data may represent, at a high level, the concepts present in the electronic document the respective of the lexicon in which they are described in the document. In addition, the concept data may, in some implementations, reduce the computing resources required to compare, categorize, or search large groups of electronic documents. In some implementations, the concept data may include binary data associated with each concept represented in a concept library, and the binary data may indicate that a respective concept is either present or not present in an electronic document. In some implementations, the concept data may include a concept score for each concept represented in the concept library, and each concept score may indicate a frequency or strength with which each respective concept is conveyed in an electronic document. In some examples, the concept data may be stored in one or more databases on other persistent storage devices accessible through a network. In some implementations, the concept data may be stored in one or more cloud databases on other persistent storage devices accessible through a network.

Some implementations may permit a user to compare two or more electronic documents for similar concepts. For example, a graphical user interface (GUI) that allows a user to select two or more documents for comparison may be presented to a user. Upon receiving a user input selecting two or more documents to be compared, a set of concepts that are conveyed in both a first and a second electronic document may be determined based on comparing concept data associated with the first electronic document to concept data associated with the second electronic document. Then, in response to receiving the user request to compare the first and the second electronic document, the set of concepts that are conveyed in both a first and a second electronic document may be provided for display to the user, for example, within the GUI.

Some implementations may permit a user to compare two or more electronic documents for dissimilar concepts (e.g., contrast two or more documents). For example, a GUI that allows a user to select two or more documents for comparison may be presented to a user. Upon receiving a user input selecting two or more documents to be compared, a set of concepts that are conveyed in a first electronic document but that are not conveyed in a second electronic document based on comparing concept data associated with the first electronic document to concept data associated with the second electronic document. Then, in response to receiving the user request to compare the first and the second electronic document, the set of concepts that are conveyed in a first electronic document but that are not conveyed in a second electronic document, for example, within the GUI.

In some examples, concept data may be stored across multiple databases. In such situations a document index may be used identify storage locations for concept data associated with respective documents, enabling documents to be compared or contrasted across databases.

Some implementations may include a user selectable control that allows the user to choose whether to compare or contrast two or more documents. In other words, the user selectable may allow the user to determine whether the two or more documents are to be compared in order to identify similar concepts conveyed within the documents, or whether the documents are to be contrasted in order to identify concepts conveyed within one of the documents but not others.

In some implementations, a search index may be created based on concept data from a plurality of documents. A search engine may, for example, use the search index created based on the concept data to permit users to efficiently search a large group of electronic documents at a concept level as opposed searching documents using traditional keyword searches. In some implementations, multiple search indices may be created based on multiple sets of concept data. Further, the search indices and corresponding sets of concept data may be stored on separate databases or cloud databases. A search engine may, for example, perform concept level document searches on two or more of the search indices concurrently.

In some implementations, a document category index may be created based on concept data from a plurality of documents, thereby, allowing a large group of documents to be efficiently organized based on concepts described within the documents.