Patent ID: 12223441

Corresponding reference numerals indicate corresponding parts or features throughout the several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference to the accompanying drawings.

Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having,” are inclusive and therefore specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed.

When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to,” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

A method of classifying an electronic document from an electronic media source according to one example embodiment of the present disclosure is illustrated inFIG.1and indicated generally by reference number100. As shown inFIG.1, the method100includes, in block102, generating statistical data from one or more training documents and, in block104, creating classification rules. The classification rules include at least one topic model-based classification rule created using the statistical data and at least one query-based classification rule created using one or more user defined categories and the statistical data. As shown inFIG.1, the method100further includes, in block106, classifying the electronic document using the topic model-based classification rule.

By creating the classification rules using the same statistical data from the training document and the user defined categories (sometimes referred to as “ad hoc categories” or “user defined ad hoc categories”), the classification rules become focused on topics relevant to users. Because the classification rules are focused on relevant topics, classification of the electronic document using the topic model-based classification rule becomes more relevant and, in turn, data from the classified electronic document may be more effective and useful to the user.

For example, users are able to identify topical categories discussed in documents relevant to their industry, expertise, etc. and particular documents discussing their ad hoc categories related to their brand, product, etc. This allows users to identify topics and documents relevant to the users and, in turn, to utilize this information to more accurately collect data pertaining to their business, interests, etc. As a result, users may be able to monitor relevant discussions (e.g., positive and/or negative discussions regarding a product, false accusations, etc.), monitor new topics and/or issues, etc. via documents (e.g., blogs, messages, emails, etc.) representing the discussions and/or respond to disgruntled customers, alter protocols, etc. based on the discussions. This may be particularly important with the increasing popularity of social media which gives consumers a forum to share their experiences, opinions, etc. regarding products and services with other interested parties.

The topic model-based classification rule may be created by a statistical model (e.g., topic model, etc.) and the query-based classification rule may be created by a search engine model (e.g., Lucence Engine, etc.). The statistical model may generate the statistical data from a training document. For example, the statistical data may represent topics from the training document. A topic model-based classification rule is then created using the statistical data.

A query-based classification rule may be created (at least in part) by using user defined categories. For example, a user may define particular categories relevant to the user and the search engine model may then create a query-based classification rule using the statistical data (e.g., generated by the statistical model) and the user defined categories. Accordingly, the statistical model may provide relevant data to the search engine model to create the query-based classification rule. In this way, the statistical model may supplement the search engine model with statistical data to create the query-based classification rule. As a result, the classification rules become focused on topics more relevant to users.

Additionally, if both models utilize the same statistical data, the rule creating process and the classification process may become more efficient, allowing more electronic documents to be classified within a particular time period.

Alternatively, and as further explained below, the method100may include classifying the electronic document(s) using both the topic model-based classification rule and the query-based classification rule. By doing so, classification of the electronic document may be more effective and the resulting data from the classification may not only increase but also may be more relevant to a particular user.

FIG.2illustrates an example method200of creating one or more classification rules. In the example ofFIG.2, the method200is executed offline (e.g., disconnected from a communication network). Alternatively, the method200may be executed online (e.g., connected to a communication network).

As shown inFIG.2, the method200includes receiving training documents (block202). In some embodiments, the training documents may include unlabeled training documents that are not previously analyzed, classified, etc. into a certain category by individuals. Accordingly, if unlabeled training documents are employed, previous knowledge about the documents may not be necessary and/or provided.

Although the example ofFIG.2illustrates receiving multiple training documents, only one training document may be received without departing from the scope of the present disclosure. Preferably, multiple training documents are employed so that the classification rules are more reliable. Additionally, the training documents may be representative of electronic documents that will be classified based on the classification rules (further described below). This may result in more effective classification rules.

In the example ofFIG.2, the training documents are processed in block204. For example, each training document may be filtered to clean textual content, including, for example, translating words to a desired language, stemming words to a root word, removing stop words (e.g., the, is, at, which, on, etc.) and/or low frequency words, etc. By filtering the training documents, efficiency of the statistical model and the search engine model may be increased. Alternatively, the training documents may not be processed and instead the method200may skip block204and utilize unprocessed training documents to create the classification rules.

As shown in block206, the method200employs a topic model engine as the statistical model. The topic model engine receives the processed training documents (of block204) and is able to discover topical categories discussed in the processed training documents. The number of topical categories may be preset to a constant value for each training document. The topic model engine may generate statistical data (e.g., using the topical categories) from the processed training documents. In the example ofFIG.2, the statistical data includes a document-topic matrix (shown in block208) and a topic-term matrix (shown in block210).

Because the topic model engine discovers the topical categories discussed in the training documents, the topical categories in the training documents do not need to be manually classified prior to inputting the training documents into the topic model engine. Accordingly, knowledge regarding potential topical categories found in the training documents may not be required in advance. Thus, unlabeled training documents (as explained above) may be employed. Additionally, because no advance knowledge regarding potential topical categories may be needed (besides the number of topical categories as explained above), the topic model engine can adapt to topical changes and discover new topical categories as needed.

In the example ofFIG.2, the topic model engine may be any suitable topic model engine including, for example, a probabilistic latent semantic indexing (PLSI) model, a latent Dirichlet allocation (LDA) model, etc.

The topic-term matrix (shown in block210) generated from the topic model engine is a probabilistic distribution of each topical category within the processed training documents over the terms (e.g., words) in these training documents. These probabilities put a weight on each term in the training document with regard to its relevance to the topical categories. For example, a term having a higher probability value is more likely to be chosen by an author discussing a given topical category than a term having a lower probability value.

FIG.4illustrates an example topic-term matrix400that may be created by the topic model engine. In the example ofFIG.4, the topic-term matrix contains fifteen topical categories in random order (shown as Topics1-15). A name for each topical category may be manually added based on the terms provided in a particular topical category. The names may be general so that more than one topical category has the same name. For example, as shown inFIG.4, the Topic4and the Topic5are both named “Energy.” Alternatively, the names may be specific. For example, the Topic4includes terms related to traditional energy and therefore may be named “Traditional Energy” and the Topic5includes terms related to green energy and therefore may be named “Green Energy.”

For each topical category, twenty terms are provided in a column. Alternatively, more or less topical categories and/or terms may be employed. The terms are ordered according to the probability that a particular term is used when discussing a particular topical category. For clarity, the probability value for each term is not shown. For example, the term “food” has a higher probability of being used when discussing Topic3than the term “organic.”

Referring back toFIG.2, the document-topic matrix (block208) generated from the topic model engine (block206) may be a distribution of the topical categories from the topic-term matrix (e.g., shown inFIG.4) for each training document. The prominence of a particular topical category in a document may be represented with a probability value from zero (0) to one (1). A topical category having a higher probability value means it has a higher relevance to a particular document.

FIGS.5aand5bare graphs500and502, respectively, representing sample document-topic matrixes generated from the topic model engine. The graphs500,502illustrate probabilities of particular topical categories of the topic-term matrix ofFIG.4being found in example documents. As shown in the example ofFIG.5a, the graph500illustrates a distribution of topical categories for the following example document (from, e.g., a media source):Starbucks Via Ready Brew Coffee costs about 1/10th of the price of a cup of coffee from a Starbucks store and contains anywhere from zero calories to upwards of 100 calories, depending on the flavor you get. The portable packets make it easy to enjoy a cup of coffee anywhere! Simply pour the packet into a cup, add hot water and stir! The Ready Brew line of coffee comes in 14 different flavors, from your regular Italian Roast to Ice Caramel to French Roast! Save calories, time and money! Product Features 100% Arabica coffee only 59 cents per packet from Amazon.com! Just add hot water and stir. Available in 14 flavors. Where to buy: Ready Brew coffee is available at your nearest Starbucks or you can have them shipped to you for FREE from Amazon.com! Also check out: 3 Tips To Healthy & Delicious Coffee It's no secret that I'm a big fan of coffee!
As shown in the example ofFIG.5a, the above document includes three topical categories (from the topic-term matrix400ofFIG.4) having a relatively high probability. The topical categories include Topic15(“Food”) having a probability value of about 0.475, Topic13(“Health”) having a probability value of about 0.25, and Topic9(“Online Shopping”) having a probability value of about 0.21. Thus, the example document is deemed relevant to the Topic “Online Shopping”, more relevant to the Topic “Heath”, and even more relevant to the Topic “Food.”

The example graph502ofFIG.5billustrates a topical category distribution of the following example document:Chrysler to sell natural gas-powered truck: Chrysler aims to be the first U.S. automaker to produce a factory-built pickup truck that is powered mainly by natural gas. The privately held company said Tuesday that its new Ram 2500 Heavy Duty CNG truck will be sold to commercial customers that operate truck fleets. The company expects to deliver the first trucks in July. The truck will have natural gas tanks and an 8-gallon fuel tank for gasoline. Chrysler said a small amount of gasoline is needed to start the truck, but after ignition it runs entirely on natural gas. If the natural gas tanks run out, the engine can switch to gasoline. Natural gas prices have dropped steeply over the last year thanks to higher production. Chrysler said the gas-powered trucks will save money over the long term.
As shown in the example ofFIG.5b, the above document includes two topical categories (from the topic-term matrix400ofFIG.4) having a relatively high probability. The topical categories include Topic5(“Energy”) having a probability value of about 0.475 and Topic12(“Auto”) having a probability value of about 0.4. Thus, this example document is deemed relevant to the Topic “Auto” and more relevant to the Topic “Energy.”

Referring back toFIG.2, the method200further includes (in block212) creating one or more topic model-based classification rules based on the document-topic matrix (block208) and the topic-term matrix (block210). For example, the topic model-based classification rules may be created by analyzing semantics of each topical category based on relevant terms (of each topical category) having the highest probability from the topic-term matrix (block210) and by analyzing a distribution of each topical category from the document-topic matrix (block208). Particular topical categories may then be identified and placed into one or more categories of interest. A category of interest may include particular topical categories that may be relevant to a particular user. The category of interest may be broad (e.g., entertainment, sports, etc.) or more specific (e.g. movie, baseball, etc.). For example, if a user is interested in the entertainment industry, a category of interest may include Topic6(“Entertainment”) and Topic10(“Entertainment”) of the topic-term matrix ofFIG.4.

After identifying the topical categories relevant to a category of interest, one or more topic model-based classification rules may be created to assign an electronic document to the particular categories of interest. For example, a topic model-based classification rule may be “IF the probability value of Topic6OR Topic10is above a certain threshold for document A, THEN document A is assigned to the Entertainment category of interest.”

The topic model-based classification rules created in block212may then be applied to and evaluated on one or more test documents. This is shown in blocks214and216. The test documents may be randomly selected from the training documents described above. Based on the topic model-based classification rules created in block212, each test document may be classified (e.g., assigned to a category as further explained below). This is shown in block214.

The classification results from block214may be evaluated in block216to ensure the topic model-based classification rules are suitable. In some embodiments, an individual may analyze the text of the test documents and ensure each test document is correctly classified. The topic model-based classification rules may then be evaluated by utilizing, for example, precision and/or recall equations to ensure the topic model-based classification rules meet a defined threshold. Equations (1) and (2) below are example equations for determining precision and recall, respectively.

Precision=N⁡(dc+)N⁡(dc+)+N⁡(dc-)(1)Recall=N⁡(dc+)N⁡(dc)(2)
In the example equations (1) and (2), N(dc+) is the number of test documents correctly classified to category C based on the topic model-based classification rules, N(dc−) is the number of test documents incorrectly classified to category C based on the topic model-based classification rules, and N(dc) denotes the total number of test documents that should be classified to category C based on human judgment.

If the results of the precision equation and/or the recall equation are below the defined threshold, the topic model-based classification rules may be deemed unsuitable. As a result, the topic model-based classification rules may be modified. The modified classification rules may be applied to the one or more test documents and evaluated again as explained above. This process of evaluating the topic model-based classification rules may be repeated until the topic model-based classification rules meet the defined threshold.

The topic model-based classification rules determined to be suitable may be identified and stored for future use as further explained below. For example, the topic model-based classification rules may be formatted and saved in XML files.

FIG.6illustrates a screen shot600of example topic model-based classification rules in XML format. As shown inFIG.6, the screen shot600includes three topical categories (Energy, Auto and Entertainment). A description of each topical category, a list of matched topics and a probability threshold for each matched topic are shown below each topical category. For example, the Energy category includes a description of “Topics about both traditional and green energy” and has two matched topics. The matched topics include Topic4and Topic5(ofFIG.4) that refer to green energy and traditional energy, respectively. The probability threshold for Topic4and Topic5is 0.20 and 0.30, respectively. Thus, if a document has a probability value above 0.20 for Topic4or above 0.30 for Topic5, the document is classified under the Energy category. Therefore, based on this topic model-based classification rule, the example document ofFIG.5bwould be classified into the Energy category because the document includes a probability value of about 0.475 for Topic5(as explained above).

Referring back toFIG.2, the method200further includes creating one or more query-based classification rules. In the example ofFIG.2, the method200employs a Lucene Engine (created by Apache Lucene™) as the search engine model to create the query-based classification rules. This is shown in block226. Therefore, each query-based classification rule is a Boolean Lucene query defined by Lucene query grammar.

Alternatively, any suitable search engine model may be employed to create the query-based classification rules without departing from the scope of the present disclosure. For example, the method200may employ a Sphinx search engine, a DataparkSearch search engine, a Terrier search engine, an OpenSearchServer search engine, etc.

As shown in block222, query-based classification rules may be created using one or more user defined ad hoc categories from block220. For example, for each user defined ad hoc category, a query-based classification rule may be created. The user defined ad hoc categories may be created for a particular user so that the categories are relevant to that particular user. For example, a user defined ad hoc category may be about a specific product or brand, an event regarding a product (e.g., a launch, a campaign, recall, etc.), etc.

The query-based classification rules are also created using the topic-term matrix generated in block210. In this way, relevant topics may be identified from the topic-term matrix for the query-based classification rule. Additionally, using the topic-term matrix assists in identifying terms used by authors when writing about the relevant topics.

In the example ofFIG.2, one or more test documents may be generated based on the processed training documents (block224). In some embodiments, the document-topic matrix (block208) and the topic-term matrix (block210) may also be utilized to generate the test documents. For example, the document-topic matrix and the topic-term matrix may identify general topics of each training document. This information may be used to generate more relevant, targeted, etc. test documents from the training documents.

As shown in block226, the Lucene Engine receives the query-based classification rules created in block222and the test documents generated in block224. The Lucene Engine indexes and matches the test documents to the query-based classification rules. The Lucene Engine then outputs results (e.g., a relevance score) reflecting the relevance of the test documents to the user defined ad hoc categories using the query-based classification rules.

The results from the Lucene Engine may be evaluated in block228to ensure each query-based classification rule is suitable (e.g., meets a defined threshold). For example, the generated test documents may be analyzed by an individual and marked as relevant or irrelevant for each user defined ad hoc category (from block220). Then, the query-based classification rule may be evaluated by utilizing precision and/or recall equations as explained above. If the results are unsuitable, the classification rule may be modified and evaluated again as explained above.

The query-based classification rules determined to be suitable may be identified and stored for future use as explained below. For example, the query-based classification rules may be formatted and saved in XML files. One example screen shot700of three query-based classification rules in XML format is shown inFIG.7.

In the example ofFIG.7, the three query-based classification rules are shown as Boolean Lucene Queries (“LuceneQuery”). The screen shot700also includes user defined ad hoc categories (e.g., from block220as explained above) and relevant topical categories from the topical categories generated by the topic model engine (block206). For example, the topical categories may be analyzed (e.g., manually by a research analyst) in view of the query-based classification rules and then relevant topical categories may be identified and saved as an element of one or more user defined ad hoc categories.

In the example ofFIG.7, the user defined ad hoc categories include “New England Auto Show,” Carbon Emissions” and “Car Recall.” The relevant topical categories for “Carbon Emission” are “Auto” (Topic12ofFIG.4) and “Green Energy” (corresponding to Topic4ofFIG.4as explained above). Therefore, only documents classified to topical categories “Auto” and “Green Energy” (as explained above with reference toFIG.6) may be matched to the Boolean Lucene query (i.e., the query-based classification rule) for “Carbon Emissions.”

FIG.3illustrates a flowchart of an example method300for classifying at least one electronic document using the topic model-based classification rules and/or the query-based classification rules created in the method200ofFIG.2.

As shown inFIG.3, the method300includes receiving electronic documents to be classified (block302). For example, the electronic documents may be received as streaming data. The electronic documents are then processed in block304. For example, each electronic document may be filtered to clean textual content as explained above. Alternatively, the electronic documents may not be processed and instead the method300may classify unprocessed electronic documents.

In some embodiments, the electronic documents may be processed (block304) in substantially the same manner as the training documents were processed in block204ofFIG.2. By processing the electronic documents in substantially the same manner, the accuracy of classifying the electronic documents may increase.

The processed electronic documents along with the topic-term matrix (block210) and the topic model-based classification rules (block218) created in method200may be provided to a topic predictor (shown in block306). In some embodiments, the topic predictor (block306) may be similar to the topic model engine (bock206) ofFIG.2. However, the topic predictor may not generate statistical data (e.g., the topic-term matrix and/or the document-topic matrix) as may be done by the topic model engine ofFIG.2. Instead, the topic predictor may receive the topic-term matrix (block210) as explained above.

In the example ofFIG.3, the topic predictor may receive the processed electronic documents from one or more other devices (e.g., servers, etc.) via a communication network (e.g., internet, intranet, etc.). In this way, the topic predictor may operate in real-time. Alternatively, the topic predictor may receive the processed electronic documents in any suitable manner including, for example, manually, via memory storing the processed electronic documents, etc.

The topic predictor may classify the processed electronic documents by assigning one or more topical categories to the electronic documents based on the topic model-based classification rules. In this way, more than one topical category may be assigned to a particular electronic document.

Alternatively, any suitable topic model engine may be employed to classify the processed electronic documents without departing from the scope of the present disclosure. For example, the method300may employ a probabilistic latent semantic indexing (PLSI) model, a latent Dirichlet allocation (LDA) model, etc.

The topic predictor may assign the topical categories by establishing a distribution of topical categories of a particular electronic document based on the topic-term matrix (block210). Then, based on the established topical distribution, the topic predictor assigns at least one of the topical categories of the topic-term matrix to the particular electronic document based on the topic model-based classification rules (block218). In this way, the method300classifies the electronic documents via the topic predictor.

As shown in block312, the method300then outputs the assigned topical categories for the particular electronic document.

In the example ofFIG.3, the method300employs the Lucene Engine (shown in block226) as used in the method200. The Lucene Engine may be employed to classify the processed electronic documents based on the query-based classification rules stored during the method200explained above. Alternatively, any suitable search engine model may be employed without departing from the scope of the present disclosure. For example, the method300may employ a Sphinx search engine, a DataparkSearch search engine, a Terrier search engine, an OpenSearchServer search engine, etc.

In the example ofFIG.3, the Lucene Engine receives one or more relevant query-based classification rules. As shown in block308, the relevant query-based classification rules are selected from the query-based classification rules (block230) based on the assigned topical categories from the topic predictor (block312). For instance, if an assigned topical category of an electronic document is “fashion” or “sport,” then it may be unnecessary to check the relevance of the electronic document with query-based classification rules regarding “car recall.” However, if an assigned topical category of the electronic document is “auto,” then query-based classification rules related to “car recall” should be selected. In the example ofFIG.3, the relevant query-based classification rules (block308) are selected automatically based on analysis completed when the query-based classification rules are created (e.g., during method200). Alternatively, the relevant query-based classification rules may be selected in any suitable manner.

For example, the method300may identify the relevant query-based classification rules by comparing the assigned topical categories from the topic predictor (block312) with the relevant topical categories saved as elements of the user defined ad hoc categories as explained above with reference toFIG.7. If an assigned topical category is the same as a relevant topical category, the method300selects the query-based classification rules (e.g., labeled “LuceneQuery” inFIG.7) created for the user defined ad hoc categories from which the relevant topical category is saved under. For example, if one of the assigned topical categories from the topic predictor is “Auto”, the method300selects the query-based classification rules (shown inFIG.7) created for the user defined ad hoc categories “New England Auto Show,” Carbon Emissions” and “Car Recall” ofFIG.7. By selecting only the relevant query-based classification rules, efficiency of the classification process may improve, accuracy of the classifying electronic documents may improve (e.g., by reducing query ambiguity caused by homonym words, etc.), etc.

The Lucene Engine then may categorize the particular electronic document from block304into at least one specific user defined ad hoc category (from the user defined ad hoc categories) using the selected relevant query-based classification rules (from block308). Therefore, the particular electronic document may be classified by both assigning topical categories to the particular electronic document and by categorizing the particular electronic document in specific user defined ad hoc categories in parallel.

The method300then outputs the specific user defined ad hoc categories for the processed electronic documents. This is shown in block310.

For example, if the selected relevant query-based classification rules correspond to the user defined ad hoc categories “New England Auto Show,” Carbon Emissions” and “Car Recall,” the Lucene Engine may categorize the particular electronic document into any of the three user defined ad hoc categories if the selected relevant query-based classification rules are satisfied. The Lucene Engine then outputs the user defined ad hoc categories used to categorize the particular electronic document. These user defined ad hoc categories are known as the specific user defined ad hoc categories.

Alternatively, if the particular electronic document does not satisfy any of the selected relevant query-based classification rules, the particular electronic document may not be categorized. As a result, the particular electronic document may be classified only by assigning topical categories to the particular electronic document (as explained above). Thus, the particular electronic document is classified only based on the topic model-based classification rules.

The assigned topical categories (block312) from the topic predictor and the specific user defined ad hoc categories, if any, (block310) from the Lucene Engine may be employed by a user to identify topics and documents relevant to that particular user. In turn, the user may utilize this information to more accurately collect data pertaining to their business, interests, etc. As a result, the user is able to monitor relevant discussions (e.g., positive and/or negative discussions regarding a product, false accusations, etc.), monitor new topics and/or issues, respond to disgruntled customers, alter protocols, etc.

Additionally, the created topic model-based classification rules and/or the query-based classification rules disclosed herein may be modified at any time during the process, including during or after the rule creating process (e.g., method200ofFIG.2), during or after the electronic document classification process (e.g., method300ofFIG.3), etc. For example, the topic model-based classification rules and/or the query-based classification rules may be modified after a period of time (e.g., after a period of minutes, hours, days, months, etc.). This in turn allows the topic model-based classification rules and/or the query-based classification rules to more accurately reflect current topics discussed in electronic media sources.

FIG.8illustrates an example computer system800for classifying an electronic document from an electronic media source. The computer system includes a communication network802and a computer server804in communication with the communication network802. The computer server804receives electronic documents from electronic media sources via the communication network802. In the example ofFIG.8, the computer server804includes memory806and a processor808.

The memory806may include training documents and user defined ad hoc categories as described above. The memory806may include onboard and/or external memory, including removable and/or remote memory. For example, the memory806may include cloud storage, disks, DVD, CD-ROMs, ROMs, RAMs, EPROMS, EEPROMs, DRAMs, VRAMs, flash memory devices, etc.

In the example ofFIG.8, the processor808generates statistical data from the training documents and creates at least one topic model-based classification rule using the statistical data and at least one query-based classification rule using the user defined ad hoc categories and the statistical data. Then, the processor808classifies the electronic document using the topic model-based classification rule. In some examples, the processor808may classify the electronic document using both the query-based classification rule and the topic model-based classification rule as explained above.

Alternatively, the processor808may only classify the electronic document using a topic model-based classification and, if applicable, a query-based classification rule created external to the system800. For example, the classifications rules may be created external to the system800(including the processor808) and saved in XML files for future use as explained above. If desired, the XML files may be stored in the memory806and the processor808may then classify the electronic document using the topic model-based classification and, if applicable, the query-based classification rule saved in the XML files.

Additionally, although not specifically described, the processor808may execute any of the example methods described herein. For example, the processor808may evaluate the classification rules on test documents stored in the memory806and then modify the classification rules if needed as explained above.

Further, the example methods disclosed herein may be implemented as computer-executable instructions. These computer-executable instructions may be stored on a non-transitory computer readable medium. In this way, the methods may be implemented in a computer program product stored on a computer readable medium. The computer readable medium may include for example, any type of disk including floppy disks, optical discs, DVD, CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems, including molecular memory ICs, or any type of system or device suitable for storing instructions and/or data.

For example, the non-transitory computer readable medium may include instructions for generating statistical data from one or more training documents, creating at least one topic model-based classification rule using the statistical data, and creating at least one query-based classification rule using one or more user defined categories and the statistical data. The instructions stored on the non-transitory computer readable medium may then be employed in a system (e.g., the system800) to classify an electronic document from an electronic media source.

The documents disclosed herein may include any suitable electronic document representing, for example, social media articles (e.g., a blog, a message, etc.), news articles (print or electronic), emails, etc. Additionally, the electronic media sources disclosed herein may include any suitable electronic media source including, for example, social media, news outlets, etc.

The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.