Leveraging word patterns in the language of popular influencers to predict popular trends

Popular trends are predicted by leveraging the language of influencers as found in their electronic publications such as social media, blogs, etc. A list of influencers in a given field is curated along with a lexicon of the field which includes product names and associated modifiers. Natural language processing is performed on the current publications to identify a particular word combination based on syntactic relationships. The current usage frequency of the particular word combination is compared to a historical usage frequency derived from a baseline. If the current usage frequency is significantly higher, an alert is generated indicating that the particular word combination represents a candidate trend. The word combination may be a syntactic n-gram. The current usage frequency is based on a first, recent time window, and the historical usage frequency is based on a second time window preceding the first time window.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention generally relates to natural language processing, and more particularly to a method of monitoring and analyzing the communications of influencers in a field to identify potential product trends.

Description of the Related Art

Predictive modeling is a well-established methodology across a diverse problems space. For example, impending mechanical failure of a complex system such as a diesel generator can be predicted by application of failure models to performance data streaming in real time. Collaborative and cooperative filters enable recommendation of consumer products to users based on retailer knowledge of prior consumer spending.

Other events are less conducive to prediction. For example, there is a cottage industry to predict commercial success of a new song, movie or book. While there is no doubt that previous commercial success of an artist suggests a ready audience for subsequent work (a new John Grisham novel often debuts as #1 on the New York Times bestseller list), much of this prediction is left to intuition of people with enormous depth in the industry. This approach is hardly scientific.

Other trends or fads can be detected on social media. A trending Twitter subject can be an indicator of enormous commercial success in the near term (“OMG, you simply MUST try x”). This source, however, is not useful in prediction or preparation; once the subject is trending on Twitter or Instagram the popularity wave is already cresting.

Although there is ample discussion of social media analytics, there is a very limited body of work on the use of media communications to predict medium term trend setting in the market place. The article “Fashion Supply Chains and Social Media: Examining the Potential of Data Analysis of Social-Media Texts for Decision Making-Processes in Fashion Supply Chains” by Beheshti-Kashi et al. considers the impact of blog posts on color choices by retail store buyers and customers. Their work suggests a strong relationship between blog posts and choices made by buyers for retail stores. Interestingly, their work also finds that blog information corresponds with real world customer demand. H. J. Fisher, in a master's thesis entitled “Food stylists' food image creation for print media and consumer interpretation: an exploratory investigation”, investigated the psychometric connections between choices made by professional food stylists and consumer food choice. He finds that food stylists, through non-verbal communication, can create images in print media that have the impact of altering consumer behavioral intent and eventual purchasing decisions. The paper “Social Media Competitive Analysis and text mining: A case study in the pizza industry” by He et al. (focused mainly on retail-level social media like Facebook) notes that more than half the people responding to a consumer survey by Market Force Information chose food options by reviewing online comments and reviews.

As analysis of social media becomes more complex, it is increasingly important to have a set of tools that provide a more intuitive understanding of user communications. As part of this effort, many systems employ some form of natural language processing. Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation allowing computers to respond in a manner familiar to a user. For example, a non-technical person may input a natural language question to a computer system, and the system intelligence can provide a natural language answer which the user can hopefully understand. Examples of an advanced computer systems that use natural language processing include virtual assistants, Internet search engines, and deep question answering systems such as the Watson™ cognitive technology marketed by International Business Machines Corp.

Deep question answering systems can identify passages from text documents (corpora) and analyze them in various ways in order to extract answers relevant to a query; answers can be scored on a number of factors, and the highest score indicates the “best” answer. Models for scoring and ranking the answer are trained on the basis of large sets of question and answer pairs.

One method of analyzing a natural language sentence is to construct a parse tree for the sentence. As the name suggests, a parse tree is a tree-like construct having branches and nodes (including a root node, interior or non-terminal nodes, and leaf or terminal nodes) whose arrangement and elements reflect the syntax of the input language. Syntax generally pertains to rules that govern the structure of sentences, particularly word order. Syntax is one set of rules that make up the grammar of a language. Grammar includes additional rules such as morphology and phonology. Syntax can help define relations between words in a statement, such as a noun being associated with an adjective or a prepositional phrase.

One aid in NLP involves the use of syntactic n-grams. An n-gram is a sequence of n items from text or speech (two items is a bi-gram, three items a tri-gram, etc.). Syntactic n-grams are n-grams defined by paths in syntactic dependency or constituent trees rather than the linear structure of the text. The paper “Syntactic N-grams as Machine Learning Features for Natural Language Processing” by Sidorov et al. promotes the use of syntactic n-grams over regular n-grams. Words in the n-gram are determined by syntactic relations in a parse tree rather than physical word order lifted directly from the text. This preserves “real” relations between words in a sentence, and lifts arbitrary constraints imposed by surface sentence structure.

SUMMARY OF THE INVENTION

The present invention in at least one embodiment is generally directed to a method of identifying a potential product trend in a consumer field by monitoring electronic media over time for current electronic publications of influencers in the field, performing natural language processing on the current publications to identify a particular word combination which has one product name and one or more modifiers based on a lexicon provided for the field, determining that a current usage frequency of the particular word combination is greater than a historical usage frequency derived from a historical baseline of word combinations, and responsively generating an alert indicating that the particular word combination represents a candidate trend. The natural language processing can identify the particular word combination based on a syntactic relationship between a particular product name and its modifiers, such as the word combination being a syntactic n-gram. The historical baseline can be based on occurrences of the word combinations from previous electronic publications of the influencers. In an exemplary implementation, to avoid spurious alerts the current usage frequency must be greater than a predetermined multiple of the historical usage frequency, and must also be greater than a sum of the historical usage frequency and a predetermined threshold value. The current usage frequency is based on a first time window, e.g., of no more than six months from the current date, and the historical usage frequency is based on a second time window preceding the first time window, e.g., of at least twelve months. Influencers can be selected based on different criteria, particularly geographic region.

The above as well as additional objectives, features, and advantages in the various embodiments of the present invention will become apparent in the following detailed written description.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Predicting new consumer trends is difficult for a variety of reasons. Desired data sources are inherently noisy, and a core requirement is a very large dataset spanning a considerable time period. Candidate detection search space is inherently large and sparse—there are no detectable clustering characteristics, although over time and with the right amount of data, precision over categories can be defined. Because of the sparseness of the data, standard forecasting/prediction methods such as ARIMA (Autoregressive Integrated Moving Average) models or ETS (Error, Trend, Seasonal) models are difficult to apply.

Part of the requirement for using predictive models with a time dimension is a steady time series of values taken at equally distant time intervals, which is impossible in the context of social media. It would, therefore, be desirable to devise an improved method of predicting consumer trends that are not especially conducive to prediction by standard analytics methods. It would be further advantageous if the method could offer early indications of an impending peak in popularity for a product, providing sufficient warning for manufacturers and retailers to prepare for increased consumer demand.

The present invention in various implementations meets these and other objectives by using natural language processing to extract certain word combinations from influencers' publications based on a product lexicon, and the compares a current usage frequency of a given word combination to a historical baseline. If the current usage frequency is significantly greater than the baseline, an alert can be generated to flag the word combination as a candidate product trend.

The method of the present invention can be generalized to predict any type of consumer trend. For purposes of explanation this disclosure uses the food industry as an exemplar but this topic should not be construed in any limiting sense. Food is a particularly useful example because it gets ample exposure through electronic media, particularly social media such as Instagram, Twitter, Facebook, etc., which can be automatically monitored by a computer system. The set of thought leaders, bloggers, industry influencers whose work is generally available in this case would include food critics for the local paper, food bloggers, columnists, etc. A lexicon of common food/flavor/ingredient names is generated as a source for the patterns. These names will be common ingredients or food product items such as beef, chicken, ice cream, eggs, etc. A list of sample modifiers for the product names is used to seed the process of syntactic n-gram extraction. Statistical entity and n-gram extraction, along with the cumulative nature of trend tracking according to the present invention, enable this functionality without excessive overhead. A historical baseline from the influencer list is also generated with corresponding time stamps. Food/flavor combinations are rarely truly novel so establishing a baseline is important. Ongoing monitoring of the published output of the influencer list (with corresponding time stamps) allows automatic detection of words on the source list, and extraction of every mention with adjacent combinations. The frequency of each extracted combination is noted, and when a marked increase is found for a particular combination over a relatively narrow time window (compared against baseline), it can be flagged for review by human experts.

Automatic detection and discovery of trend candidates from raw textual data on this scale is not possible today. Great human effort is required to do retrospective searching and curation, and that effort spent still does not allow for the wide perspective and aperture needed to identify trending items prior to mainstream awareness. Identifying a trend prior to mainstream awareness can provide companies with a competitive edge and lead time in bringing the appropriate offering to market.

With reference now to the figures, and in particular with reference toFIG. 1, there is depicted one embodiment10of a computer system in which the present invention may be implemented to carry out predictive analysis of consumer trends. Computer system10is a symmetric multiprocessor (SMP) system having a plurality of processors12a,12bconnected to a system bus14. System bus14is further connected to and communicates with a combined memory controller/host bridge (MC/HB)16which provides an interface to system memory18. System memory18may be a local memory device or alternatively may include a plurality of distributed memory devices, preferably dynamic random-access memory (DRAM). There may be additional structures in the memory hierarchy which are not depicted, such as on-board (L1) and second-level (L2) or third-level (L3) caches. System memory18has loaded therein one or more applications and/or data modules in accordance with the present invention, such as natural language processing, a historical baseline of word combinations, a list of available sources for influencers, and a monitor for such media.

MC/HB16also has an interface to peripheral component interconnect (PCI) Express links20a,20b,20c. Each PCI Express (PCIe) link20a,20bis connected to a respective PCIe adaptor22a,22b, and each PCIe adaptor22a,22bis connected to a respective input/output (I/O) device24a,24b. MC/HB16may additionally have an interface to an I/O bus26which is connected to a switch (I/O fabric)28. Switch28provides a fan-out for the I/O bus to a plurality of PCI links20d,20e,20fThese PCI links are connected to more PCIe adaptors22c,22d,22ewhich in turn support more I/O devices24c,24d,24e. The I/O devices may include, without limitation, a keyboard, a graphical pointing device (mouse), a microphone, a display device, speakers, a permanent storage device (hard disk drive) or an array of such storage devices, an optical disk drive which receives an optical disk25(one example of a computer readable storage medium) such as a CD or DVD, and a network card. Each PCIe adaptor provides an interface between the PCI link and the respective I/O device. MC/HB16provides a low latency path through which processors12a,12bmay access PCI devices mapped anywhere within bus memory or I/O address spaces. MC/HB16further provides a high bandwidth path to allow the PCI devices to access memory18. Switch28may provide peer-to-peer communications between different endpoints and this data traffic does not need to be forwarded to MC/HB16if it does not involve cache-coherent memory transfers. Switch28is shown as a separate logical component but it could be integrated into MC/HB16.

In this embodiment, PCI link20cconnects MC/HB16to a service processor interface30to allow communications between I/O device24aand a service processor32. Service processor32is connected to processors12a,12bvia a JTAG interface34, and uses an attention line36which interrupts the operation of processors12a,12b. Service processor32may have its own local memory38, and is connected to read-only memory (ROM)40which stores various program instructions for system startup. Service processor32may also have access to a hardware operator panel42to provide system status and diagnostic information.

In alternative embodiments computer system10may include modifications of these hardware components or their interconnections, or additional components, so the depicted example should not be construed as implying any architectural limitations with respect to the present invention. The invention may further be implemented in an equivalent cloud computing network.

When computer system10is initially powered up, service processor32uses JTAG interface34to interrogate the system (host) processors12a,12band MC/HB16. After completing the interrogation, service processor32acquires an inventory and topology for computer system10. Service processor32then executes various tests such as built-in-self-tests (BISTs), basic assurance tests (BATs), and memory tests on the components of computer system10. Any error information for failures detected during the testing is reported by service processor32to operator panel42. If a valid configuration of system resources is still possible after taking out any components found to be faulty during the testing then computer system10is allowed to proceed. Executable code is loaded into memory18and service processor32releases host processors12a,12bfor execution of the program code, e.g., an operating system (OS) which is used to launch applications and in particular the predictive analysis program of the present invention, results of which may be stored in a hard disk drive of the system (an I/O device24). While host processors12a,12bare executing program code, service processor32may enter a mode of monitoring and reporting any operating parameters or errors, such as the cooling fan speed and operation, thermal sensors, power supply regulators, and recoverable and non-recoverable errors reported by any of processors12a,12b, memory18, and MC/HB16. Service processor32may take further action based on the type of errors or defined thresholds.

Computer system10carries out program instructions for a predictive analysis process that uses novel natural language techniques to identify candidate trends. Accordingly, a program embodying the invention may additionally include conventional aspects of various NLP tools, and these details will become apparent to those skilled in the art upon reference to this disclosure.

Referring now toFIG. 2, there is depicted a high-level representation50of a predictive analysis system for identifying candidate consumer trends in accordance with one implementation of the present invention. System50is given a lexicon52for the particular product field which is the subject of the analysis. Lexicon52includes product names or keywords, and potential modifiers for those product names. Lexicon52may be created by one or more subject matter experts. In the food arena, lexicon52includes primary ingredient variants (steak, chicken, salmon, bananas, etc.). Since even a cursory look at food trends indicates broad coverage of ingredients, this list should be comprehensive. A product name can itself be more than one word, not considered as any modifier, like “potato chip” or “pork chop”.

System50monitors electronic media for publications54from a set of known influencers for the particular product field which is the subject of the analysis. The publications may include unstructured textual data. The sources of publications54may vary considerably. They can include web-based social media, blogs, etc., of the influencers themselves, or can be third-party (host) sites which the influencers may utilize, such as a restaurant review enterprise like Zomato. The cognitive system implementing the natural language processing used in the present invention can identify the names of reviewers to figure out that a particular review was generated by a known influencer. While the electronic media monitored is primarily from Internet sources, it could include other sources and even services that can electronically scan hard copy printouts to yield computer-readable information. Influencers can be anyone in a position of authority on the subject matter, including industry leaders, bloggers, newspaper critics, subject matter reviewers, periodical authors, and others having a high social profile in the field. The original list of influencers is preferably based on geographic boundaries (e.g., country, region, or state) as well as subject (e.g., food), and possibly based on subtopics (e.g., vegan or beverages).

A natural language processing system56(such as IBM's Watson) receives influencer publications54and automatically identifies word combinations therein pertaining to lexicon52based on syntactic relationships. It is virtually impossible to come up with a fully comprehensive dictionary of all possible word combinations in a field, in part because the standard rules-based technique falls apart with colloquialisms (“freek shake”) and longer descriptors with filler words (“cheese fries WITH ranch dressing AND A SIDE OF mash”). The goal with syntactic relations is to eventually create (with enough data from a defined domain) a model that predicts the type of syntactic relationships expected, which can then be used to perform extractions.

The number of allowable modifiers for a product name can vary according to implementation. Empirical evidence suggests that mentions of foods made popular by such influencers will follow a general language pattern, where the name of a food will be adjacent to 1-3 modifiers. The invention can leverage work on syntactic n-grams, allowing NLP system56to detect relationships between key ingredient mentions and their modifiers regardless of word choices by influencers. Thus, if a word pattern is found as a syntactic n-gram and that n-gram has one product name according to the lexicon and 1-3 modifiers according to the lexicon, it becomes a word combination whose usage is recorded. This is an especially important feature in the earliest stages of trend emergence, when individual influencers may not have agreed on a standard reference to a new combination of flavors.

As particular word combinations58are discovered in influencer publications54, their usage frequency can be computed by a monitor60, e.g., how many times that word combination appears per month. The time window for current usage frequency can vary according to implementation. For example, current usage frequency may be based on the number of times a particular word combination appears over the last 2 months or over the last 6 months. A comparator64analyzes the current usage frequency in light of a historical usage frequency from a baseline62. The time window for the historical usage can also vary, e.g., over the 12 months preceding the last 2 months, or over the last 24 months. The two time windows (current and historical) can overlap or can be mutually exclusive. The frequency comparison may consider multiple baseline time windows. Comparator64may allow for partial matches of word combinations, like just the two words “pork” and “salty” appearing together regardless of other modifiers (sub segmentation).

If the current usage frequency for a particular word combination is significantly higher than the historical baseline for that combination, an alert is generated informing an administrator66of a candidate trend. The determination of when current usage frequency is sufficiently higher than the baseline can also vary according to implementation. For example, a doubling of the usage frequency may be considered insufficient to generate an alert, but if the current usage frequency is more than ten times greater (or some other predetermined multiplier) than the historical frequency than a trend can be assumed. The system designer may want to avoid spurious alerts when the historical baseline shows a very small usage for some word combination and the current usage is comparatively higher but still not particularly significant, so may require that the absolute increase in usage is still above some predetermined threshold value. The breakout candidates can then go through further due diligence by product development or marketing teams.

FIG. 3illustrates an example of four statements from various influencer publications54that might be encountered when trying to predict a trend in fine dining. Each publication has a corresponding time stamp. Parsing these statements yields the following word combinations: “pork chip” with “cranberry”; “Bordeaux” with “grapefruit”; “quail egg” with “champagne vinegar”; and “lamb” with “sweet” and “chile de arbol” (the term according to lexicon52, which is equated to the phrase “arbol chile” in the influencer statement). Each occurrence of a word combination is recorded with its corresponding timestamp.FIG. 4shows how these word combinations might be analyzed in comparison to the historical baseline. Current usage frequency of the word combination “pork chop” plus “cranberry” is only slightly higher than historical usage, and not deemed to indicate any trend. Current usage frequency of the word combination “Bordeaux” plus “grapefruit” is slightly lower than historical usage, and also not deemed to indicate any trend. Current usage frequency of the word combination “quail egg” plus “champagne vinegar” (three per month) is more than double the historical usage (one per month), but the absolute number of occurrences is still small so not deemed to indicate any trend. Current usage frequency of the word combination “lamb” with both “sweet” and “chile de arbol” is roughly one hundred times higher than historical usage, so this word combination is flagged for further review as a likely trend.

The invention thereby enhances and extends the judgement of human experts in a field where experience is an essential element in trend prediction. It automatically extracts and flags for inspection sudden changes in popularity and discussion of an item by influencers in a field. Leveraging this impact can help manufacturers and distributors reliably predict the timing and intensity of market peaks ahead of general market awareness.

The present invention may be further understood with reference to the chart ofFIG. 5which illustrates the logical flow for a consumer trend prediction process80. Process80, which may be carried out on computer system10, begins by receiving the product lexicon (82) and receiving the list of influencers or electronic media sources (84). The cognitive system monitors the influencer publications (86), and performs NLP on those publications to identify particular word combinations based on syntactic relationships (88). A determination is then made as to whether the current usage frequency of any identify word combination is significantly greater than the historical frequency obtained from the baseline (90). If so, the alert is generated to flag a candidate trend (92).

Understanding and anticipating changes in consumer trends is central to most companies' continued successful growth and development. The present invention allows companies to understand the developing trends in a given market ahead of the curve. The system can continually process and update fresh results as new data is available, identifying breakout trend candidates that have a strong probability of gaining a disproportionate share of the consumer mindshare. The present invention scales this solution so that a comprehensive range of influencers can be monitored in near real time, and sudden changes in frequency of discussion referred to human experts for possible action.

Although the invention has been described with reference to specific embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. For example, the invention has been described in the context of the food industry but it is applicable to any product, not just goods but services as well. It is therefore contemplated that such modifications can be made without departing from the spirit or scope of the present invention as defined in the appended claims.