Providing alerts based on unstructured information methods and apparatus

A system, method, and apparatus for providing alerts based on unstructured information are disclosed. An example method includes receiving a data item from a remotely located information source, the data item including unstructured information. The method also includes determining a threat score for the data item by matching information associated with the data item to pre-identified information associated with a numerical value. The method further includes responsive to the threat score exceeding a predetermined threshold, creating a Common Alerting Protocol data structure that includes at least a portion of the information associated with the data item and transmitting the Common Alerting Protocol data structure.

BACKGROUND

Known moderation systems are oftentimes configured to filter and flag content from one source of information. This enables moderation systems to be uniquely configured based on the source of the information. Other known moderation systems are configured to filter and flag information from multiple information sources. However, these known moderation systems are configured to process only structured information generated from the sources.

For example, some information sources (e.g., the Department of Homeland Security and the National Oceanic and Atmospheric Administration) generate information according to a Common Alerting Protocol (“CAP”) format. This protocol defines a template for classifying information included within alerts that includes clearly labeled data fields that specify an urgency of an alert, a severity of an alert, a category of an alert, and a certainty of an alert. The protocol also defines how geographic locations associated with the alert are to be specified (e.g., by latitude and longitude). Similarly, other information sources may structure information according to a Specific Area Message Encoding (“SAME”) format or a Commercial Mobile Alert System (“CMAS”).

Current moderation systems use the predefined formats/templates of the information sources to effectively identify relevant information. For example, known moderation systems include algorithms that search for information labeled in CAP data fields (e.g., urgency, severity, certainty, and category) to determine how to score and route the information. These known algorithms are effective because the information is received in a known format.

However, these known moderation algorithms are ineffective processing unstructured information because there is no defined data format that identifies the importance of the information. As a result, known moderation systems disregard unstructured information. There accordingly exists a need to provide moderation of unstructured information.

SUMMARY

The present disclosure provides a new and innovative system, method, and apparatus for providing alerts based on unstructured information. In an example embodiment, a method includes receiving a data item from a remotely located information source, the data item including unstructured information. The method also includes determining a threat score for the data item by matching information associated with the data item to pre-identified information associated with a numerical value. The method further includes responsive to the threat score exceeding a predetermined threshold, creating a CAP data structure that includes at least a portion of the information associated with the data item and transmitting the Common Alerting Protocol data structure.

In another embodiment, a moderation system includes an interface configured to receive an unstructured data item from a data source and a semantic classifier configured to determine a first threat score by determining content within the data item that substantially matches pre-identified semantic information. The moderation system also includes a dictionary classifier configured to determine a second threat score by determining the content within the data item that substantially matches pre-identified dictionary information and a location identifier configured to determine a third threat score by determining the content within the data item that substantially matches geographic location information. The moderation system further includes a content scorer configured to determine whether the combination of the first, second, and third threat scores is greater than a predetermined threshold and a processor configured to create a Common Alerting Protocol data structure based on at least some of the content within the data item and the combined threat score responsive to the content scorer determining that the combined threat score is greater than the predetermined threshold.

While the example embodiments disclosed herein include generating CAP data structures from unstructured data items, it should be appreciated that other types of protocol-based data structures rather than CAP could be generated. Further, while the example embodiments disclosed herein use CAP data structures to provide alerts to clients, it should be appreciated that the example system may provide communication pattern identification for any type of unstructured information that may be reviewed by one or more moderators. The disclosed system may be configured to provide certain structured data outputs based on the context in which the system is implemented.

Additional features and advantages of the disclosed system, method, and apparatus are described in, and will be apparent from, the following Detailed Description and the Figures.

DETAILED DESCRIPTION

The present disclosure relates in general to a method, system, and apparatus to provide structured information, and in particular, to provide structured information based at least in part on unstructured information. The conversion from unstructured to structured information (and corresponding classification) enables data structures to be generated that are compatible with multiple structure-dependent systems. While the outputs are disclosed herein as CAP data structures (e.g., CAP alerts), it should be appreciated that in other embodiments the output can include Rich Site Summaries (“RSS”), flat files, databases, and/or XML.

As disclosed herein, unstructured information is information (e.g., text, pictures, graphs, charts, video, audio, images, maps, etc.) that is generally assembled in a document or data structure without identifying or labeling particular portions of the information. Unstructured information includes, for example general webpage information, audio/visual information, metadata, and social media content. An example of unstructured information (i.e., an online news article) is shown inFIG. 6.

For comparison, structured information is configured based on a template or a known format. Structured information often includes labels or data fields that identify how certain words or phrases within the information are to be interpreted. The labels are specifically named so that the structured information can be easily identified by an automated interpretation system. For example, an alert configured for a CAP format identifies the category of the alert and a geographic location affected by the alert. In contrast, an unstructured news article generally includes text and graphics. There is no indication as to the relevance of any of the text of graphics. Furthermore, individual terms or phrases within the article are not individually labeled or identified as pertaining to certain types of information.

In an example of unstructured information, an online media company publishes online articles to a webpage. While the text and pictures (e.g., information) within the article are formatted for viewing, the text and pictures are not assigned a label based on any protocol. In other words, the text and pictures are unstructured information. For instance, a severity or urgency of the information within the article is not labeled. At most, some online media companies may provide hyperlinks for certain words or phrases within the article to other articles or graphics. Additionally, some articles may include key words associated with the article or metadata embedded within the article. However, these key words and metadata are for search engines and are not structured information.

Unstructured information is generated by information sources including, for example, information channels and data feeds (e.g., RSS). Information channels are websites that provide information for display in a web browser. For example, media channels can include CNN, BBC, and MSNBC, security channels can include CHP, the Department of Homeland Security, and the Central Intelligence Agency, and weather channels can include the World Meteorological Organization, the Weather Channel, and the National Hurricane Center. Information channels can also include social media websites operated by, for example, Facebook, Twitter, Foursquare, Google, etc. Data feeds are streams of information provided by information sources. In some instances, an information source can function as an information channel and as a data feed.

In an example embodiment, a moderation system receives (or requests) unstructured information from many different information sources. The moderation system may also receive structured information. For each piece of unstructured information (e.g., a data item) the example moderation system performs an analysis to determine a threat score. The analysis includes a semantic comparison of words or phrases within the data item to pre-identified semantic words or phrases, which are each assigned a weighted score. The analysis also includes a dictionary comparison of words or phrases within the data item to pre-identified dictionary words or phrases, which are also each assigned a weighted score. The analysis may further include a determination of one or more geographic (or logical) locations associated with the data item.

The difference between the semantic comparison and the dictionary comparison is based on the context usage of the terms or phrases. For instance, a dictionary comparison searches for particular words (e.g., Nuclear, Hostage, War, Hurricane, etc.) regardless of the context. As long as there is a match in a dictionary comparison, a weighted threat score is assigned to the matching terms in the data item.

In contrast, a semantic comparison analyzes the usage of terms and phrases based on the context of and patterns within the unstructured information. Semantic comparisons can determine, for example, a tone or theme of unstructured information and assign a corresponding score. Semantic comparisons can also be applied to non-English based unstructured information for assigning threat scores. For example, a terrorist organization may use a hybrid version of multiple languages (e.g., a complex mélange of colloquialisms, cover terms, and natural language) to communicate information. After adequate calibration using representative examples of the unstructured information, semantic comparisons may be performed on this unstructured information despite the fact that the precise definitions of individual terms is unknown. As disclosed herein, the pre-identified information used for semantic comparisons is periodically updated to reflect new usage and patterns in human communication.

Returning to the example embodiment, after a threat score is determined for a data item, a moderation system determines whether the data item should be promoted to an alert. To make this determination, the moderation system compares the threat score to one or more thresholds. Responsive to the threat score exceeding a certain threshold, the moderation system formats at least some of the information within the data item to a CAP formatted data structure. In other words, the moderation system uses information that generated the greatest threat scores within the data item as the information that is included within the CAP data structure. Additionally, the moderation system creates labels with the CAP data structure including, for example, ‘urgency,’ ‘severity,’ ‘category,’ ‘certainty,’ and other specified labels based on the information within the data item and/or the threat score. The example moderation system transmits the CAP data structure to a decision system, which determines whether to alert one or more clients.

However, if the threat score does not exceed a threshold, the moderation system routes the data item to a moderator. The moderator then determines whether the data item should be promoted as an alert or discarded. Alternatively, the moderation system, instead of the moderator, may discard the data item if the threat score is below a certain threshold.

While the disclosed moderation system is described in conjunction with identifying information that poses a threat to a client, it should be appreciated that the moderation system can be used by any system that moderates or processes unstructured information. In other words, data structures generated by the moderation system may be created for virtually any format for direct throughput to databases, flat files (e.g., Excel files), RSS feeds, or other format schema (e.g., XML). For example, social media companies may use the example moderation system to quickly identify and remove objectionable content. In another example, publishers of media content oftentimes include a comments section at the bottom of an online article. The example moderation system disclosed herein can be used by these media publishers to identify objectionable comments to avoid offending viewers of the article.

Moderation Environment

The disclosed system may be readily realized in a moderation environment. A high level block diagram of an example moderation environment100is illustrated inFIG. 1. The illustrated system100includes a moderation system101that receives information from data sources102via a network103(e.g., the Internet). The information includes structured and unstructured information. The data sources102can include any host, server, processor, computer, cloud computing network that generates information. Each portion of received information is referred to herein as a data item104.

A data item104includes, for example, a compartmentalized piece of information that has a unique web address (e.g., a webpage), was generated during a session (e.g., a social media message), and/or was generated as a single instance of information. The data item104can also include a periodic stream of data from, for example, an RSS data feed. The data item104can further include any video, audio, image, map, graphical representation, and/or meme that is accessible via the Internet.

The example moderation system101may subscribe to one or more data sources102to automatically receive data items104. Alternatively, the moderation system101may query the data sources102for data items104. The query may be performed hourly, daily, weekly, monthly, etc.

As described in conjunction withFIG. 2, the example moderation system101classifies unstructured data from the data sources102to determine a threat score. Based on the threat score, the moderation system101either generates a CAP alert106that includes information included within a scored data item104, discards the data item104, or routes the data item104to a moderator. The moderator then decides whether the data item should be promoted to an alert or discarded.

Any CAP alerts106generated by the moderation system101are transmitted to a decision system108. The example decision system108aggregates CAP alerts106to determine whether there is a security or safety risk for one or more subscribing clients. The decision system108may operate as a combination of machine intelligence with human operators to determine when a situation warrants action. If the decision system108determines that a client is at risk, the decision system108transmits one or more alerts to, for example, a client device110(e.g., any smartphone, tablet computer, laptop, personal computer, server used by a client), a client112(e.g., individuals or an organization), or a security agency114hired to protect the client112. The security agency114can also include a public security force.

While the moderation system101is shown as a single entity, it should be appreciated that the system may be comprised of one or more servers, processors, etc. Further, the moderation system101may be distributed within a cloud computing network. Alternatively, the moderation system101may operate in conjunction with an application server or other host that includes content that could be moderated.

Moderation System

FIG. 2shows an example functional diagram of the moderation system101ofFIG. 1. It should be appreciated that the diagram shown inFIG. 1is only one example of implementing the moderation system101. In other embodiments, the functional blocks may be combined, removed, rearranged or expanded. For instance, the moderation system101may additionally include components to format information from data items for other types of protocols rather than CAP.

To compile information from data sources102, the example moderation system101includes a content query interface202and a content listener interface204. The example content query interface202is configured to search websites (and other Internet accessible content) for information (structured and unstructured) generated by the data sources102a. The content query interface202may also subscribe to data sources102afrom which data items are periodically pulled. For example, the content query interface202may be configured to access and pull data items from different data sources at different time intervals such that more relevant websites and data feeds are accessed more frequently. It should be appreciated that this configuration is useful for gathering information from data sources102athat frequently publish information.

The example content listener interface204is configured to receive content (e.g., data items) pushed from data sources102b. For example, the content listener interface204may register to receive data items from one or more Twitter users, hashtags, and/or keywords. Thus, the content listener interface204receives data items as the information is generated. This configuration is particular useful for receiving data items from numerous accounts that may only periodically publish information.

The example content query interface202and the example content listener interface204are configured by personnel of the moderation system101to access specifically identified data sources102. For example, personnel may specify that the content query interface202is to access a specific list of websites and data feeds for data items. In the same manner, personnel specify which social media accounts are to be subscribed to by the content listener interface204. Additionally or alternatively, the content query interface202and/or the content listener interface204may include one or more algorithms that select which data sources102are accessed based on determinations as to which data sources previously provided relevant data items. For instance, after a period of time, the moderation system101compiles a data structure of the top100data sources102that generated data items used in alerts. The interfaces202and204are accordingly configured to access at least (or at most) these top100data sources.

To process unstructured data items104compiled by the interfaces202and204, the example moderation system101includes analysis components (e.g., a semantic classifier206, a dictionary classifier208, a location identifier210, and a content scorer212). These components are configured to analyze the data items for content that matches pre-identified content to determine a threat score. In some embodiments, each component may assign a score. Alternatively, each data item is assigned one threat score based on the analysis of the components206to212.

It should be noted that structured data is not processed through the components because the relevant information is already labeled and identified. Instead, the moderation system101routes structured data items directly for generation into a CAP (or other specified type) data structure. Alternatively, the moderation system101may include a separate module for receiving and processing structured data items.

In the illustrated example ofFIG. 2, the pre-identified content is selected based on known terms, phrases, and semantics that correspond to conditions or events that could be a concern to subscribing clients. The pre-identified content may also be selected or acquired from external resources (e.g., online dictionaries). In some examples, the pre-identified content may be specific for a particular client. Alternatively, the pre-identified content is common (or at least considered) for all clients.

It should be appreciated that these components are configured to process many thousands of data items within relatively short periods of time. For example, the interfaces202and204typically compile thousands of data items a minute. The components206to212accordingly have to process these thousands of data items within the same time period. Otherwise, the moderation system101would generate a backlog of data items, thereby reducing the effectiveness of the system.

In particular, the semantic classifier206is configured to analyze data items based on context, language, and patterns of terms and phrases to assign a threat score. The semantic classifier206uses one or more machine learning algorithms to identify relevant content within the data items. For example, the semantic classifier206may include a Waikato Environment for Knowledge Analysis (“Weka”) algorithm to match pre-identified content to content within data items.

The example semantic classifier206is calibrated by a calibration processor214, which uses days, weeks, or years of training data (e.g., previously received data items and/or specifically designed data items) to determine how information is to be classified. For example, the calibration processor214compiles two or three weeks of data items associated with a particular issue (e.g., weather, terror, political instability, etc.). The calibration processor214then analyzes these data items using natural language algorithms to identify trends, patterns, and semantics. Moderators may also tune the calibration processor214based on initial results to provide more accurate classifications of information. For example, moderators may instruct the calibration processor214to consider one or more phrases that were not initially flagged or flagged with a relatively low threat score. Additionally, moderators may review the data items with the 100 greatest threat scores and determine if the threat score is warranted. The calibration processor214receives feedback of any changes to threat scores and adjusts weighting of pre-identified content, removes pre-identified content, and/or adds pre-identified content as appropriate.

In addition to defining pre-identified content, the example calibration processor214determines a threat score for each piece of content. The threat scores are weighted numerical values that correspond to how likely that particular term, phrase, or data item in general is a threat to one or more clients. The semantic weighting of pre-identified information means that, for example, a phrase that includes the word ‘nuclear’ is weighted relatively higher when the context is North Korea or a terrorist organization compared to when the context is power generation. Further, threat scores may be generated for each label of a CAP data structure, including generating a separate score for urgency, severity, category, and certainty.

For example, the calibration processor214may perform a calibration routine for pirate activity on the Horn of Africa. The calibration processor214analyzes data items to determine common usage of terms, tone, and patterns for identifying which data items include relevant new information of concern (e.g., a government alert reporting a recent hijacking of a cargo vessel), which data items include reports of past events (e.g., an online news article recapping a hijacking attempt), and which data items are irrelevant (e.g., a sports article about the Pittsburgh Pirates). In this instance, the calibration processor214may search for dates in comparison of a date the data item was generated, the tense (present or past) of terms, and patterns regarding how breaking news is worded compared to how general reports are worded.

The calibration processor214also identifies terms which indicate that the data item is not relevant. This can include, for example, determining that an article is related to baseball by locating semantic pre-identified baseball information within the data item. This matching pre-identified baseball information may be associated with a negative or relative low threat score when the context of the analysis is illegal pirate activity in Africa. The low threat score applied to the article by the semantic classifier206causes the data item to be discarded.

In another example, the calibration processor214may operate a calibration routine for a new pseudo-language communications being employed by a terrorist group. The calibration processor214analyses messages, for example, over social media to identify patterns and determine meanings of phrases (e.g., determine pre-identified information). The semantic classifier206uses this information to then assess whether a data item corresponds to the terrorist group. As a result, an increase in alerts (corresponding to a detected increase in data items generated by the terrorist group) may signal that a terrorist event is about to occur. Thus, while the exact translation of the data items is not available, the calibration processor214, in conjunction with the semantic classifier206, uses compiled pre-identified information to assign an appropriate threat score.

In some embodiments, the semantic classifier206may include additional analytic algorithms including, for example, algorithms provided by Entegra, Analysts Notebook, Palantir, FalconView, etc. to provide further analysis of information within the data item. For instance, these algorithms may perform link/network analysis and/or text extraction to identify, for example, meanings within pseudo-language communications to determine threat score(s). Alternatively, CAP data structures may be analyzed by these algorithms.

After being calibrated, the example semantic classifier206analyzes data items by comparing the data items to pre-identified semantic information. The content scorer212then determines a threat score of the semantic classification based on the matching pre-identified information. In some instances, the content scorer212references threat scores for the matching pre-identified information and adds each of the threat scores to determine a total semantic threat score for the data item. Alternatively, the content scorer212analyzes the matching pre-identified information and determines a total semantic threat score for the data item based on one or more algorithms configured to quantify information

The example content scorer212also routes the data item after determining a threat score for the data item. In some embodiments, the content scorer212routes the data item after determining only the semantic threat score. For example, responsive to determining that the semantic threat score exceeds a threshold, the content scorer212routes the data item to be structured into a CAP alert. In this instance, the data item is created into an alert without the dictionary classifier208analyzing the data item. Further, if the semantic threat score is below a threshold, the content scorer212routes to the data item to the dictionary classifier208and/or the location identifier210. Alternatively, the content scorer212may route the data item to a moderator216.

The example dictionary classifier208ofFIG. 2is configured to compare terms (or phrases) with data items to a data structure218that includes pre-identified terms (or phrases). The pre-identified terms are selected by personnel (e.g., personnel working in conjunction with the moderator216) based on previous data items and/or trending current events. The pre-identified terms may also include names, slang, initials, or other groupings of characters.

For instance, the moderator216updates the data structure218with the name of a newly elected world leader or a newly formed terrorist organization. The data structure218is dynamically updated for each processed data item. Additionally, the data structure210may include a thesaurus, which is similarly updated by collective moderation and/or administration. The example data structure218ofFIG. 2may be implemented by any computer-readable medium, including RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media.

The example dictionary classifier208operates in conjunction with the content scorer212to determine either a total threat score or a dictionary threat score. The content scorer212determines a dictionary score based on individually weighted scores for matching pre-identified terms, phrases, or expressions. The scores may be weighted based on a number of occurrences of, for example, a matching term within a data item. The scores may also be weighted based on the term or phrase. For example, the term ‘ballistic missile’ corresponds to a greater weight than the term ‘protest.’

In addition to performing semantic and dictionary classification, the example moderation system101ofFIG. 2analyzes data items using the location identifier210. The example location identifier210is configured to analyze unstructured data items for terms or expressions that identify geographic locations. The place names of the geographic locations may take the form of any combination of organically evolved, web service, and/or other external lexicon, and may be referenced on a pre-determined and/or prioritized basis. For example, the location identifier210searches the text and metadata of a data item for the names of cities, states, countries, regions, etc.

In some embodiments, the location identifier210operates in conjunction with the content scorer212to determine a location threat score. In these embodiments, certain locations (e.g., Iran, North Korea, Egypt, etc.) may correspond to a greater threat score than other locations. Alternatively, the location identifier210operates in conjunction with the classifiers206and208to determine a location in addition to semantics and dictionary terms.

As discussed above, the content scorer212routes a data item based on a threat score. Data items that have a threat score that is below a certain threshold are routed to a content remover220, which is configured to purge or delete the data item. In some embodiments, the content remover220caches data items220for a predetermined time period before deleting them. For example, the content remover220may cache data items for three days to ensure that a data item was not inadvertently discarded by the content scorer212. During this time, personnel working in conjunction with the moderator216analyze the cached data items to determine if any should have instead been promoted to an alert.

Data items that have threat scores greater than the discard threshold but less than a promotion threshold are routed by the content scorer212to the moderator216. The example moderator216includes one or more personnel that manually review data items. The moderator216ofFIG. 2may include components to manage how personnel review the data items. For instance, the moderator216can include queue management that routes data items to the appropriate personnel (e.g., based on location information, threat category, threat score, etc.) and places data items within a queue based on, for example, the threat score.

Personnel associated with the moderator216review and determine whether data items are routed to the content remover220or promoted to a CAP alert. The moderator216may also transmit messages to the calibration processor214and/or the data structure218based on which data items were removed or promoted. The messages can include an identifier of the data item and/or the contents of the data item in conjunction with a threat scores (or semantic threat score separated from the dictionary threat score).

To provide CAP alerts to the decision system108ofFIG. 1, the example moderation system101ofFIG. 2includes a CAP processor222and a CAP transmitter224. The example CAP processor222is configured to convert a data item (and/or the corresponding threat score) into a CAP data structure used as an alert by the decision system108. To create a CAP data structure, the example CAP processor222formats at least some of the information within the data item into a CAP template.

As discussed above, a CAP template includes an ‘urgency’ field, a ‘severity’ field, a ‘certainty’ field, and a ‘category’ field. In addition, the CAP template includes a ‘geographic location’ field, a ‘headline’ field, an ‘information source field’, and a ‘description’ field. The example CAP processor222selects information that was identified in the classifiers206,208, and210to use as information for the fields. The CAP processor222may also use the threat score to determine how descriptors for the urgency field, severity field, certainty field, and category field are selected. Alternatively, the CAP processor222uses individually determined threat scores for each of the fields to determine the descriptor.

In an example, the severity field can be labeled as one of ‘Concern,’ ‘Caution,’ ‘Severe,’ ‘Disastrous,’ etc. The CAP processor222selects the correct label either based on a threat score corresponding to the severity level (e.g., a severity threat score corresponding to the ‘Caution’ label) or information within the data item indicative of the severity (e.g., terms such as ‘wide spread damage,’ ‘winds exceeding 100 miles an hour,’ etc.). Similarly, the CAP processor222selects labels for other fields within the CAP data structure.

The CAP processor222selects which information is to be included within a ‘description’ field based on which portions of the data item have the most terms or phrases that match pre-identified content. Additionally or alternatively, the CAP processor222selects introductory portions of the data item for the ‘description’ field. Further, the CAP processor222uses location information identified by the location identifier210to include within the ‘geographic location’ field. In some instances the CAP processor222may determine a latitude and longitude for the ‘geographic location’ field using the identified location information.

Moreover, the CAP processor222may append any images, video, audio, etc. to the CAP data structure. The CAP processor222may append the multimedia content by including a link (e.g., a hyperlink) to the content within the CAP data structure. Alternatively, the CAP processor222may attach (or store) the multimedia content to the CAP data structure.

The CAP processor222ofFIG. 2may also be configured to filter redundant CAP data structures from being transmitted to, for example, the decision system108. For example, the CAP processor222may cache for a predetermined period of time (e.g., days, weeks, months, etc.) data items and/or CAP data structures that were sent to the decision system108. The CAP processor222may compare each newly received data item (or newly created CAP data structure) to the cached information to determine if the same instance of information has already been transmitted. Any information that is redundant may be disposed by the CAP processor222, thereby reducing the amount of redundant information processed by the decision system108. For example, Associated Press or Reuters articles are republished by different online publishers. The CAP processor222accordingly filters redundant data items from different data sources102based on the same news story.

The example CAP processor222may further be configured to determine which client(s) is associated with a CAP data structure. To determine the client, the CAP processor222may match client employee names, client facility locations, client product names, etc. to information within a data item. For example, the CAP processor222may identify that all CAP data structures associated with Nogales, Mexico are to be transmitted to client(s) with a manufacturing plant within that city. Alternatively, the decision system108associates CAP data structures with particular clients. In these alternative embodiments, the CAP processor222may include within the CAP data structure the name(s) of the associated clients.

The example CAP transmitter224of the illustrated example is configured to transmit CAP data structures to the decision system108ofFIG. 1. The CAP data structures are sent as CAP alerts to be further processed by the decision system108to determine if an action is to be performed. The example CAP transmitter224may be configured to transmit CAP data structures as they are received from the CAP processor222. Alternatively, the CAP transmitter224may periodically transmit CAP data structures (e.g., every five minutes, ten minutes, etc.).

It should be appreciated that while the moderation system101ofFIG. 2was described as processing text, the system is also configured to generate CAP data structures by analyzing data items that include unstructured video, images, audio, maps, etc. For example, the location identifier210may use image recognition to determine a geographic location from one or more images. In another example, the semantic classifier206may be configured to analyze video to identify a riot. Moreover, the dictionary classifier208may be configured to convert audio to text (and textual sounds) and compare this converted audio to pre-identified text.

It should be appreciated from the moderation system101ofFIG. 2that relatively greater amounts of processing and analysis are needed to determine whether an alert should be generated based on unstructured data. For instance, the moderation system101analyzes substantially all content within a data item to determine a threat score. This includes metadata, text, pictures, video, audio, etc. In comparison, structured information already has the appropriate information configured into known template fields and labeled. As a result, relatively less processing is necessary to determine whether an alert should be generated based on structured information.

Processor

A detailed block diagram of electrical systems of an example computing device (e.g., a moderation system101, a decision system108, and/or components202-224) is illustrated inFIG. 3. In this example, the moderation system101, the decision system108, and/or any one of the components202-224includes a main unit302which preferably includes one or more processors304communicatively coupled by an address/data bus306to one or more memory devices308, other computer circuitry310, and one or more interface circuits312. The processor304may be any suitable processor, such as a microprocessor from the INTEL PENTIUM® or CORE™ family of microprocessors. The memory308preferably includes volatile memory and non-volatile memory. Preferably, the memory308stores a software program that interacts with the other devices in the moderation environment100, as described above. This program may be executed by the processor304in any suitable manner. In an example embodiment, memory308may be part of a “cloud” such that cloud computing may be utilized by the moderation system101, the decision system108, and/or any one of the components202-224. The memory308may also store digital data associated with pre-identified content/information (and corresponding threat scores), data items, CAP data structures, etc. retrieved from the moderation system101, the decision system108, any one of the components202-224and/or loaded via an input device314.

The example memory devices308store software instructions323, data items104, data source interface features, CAP data structures, permissions, protocols, configurations, pre-identified content, calibration instructions, and/or application interface information326(e.g., information for accessing and using data items from data sources102). The memory devices308also may store network or system interface features, permissions, protocols, configuration, and/or network interface information328(e.g., information associated with the interfaces202and204) for use by the moderation system101, the decision system108, and/or any one of the components202-224. It will be appreciated that many other data structures and records may be stored in the memory device308to facilitate implementation of the methods and apparatus disclosed herein. In addition, it will be appreciated that any type of suitable data structure (e.g., a flat file data structure, a relational database, a tree data structure, etc.) may be used to facilitate implementation of the methods and apparatus disclosed herein.

The interface circuit312may be implemented using any suitable interface standard, such as an Ethernet interface and/or a Universal Serial Bus (USB) interface. One or more input devices314may be connected to the interface circuit312for entering data and commands into the main unit302. For example, the input device314may be a keyboard, mouse, touch screen, track pad, track ball, isopoint, image sensor, character recognition, barcode scanner, microphone, and/or a speech or voice recognition system.

One or more displays, printers, speakers, and/or other output devices316may also be connected to the main unit302via the interface circuit312. The display may be a cathode ray tube (CRTs), a liquid crystal display (LCD), or any other type of display. The display generates visual displays generated during operation of the moderation system101, the decision system108, and/or any one of the components202-224. For example, the display may provide a user interface and may display CAP data structures provided by the moderation system101. A user interface may include prompts for human input from personnel associated with the system101including links, buttons, tabs, checkboxes, thumbnails, text fields, drop down boxes, etc., and may provide various outputs in response to the user inputs, such as text, still images, videos, audio, and animations.

One or more storage devices318may also be connected to the main unit302via the interface circuit312. For example, a hard drive, CD drive, DVD drive, and/or other storage devices may be connected to the main unit302. The storage devices318may store any type of data, such as pre-identified content, threat scores, security data, etc., which may be used by the moderation system101, the decision system108, and/or any one of the components202-224.

The moderation system101, the decision system108, and/or any one of the components202-224may also exchange data with other network devices320via a connection to the Internet or a wireless transceiver322connected to the Internet. Network devices320may include one or more servers, which may be used to store certain types of data, and particularly large volumes of data which may be stored in one or more data repository. A server may include any kind of data including databases, programs, files, libraries, records, images, documents, requirements, transaction data, operations data, configuration data, index or tagging data, data items, unstructured information, statistical data, security data, etc. A server may store and operate various applications relating to receiving, transmitting, processing, and storing the large volumes of data. It should be appreciated that various configurations of one or more servers may be used to support and maintain the environment100. For example, servers may be operated by various different entities. Also, certain data may be stored in the moderation system101, the decision system108, and/or any one of the components202-224which is also stored on a server, either temporarily or permanently, for example in memory308or storage device318. The network connection may be any type of network connection, such as an Ethernet connection, digital subscriber line (DSL), telephone line, coaxial cable, wireless connection, etc.

Access to the moderation system101, the decision system108, and/or any one of the components202-224can be controlled by appropriate security software or security measures. An individual users' access can be defined by the moderation system101, the decision system108, and/or any one of the components202-224and limited to certain data and/or actions. Accordingly, users of the moderation environment100may be required to register with one or more of the moderation system101, the decision system108, and/or any one of the components202-224.

Flowchart of Example Data Item Analysis

FIGS. 4 and 5show a flow diagram illustrating example procedures400,420, and460to determine whether an unstructured data item should be promoted to a CAP alert, according to an example embodiment of the present invention. The example procedures400,420, and460may be carried out by, for example, the data source102, the moderation system101, and/or the decision system108described in conjunction withFIGS. 1 to 3. Although the procedures400,420, and460are described with reference to the flow diagrams illustrated inFIGS. 4 and 5, it will be appreciated that many other methods of performing the functions associated with the procedures400,420, and460may be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. For instance, the example procedure420could include a step of identifying a client associated with data item and/or a CAP data structure prior to transmitting the CAP data structure.

The procedure400begins when a data source102receives a request message401to transmit one or more data items to moderation system101(block402). The request message401can identify webpage addresses, accounts, or any other identifier of Internet accessible information. The data source102then identifies and transmits the data item(s)104to the moderation system101(block404). For example,FIG. 6shows an example of an unstructured data item104transmitted to the moderation system101. The example data item104is a news article posted to a website that includes text and a map. In other examples, the data item104can be an RSS feed of information, social media messages, or any other piece of unstructured information available via the Internet.

The example procedure400then returns to block402when another request for data item(s) is received. In other embodiments, the data source102periodically transmits data items to the moderation system101based on the request message401. Alternatively, the example procedure400ends.

The example moderation system101then performs a semantic classification, a dictionary classification, and location identification, as described above in conjunction withFIG. 2(blocks426,428, and430). The moderation system101also determines a threat score based on the classifications (block432).FIG. 7shows a diagram of an example data structure700that graphically shows how threat scores are compiled and calculated for the data item104ofFIG. 6. It should be noted that the data structure700is only representative in nature to graphically illustrate how threat scores are compiled based on a data item104. In other embodiments, the moderation system101may store and/or associate a determined threat score with the data item104instead of creating a separate data structure.

In this example, the semantic classifier206scans the data item104for patterns, tones, and language that match pre-identified content/information. For instance, the semantic classifier206determines that the phrases from the data item104ofFIG. 6shown inFIG. 7under the ‘Schematic’ section match pre-identified content. Next to each phrase is a corresponding threat score that is associated with the matching pre-identified content. In this example, the semantic threat score is 12.

The data structure700ofFIG. 7also includes a ‘Dictionary’ section that identifies terms from the data item104ofFIG. 6that match pre-identified terms having associated threat scores. The numbers in parenthesis correspond to a number of times the term was included within the data item104, which may be used to weight the threat score. In this example, the dictionary threat score is 21.5.

The data structure700also includes a ‘Location’ section that identifies geographic locations referenced in the data item104. It should be noted that the location identifier210also analyzes the map shown inFIG. 6in addition to the text of the article. In this example, the location threat score is 12.9.

After determining threat scores for each section, the moderation system101combines the threat scores to generate a total threat score (e.g.,46.4). In other embodiments, the moderation system101determines a total threat score without determining threat scores for separate sections. In yet other embodiments, the moderation system101determines a threat score for each field (e.g., certainty, severity, category, and urgency) associated with a CAP template.

Returning to the example procedure420ofFIG. 5, after determining the threat score, the moderation system101determines whether the threat score is greater than a first threshold (block434). Responsive to determining that the threat score is greater than the first threshold, the moderation system101creates a CAP data structure435based on at least some of the content or information within the data item104(block436).FIG. 8shows a diagram of a graphical representation of the CAP data structure435. It should be noted that in this example the CAP data structure435is only a representative embodiment and was not actually generated from the data item104ofFIG. 6. It should also be noted that the CAP data structure435includes fields for certainty, severity, category, and urgency. In addition, the CAP data structure435includes a geographic field, a title field, and a description field that includes text from, for example, a data item. Moreover, the CAP data structure435includes a field for displaying a map, which corresponds to a map shown in a data item. Alternatively, the moderation system101may generate the map shown in the CAP data structure435based on identified geographic information. In other embodiments, the CAP data structure435includes a section for displaying video or playing audio that was included within a data item.

In the illustrated embodiment ofFIG. 5, the moderation system101transmits the generated CAP data structure435as a cap alert (block438). In some embodiments, the CAP data structure435is transmitted to the decision system108ofFIG. 1. Alternatively, the CAP data structure435is transmitted to a client112or a client device110. After transmitting the CAP data structure435, the example procedure420returns to block422to request additional data items104. Alternatively, the example procedure420ends. It should be appreciated that in some embodiments the moderation system101receives data items while at the same time classifying other data items and creating CAP data structures from yet other data items.

Returning to block434, if the threat score is not greater than the first threshold, the moderation system101determines if the threat score is less than a second threshold, which is lower than the first threshold (block440). Responsive to determining that the threat score is less than a second threshold, the moderation system101discards the data item104(block442). The example procedure420then returns to block422to request additional data items.

However, in block440, if the threat score is greater than the second threshold, the moderation system101transmits the data item104to personnel working in conjunction with moderator216(block444). Based on an analysis by the personnel, the moderation system101determines whether the data item is to be promoted to an alert (block446). If the data item104is not to be promoted, the moderation system101provides feedback regarding the data item to be discarded, thereby causing the calibration processor214and/or the classifiers206,208, and210to amend the appropriate threat scores and/or pre-identified content/information (block448). The example procedure420then discards the data item (block442) and returns to block422to request additional data items.

In contrast, if the data item is to be promoted, the moderation system101provides feedback regarding the promotion, thereby causing the calibration processor214and/or the classifiers206,208, and210to amend the appropriate threat scores and/or pre-identified content/information (block450) based on the promotion. In other words, the calibration processor214and/or the classifiers206,208, and210increase a threat score (e.g., a weight of a threat score) for the pre-identified information that matches information within the data item104. The example procedure420then executes blocks436and438to create and transmit the CAP data structure435. The example procedure420then returns to block422to request additional data items.

The example procedure460ofFIG. 4begins when a decision system108receives a CAP data structure435from, for example, the moderation system101(block462). The decision system108then analyzes the CAP data structure435as an alert in conjunction with other alerts (block464). It should be appreciated that the alerts include other CAP data structures generated from previously processed unstructured data items. The alerts can also include CAP data structures received from structured data items.

In this embodiment, the example decision system108identifies and notifies a client based on the alerts (block466). The procedure460then returns to block462for the next CAP data structure. In other embodiments, the decision system108may not notify a client if the aggregation of the alerts is not serious for that client. The example procedure460continues to operate as new CAP data structures are received.

CAP Data Structure Embodiment

As discussed above, a CAP data structure includes content from a data item accordingly labeled based on predefined template data fields.FIG. 8shows a graphical example of one such CAP data structure. The XML code shown below includes a CAP data structure implemented as a series of instructions. It should be noted that the instructions include labels for the different fields, including urgency, certainty, category, and severity. In addition, the data structure includes fields for a description, a headline, a time a corresponding data item was transmitted (e.g., <sent>) from a data source, and an identifier of the data source (e.g., <senderName>). The data structure also includes hyperlinks to additional information, which may be displayed as links in a graphical representation of the CAP data structure. Alternatively, the linked content may be shown in conjunction with the information provided in the CAP data structure.

CONCLUSION

It will be appreciated that all of the disclosed methods and procedures described herein can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer-readable medium, including RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be configured to be executed by a processor, which when executing the series of computer instructions performs or facilitates the performance of all or part of the disclosed methods and procedures.