System and method for analyzing items using lexicon analysis and filtering process

The present invention is a method or system of analyzing items using lexicon analysis and filtering process. The present invention drives the analysis without a user being required to provide search terms and Boolean queries. Further, the present invention merges the use of unstructured data and structured data using a lexicon as a focus. Structured data is mapped to lexicon terms through one analysis process, while unstructured data is mapped to lexicon terms through a separate analysis process. The result is an analysis that merges the two different sets of data, which enables a user to navigate through the items using lexicons. Analysis results are displayed simultaneously in a tabular view, geospatial view, and lexicon histogram.

FIELD OF THE INVENTION

The present invention generally relates to a method or system of analyzing items by using lexicon analysis and filtering process.

BACKGROUND OF THE INVENTION

It is often very tedious and time consuming to search through a large pile of documents or reports when only one or a few documents are likely to possess most of the information needed. Simple key word searches or conventional queries would not provide a solution to this problem because a user is often not familiar with the subject matter that is at issue, and thus, is not able to figure out a fast and easy way to find a proper key word. The need for a fast search and easy identification of required information is manifest in many sectors of industry.

A lexicon analysis has been developed to assist non-experts perform an efficient search and analysis of documents in a particular field. However, lexicon analysis alone fails to resolve the problem presented above because there still exists a technical issue as to how to put various types of data into a single lexicon analysis and how to present various types of data to an end user for an efficient analysis. Moreover, conventional lexicon analyses separated lexicons themselves from the application of the lexicons. This caused additional problems such as difficulty in adjusting or updating old lexicons and adapting to the user's criteria or need generated after the creation of lexicons.

The present invention is directed to overcoming one or more of the problems set forth above.

SUMMARY OF INVENTION

An aspect of this invention provides a system for analyzing items using a lexical analysis and filtering process that includes a data storage unit storing source data comprising a plurality of items, wherein the source data is one of structured data and unstructured data; a processor associated with the data storage unit, wherein the processor extracts data from each item of the source data and the extracted data is associated with an item from which the data is extracted and maps each of the plurality of items with at least one lexicon term and the extracted data is analyzed to identify a match between the plurality of items and lexicon terms and the lexicon terms are represented as a network of a plurality of nodes and each node represents a lexicon term, which includes at least one of a parent node to which the lexicon term is subordinate and a child node to which the lexicon term is superordinate; and an electronic display for presenting the source data utilizing the processor configured to present a first group of items based on the extracted data of items in at least one of tabular view and geospatial view and first group of items represent items having at least one common characteristic of the extracted data, and present a second group of items and second group of items represent the items mapped to a first matching lexicon terms of the plurality of lexicon terms in at least one of the tabular view and the geospatial view.

Another aspect of this invention provides a method for analyzing items using lexical analysis and filtering process, which includes storing a plurality of items in a data storage unit, wherein the plurality of items are represented by source data and the source data is one of structured data and unstructured data; processing the plurality of items to extract data from each item of the plurality of items, wherein the extracted data is associated with an item of the plurality of items; displaying a first group of items based on the extracted data of the items on an electronic display presented in at least one of a tabular view and a geospatial view, wherein the electronic display is electrically connected to at least one processor in electronic communication with the source data, wherein the first group of items represents items having at least one common characteristic of the extracted data; mapping each of the plurality of items with at least one lexicon term, wherein the extracted data is analyzed to identify a match between the plurality of items and lexicon terms, wherein the lexicon terms are represented as a network of a plurality of nodes and each node of the plurality of nodes represents a lexicon term, which is subordinate to at least one of a parent node and a child node; and displaying a second group of items, wherein the second group of items represent the items mapped to a first matching lexicon term of the plurality of lexicon terms on the electronic display presented in at least one of the tabular view and the geospatial view, wherein the electronic display is electrically connected to at least one processor in electronic communication with the source data.

Still another aspect of this invention provides a system for presenting a plurality of items in multiple views that includes an electronic display; a processor that is electrically connected to the electronic display; and a data storage unit associated with the processor and in electronic communication therewith, wherein the processor is able to present a tabular view of a plurality of items to the electronic display, wherein the tabular view comprises certain information obtained from the items and the information represents certain characteristics of the items and the electronic display presents a geospatial view of the items to the user, wherein the geospatial view comprises a geographical map displaying a geographical representation associated with the location information obtained from each item, wherein the geospatial view is concurrently displayed with the tabular view; and the electronic display presents a lexicon histogram view of the items to the user, wherein the lexicon histogram view comprises a plurality of matching lexicon terms and corresponding match count for each of the plurality of matching lexicon terms, wherein the matching lexicon term represents a lexicon term mapped to each item and the match count represents a number of matches between each of the matching lexicon terms and a certain group of the items wherein the lexicon histogram view that is concurrently displayed with the tabular view and the geospatial view on the electronic display, wherein the lexicon terms are represented as a network of a plurality of nodes and each node represents a lexicon term which includes at least one of a parent node to which the lexicon term is subordinate and a child node to which the lexicon term is superordinate.

Yet another aspect of this invention provides a method for presenting a plurality of items in multiple views, which includes presenting a tabular view of a plurality of items on an electronic display that is electronically connected to a processor, wherein the processor is configured to present the tabular view on the electronic display with select information obtained from the plurality of items and the select information represents certain characteristics of the items; presenting a geospatial view of the plurality of items on the electronic display that is electronically connected to the processor, wherein the processor is configured to present the geospatial view on the electronic display and the geospatial view includes a geographical map displaying a geographical representation associated with location information obtained from each item of the plurality of items and the geospatial view is concurrently displayed with the tabular view on the electronic display; and presenting a lexicon histogram view of the plurality of items on the electronic display, wherein the processor is configured to present the lexicon histogram view that includes a plurality of matching lexicon terms and corresponding match count for each of the plurality of matching lexicon terms and each matching lexicon term of the plurality of lexicon terms is mapped to each item of the plurality of items generating a count representing a number of matches between each of the matching lexicon terms and a certain group of the items and the lexicon histogram view is concurrently displayed with the tabular view and the geospatial view on the electronic display and the lexicon terms are represented as a network of a plurality of nodes and each node represents a lexicon term, which includes at least one of a superordinate parent node and a subordinate child node.

In still yet another aspect of this invention provides a system for presenting a plurality of items in using lexical analysis that includes an electronic display; a processor that is electrically connected to the electronic display; and a data storage unit associated with the processor and in electronic communication therewith, wherein the data storage unit includes source data comprised of a plurality of items and the source data is structured data, wherein the processor is able to process the source data to extract data from each item of the source data, wherein the extracted data is associated with an item from which the data is extracted; create a plurality of tuple-structures for each item and the tuple-structure comprises a source, predicate, and object; and then map the item with at least one lexicon term and at least one of the source, predicate, and object of the tuple-structure is analyzed to find a matching lexicon term for viewing on an electronic display.

In yet another aspect of this invention provides a method for analyzing items using lexical analysis and filtering process, which includes storing a plurality of items in a data storage unit associated with a processor, wherein the plurality of items are represented by source data and the source data is one of structured data and unstructured data; processing the plurality of items to extract data from each item of the plurality of items, wherein the extracted data is associated with an item of the plurality of items and the processor is configured to extract textual data from the unstructured data and is configured to extract data from the structured data in order to create a plurality of tuple-structures for each the item of the structured data and the tuple-structure comprises at least one of a source, a predicate, and an object; mapping each of the plurality of items with at least one lexicon term, wherein the processor is configured to find a match between the plurality of items and lexicon terms and the processor is configured to find a match between the textual data and lexicon terms and also between the textual data and synonyms for the lexicon terms, and further configured to find a match between the tuple-structure and lexicon terms and at least one of the source, predicate, and object of the tuple-structure is analyzed to find a matching lexicon term.

Another aspect of this invention provides a system for analyzing items using lexical analysis and filtering process that includes a data storage unit storing source data comprising a plurality of items and the source data is one of structured data and unstructured data; and a processor associated with the data storage unit, wherein the processor extracts data from each item of the plurality of items, wherein the extracted data is associated with an item of the plurality of items, wherein the processor is configured to extract textual data from the unstructured data and is configured to extract data from the structured data in order to create a plurality of tuple-structures for each the item of the structured data and the tuple-structure comprises at least one of a source, a predicate, and an object and maps each of the plurality of items with at least one lexicon term wherein the processor is configured to find a match between the plurality of items and lexicon terms, wherein the processor is configured to find a match between the textual data and lexicon terms and also between the textual data and synonyms of the lexicon terms, and further configured to find a match between the tuple-structure and lexicon terms and at least one of the source, predicate, and object of the tuple-structure is analyzed to find a matching lexicon term.

In yet another aspect of this invention provides a method for analyzing items using lexical analysis and filtering process, which includes storing a plurality of items in a data storage unit associated with a processor, wherein the plurality of items are represented by source data and the source data is one of structured data and unstructured data; processing the plurality of items to extract data from each item of the plurality of items, wherein the extracted data is associated with an item of the plurality of items and the processor is configured to extract textual data from the unstructured data and is configured to extract data from the structured data in order to create a plurality of tuple-structures for each the item of the structured data and the tuple-structure comprises at least one of a source, a predicate, and an object; mapping each of the plurality of items with at least one lexicon term, wherein the processor is configured to find a match between the plurality of items and lexicon terms and the processor is configured to find a match between the textual data and lexicon terms and also between the textual data and synonyms for the lexicon terms, and further configured to find a match between the tuple-structure and lexicon terms and at least one of the source, predicate, and object of the tuple-structure is analyzed to find a matching lexicon term

These are merely some of the innumerable aspects of the present invention and should not be deemed an all-inclusive listing of the innumerable aspects associated with the present invention. These and other aspects will become apparent to those skilled in the art in light of the following disclosure and accompanying drawings. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

Reference characters in the written specification indicate corresponding items shown throughout the drawing figures.

DETAILED DESCRIPTION OF THE INVENTION

1. Description of Overall System

FIG. 1is a schematic block diagram of an illustrative, but nonlimiting, system100for an exemplary embodiment. The illustrative, but nonlimiting, exemplary system ofFIG. 1may include a plurality of user computers110, a processor, e.g., server,120, a data storage unit130, a network140, a data link150, and an external computer system160. The user computers110are connected to the processor, e.g., server,120through the network140. The processor, e.g., server,120is connected to the data storage unit130through the data link150. The user computers110can be any device with sufficient computational and network-connectivity capabilities to interface with the processor, e.g., server,120for the purposes described herein. For example, the user computers110can be a mobile device such as a smartphone, e.g., iPhone, a Google Android device, a BlackBerry device, and so forth, tablet computer, e.g., iPad, or the like. The user computers110can also take forms such as a personal computer, e.g., a desktop computer or laptop computer. The processor, e.g., server,120can be any computer with sufficient computational and network-connectivity capabilities to interface with the user computers110and the data storage unit130. The processor, e.g., server,120and the user computer110can be configured to be or reside on the same computer. It should be understood that the processor, e.g., server,120may comprise multiple servers or processors.

The data storage unit130can be any memory storage medium, computer memory, database, or database server suitable to store electronic data. The data storage unit130can be a separate computer independent from the processor, e.g., server,120. The data storage unit130can also be a relational data storage unit. Furthermore, the data storage unit130can reside on the processor, e.g., server,120as an internal memory, or alternatively, can be configured to reside on more than a single location.

The network140can be any data communications network capable of communicating data between the processor, e.g., server,120and any of the user computers110. An example of a suitable network is the Internet. However, it should be understood that the network140can comprise a plurality of networks that interconnect to form a larger network, including networks such as cellular data networks and other wireless or wired data networks.

The data link150is preferably an internal data communication link between the processor, e.g., server,120and the data storage unit130. However, it should be understood that as the processor, e.g., server,120and the data storage unit130can be connected through an outside network, e.g., the Internet, the data link150can be any network suitable for the transmission of electronic data. Preferably, the user computers110provide an electronic display along with an input/output capability. Alternatively, a separate electronic display111and input/output device, e.g., keyboard,112can be utilized in direct electronic communication with the processor, e.g., server,120. Any of a wide variety of electronic displays and input/output devices may be utilized with the present invention.

The external computer system160can be any computer system with sufficient computational and network-connectivity capabilities to interface with the server120. The external computer system160can comprise multiple computer systems.

The system100provides an analysis method for a certain type of items, e.g., reports or documents, which centers on the use of a lexicon. The system100drives the analysis without a user being required to provide search terms and Boolean queries. This provides a significant improvement over conventional search methods. Further, the system100merges the use of unstructured data and structured data using a lexicon as a focus. Structured data is mapped to lexicon terms through one analysis process, while unstructured data is mapped to lexicon terms through a separate analysis process. The result is an analysis that merges the two different sets of data, which enables a user to navigate through the items using lexicons.

Analysis results are preferably, but not necessarily, displayed simultaneously in a tabular view, geospatial view, and lexicon histogram. The tabular view can present certain information about items, e.g., reports or documents, of either structured or unstructured nature. The geospatial view presents geographical information about items. The lexicon histogram allows both navigation of lexicons and analysis of frequency of lexicon terms mapped to the items. The fusing of multiple source data, source data types, and concurrent display of different types of information into one cohesive whole is the primary value proposition of the present invention.

In operation, the system100can import lexicon data from an outside source (not shown). Any available means for transporting lexicon data to the system100can be used. For example, lexicon data can be delivered to the system100as a form of hard copies. The lexicon data can also be stored in an electronic medium such as a CD, DVD, USB thumb drive, or the like and delivered to the system100. In one embodiment, lexicon data is stored as a Java Script Notation (JSON) file and electronically transmitted to the system100. In this embodiment, the processor, e.g., server,120is connected to a supplier of the lexicon data through a network. However, it should be understood that any other suitable type of transportation means can be employed for the delivery of lexicon data from a lexicon source to the system100and any form of lexicon data can be implemented for the purposes described herein. The system100stores the imported lexicon data in the data storage unit130for further analysis as discussed below.

The imported lexicon data represents conceptual definitions of certain terminologies in a particular field that were developed by Subject Matter Experts (SMEs). In one illustrative, but nonlimiting, embodiment, a lexicon may contain conceptual definitions of a plurality of lexicon terms concerning an improvised explosive device (IED). In this specific embodiment, the system100can import lexicon data developed for Weapons Technical Intelligence (WTI). It should be understood, however, that the present invention is not limited to a particular field or expertise for purposes of importing and analyzing lexicon data but a wide spectrum of applications are applicable.

Once the system100imports the lexicon data, the system100further processes the lexicon data to add detail information to the lexicon data. In the illustrative, but nonlimiting, exemplary embodiment, the system100solicits synonyms for each lexicon term in the lexicon data from SMEs. Referring now toFIG. 2, a screenshot of the illustrative, but nonlimiting, exemplary embodiment of the system100is depicted in which a lexicon for an improvised explosive device (IED) is displayed. InFIG. 2, the system100solicits synonyms for the lexicon term “Carbon Rod”202. The lexicon term “Carbon Rod” is tagged with solicited synonyms such as “Graphite Rod,” “D-Cell Battery Core,” and “LMS”204.

Next, the system100builds a data structure of the lexicon data by creating a network of nodes where each node represents each lexicon term contained in the lexicon data. An example of this network structure is shown inFIG. 3and generally indicated by numeral302. In a comparable manner asFIG. 2,FIG. 3depicts a screenshot of the system100in which a lexicon for an illustrative, but nonlimiting example, e.g., IED, is displayed. As shown inFIG. 3, each node in the network indicates a lexicon term obtained from the lexicon data, which is subordinate to any referenced parent node and is superordinate to any referenced child node. The hierarchical relationship among the nodes is encoded and represented in data stored in the data storage unit130of the system100. The solicited synonyms for each lexicon term can also be associated with the node that corresponds to a lexicon term for which such synonyms are solicited.

In one illustrative, but nonlimiting, embodiment, the system100provides an interface for a user to edit the lexicon. As shown inFIG. 3, the system provides a lexicon editor that allows user modification and creation of lexicons. For example, users can add a new node to the existing network, delete a preexisting node from the network, or change a preexisting node of the network. The modified or revised version of the lexicon can be saved by a user into the data storage unit130. Accordingly, the updated topology of the network will be stored in the data storage unit130. Furthermore, the lexicon editor allows a user to change, modify, add, or delete synonyms and mapping rules that are associated with the nodes in the network.

(2) Source Data

In the illustrative, but nonlimiting, exemplary embodiment, the system100collects various types of source data that can be mapped with a lexicon from outside sources. The processor, e.g., server,120can receive source data from the external system160. For example, the source data can be pushed to the processor, e.g., server,120by the external system160or pulled from the external system160by the processor, e.g., server,120through a network170, as shown inFIG. 1. In the latter case, the processor, e.g., server,120sends a set of queries to the external system160for purposes of pulling data from the external system160or registers certain web-services with the external system160such that the data generated on the external system160can be periodically pushed over to the system100. Like the network140, the network170can be any data communications network capable of communicating data between the processor, e.g., server,120and the external system160. The source data can represent any item of interest that can be matched against lexicons. In the illustrative, but nonlimiting, exemplary embodiment, the source data represents a set of reports describing activities that may involve IEDs. Alternatively, the system100can use any data stored in the data storage unit130as the source data.

Each supplier of source data is responsible for providing unique identification information for the source data. Each item in the source data can be associated with a unique identifier, e.g., a file path to the source data. In the alternative, a user, or the processor, e.g., server,120can provide a unique identifier to each item. Each supplier of source data can also provide information regarding a data type. The information regarding a data type is utilized to select the proper parser configuration and mapping rules for processing the source data and also for mapping lexicons to the source data, which is discussed below in reference toFIG. 4throughFIG. 7.

In the illustrative, but nonlimiting, exemplary embodiment, the source data can take a form of structured data and unstructured data. Any data such as XML (Extensible Markup Language), CSV (Comma-Separated Values), JSON (Java Script Notation), and RDF (Resource Description Framework) data can be used as structured data. As discussed in further detail below, the system100is designed to be capable of importing and processing both unstructured and structured data.

(3) Processing Source Data

The system100handles both unstructured and structured data received from the suppliers of source data. For unstructured data, the processor, e.g., server,120extracts textual data, e.g., text or key word, from each item in the source data. For each item, the parser122of the processor, e.g., server,120extracts at least one textual data along with any available structured supporting information such as metadata. The structured supporting information represents certain characteristics of an item. The parser122can be a software program residing on the processor, e.g., server,120or any combination of software and hardware that reside on the processor, e.g., server,120configured for the purposes described herein. Examples of the structured supporting information include information concerning a data type, a location, a date/time, and any description or summary contained in an item. In the illustrative, but nonlimiting, exemplary embodiment in which an item is a report about an IED, the structured supporting information can include the location information of where an IED was found, date/time when an IED was found or reported, data type information, and any description or summary about an IED itself. Any structured supporting information can be associated with the corresponding item or the source data as part of an analysis effort. It should be understood that the above example of IEDs was given only as merely an illustrative, but nonlimiting, example and other types of reports or documents can be used as an item for the illustrative, but nonlimiting, exemplary embodiment of the present invention.

Once the textual data is isolated, the processor, e.g., server,120is configured to find a match between the text represented by the textual data and lexicons. This process maps the text represented by the textual data extracted from the items to the lexicon terms. In addition, the processor, e.g., server,120can be configured to find a match between the text represented by the textual data and synonyms solicited for lexicon terms. The scan for synonyms uses a combined word list from the textual data and the lexicon to avoid redundant scans of the input.

Once a set of lexicon term matches are identified, the processor, e.g., server,120aggregates and stores certain information such as match counts, e.g., a number of matching lexicon terms identified for a certain group of the items, stores an identifier associated with each match, the offset for the match, and the structured supporting information associated with each item or source data. The aggregated information or data is later used for the display of the lexicon histogram view and for the filtering of the data source sets as discussed in further detail below with reference toFIG. 8.

The handling of structured data is more comprehensive and requires more involvement of the parser122than unstructured data as the mapping of structured data requires the flexibility to extract parts of the incoming source data and associate values in those fields to lexicon terms in a flexible manner. The first part of handling structured data is parsing. The parser122extracts data for each item. Unlike the unstructured data parsing, the parser122can be configured to extract any type of data (not just textual data) from each item. The result of this stage is a set of associated data ideally in the form of tuples, preferably, but not necessarily, having at least one of a source, predicate, and object obtained for each piece of data from each item in the source data as further discussed below with regard toFIG. 4throughFIG. 7. This tuple-structure allows flexible mapping of fields, parts of fields, or specific values in fields to lexicon terms. Once a tuple-structure is formed for each item, the processor, e.g., server,120, or the rules engine124, processes a mapping between the tuple-structure and the lexicon. The rules engine124can be a software program that resides on the processor, e.g., server,120or any combination of software and hardware that can be running on the processor, e.g., server,120that is configured for purposes described herein. The rules engine124can also be configured to reside on the external computer system160. Preferably, a set of predefined rules are used by the processor, e.g., server,120or the rules engine124to map each item based on the information parsed from the source data. In the illustrative, but nonlimiting, exemplary embodiment, the processor, e.g., server,120or the rules engine124is configured to use at least one of the information contained in the subject, predicate, and object; however, it should be understood that the tuple-structure can be replaced with a non-tuple structure form having less than or more than three data elements and that any type of information can be represented by that structure for the purposes described herein.

A result of the parsing can take a form of Resource Descriptive Framework (RDF) data. RDF data can be generated for a group of the items. However, it should be understood that other forms of data can also be used that is compatible and suitable with the purposes described herein. Configuration parameters for the parser and rules engine are associated with each type of item, which is further discussed below with reference toFIGS. 4 through 7. Ideally such configuration information would also be represented as RDF tuples. However, it should be understood that other forms of data can also be used.

(4) Presenting Lexicon Analysis Result

Once each item from the source data has been parsed and mapped to a lexicon term(s), the system100presents a graphical user interface to the user for the presentation of the analysis result as shown inFIG. 1. There are numerous ways to implement a graphical user interface. First, the user can utilize the user computer110to access a web-page hosted by the processor, e.g., server,120through the network140. In this first embodiment, the processor, e.g., server,120provides a web-page the user can access by using a conventional web browser or viewer, e.g., Safari, Internet Explorer, and so forth, that can be installed on the user computer110. The lexicon analysis result can be presented to the user through this web-page. Second, the system100provides a computer application that can be downloaded to the user computer110. In this second embodiment, the user can access a web-page hosted by the processor, e.g., server,120to download the computer application provided by the system100. The computer application can be installed in the user computer110, which provides the interface for the user to view the analysis result. In the third embodiment, the user computer110is a mobile device, e.g., an iPhone, which can access the processor, e.g., server,120through the network140. In this embodiment, the network140can be a cellular network or a wireless internet. The system100provides a mobile application that can reside on the user's mobile device and which can provide an interface for the user to view the analysis result presented by the processor, e.g., server,120. In yet a fourth embodiment, there can be the input/output device, e.g., keyboard,112that is in direct electronic communication with the processor, e.g., server,120with a separate electronic display111. It should be understood that the above descriptions of the preferred embodiments of the system100are given only as an example, and should not be used to limit the scope of the present invention.

Regardless of which embodiment the system100employs, the user will be provided with an appropriate interface which enables the user to see the analysis result as shown inFIG. 8, which is generally indicated by numeral802. The analysis result is preferably displayed simultaneously in a tabular view, geospatial view, and lexicon histogram, which is discussed further below in reference toFIG. 8.

2. Processing Structured Data

Because processing structured data is not same as that of unstructured data and requires more involvement of the parser122as shown inFIG. 1, further descriptions regarding the processing of structured data will be discussed herein with an illustrative, but nonlimiting, exemplary embodiment in which an item is a report about an IED. However, it should be understood that the given example is only intended to be an example and should not be considered as the only embodiment of the present invention.

As discussed above in Section 1, the first step of processing structured data is parsing. A primary purpose of parsing is to extract meaning from the source data. The parsing process can be adapted to any structured source data scheme but XML data will be utilized herein to provide an example of how the parsing occurs.FIG. 4depicts exemplary source data in a form of XML data that is generally indicated by numeral402. The data set shown inFIG. 4is an input to the parser122of the processor, e.g., server,120, shown inFIG. 1.

FIG. 5depicts an actual parsing logic in a pseudo code form that is generally indicated by numeral502. Preferably, this pseudo code (or the like) can be encoded into the parser122of the processor, e.g., server,120, shown inFIG. 1. As shown inFIG. 5, the pseudo code is designed to parse each input element. The element can be any type of data that can be extracted from an item. The term “resource” in the pseudo code is used to refer to any object/element in the source data that has property data. For example, if “ArmingSwitchTier1” inFIG. 4has any property data, the element “ArmingSwitchTier1” will be considered as a new resource according to the pseudo code, and the term “ArmingSwitchTier1” will be put into a tuple and will also be associated with such property data.

FIG. 6depicts an exemplary result of parsing in a form of RDF data, e.g., tuple, and is generally indicated by numeral602. As shown inFIG. 6, the source XML data set has been parsed to generate tuple structures. Each tuple starts with a unique identifier that is indicated by numeral610. It should be understood that the unique identifiers610ofFIG. 6are given as only examples and thus can be in a different form in the actual system. As can be seen inFIG. 6, resources are related from enclosing element, e.g., “<ExploitationIEDLevel1 . . . >”, to enclosed element, e.g., “<IEDDevice . . . >”, using the element name of the enclosed element, e.g., “IEDDevice”, and fixed to a ‘parent’ property from enclosed to enclosing resources. For example, as shown inFIG. 6, the resource for “IEDDevice” includes a tuple structure, which indicates that the parent is the resource “IED Level 1.” The tuple “_2 parent _1” shows that the resource “IED Level 1” is the parent of the resource “IED Device.” The actual property names used in RDF can be URI values with namespaces.

The property names can be directly borrowed from the input element names. A method for generating property names can be controlled by manipulating a pseudo code for the parsing logic. For example, inFIG. 5, the pseudo code is expressly written to exclude any resource that represents the top-level resource, e.g., “Remove the top resource from the stack.” More fundamentally, the generation of property names can also be configured at the parameter level of parsing. The user can set up the parser122to exclude certain elements from the source XML. The parsing process can be set up by a user through configuration parameters. For example, the user can set up certain parameters that define which elements are to be ignored, which elements are to be used as new resources, and which elements are to be grouped into new resources. These parameters can be encoded into the parser122. As such, in the above example, the element “ExploitationIEDLevel1Reports” was excluded inFIG. 6because the parser122was configured to ignore such element.

The tuple-structure preferably comprises at least one of a subject, predicate, and object with the most preferred tuple-structure utilizing all three elements. The subject preferably includes a general concept or conceptual information representing an item. The predicate preferably includes information representing one or more categories or subcategories of an item. The object preferably includes information representing the extracted data itself, e.g., resource.

Once the parsing is completed, and RDF data is generated as a result of the parsing, the RDF data is augmented with mapping data taken from the mapping rules. The mapping rules are predefined rules that map the resources, e.g., an element of an item, to lexicons. In particular, these mapping rules generate new RDF tuples that connect the resources to predefined resources that represent each lexicon term in the data storage unit130as shown again inFIG. 1. In one embodiment, the parser122uses the predicate and object of the tuple to generate new RDF tuples for mapping. However, any one of the subject, predicate, and object, or any combination thereof can be used to generate new RDF for mapping purposes. In the illustrative, but nonlimiting, exemplary embodiment, these rules are processed using the JENA generalized rule engine which takes tuple patterns to match, and tuple patterns to be generated on a match. In the preferred embodiment, the rules engine124ofFIG. 1can be configured as the JENA generalized rule engine.

FIGS. 7(a) and 7(b)depict exemplary mapping rules that augment the RDF data with mapping data. The rule shown inFIG. 7(a)that is generally indicated by numeral702indicates that a “MainCharge” with “MainChargeTier2” value “Commercial” and “MainChargeTier1” value “High Explosive” should produce new tuples for the resource (?mc) that maps to the lexicon term “HE_CommercialExplosives” and should match the terms “Commercial” and “High Explosives” in term highlighting. Highlighting is defined as any type of visual emphasis, e.g., background or text color change, colored border, and so forth. In addition to use in lexicon and term highlighting, the mapping rules can indicate that values are available under predefined property names to allow unification of different input terminology as shown inFIG. 7(b)that is generally indicated by numeral740. The rule inFIG. 7(b)takes the key (?k) indicated by the “MainLocationKey” element in the input, and matches it to the “LocationKey” element of the resource (?l). After that, the rule extracts the “Latitude” property from that resource and assigns it to the “incident_lat” property of the original resource (?r).

Once the RDF for each input item, e.g., report, is produced, the resources in the RDF are grouped into several categories of grouping and then aggregated into RDF for those groups. This process uses explicit links in the input, and fuzzy matching of values in the RDF data. In the preferred embodiment, location (latitude/longitude), and date/time group (DTG) are the values used for fuzzy matching, but the system100is designed to allow matching based on any shared value in the RDF resources including lexicon terms or values generated from the mapping rules. Because each report is represented by a top-level resource, and all resources are uniquely identified in RDF, the aggregation is a simple append operation merging the RDF statements from all reports in the group. This facilitates processing in a cloud or HADOOP type environment as all processing of reports or groups is working on local data and does not require access to other objects for processing. The aggregation of group RDF can be dynamic (on the fly), or static (performed at the time the report is initially received). In the preferred embodiment, the latter was selected, while in a cloud environment this aggregation may be performed dynamically by merging RDF in a reduce phase that then collects the values and combines them to form the desired output. This allows each system in the cloud cluster to operate on local data while then combining their collective output with minimal network traffic.

3. Presentation of Lexicon Analysis Result

As discussed above in Section 1(4), the illustrative, but nonlimiting, exemplary embodiment of the system100provides a graphical user interface to a user for purposes of presenting the lexicon analysis.FIG. 8depicts a graphical user interface provided by the illustrative, but nonlimiting, exemplary embodiment of the system100. As shown inFIG. 8, the system100provides three different displays: a tabular view810, a geospatial view820, and a lexicon histogram view830as generally indicated by numeral802.

The tabular view810displays certain information of the items of the source data. The user can input initial criteria for selecting a certain group of the items. The user can also select the whole source data in which case the tabular view810displays all the items contained in the source data. In the illustrative, but nonlimiting, exemplary embodiment of the system100, the tabular view810can be configured to display such information as a data type, date/time, location, and any description or summary contained in the items. In the illustrative, but nonlimiting, exemplary embodiment in which an item is a report about an IED, the tabular view810is configured to display “Report Type” section811, “Date & Time” section812, “MGRS” section813, and “Description” section814. The “Report Type” section811shows the types of reports. The “Date & Time” section812shows the date/time an IED was found or reported. The MGRS section813shows a geo-coordinate or a grid reference for an IED. The “Description” section shows any description or summary stated in an IED report. The user can click on one of the reports shown in the tabular view810to select one. If the user double clicks one report, a “Report Summary” window815pops up. The “Report Summary”815shows more detail information of the selected report.

The geospatial view820shows a geographical map displaying a geographical representation associated with location information extracted from each item of the source data. In the illustrative, but nonlimiting, exemplary embodiment in which an item is a report about an IED, the map in the geospatial view820can be configured to display the locations where IEDs were found or reported. In addition, the geospatial view820can be configured such that certain items with similar characteristics can be grouped together by the same color in the map. For example, as shown inFIG. 8, the “Legend”822indicates that all red colored reports in the map represent ‘A’ type IEDs and all green colored reports in the map represent ‘B’ type IEDs. Furthermore, the geospatial view820can be configured to color the reports based on their type, location, date/time, and any descriptive information contained in the reports. The reports with the same type, same location, same date/time, or same or similar description can be shown in the map with the same color. The geospatial view820and the tabular view810are configured to be displayed concurrently.

The lexicon histogram view830contains preferably two sections. A “Lexicon” section832lists lexicon terms that are mapped with the items shown in the tabular view810. Alternatively, the user can select a certain lexicon term(s). A “Reports” section834shows a histogram which represents a match count for each of the lexicon terms listed in the “Lexicon” section832. The match count refers to a total number of matches found in the items shown in the tabular view810for each lexicon term appearing in the “Lexicon” section832. In addition, the match count can be configured to represent a total number of matches found in the items for a particular lexicon term appearing in the “Lexicon” section832plus a total number of matches with all of the lexicon terms that are subordinate to that particular lexicon term shown in the “Lexicon” section832. For example, the lexicon histogram view830ofFIG. 8shows that there are 97 matches found in this set of the items shown in the tabular view810for the lexicon term “Forensics.” The number97includes the number of matches for the term “Forensic” and the number of matches for all of the terms that are subordinate to the “Forensic.” Furthermore, the user can click on each of the lexicon terms appearing in the “Lexicon” section832. If the user clicks on one of the lexicon terms in the “Lexicon” section832, all the reports in the tabular view810(or alternatively, the source data or any group of items chosen by the user) that were found by the parser122to be matching with the selected lexicon term will appear in the tabular view810. This change will occur automatically by the system100. In addition, the geospatial view820will also be updated with a new map indicating the locations of only those reports found to be matching the lexicon term selected by the user. In this way, the user is provided with an interactive display that automatically adapt to the user's filtering criteria.

The lexicon histogram view830also provides a navigation function to the user, which enables the user to search through the network of lexicon nodes. For example, the lexicon histogram view830can be configured such that if the user clicks on one of the buttons displayed in a “Menu”836” section, the lexicon histogram view830will update its display with a list of only those lexicon terms that are defined to be a superordinate node of the lexicon term(s) selected by the user. Accordingly, a new set of histograms will show up in the “Reports” section834reflecting the changes made to the “Lexicons”832. Moreover, the tabular view810and the geospatial view820can also be configured to reflect the change made by the user. For example, the tabular view810can be configured to display only those reports that were found to be matching the lexicon term(s) selected by user. The geospatial view820can also be configured to display the geographical representations of only those lexicon terms that are selected by the user.

The lexicon histogram view830can also be configured such that if the user clicks on one of the buttons displayed in the “Menu”836section, the lexicon histogram view830will update its display with only those lexicon terms that are defined to be a subordinate node of the lexicon term(s) selected by the user. Accordingly, a new set of histograms will show up in the “Reports” section834reflecting the changes made to the “Lexicons”832. Moreover, the tabular view810and the geospatial view820can also be configured to reflect the change made by the user. In this way, the user can freely navigate through the lexicon network without having to do a manual search for a particular lexicon term in the network.

It should be also understood thatFIG. 8is intended to provide only an illustrative, but nonlimiting, example and should not be construed to limit the display to only those described inFIG. 8. One of ordinary skill in the art would understand that other types of information obtained from the source data can be displayed inFIG. 8in addition to the information already described inFIG. 8.

Additionally, the illustrative, but nonlimiting, exemplary embodiment of the system100provides a filtering interface to the user. The preferred embodiment provides two types of filtering interfaces with respect to the tabular view810ofFIG. 8. First is a source data filtering interface. As shown inFIG. 8, a “Source Data Filtering” interface816enables the user to filter the reports listed in the tabular view810by a report type, date/time, location, or description. For example, the user can click on the “Source Data Filtering”816in which case a new window with an appropriate number of data fields pops up in a user screen. The user can enter filtering criteria into the appropriate fields. For example, if the user wants to see the reports that were produced between Jan. 1, 2007 and Jan. 1, 2008, the user can put that filtering criteria into the appropriate fields of the window. The tabular view810updates its display according to the user's filtering criteria by displaying only those reports that meet the user's filtering criteria. The geospatial view820and the lexicon histogram view830concurrently update the display to reflect the changes made in the tabular view810in consistent with the descriptions stated above.

The second filtering interface provided by the tabular view810is a lexicon filtering interface817. The user is enabled to filter the reports listed in the tabular view810by lexicons. If the user clicks on the “Lexicon Filtering”817, a “Lexicon Filter Builder” window910containing an appropriate number of data fields pops up, as shown inFIG. 9. The user can enter a lexicon term(s) into the appropriate field. For example, if the user wants to see only those reports that contain a lexicon term “Switch,” then the user can enter the term “Switch” into the appropriate field, as shown inFIG. 9. The system100will update the tabular view810to reflect the user's input, i.e., displays only those reports mapped to the term “Switch.” Likewise, the geospatial view820concurrently updates its display to reflect this change made by the user. Alternatively, the “Lexicon Filter Builder” window can be accessed from the lexicon histogram view830. For example, the lexicon histogram view830can be configured such that if the user clicks on the “Reports” section834or any button in the “Menu” section836, the “Lexicon Filter Builder” window pops up.

The lexicon filtering interface817further provides a Boolean search option. As shown inFIG. 9, the “Lexicon Filter Builder”910provides additional interface for the user to enter in a Boolean logic. For example, the user can use a pull-down menu bar, as shown inFIG. 9to select one of the Boolean logics, e.g., AND, OR, Except, and so forth. As shown inFIG. 9, the user can filter the reports that contain the lexicon term “Switch”912while excluding the reports that contain the lexicon term “MainCharge”914.

The illustrative, but nonlimiting, exemplary embodiment also provides color-based displays with respect to the geospatial view820ofFIG. 8. For example, the user can change the settings of the “Legend”822from one setting to another. First, the “Legend”822can be configured to list a data type, date/time, and description. If the user selects a data type setting, the geospatial view820can be configured to color the reports by the type of each report, e.g., red for ‘A’ type IEDs and blue for ‘B’ type IEDs. Likewise, if the user selects a date/time setting, the geographical view820can be configured to color the reports by the date/time information contained in each report. The user is allowed to switch from one color group to another by interacting with the settings in the “Legend”822.

Second, the user can change the settings of the “Legend”822to lexicons. In this example, the geospatial view820is configured to color the reports by lexicon terms that are mapped to the reports. For example, all of the reports mapped to the lexicon term “Forensics” can be displayed in red while all of the reports mapped to the lexicon term “Attack the Network” can be displayed in blue. The user is allowed to switch from one color group to another by interacting with the “Legend”822.

It should be understood that the interfaces described above are given only as examples and should not be construed to limit the scope of the present invention. One of ordinary skill in the art can come up with different ways to implement the disclosed filtering interfaces that are consistent with the descriptions provided above.

Once the analysis data is complete and represented in the system100, the user can select a working set of data, e.g., a case file, which matches some set of initial criteria chosen by the user. The initial criteria may not include lexicon-based criteria. For example, the user can select a case file by filtering the reports by certain location, data type, date/time, and description. Within that working set, the user can add or remove filters by interacting with three views810,820, and830consistent with the descriptions stated above. Furthermore, the user can select a case file that matches lexicon based criteria. For example, the user can select a case file by filtering the reports by a certain lexicon term(s). In this example, all of the reports mapped to that lexicon term can be shown through the tabular view810, geospatial view820, and/or lexicon histogram view830, in consistent with the descriptions stated above inFIG. 8.

In an alternative embodiment, the system100can be configured to group the source data by certain incidents. For example, the system100can be configured to group the IED reports concerning an explosion incident occurred on a certain date. All the reports related to that particular incident can be grouped together and presented to the user with the views810,820, and830. The tabular view810can be configured to display the reports related to that particular incident. The geospatial view820can be configured to display the geographical representations of the reports related to that particular incident. The lexicon histogram view830can be configured to display the lexicon terms and match counts for the lexicon terms that are mapped to the reports related to that particular incidents. Furthermore, the user can filter the collected reports by report type, date/time, description, location, and lexicons terms consistent with the descriptions stated above. In addition, in this alternative embodiment, the user can create a case file that includes many different types of incidents that meet search criteria. For example, the user can create a case file comprising an incident ‘A’, e.g., an explosion occurred on a certain date, and incident ‘B’, e.g., an explosion occurred in a certain area. The case file in this example would contain a group of the reports that meet the incident ‘A’ criteria and another group of reports that meet the incident ‘B’ criteria. Furthermore, the system100can be configured to filter the source data by incidents (not by reports). The user can filter the incidents in the selected case file by data type, date/time, description, location, and lexicon terms consistent with the descriptions stated above. For example, if the user filters the incidents by a location then all the incidents that meet the selected location criteria can be configured to be displayed through the views810,820, and830. Additionally, the lexicon histogram view830can be configured to display the number of matching groups (i.e., number of incidents) for each lexicon term in the histogram, instead of the number of matching reports. The lexicon histogram view830can also be configured to display the number of matching sub-groups if the reports included in the selected incident can further be divided into sub-groups based on the information obtained from the tuple-structure analysis.

FIG. 10depicts a flowchart of a method of analyzing items using lexical analysis and filtering process. In the description of the flowcharts, the functional explanation marked with numerals in angle brackets, <nnn>, will refer to the flowchart blocks bearing that number.

At step <1010>, the system100imports lexicon data from an outside source. Alternatively, the system100can use any internal lexicon data stored in the data storage unit130. The imported lexicon data is processed at step <1020> to add more value to the data. For example, the system100solicits synonyms for each lexicon term as described in Section 1. At step <1030>, the system100builds a data network comprising a plurality of nodes in which each node represents a lexicon term.

Once the lexicon hierarchy is created at step <1030>, the system100imports a source data at step <1040>. In the illustrative, but nonlimiting, exemplary embodiment, the system100receives the source data from the external system160. The source data can be both unstructured and structured data. At step <1050>, the parser122of the system100parses the source data. In case of unstructured data, the parser122simply parses textual data, e.g., text or key word, from the source data. In case of structured data, the system100parses any type of data from the source data and creates a tuple structure, which is described above in Section 2.

At step <1060>, the system100maps the items with lexicons. For unstructured data, the system100maps the text represented by the textual data of the items with lexicons. In addition, the system100can be configured to map the text represented by the textual data with the synonyms for lexicons solicited at step <1020>. For structured data, the system100uses the rules engine124to map the tuples with lexicons, which is previously described above in Section 2.

At step <1070>, the system100is now ready for presenting the lexicon analysis to the user. The system100can be configured to present different types of display modes to the user as discussed above in Section 3. Furthermore, the user can, at step <1080>, filter the items or lexicons by setting up various filtering criteria as discussed in Section 3.

Furthermore, it should be understood that when introducing elements of the present invention in the claims or in the above description of the preferred embodiment of the invention, the terms “have,” “having,” “includes” and “including” and similar terms as used in the foregoing specification are used in the sense of “optional” or “may include” and not as “required.” Similarly, the term “portion” should be construed as meaning some or all of the item or element that it qualifies.

Thus, there have been shown and described several embodiments of a novel invention. As is evident from the foregoing description, certain aspects of the present invention are not limited by the particular details of the examples illustrated herein, and it is therefore contemplated that other modifications and applications, or equivalents thereof, will occur to those skilled in the art. Many changes, modifications, variations and other uses and applications of the present construction will, however, become apparent to those skilled in the art after considering the specification and the accompanying drawings. All such changes, modifications, variations and other uses and applications which do not depart from the spirit and scope of the invention are deemed to be covered by the invention which is limited only by the claims that follow.