Method and system for mapping a hypothesis to an analytical structure

A method for mapping a hypothesis to an analytical structure includes receiving the hypothesis from an interface and generating a representation of the hypothesis. The method includes generating a script of the representation by mapping the representation into a neutral grammar and transmitting the script to an analytical tool. The method may also include analyzing the script to form an analytical structure. The interface may comprise a human interface or a machine interface. The method may also include generating a linguistic expression of the hypothesis.

TECHNICAL FIELD OF THE INVENTION

This invention generally relates to the field of data analysis and, more particularly, to a method and system for mapping a hypothesis to an analytical structure.

BACKGROUND OF THE INVENTION

Advancements in technology have reduced the cost of computers to the point where many events in one's day are recorded by a computer. Events recorded by computer are numerous and include, for example, transactions made by an individual. Computers store the data associated with the transactions they process resulting in very large databases of information. Also, companies and individuals frequently use computers to record events related to a specific domain. For example, a meteorologist may enter into a computer database many records of data relating to weather occurrences.

One problem arises of how to make efficient use of the tremendous amount of information in these databases. When the number of records in a database rises to a certain level, simply sorting the information in the database provides no meaningful results. While statistical analysis of the records in a database may yield useful information, such analysis must generally be performed by persons with advanced training in math or computer science. Typically, these people are also needed to understand the results of the analyses. Additionally, translation of the statistical analysis of the information in a large database into a useful form is also difficult. For example, a strategic business activity such as marketing may require analytical information to be converted into a form specifically suited to the activity of marketing. Difficulties in providing or obtaining information in a useful form may prevent the effective use of the information in a database and preclude the use of a possibly valuable data resource.

Organizations of all types commonly collect and store business and technical data in various types of databases. Strategic and/or technical knowledge may be contained in the databases. In some instances, based on many years of experience, experts are able to glean knowledge from databases existing in their particular domain of expertise. In the absence of such experts, however, strategically useful information may not be available to the organization controlling or accessing a given database. The inability to obtain this knowledge may be detrimental to the business objectives of the organization. For example, if a business cannot extract useful knowledge from the data it possesses, it will likely be at a competitive disadvantage compared to a business that can discover such knowledge. Thus, the ability to discover knowledge from data contained in databases would be a valuable asset to any organization.

Certain tools are available which assist a non-expert to gain some knowledge from a database, such as data mining tools. Certain tools are also available to assist analysts to validate hypotheses through interactive exploration, such as OLAP and multidimensional database analysis tools. For example, some data analysis tools respond to queries input by the user. A query might be: “How many people within the database are between the ages 30 and 35?” The data analysis tool looks to all the records in which an age field meets the age range requirement of the query. Then, the tool simply counts the number of records. Query tools require the user to have an extensive knowledge of the database domain and the queries generally are very rigid in their structure. One example of a data mining tool is described in U.S. Pat. No. 5,933,818, entitled “Autonomous Knowledge Discovery System and Method.”

Data analysis tasks typically require skilled analysts and significant time and also may introduce opportunities for errors due to steps that require manual intervention.

SUMMARY OF THE INVENTION

The present invention provides a method and system for mapping a hypothesis to an analytical structure that substantially eliminates or reduces at least some of the disadvantages and problems associated with previous methods and systems.

In accordance with a particular embodiment of the present invention, a method for mapping a hypothesis to an analytical structure includes receiving the hypothesis from an interface and generating a representation of the hypothesis. The method includes generating a script of the representation by mapping the representation into a neutral grammar and transmitting the script to an analytical tool. The method may also include analyzing the script to form an analytical structure. The interface may comprise a human interface or a machine interface. The method may also include generating a linguistic expression of the hypothesis.

In accordance with another embodiment, a system for mapping a hypothesis to an analytical structure includes a memory comprising a hypothesis representation module operable to receive the hypothesis from an interface. The hypothesis representation module is further operable to generate a representation of the hypothesis. The memory also includes a hypothesis mapping module operable to generating a script of the representation by mapping the representation into a neutral grammar. The system also includes a processor coupled to the memory. The processor is operable to transmit the script to an analytical tool. The memory may also include analytical tool. The analytical tool may be operable to analyze the script to form an analytical structure. The interface may comprise a human interface or a machine interface. The hypothesis representation module may be further operable to generate a linguistic expression of the hypothesis.

Technical advantages of particular embodiments of the present invention include the automatic mapping of a hypothesis into a script. The script comprises a neutral grammar that any of a number of analytical tools may understand. Thus, substantial time and resources may be saved because additional or substantial customization is not needed to enable the analytical tool to read and parse the script. Moreover, the hypothesis may be entered by a machine interface or a human interface in any number of ways or formats. This saves additional time and resources since extensive training of users relating to how hypotheses should be entered may be reduced.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1is a schematic diagram illustrating a process for mapping a hypothesis to an analytical structure. A hypothesis18is formulated by a human computer interface14or by a machine, such as a data mining tool16. When formulating hypothesis18, a data source12is selected for use in testing hypothesis18. A hypothesis mapping function20generates a representation of hypothesis18and maps the hypothesis representation into a script22. Script22is an expression of the hypothesis representation that an analytical tool24can understand. Analytical tool24reads and parses script22and uses data source12to test the hypothesis. Analytical tool24generates an analysis report26of the results. A user may evaluate analysis report26and refine the process or release analysis results28. The illustrated process provides a manner whereby a user may enter a hypothesis which is automatically mapped into a script of a neutral grammar that any of a number of analytical tools may understand and analyze without additional or substantial customization.

Data source12may be a database that includes information of various domains and formats. The data formats of data source12may include for example text, numeric, date/time, boolean and other data formats. Data source12may include only the data selected to be used in testing hypothesis18, or it may include a larger set of data from which the data to be used for hypothesis testing is selected.

As stated above, the formulation of hypothesis18may be undertaken by human computer interface14or by a machine such as data mining tool16. Data mining tool16is an automated hypothesis generation tool.

In particular embodiments hypothesis mapping function20also formulates a linguistic expression of the hypothesis. The linguistic expression is a formulation of the hypothesis that a user can understand, for example, in plain or natural language. In particular embodiments the linguistic expression of the hypothesis may be similar to the original hypothesis formulated by human computer interface14. However, in some situations when multiple human computer interfaces formulate hypotheses, such hypotheses may not be formulated in the same format. Thus, the linguistic expression generated by hypothesis mapping function20provides a consistent format for different hypotheses to be expressed in a way that users may understand.

As stated above, script22is an expression of the hypothesis representation in a language that an analytical tool may understand. Examples of such analytical tools are discussed below. The language used for script22may be a neutral format such that analytical tools of various types may be able to read and parse script22. Script22may include a tag-based language, such as eXtensible Markup Language (XML). In such cases, each hypothesis, data source and other parameters will be mapped into a tag. Default values may be used for unspecified parameters of the script.

As stated above, analytical tool24reads and parses script22and uses data source12to test the hypothesis and generate an analysis report26of the results. Analytical tool24tests hypothesis18by analyzing data source12and creating value by drawing appropriate conclusions that either support or negate hypothesis18. Analytical tool24may be of a type that is commercially available such as Cognos Powerplay™, MicroStrategy™, SAS EnterpriseMiner™, Hyperion Essbase™ or Databeacon™. Other analytical tools may also be used, such as custom applications.

FIG. 2illustrates a system50for mapping a hypothesis to an analytical structure in accordance with an embodiment of the present invention. System50includes a memory52, a database54, a processor56, an input device58, and an output device60. Processor56is typically a microprocessor, controller or any other suitable computing device or resource. Processor56is adapted to execute various types of computer instructions in various computer languages for implementing functions available within system50. Memory52will usually be any form of volatile or non-volatile memory including, without limitation, magnetic media, optical media, random access memory (RAM), read-only (ROM), removable media or any other suitable memory component. Memory52includes components or software executable by processor56. Components of memory52may be otherwise combined and/or divided for processing within the scope of the present invention. Memory52includes a hypothesis representation module62, a hypothesis mapping module64and an analytical tool66. In particular embodiments hypothesis representation module62, hypothesis mapping module64and analytical tool66may be combined into or encompassed in any number of components.

Database54acts as a storage vehicle for system50. Database54may include various types of data and information used in the testing and analysis of a hypothesis. For example, database54may be a customer database of a business that includes information on the types of products purchased, frequency of purchase, quantity of purchase, and other general information on customers, e.g., age, gender, marital status, etc. As another example, database54may include accident records for an automobile maker, such as dates of accidents, automobile types involved in the accidents, ages of drivers involved in the accidents, weather at the times of the accidents, etc.

The information included in database54may be compiled using records and dimensions. Records are domains of information. Dimensions are fields or attributes. The dimensions may be discrete, continuous or another type. For example, a discrete dimension may be a color like red, blue or green. A continuous dimension may be, for example, age.

Hypothesis representation module62is adapted to generate a representation of a hypothesis received through a human computer interface or a machine. An example form of such hypothesis representation is further discussed below.

Hypothesis mapping module64is adapted to generate a script by mapping the hypothesis representation generated by hypothesis representation module62into grammar of a neutral format to be read and understood by analytical tool66. The neutral format of the grammar enables analytical tools of various types to understand the script without additional or substantial customization. The script may include the hypothesis to be tested, the data associated with such hypothesis and any relevant conditions. The mapping of the hypothesis representation into grammar for the script may be a one-to-one mapping (i.e., data source to data source, record to record, dimension to dimension, range within a dimension to range within a dimension).

Analytical tool66is able to analyze the script generated by hypothesis mapping module64and test the hypothesis represented by the script using data from database54to create an analytical structure. Analytical tool66may be a type that is commercially available, such as Cognos Powerplay™, MicroStrategy™, SAS EnterpriseMiner™, Hyperion Essbase™ or Databeacon™, or a customized type.

System50also includes an input device58and an output device60. Input device58may be a keyboard, mouse, touch pad or any other suitable component for inputting information into the system. Output device60may be a disk drive, printer, display or any other component for outputting information such as an analysis report of the results of hypothesis testing. System50may include other components, such as a modem for making connections to external communication media.

FIG. 3illustrates a data representation, in accordance with an embodiment of the present invention. Data may be represented in the illustrated manner in a data source or a database. Data representation75includes tables80,82and84. Each table includes rows (r) of records and columns (d) of dimensions. As stated above, records are domains of information. Dimensions are fields or attributes. The dimensions may be discrete, continuous or another type. For example, a discrete dimension may be a color like red, blue or green. A continuous dimension may be, for example, age.

In a particular embodiment, R denotes the set of data records in all tables (R={R1U R2U . . . Rm}). Ridenotes the set of records in table i (Ri={ri1, ri2, . . . rik}). D denotes the set of dimensions in all tables (D={D1, D2, . . . Dg}). Didenotes the set of dimensions in table i with m dimensions Di={di1, di2, . . . din|dijis of discrete, continuous or other type}). λd is a subset of values of dimension d as a function of d's type, for example, discrete, continuous, etc. such that D′i={λdij|dijεDiwhere j=1 . . . n}. For example, if a dimension type is discrete and the dimensions are people's first names such as John, Joe, Beth and Jane, then a λd may represent, for example, the male names (John, Joe) and not the female names. As another example, if a hypothesis is analyzing car accidents involving people of a particular age group, or example ages 18–20, then the appropriate λd for the hypothesis would be the subset of people who are 18 to 20 years old out of a set of people of all ages. As another example a hypothesis that involved particular dates might be limited to dates in the months of December and January only. Thus, λd is a subset of the data in any of the dimensions, and λd depends on the type of such data, whether discrete, continuous or otherwise.

FIG. 4is a flowchart illustrating a method for mapping a hypothesis to an analytical structure in accordance with an embodiment of the present invention. The method begins at step100where a hypothesis is received from an interface. The interface may be a human interface or a machine interface. The hypothesis may be received in any number of formats. For example, a hypothesis received from a human interface may be in a different format than a hypothesis received from another human interface or a machine interface. Likewise, a hypothesis received from a machine interface may be in a different format than a hypothesis received from another machine interface or a human interface. The ability to receive hypothesis in different formats saves time and resources since the amount of training relating to how hypotheses should be entered is reduced. The source of data against which the hypothesis may be tested may also be received or indicated.

At step102, a representation of the hypothesis is generated which is internal (as opposed to a linguistic expression of the hypothesis which is external). The representation of the hypothesis comprises a format from which a script for an analytical tool can be subsequently generated. In particular embodiments, the representation indicates data against which the hypothesis may be tested. An example of the form of a hypothesis representation, such as a hypothesis representation generated by hypothesis representation module62ofFIG. 2is as follows:
Hi={{r|rεRk, Rk⊂R}∩{λd|λdεD′j⊂D}}
where r defines the domain, records or prior observations used to test the hypothesis and where λd defines the relevant dimensions and value ranges. As an example, a hypothesis might include a statement such as “Drivers of ages 18–20 in midwestern states are more likely to kill themselves when driving SUVs.” In this example, the domain (r) may include a set of all automobile accidents. The relevant dimensions or value ranges (λd) may include fatal automobile accidents, automobile accidents in midwestern states, and automobile accidents involving drivers of ages 18–20.

At step104, a linguistic expression of the hypothesis is generated. The linguistic expression is a formulation of the hypothesis that a user can understand. In particular embodiments, the linguistic expression of the hypothesis can be formulated by filling in blanks of a template with actual data. Many types of templates may be used for various hypotheses. Each template expresses a unique concept in natural language. For example, a sample linguistic template may be as follows:When <field> is <operator> <value> and <field> is <value>, then <field> is likely to be <value> approximately <precision> of the time.
A sample linguistic expression of a hypothesis wherein the template includes actual data may be as follows:When temperature is greater than 45 and humidity is high, then defect is likely to be run-in-paint approximately 60% of the time.
Particular embodiments may not include generating a linguistic expression of the hypothesis.

At step106, a script of the hypothesis representation is generated by mapping the representation into a neutral grammar. The script is generated in a neutral grammar for automatic understanding by any of a number of analytical tools without additional or substantial customization. As stated above, the mapping of the hypothesis representation into grammar for the script may be a one-to-one mapping. The automatically generated script of a neutral grammar saves substantial time and resources because the hypothesis will not have to be entered in a format that is specific to a particular analytical tool.

In particular embodiments, the grammar used for the script may be tag-based. Table A below illustrates example script tags which may be used in particular embodiments.

In particular embodiments, the following features from Table A may be part of the r (records) portion of a hypothesis: data source, data source type, data source path, database, table/query, database field(s), database record(s), field type and SQL statement. It should be understood that in other embodiments, some of the features discussed above may be part of the d (dimensions) portion of a hypothesis. In particular embodiments, the following features from Table A may be part of the d (dimensions) portion of a hypothesis or part of another portion of a hypothesis: hypothesis, hypothesis dimension, record sample rate, hypothesis member(s), hypothesis description, dimension subset, function library, if condition, else condition, then condition, variable and expression. It should be understood that in other embodiments, some of the features discussed above may be part of the r (records) portion of a hypothesis or part of another portion of a hypothesis.

An example script generated by hypothesis mapping module64ofFIG. 2using some of the example script tags above from Table A is as follows:

<analysis><title> Root causes of vehicle paint defects </title><description> Using one months of data from the paintshop at GM's Saturn plant in TN, a number of conceptswere developed to study the causes of defects inpaint </description><ds> Saturn data, 2/1997<dstype>Validation</dstype><dspath> \\server5\ <\dspath> <db>sat-pnt-0297.mdb<table>week1</table><table>week2</table><table>week3</table> <table>week4</table></db></ds><hypothesis><hdescription>When temperature is highand humidity is low, there is a 50% chance for pop inpaint<samplerate>45</samplerate></hdescription><hdimension>Temperature<hdsub>45–100></hdsub></hdimension><hdimension>Humidity<hdsub>15–25</hdsub></hdimension><hdimension>Defect<hmember>pop</hmember></hdimension><hdimension>Panel</hdimension></hypothesis></analysis>
In the example script above, the title is “Root causes of vehicle paint defects.” The description is “Using one month of data from the paint shop at GM's Saturn™ plant in Tennessee, a number of concepts were developed to study the causes of defects in paint.” The data source is “Saturn data” from February, 1997. The data source type is validation. The database is identified as “sat-pnt-0297. mdb,” and four tables are selected (“week 1,”“week 2,” “week 3” and “week 4”). The hypothesis is then described as “When temperature is high and humidity is low, there is a 50% chance for pop in paint.” The sample rate 45%. Dimensions (or fields) of “Temperature” within a range of 45–100 will be used, and dimensions of “Humidity” within a range of 15% to 25% will be used to test the hypothesis. The script also refers to dimensions “Defect” and “Panel;” however, nothing is pulled from them because the hypothesis is attempting to conclude what types of defects arise and the panels on which they arise. Thus, the example script above describes the hypothesis and the data against which the hypothesis will be tested.

At step108, the script is transmitted to an analytical tool for analysis. The analytical tool to which the script is transmitted may be one that is commercially available or one of a customized type.

At step110, an analytical tool analyzes the script to form an analytical structure. The analytical structure is a representation of the hypothesis that is suitable for analysis by the analytical tool.

An example of the form of an analytical structure, such as an analytical structure formulated by analytical tool66ofFIG. 2is as follows:
Ai={{f(d)|dεDk, Dk⊂D}, {r′:r′εRj, Rj⊂R}, Ci}
where f(d) is a function operating on d, r′ is the test data records, and Ciis the statistical confidence of Ai. Particular embodiments of the present invention may not include step110.

Although the present invention has been described in detail, various changes and modifications may be suggested to one skilled in the art. It is intended that the present invention encompass such changes and modifications as falling within the scope of the appended claims.