Search indexing using discourse trees

Systems, devices, and methods of the present invention create a searchable index that includes informative portions of text. In an example, a computer-implemented method creates a discourse tree from a body of text. For each non-terminal node in the discourse tree, the method identifies a rhetorical relationship associated with the non-terminal node. The method labels each terminal node associated with the non-terminal node as either a nucleus or a satellite. The method further accesses a rule associated with the rhetorical relationship, and selects, based on the rule, selects the fragment associated with the nucleus. The method creates a searchable index including the selected fragments.

TECHNICAL FIELD

This disclosure is generally concerned with linguistics. More specifically, this disclosure relates to using communicative discourse trees to improve search indexing.

NOT APPLICABLE

BACKGROUND

Much online content is available via question-answer pairs such as frequently-asked questions stored on customer portals or internal company portals. Question-answer pairs can be an efficient manner to familiarize a user with content. In some cases, autonomous agents (ChatBots) can import such question-answer pairs in order to field user questions.

But such question-answer pairs can contain content that is not central to a topic of an answer. For example, content can include text that is irrelevant or misleading, non-responsive to the particular question, or is neutral and not helpful. If irrelevant text is indexed by a keyword-based search engine, the precision of the search engine is lowered. Moreover, an autonomous agent attempting to answer a user question based on erroneously-indexed text may answer the question incorrectly, resulting in lowered user confidence in the agent.

As such, solutions are needed for identifying informative text from text to be indexed.

BRIEF SUMMARY

Generally, systems, devices, and methods of the present invention use discourse trees to determine informative parts of an answer. In an example, a method accesses a body of text including fragments. The method creates a discourse tree from the body of text. The discourse tree includes nodes. Each nonterminal node represents a rhetorical relationship between two of the fragments. Each terminal node is associated with one or more fragments and is associated with a non-terminal node. The method identifies a rhetorical relationship associated with the non-terminal node for each non-terminal node. The method labels each terminal node associated with the non-terminal node as either a nucleus or a satellite, accesses a rule associated with the rhetorical relationship. The rule specifies accessing a fragment of text associated with one or more of the nucleus or the satellite. The method selects, based on the rule, one or more of the fragment associated with the nucleus or the fragment associated with the satellite. The method creates a searchable index including multiple entries, each entry corresponding to a selected fragment.

In an aspect, each entry can represent a question that can be answered by the body of text or a phrase to be matched to the question.

In an aspect, the rule specifies selecting the fragments associated with a nucleus and discarding fragments associated with a satellite.

In a further aspect, one or more fragments comprise clauses of compound sentences.

In yet another aspect, each rhetorical relationship is one of an elaboration, enablement, condition, contrast, or attribution.

In yet another aspect, the method, for at least one non-terminal node, identifies the rhetorical relationship as either a same unit or a joint relationship and selects both the nucleus and the satellite. The rule specifies accessing both the nucleus and the satellite.

In yet another aspect, the method, for at least one non-terminal node, identifies the rhetorical relationship as an attribution relationship, and responsive to determining that the rhetorical relationship does not represent a query by an author, selects the nucleus.

In yet another aspect, the method creates an index of additional questions by determining, for each entry of the searchable index, an additional question that is answered by the entry. The method receives a query from an external device. The method searches in each of the questions of multiple question-answer pairs, for matches corresponding to the query. The method, responsive to determining that fewer than a threshold number of matches are available, searches, in the index of additional questions, for additional matches to the query. The method, responsive to determining that fewer than a threshold number of additional matches are available in the index of additional questions, searches for further matches in each answers of the plurality of question-answer pairs. The method provides a response to the external device.

In yet another aspect, the method accesses question-answer pairs, each answer includes fragments. The method creates, for each answer, a discourse tree. The discourse tree includes nodes, each nonterminal node representing a rhetorical relationship between two of the fragments and each terminal node is associated with one or more fragments. For each non-terminal node in each answer, the method identifies a rhetorical relationship associated with the non-terminal node. The method labels each terminal node associated with the non-terminal node as either a nucleus or a satellite. The method accesses a rule associated with the rhetorical relationship. The rule specifies accessing a fragment of text associated with either the nucleus or the satellite. The method selects, based on the rule, one or more of the fragment associated with the nucleus or the fragment associated with the satellite. The method creates a searchable index of additional questions, the searchable index including multiple entries, each entry corresponding to one of the selected fragments for the answers.

In yet another aspect, the method receives a query from an external device. The method searches in each of the questions of multiple question-answer pairs, for matches corresponding to the query. The method, responsive to determining that fewer than a threshold number of matches are available, searches in the searchable index of additional questions, for additional matches to the query. The method, responsive to determining that fewer than a threshold number of additional matches are available in the searchable index of additional questions, searches for further matches in each answer of the plurality of question-answer pairs. The method provides a response to the external device.

In yet another aspect, a method trains a classification model to determine text to index. The method accesses a set of training data including a set of training pairs. Each training data pair includes text and an expected classification. The set of training data includes both a first training data pair that has an expected classification of a nucleus and a second training data pair that has an expected classification of a satellite. The method trains a classification model by iteratively providing one of the training data pairs to the classification model, receiving, from the classification model, a determined classification, calculating a loss function by calculating a difference between the determined classification and the expected classification, adjusting internal parameters of the classification model to minimize the loss function.

The above methods can be implemented on tangible computer-readable media and/or operating within a computer processor and attached memory.

DETAILED DESCRIPTION

As discussed above, existing search solutions index entire bodies of text without regard to which parts of the text are informative, thereby causing erroneous results and lower search precision. In contrast, aspects described herein use discourse trees to identify informative fragments of text for indexing. In so doing, certain aspects improve the precision of search engines and autonomous agents that answer questions from users. Precision refers to a fraction of informative instances among retrieved instances.

Other aspects use trained machine learning models to identify informative fragments of text for indexing, permitting the application of sets of rules that dictate which parts of a discourse tree are informative for indexing. Further aspects generate alternative questions from the indexed text that can be answered by the body of text, thereby improving search recall (the fraction of informative instances that have been retrieved of the total amount of informative instances).

Discourse trees originate from Rhetorical Structure Theory (RST). RST models a logical organization of text employed by an author, relying on relations between parts of text. RST simulates text coherence by forming a hierarchical, connected structure of texts via discourse trees. Rhetoric relations are split into the classes of coordinate and subordinate; these relations hold across two or more text spans and therefore implement coherence. These text spans are called elementary discourse units (EDUs). Clauses in a sentence and sentences in a text are logically connected by the author. The meaning of a given sentence is related to that of the previous and the following sentences.

The leaves of a discourse tree correspond to EDUs, the contiguous atomic text spans. Adjacent EDUs are connected by coherence relations (e.g., attribution, sequence), forming higher-level discourse units. The leaves of a particular EDU are logically related. This relationship is referred to as the coherence structure of the text. Example relations include elaboration and enablement. As used herein, “nuclearity” refers to which text segment, fragment, or span, is more central to an author's purpose. A “nucleus” refers to a span of text that is more central to an author's purpose than a “satellite,” which is less central to an author's purpose. Certain aspects use the determined EDUs of a discourse tree for to a body of text and the relations between the EDUs to determine which EDUs should be indexed for search. Different relations (e.g., elaboration, contrast, etc.) can employ different rules.

The following non-limiting example is provided to introduce certain aspects. An indexing application receives the following text that is a review of a digital camera: “it shoots well in low light, and I made a few good shots on a boat at night.” This text would be a good answer for the following questions: “Which camera suits well in low light?,” “How to shoot in low light,” “low light camera,” and “low light conditions.” In contrast, this text is not a good answer for search queries such as “good shorts at a boat,” “night boat,” “boat at night,” “good shots,” “good boat,” or “good night.”

Creating a discourse tree for the above compound sentence reveals nucleus “it shoots well in low light,” and satellite “I made a few good shots on a boat at night.” The nucleus and satellite are related by an “elaboration” relation because the satellite fragment elaborates the nucleus statement. As can be seen with the “elaboration” relationship, the nucleus fragment provides useful information while the satellite fragment does not. In particular, the nucleus phrase is more important text is strongly correlated with the potential question. In contrast, the satellite phrase provides less important details which are less likely to be asked directly. If this less important text is indexed, it might trigger this answer as a response to question on something totally different. Certain aspects use trained machine-learning models to predict which fragments within a body of text should be indexed for search based in part on the type of rhetorical relationship associated with the particular fragment. These aspects train machine learning models to predict informative EDUs to be indexed.

Technical advantages the present disclosure include improved search engine performance (precision, recall) and improved autonomous agents as compared to traditional statistical-based approaches. As discussed, existing keyword-based solutions erroneously index text that is not only non-informative but irrelevant. Considering the above example, existing search techniques may identify words and phrases from the original text such as “boat” or “good shots” as significant, thereby causing a search engine to erroneously index and rely on these phrases.

Certain Definitions

As used herein, “rhetorical structure theory” is an area of research and study that provided a theoretical basis upon which the coherence of a discourse could be analyzed.

As used herein, “discourse tree” or “DT” refers to a structure that represents the rhetorical relations for a sentence of part of a sentence.

As used herein, a “rhetorical relation,” “rhetorical relationship,” or “coherence relation” or “discourse relation” refers to how two segments of discourse are logically connected to one another. Examples of rhetorical relations include elaboration, contrast, and attribution.

As used herein, a “sentence fragment,” or “fragment” is a part of a sentence that can be divided from the rest of the sentence. A fragment is an elementary discourse unit. For example, for the sentence “Dutch accident investigators say that evidence points to pro-Russian rebels as being responsible for shooting down the plane,” two fragments are “Dutch accident investigators say that evidence points to pro-Russian rebels” and “as being responsible for shooting down the plane.” A fragment can, but need not, include a verb.

As used herein, “index” is a table, data structure, pointer, or other mechanism that links two keywords, data, or parts of text for purposes of data retrieval. For example, an index can include links to keywords, data or parts of text. Index can be a string match or an inverse index.

Turning now to the figures,FIG. 1shows an exemplary indexing environment in accordance with an aspect.FIG. 1depicts computing device101, input text130, user question140, answer150, data network104, server160, and mobile device170. Computing device101includes one or more of indexing application102, question-answer index135, machine-learning model120, and training data125. Computing device101can receive text and use discourse trees and/or machine learning models to determine informative parts of text to index, and can use an index to more accurately answer questions received via an autonomous agent.

More specifically, indexing application102can determine informative fragments of input text130, index the determined informative fragments, generate additional questions, generate a question-answer index135, or generate answer150using question-answer index135.

In an example, indexing application102receives a body of input text130that is directed to a particular news story with comments. Input text130contains multiple sentences that broadly speaking, address the topic of Hurricane “Lane” that made land on Hawaii, but the comments include irrelevant, opinionated, or incorrect statements such as “residents of Hawaii deserve this, who lives on an island?” Indexing application102creates a discourse tree from input text130. The discourse tree includes identification of different fragments of text as nuclei or satellites, some of which are as shown in Table 1 below.

TABLE 1NucleusSatelliteOfficials also expressedwhich they said have already scorched moreconcern about thethan 2,000 acres and destroyed severalspread of brush fires.homes.Dozens of residents havewading through waist-high water, andbeen forced to evacuatesometimes spreading brush fires, to safety,their homes over thewhile thousands of others remain in shelterspast two days.Federal Emergencymostly in OahuManagement officialssaid about 2,000 peopleremain in shelters.

Indexing application102applies a set of rules that govern whether, for a specific rhetoric relation, a nucleus or a satellite EDUs should be indexed. Indexing application102extracts the EDUs (fragments of text) that are informative. Indexing application102creates a question for each indexed fragment of text that can be answered by the particular fragment of text. Indexing application creates question-answer index135, which includes multiple entries. Each entry includes an indexed fragment of text and the corresponding generated question.

Subsequently, indexing application102receives a question from a user via a an autonomous agent (ChatBot). The question states “how many people remain in shelters?” Indexing application102accesses the question-answer index135, identifying the corresponding EDU “Federal Emergency Management officials said about 2,000 people remain in shelters.” Indexing application102provides this answer to the user device.

Mobile device170can be any mobile device such as a mobile phone, smart phone, tablet, laptop, smart watch, and the like. Mobile device170communicates via data network104to server160or computing device101. In this manner, mobile device170can provide questions from and output answers to a user, to server160or computing device101. Data network104can be any public or private network, wired or wireless network, Wide Area Network, Local Area Network, or the Internet.

In a further aspect, indexing application102uses a trained machine-learning model120to predict informative parts of fragments to index. An example of machine-learning model120is a classification model. Machine-learning model120is trained with training data125, which can include sets of positive-negative training pairs. Positive training pairs include text or text fragments identified as informative for indexing and negative training pairs include text or text fragments identified as not informative.

Rhetoric Structure Theory and Discourse Trees

Linguistics is the scientific study of language. For example, linguistics can include the structure of a sentence (syntax), e.g., subject-verb-object, the meaning of a sentence (semantics), e.g. dog bites man vs. man bites dog, and what speakers do in conversation, i.e., discourse analysis or the analysis of language beyond the sentence.

The theoretical underpinnings of discourse, Rhetoric Structure Theory (RST), can be attributed to Mann, William and Thompson, Sandra, “Rhetorical structure theory: A Theory of Text organization,” Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243-281, 1988. Similar to how the syntax and semantics of programming language theory helped enable modern software compilers, RST helped enabled the analysis of discourse. More specifically RST posits structural blocks on at least two levels, a first level such as nuclearity and rhetorical relations, and a second level of structures or schemas. Discourse parsers or other computer software can parse text into a discourse tree.

Rhetoric Relations

As discussed, aspects described herein use rhetorical relations and discourse trees. Rhetorical relations can be described in different ways. For example, Mann and Thompson describe twenty-three possible relations. C. Mann, William & Thompson, Sandra. (1987) (“Mann and Thompson”). Rhetorical Structure Theory: A Theory of Text Organization. Other numbers of relations are possible. Table 2 below lists different rhetorical relations.

TABLE 2Relation NameNucleusSatelliteAntithesisideas favored by theideas disfavored by the authorauthorBackgroundtext whosetext for facilitatingunderstanding isunderstandingbeing facilitatedCircumstancetext expressing thean interpretive context ofevents or ideassituation or timeoccurring in theinterpretive contextConcessionsituation affirmedsituation which is apparentlyby authorinconsistent but also affirmed byauthorConditionaction or situationconditioning situationwhose occurrenceresults from theoccurrence of theconditioningsituationElaborationbasic informationadditional informationEnablementan actioninformation intended to aid thereader in performing an actionEvaluationa situationan evaluative comment about thesituationEvidencea claiminformation intended to increasethe reader's belief in the claimInterpretationa situationan interpretation of the situationJustifytextinformation supporting thewriter's right to express the textMotivationan actioninformation intended to increasethe reader's desire to performthe actionNon-a situationanother situation which causesvolitionalthat one, but not by anyone'sCausedeliberate actionNon-a situationanother situation which is causedvolitionalby that one, but not by anyone'sResultdeliberate actionOtherwiseaction or situationconditioning situation(antiwhose occurrenceconditional)results from thelack of occurrenceof the conditioningsituationPurposean intended situationthe intent behind the situationRestatementa situationa reexpression of the situationSolutionhooda situation or methoda question, request, problem, orsupporting full orother expressed needpartial satisfactionof the needSummarytexta short summary of that textVolitionala situationanother situation which causesCausethat one, by someone'sdeliberate actionVolitionala situationanother situation which is causedResultby that one, by someone'sdeliberate action

Some empirical studies postulate that the majority of text is structured using nucleus-satellite relations. See Mann and Thompson. But other relations do not carry a definite selection of a nucleus. Examples of such relations are shown in Table 3 below.

FIG. 2depicts an example of a discourse tree in accordance with an aspect.FIG. 2includes discourse tree200. Discourse tree includes text span201, text span202, text span203, relation210and relation211. The numbers inFIG. 2correspond to the three text spans.FIG. 3corresponds to the following example text with three text spans numbered 1, 2, 3:

1. Honolulu, Hawaii will be site of the 2017 Conference on Hawaiian History

2. It is expected that 200 historians from the U.S. and Asia will attend

3. The conference will be concerned with how the Polynesians sailed to Hawaii

For example, relation210, or elaboration, describes the relationship between text span201and text span202. Relation210depicts the relationship, elaboration, between text span203and204. As depicted, text spans202and203elaborate further on text span201. In the above example, given a goal of notifying readers of a conference, text span 1 is the nucleus. Text spans 2 and 3 provide more detail about the conference. InFIG. 2, a horizontal number, e.g., 1-3, 1, 2, 3 covers a span of text (possibly made up of further spans); a vertical line signals the nucleus or nuclei; and a curve represents a rhetoric relation (elaboration) and the direction of the arrow points from the satellite to the nucleus. If the text span only functions as a satellite and not as a nuclei, then deleting the satellite would still leave a coherent text. If fromFIG. 2one deletes the nucleus, then text spans 2 and 3 are difficult to understand.

FIG. 3depicts a further example of a discourse tree in accordance with an aspect.FIG. 3includes components301and302, text spans305-307, relation310and relation311. Relation310depicts relation310, enablement, between components306and305, and307, and305.FIG. 3refers to the following text spans:

1. The new Tech Report abstracts are now in the journal area of the library near the abridged dictionary.

2. Please sign your name by any means that you would be interested in seeing.

3. Last day for sign-ups is 31 May.

As can be seen, relation310depicts the relationship between entity307and306, which is enablement.FIG. 3illustrates that while nuclei can be nested, there exists only one most nuclear text span.

Constructing a Discourse Tree

Discourse trees can be generated using different methods. A simple example of a method to construct a DT bottom up is:

(1) Divide the discourse text into units by:(a) Unit size may vary, depending on the goals of the analysis(b) Typically, units are clauses

(2) Examine each unit, and its neighbors. Is there a relation holding between them?

(3) If yes, then mark that relation.

(4) If not, the unit might be at the boundary of a higher-level relation. Look at relations holding between larger units (spans).

(5) Continue until all the units in the text are accounted for.

Mann and Thompson also describe the second level of building block structures called schemas applications. In RST, rhetoric relations are not mapped directly onto texts; they are fitted onto structures called schema applications, and these in turn are fitted to text. Schema applications are derived from simpler structures called schemas (as shown byFIG. 4). Each schema indicates how a particular unit of text is decomposed into other smaller text units. A rhetorical structure tree or DT is a hierarchical system of schema applications. A schema application links a number of consecutive text spans, and creates a complex text span, which can in turn be linked by a higher-level schema application. RST asserts that the structure of every coherent discourse can be described by a single rhetorical structure tree, whose top schema creates a span encompassing the whole discourse.

FIG. 4depicts illustrative schemas in accordance with an aspect.FIG. 4shows a joint schema is a list of items consisting of nuclei with no satellites.FIG. 4depicts schemas401-406. Schema401depicts a circumstance relation between text spans410and411. Scheme402depicts a sequence relation between text spans420and421and a sequence relation between text spans421and422. Schema403depicts a contrast relation between text spans430and431. Schema404depicts a joint relationship between text spans440and441. Schema405depicts a motivation relationship between450and451, and an enablement relationship between452and451. Schema406depicts joint relationship between text spans460and462. An example of a joint scheme is shown inFIG. 4for the three text spans below:

1. Skies will be partly sunny in the New York metropolitan area today.

2. It will be more humid, with temperatures in the middle 80's.

3. Tonight will be mostly cloudy, with the low temperature between 65 and 70.

WhileFIGS. 2-4depict some graphical representations of a discourse tree, other representations are possible.

FIG. 5depicts a node-link representation of the hierarchical binary tree in accordance with an aspect. As can be seen fromFIG. 5, the leaves of a DT correspond to contiguous non-overlapping text spans called Elementary Discourse Units (EDUs). Adjacent EDUs are connected by relations (e.g., elaboration, attribution...) and form larger discourse units, which are also connected by relations. “Discourse analysis in RST involves two sub-tasks: discourse segmentation is the task of identifying the EDUs, and discourse parsing is the task of linking the discourse units into a labeled tree.” See Joty, Shafiq R and Giuseppe Carenini, Raymond T Ng, and Yashar Mehdad. 2013. Combining intra-and multi-sentential rhetorical parsing for document-level discourse analysis. In ACL (1), pages 486-496.

FIG. 5depicts text spans that are leaves, or terminal nodes, on the tree, each numbered in the order they appear in the full text, shown inFIG. 6.FIG. 5includes tree500. Tree500includes, for example, nodes501-507. The nodes indicate relationships. Nodes are non-terminal, such as node501, or terminal, such as nodes502-507. As can be seen, nodes503and504are related by a joint relationship. Nodes502,505,506, and508are nuclei. The dotted lines indicate that the branch or text span is a satellite. The relations are nodes in gray boxes.

FIG. 6depicts an exemplary indented text encoding of the representation inFIG. 5in accordance with an aspect.FIG. 6includes text600and text sequences602-604. Text600is presented in a manner more amenable to computer programming. Text sequence602corresponds to node502, sequence603corresponds to node503, and sequence604corresponds to node504. InFIG. 6, “N” indicates a nucleus and “S” indicates a satellite.

Examples of Discourse Parsers

Automatic discourse segmentation can be performed with different methods. For example, given a sentence, a segmentation model identifies the boundaries of the composite elementary discourse units by predicting whether a boundary should be inserted before each particular token in the sentence. For example, one framework considers each token in the sentence sequentially and independently. In this framework, the segmentation model scans the sentence token by token, and uses a binary classifier, such as a support vector machine or logistic regression, to predict whether it is appropriate to insert a boundary before the token being examined. In another example, the task is a sequential labeling problem. Once text is segmented into elementary discourse units, sentence-level discourse parsing can be performed to construct the discourse tree. Machine learning techniques can be used.

In one aspect of the present invention, two Rhetorical Structure Theory (RST) discourse parsers are used: CoreNLPProcessor which relies on constituent syntax, and FastNLPProcessor which uses dependency syntax. See Surdeanu, Mihai & Hicks, Thomas & Antonio Valenzuela-Escarcega, Marco. Two Practical Rhetorical Structure Theory Parsers. (2015).

In addition, the above two discourse parsers, i.e., CoreNLPProcessor and FastNLPProcessor use Natural Language Processing (NLP) for syntactic parsing. For example, the Stanford CoreNLP gives the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize dates, times, and numeric quantities, mark up the structure of sentences in terms of phrases and syntactic dependencies, indicate which noun phrases refer to the same entities. Practically, RST is a still theory that may work in many cases of discourse, but in some cases, it may not work. There are many variables including, but not limited to, what EDU's are in a coherent text, i.e., what discourse segmenters are used, what relations inventory is used and what relations are selected for the EDUs, the corpus of documents used for training and testing, and even what parsers are used. So for example, in Surdeanu, et al., “Two Practical Rhetorical Structure Theory Parsers,” paper cited above, tests must be run on a particular corpus using specialized metrics to determine which parser gives better performance. Thus unlike computer language parsers which give predictable results, discourse parsers (and segmenters) can give unpredictable results depending on the training and/or test text corpus. Thus, discourse trees are a mixture of the predicable arts (e.g., compilers) and the unpredictable arts (e.g., like chemistry were experimentation is needed to determine what combinations will give you the desired results).

In order to objectively determine how good a Discourse analysis is, a series of metrics are being used, e.g., Precision/Recall/F1 metrics from Daniel Marcu, “The Theory and Practice of Discourse Parsing and Summarization,” MIT Press, (2000). Precision, or positive predictive value is the fraction of informative instances among the retrieved instances, while recall (also known as sensitivity) is the fraction of informative instances that have been retrieved over the total amount of informative instances. Both precision and recall are therefore based on an understanding and measure of relevance. Suppose a computer program for recognizing dogs in photographs identifies eight dogs in a picture containing 12 dogs and some cats. Of the eight dogs identified, five actually are dogs (true positives), while the rest are cats (false positives). The program's precision is ⅝ while its recall is 5/12. When a search engine returns 30 pages only 20 of which were informative while failing to return 40 additional informative pages, its precision is 20/30=⅔ while its recall is 20/60=⅓. Therefore, in this case, precision is ‘how useful the search results are’, and recall is ‘how complete the results are.’” The F1 score (also F-score or F-measure) is a measure of a test's accuracy. It considers both the precision and the recall of the test to compute the score: F1=2×((precision×recall)/(precision+recall)) and is the harmonic mean of precision and recall. The F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.

Analyzing Request and Response Pairs

FIG. 7depicts an exemplary discourse tree for an example request about property tax in accordance with an aspect. The node labels are the relations and the arrowed line points to the satellite. The nucleus is a solid line.FIG. 7depicts the following text.

Request: “My husbands' grandmother gave him his grandfather's truck. She signed the title over but due to my husband having unpaid fines on his license, he was not able to get the truck put in his name. I wanted to put in my name and paid the property tax and got insurance for the truck. By the time it came to sending off the title and getting the tag, I didn't have the money to do so. Now, due to circumstances, I am not going to be able to afford the truck. I went to the insurance place and was refused a refund. I am just wondering that since I am not going to have a tag on this truck, is it possible to get the property tax refunded?”

Response: “The property tax is assessed on property that you own. Just because you chose to not register it does not mean that you don't own it, so the tax is not refundable. Even if you have not titled the vehicle yet, you still own it within the boundaries of the tax district, so the tax is payable. Note that all states give you a limited amount of time to transfer title and pay the use tax. If you apply late, there will be penalties on top of the normal taxes and fees. You don't need to register it at the same time, but you absolutely need to title it within the period of time stipulated in state law.”

As can be seen inFIG. 7, analyzing the above text results in the following. “My husbands' grandmother gave him his grandfather's truck” is elaborated by “She signed the title over but due to my husband” elaborated by “having unpaid fines on his license, he was not able to get the truck put in his name.” which is elaborated by “I wanted to put in my name,” “and paid the property tax”, and “and got insurance for the truck.”

“My husbands' grandmother gave him his grandfather's truck. She signed the title over but due to my husband having unpaid fines on his license, he was not able to get the truck put in his name. I wanted to put in my name and paid the property tax and got insurance for the truck.” is elaborated by;

“I didn't have the money” elaborated by “to do so” contrasted with [0097] “By the time” elaborated by “it came to sending off the title”

“and getting the tag”

“My husbands' grandmother gave him his grandfather's truck. She signed the title over but due to my husband having unpaid fines on his license, he was not able to get the truck put in his name. I wanted to put in my name and paid the property tax and got insurance for the truck. By the time it came to sending off the title and getting the tag, I didn't have the money to do so” is contrasted with

“Now, due to circumstances,” elaborated with “I am not going to be able to afford the truck.” which is elaborated with

“I went to the insurance place”

“and was refused a refund”

“My husbands' grandmother gave him his grandfather's truck. She signed the title over but due to my husband having unpaid fines on his license, he was not able to get the truck put in his name. I wanted to put in my name and paid the property tax and got insurance for the truck. By the time it came to sending off the title and getting the tag, I didn't have the money to do so. Now, due to circumstances, I am not going to be able to afford the truck. I went to the insurance place and was refused a refund.” is elaborated with

“I am just wondering that since I am not going to have a tag on this truck, is it possible to get the property tax refunded?”

“I am just wondering” has attribution to

“that” is the same unit as “is it possible to get the property tax refunded?” which has condition “since I am not going to have a tag on this truck”

As can be seen, the main subject of the topic is “Property tax on a car”. The question includes the contradiction: on one hand, all properties are taxable, and on the other hand, the ownership is somewhat incomplete. A good response has to address both topic of the question and clarify the inconsistency. To do that, the responder is making even stronger claim concerning the necessity to pay tax on whatever is owned irrespectively of the registration status. This example is a member of positive training set from our Yahoo! Answers evaluation domain. The main subject of the topic is “Property tax on a car”. The question includes the contradiction: on one hand, all properties are taxable, and on the other hand, the ownership is somewhat incomplete. A good answer/response has to address both topic of the question and clarify the inconsistency. The reader can observe that since the question includes rhetoric relation of contrast, the answer has to match it with a similar relation to be convincing. Otherwise, this answer would look incomplete even to those who are not domain experts.

FIG. 8depicts an exemplary response for the question represented inFIG. 7, according to certain aspects of the present invention. The central nucleus is “the property tax is assessed on property” elaborated by “that you own”. “The property tax is assessed on property that you own” is also a nucleus elaborated by “Just because you chose to not register it does not mean that you don't own it, so the tax is not refundable. Even if you have not titled the vehicle yet, you still own it within the boundaries of the tax district, so the tax is payable. Note that all states give you a limited amount of time to transfer title and pay the use tax.”

The nucleus “The property tax is assessed on property that you own. Just because you chose to not register it does not mean that you don't own it, so the tax is not refundable. Even if you have not titled the vehicle yet, you still own it within the boundaries of the tax district, so the tax is payable. Note that all states give you a limited amount of time to transfer title and pay the use tax.” is elaborated by “there will be penalties on top of the normal taxes and fees” with condition “If you apply late,” which in turn is elaborated by the contrast of “but you absolutely need to title it within the period of time stipulated in state law.” and “You don't need to register it at the same time.”.

Comparing the DT ofFIG. 7and DT ofFIG. 8, enables a determination of how well matched the response (FIG. 8) is to the request (FIG. 7). In some aspects of the present invention, the above framework is used, at least in part, to determine the DTs for the request/response and the rhetoric agreement between the DTs.

In another example, the question “What does The Investigative Committee of the Russian Federation do” has at least two answers, for example, an official answer or an actual answer.

FIG. 9illustrates a discourse tree for an official answer in accordance with an aspect. As depicted inFIG. 9, an official answer, or mission statement states that “The Investigative Committee of the Russian Federation is the main federal investigating authority which operates as Russia's Anti-corruption agency and has statutory responsibility for inspecting the police forces, combating police corruption and police misconduct, is responsible for conducting investigations into local authorities and federal governmental bodies.”

FIG. 10illustrates a discourse tree1000for a raw answer in accordance with an aspect. As depicted inFIG. 10, another, perhaps more honest, answer states that “Investigative Committee of the Russian Federation is supposed to fight corruption. However, top-rank officers of the Investigative Committee of the Russian Federation are charged with creation of a criminal community. Not only that, but their involvement in large bribes, money laundering, obstruction of justice, abuse of power, extortion, and racketeering has been reported. Due to the activities of these officers, dozens of high-profile cases including the ones against criminal lords had been ultimately ruined.”

The choice of answers depends on context. Rhetoric structure allows differentiating between “official”, “politically correct”, template-based answers and “actual”, “raw”, “reports from the field”, or “controversial” answers. (SeeFIGS. 9 and 10). Sometimes, the question itself can give a hint about which category of answers is expected. If a question is formulated as a factoid or definitional one, without a second meaning, then the first category of answers is suitable. Otherwise, if a question has the meaning “tell me what it really is,” then the second category is appropriate. In general, after extracting a rhetoric structure from a question, selecting a suitable answer that would have a similar, matching, or complementary rhetoric structure is easier.

The official answer is based on elaboration and joints, which are neutral in terms of controversy a text might contain (SeeFIG. 9). At the same time, the row answer includes the contrast relation. This relation is extracted between the phrase for what an agent is expected to do and what this agent was discovered to have done.

Using Discourse Trees to Determine Informative Text for Indexing

As discussed, certain aspects use discourse trees to determine informative portions of text to index. This approach increases the amount of text that is informative in answers to questions provided by search engines or autonomous agents and facilitates the generation of additional questions that can be answered by a body of text, thereby making the text more accessible via question answering. Certain aspects are described with respect toFIG. 11for example purposes.

FIG. 11illustrates a discourse tree for an example of an answer to a question, in accordance with an aspect.FIG. 11illustrates discourse tree1100, which represents an answer to the following question: “How should I plan to pay for taxes resulting from converting to a Roth IRA?”FIG. 11also depicts elementary discourse units corresponding to fragments of the text as terminal nodes1110-1125.

The answer corresponding to discourse tree100is: “[t]o help maximize your retirement savings, it's generally a good idea to consider not using the proceeds from the conversion to pay the resulting tax costs. Instead, you should consider using cash or other savings held in nonretirement accounts. Using retirement account funds to pay the taxes will reduce the amount you would have available to potentially grow tax-free in your new Roth IRA. Additionally, if you are under 59½, using funds from your retirement account could result in an additional 10% tax penalty, which may significantly reduce the potential benefit of conversion.”

The answer could be obtained from a source such as a Frequently Asked Questions (FAQ) database or a question-answer index. A question-answer index can include multiple questions and corresponding answers. But as further explained, some fragments in each answer are more informative to answering a question than other fragments. For example, the phrase “it is generally a good idea” adds little to the answer, whereas “consider not using the proceeds from the conversion” is informative to the user who posed the original question.

Additionally, each answer in the question-answer index may provide additional questions that can be answered, which are in turn indexed, increasing the usefulness of the data. For example, “at what age do I pay a penalty for using retirement funds?” could be answered by the text (e.g., “age 59½”). Certain aspects can determine informative text from a body of text and such additional questions that can be answered from the body of text.

The nodes selected for indexing are nodes1110,1112,1115,1117,1119,1123, and1124(shown with underlining). A general rule is that the EDUs representing nuclei should be indexed, and the EDUs representing satellites is discarded.

As depicted, a node that is the first, or upper, of two related nodes is a nucleus. The node that is the second, or lower, of two related nodes, is the satellite. Note that in a conditional relation, the “if” EDU is the satellite. For example, phrases from the nucleus elementary discourse units and their respective nodes are:help maximize your retirement savings (Node1110)proceeds from the conversion (Node1112)cash or other savings held in nonretirement accounts (Node1115)retirement account funds; (Node1117)using funds from your retirement account (Node1123)result in an additional 10% tax penalty (Node1124)

The satellite EDU expressions are:it's generally a good idea (Node1111)pay the resulting tax costs (Node1118)held in nonretirement accounts (Node1116)to pay the taxes (Node1118)you would have available to potentially (Node1120)if you are under 59½ . . . (Node1122)

It can be seen that generally speaking the nodes of terminal nodes1100-1125that are identified as nuclei are directly informative to the topic of the answer and therefore should be indexed. In contrast, the satellite nodes for a particular rhetorical relationship are not informative to the question and therefore should not be indexed. For example, the phrase “it is generally a good idea” (node1111) is not related to finance. The phrase “pay the resulting tax costs” (node1118) is detached from the context of the question, as is “held in nonretirement accounts” (node1116). Additionally, “you would have available to potentially” (node1120) is a counterfactual question that is unlikely to occur in a user question. Finally, “if you are under 59½” (node1122) is a condition that is not necessary directly queried.

The above hypothesis that only EDUs that are nucleus of rhetoric relations should be indexed and all satellite EDUs should not be selected for indexing is illustrated by the “elaboration” relationship. In the elaboration relationship, the nucleus expresses more important information than satellite. A satellite may express a detail of information being communicated that is unlikely to be cited by a user query. For example, nodes1124and1125are related by an elaboration relationship. A can be seen, “could result in an additional 10% penalty” is more important than “which may significantly reduce the potential benefit of conversion.” Accordingly, the satellite fragments of text should not be matched with a potential question to deliver this particular answer.

Different Rhetoric Relations

But the general rule described above can be subject to certain exceptions. For example, under certain conditions, the “contrast” and “attribution” relations can require indexing of the satellite rather than the nucleus. Additionally, for the “same-unit” and “joint” relations, both the nucleus and the satellite are indexed. Different rhetoric relations can have different rules, as shown in Table 4 below:

TABLE 4RelationExampleIndexing ruleElaborationTo achieve some stateNucleus[nucleus] | do this and that[satellite]EnablementA query may be of the formNucleus“how to achieve some state?”but less likely be of the form“what can I achieve doingthis and that?”.ConditionA query may be of the formWhen the question is of the type“how to achieve some state?”“when/where/under what condition . . .”,but less likely be of the formindex the if part (the satellite)“what can I achieve doingthis and that?”.ContrastIndex the nucleus. The satellite includesfacts which are unusual, unexpected,unanticipated.AttributionGeneral rule: index the nucleus (may occurin a factoid question). Do not index thesatellite part on whom the factoid isattributed (as it is usually a minor detail).Exception: a query by an author. In thiscase, such queries texts should betransformed into a structured way andcovered by a different kind of searchtechnology.Same-UnitIndex both nucleus and satellite because ofthe symmetric relationship of same-unit.JointIndex both nucleus and satellite because ofthe symmetric relationship of joint.

FIG. 12depicts a flowchart illustrating an example of a process for determining informative text for indexing, in accordance with an aspect.FIG. 12is discussed with respect toFIG. 11for example purposes.

At block1201, process1200involves accessing a body of text including fragments. Examples of bodies of text are user utterances, online content, stored electronic content, and scanned documents. Text can include question/answer pairs. In an example, indexing application102accesses the text associated with discourse tree1100.

At block1202, process1200involves creating a discourse tree from the body of text. Creating a discourse tree involves determining elementary discourse units and the rhetorical relations between each pair of elementary discourse units. The discourse tree includes nodes, each nonterminal node representing a rhetorical relationship between two of the fragments, and each terminal node is associated with one or more fragments and is associated with a non-terminal node. Continuing the example, indexing application102creates discourse tree1100. Any discourse tree parser can be used.

At block1203, process1200involves identifying a rhetorical relationship for each non-terminal node. The rhetorical relationship indicates the relationship of the two child nodes, which can be terminal nodes, or other non-terminal nodes. Continuing the example, indexing application102identifies various rhetorical relations as illustrated inFIG. 11such as elaboration, same unit, enablement, etc. Each terminal node1100-1125is associated with a rhetorical relation. For example, indexing application102identifies nodes1112and1113as related by an enablement relation.

At block1204, process1200involves labeling each terminal node associated with a non-terminal node as a nucleus or a satellite. Using Rhetorical Structure Theory, indexing application102labels each terminal node as a nucleus or a satellite.

At block1205, process1200involves accessing a rule associated with the rhetorical relationship. The rule specifies accessing a fragment of text associated with either the nucleus or the satellite. Continuing the example, indexing application102accesses a rule for enablement. As described in Table 4, for EDUs associated with enablement, the nucleus is indexed. Indexing application102continues block1204for each rhetorical relation.

At block1206, process1200involves selecting, based on the rule, either (i) the fragment associated with the nucleus or the (ii) fragment associated with the satellite. Using the rule identified at block1204, indexing application102selects the nucleus, node1112, for indexing. Indexing application102continues block1205for each rhetorical relation.

At block1207, process1200involves creating a searchable index with multiple entries, each entry corresponding to a selected fragment and representing a question that can be answered by the body of text. Continuing the example, indexing application102creates a searchable index with all selected elementary discourse units gathered during block1205. Specifically, indexing application102gathers nodes1110,1112,1115,1117,1119,1123, and1124. In some cases, indexing application102determines a question that can be answered by each indexed EDU and stores the question along with the indexed text in question-answer index135.

Building Additional Questions from Indexed Text

Indexing application102can use a multi-tiered approach to answering questions or queries. For example, indexing application102can use the question-answer index135(e.g., a Frequently-Asked Question database) or the entirety of the text from which question-answer index135was constructed. In the example that follows, indexing application102can first check question-answer index135for a suitable match, then check the list of alternative questions, i.e., the searchable index determined by process1200, then search all of the answers in question-answer index135.

FIG. 13depicts a flowchart illustrating an example of a process for responding to searches, in accordance with an aspect.FIG. 13can use question-answer index135, a searchable index that comprises informative fragments of text determined by process1200, or another previously-indexed text.

At block1301, process1300involves creating an index of additional questions by determining, for each entry of the searchable index, an additional question that is answered by the entry. Different methods can be used to determine questions from a particular informative fragment of text.

In an example, indexing application102receives an arbitrary sentence and outputs a set of questions. Indexing application102builds a parse tree. Indexing application102selects nodes of the parse tree that represent nouns, verbs, and adjectives. Next, for each selected node, indexing application102forms a reduction of a parse tree by removing the particular node. Indexing application102builds a question for the reduction by substituting a Wh word (what, who, where, whose, why, etc.) for the removed node. Indexing application102selects a proper Wh word following a set of rules. In an example, rules include substituting “who” or “what” for a noun, “what . . . do,” and “which way” or “how is” for an adjective.

FIG. 14depicts a parse tree for a sentence used to illustrate forming a question from text, in accordance with an aspect.FIG. 14depicts a parse tree1400for the sentence “Joe packed his tools and materials neatly.” As depicted, parse tree1400includes 11 nodes, 7 of which are terminal nodes. Table 5 below illustrates questions generated by deleting odd-numbered nodes in the sentence using the method described.

TABLE 5Node deletedQuestion Generated3What did Joe pack neatly?5Who packed his tools and materials neatly?7Whose tools and materials did Joe pack neatly?9No question: Structure words are not deleted.11How did Joe pack his tools and materials?

Returning toFIG. 13, at block1302, process1300involves receiving a query from an external device such as mobile device170. A query can be an input to an autonomous agent (ChatBot), or a search provided by a user.

At block1303, process1300involves searching, in each question of the question-answer index for a match corresponding to the query. The question-answer pairs can be a FAQ, e.g., with a question and corresponding answer.

At block1304, process1300involves responsive to determining that fewer than a threshold number of matches are available, searching for additional matches to the query in the index of additional questions. The searchable index can be generated, for example, by process1200. By using the searchable index that contains only text fragments informative to the answer, indexing application102need not parse through and potentially output less-informative text to the question.

Continuing the example, index application102determines that less than a threshold number of matches to the query are available in the index of additional questions. Example thresholds are a percentage of the total available questions, a predetermined number, or a number of questions determined by a particular domain, subject, or system configuration. In response, indexing application102searches the question-answer index135.

At block1305, process1300involves, responsive to determining that fewer than a threshold number of additional matches are available in the index of additional questions, searching for further matches in each answer of the an index of original answers. The threshold used at block1305can be different from the threshold used at block1304. Continuing the example, indexing application102searches for a match in each answer of the question-answer index.

At block1306, process1300involves providing the response to the external device. Example external devices include mobile device170or remote devices connected across data network104. Alternatively, indexing application102can provide the answer directly to a user.

Autonomous Agents

As discussed, certain aspects described herein enable improved autonomous agents (ChatBots). A conversation between Human A and Human B is a form of discourse. For example, applications exist such as FaceBook® Messenger, WhatsApp®, Slack,® SMS, etc., a conversation between A and B may typically be via messages in addition to more traditional email and voice conversations. A ChatBot (which may also be called intelligent bots or virtual assistant, etc.) is an “intelligent” machine that, for example, replaces human B and to various degrees mimics the conversation between two humans. An example ultimate goal is that human A cannot tell whether B is a human or a machine (the Turning test, developed by Alan Turing in 1950). Discourse analysis, artificial intelligence, including machine learning, and natural language processing, have made great strides toward the long-term goal of passing the Turing test. Of course, with computers being more and more capable of searching and processing vast repositories of data and performing complex analysis on the data to include predictive analysis, the long-term goal is the ChatBot being human-like and a computer combined.

For example, users can interact with the Intelligent Bots Platform through a conversational interaction. This interaction, also called the conversational user interface (UI), is a dialog between the end user and the ChatBot, just as between two human beings. It could be as simple as the end user saying “Hello” to the ChatBot and the ChatBot responding with a “Hi” and asking the user how it can help, or it could be a transactional interaction in a banking ChatBot, such as transferring money from one account to the other, or an informational interaction in a HR ChatBot, such as checking for vacation balance, or asking an FAQ in a retail ChatBot, such as how to handle returns. Natural language processing (NLP) and machine learning (ML) algorithms combined with other approaches can be used to classify end user intent. An intent at a high level is what the end user would like to accomplish (e.g., get account balance, make a purchase). An intent is essentially, a mapping of customer input to a unit of work that the backend should perform. Therefore, based on the phrases uttered by the user in the ChatBot, these are mapped that to a specific and discrete use case or unit of work, for e.g. check balance, transfer money and track spending are all “use cases” that the ChatBot should support and be able to work out which unit of work should be triggered from the free text entry that the end user types in a natural language.

The underlying rational for having a ChatBot respond like a human is that the human brain can formulate and understand the request and then give a good response to the human request much better than a machine. Thus, there should be significant improvement in the request/response of a ChatBot, if human B is mimicked. So an initial part of the problem is how does the human brain formulate and understand the request? To mimic, a model is used. RST and DT allow a formal and repeatable way of doing this.

At a high level, there are typically two types of requests: (1) A request to perform some action; and (2) a request for information, e.g., a question. The first type has a response in which a unit of work is created. The second type has a response that is, e.g., a good answer, to the question. The answer could take the form of, for example, in some aspects, the AI constructing an answer from its extensive knowledge base(s) or from matching the best existing answer from searching the intern& or intranet or other publically/privately available data sources.

FIG. 15illustrates an autonomous agent answering user questions in accordance with an aspect.FIG. 15depicts chat1500, user messages1501and1503, and agent responses1502and1504. Agent responses1502and1504can be provided by indexing application102. As depicted, the user message1501asks “at what age do I normally pay a penalty for using retirement funds?” Indexing application102identifies the answer from the body of text indexed using process1200and provides agent response1502, which states “age 59½.” User asks a follow on question1503“What is the penalty?” In response, the agent answers “An additional 10% tax penalty.”

The features depicted inFIG. 15can be implemented by computing device101or by indexing application102.

Using Classifiers to Determine Text for Indexing

Certain aspects use machine learning models to learn rules such as those depicted in Table 4. A machine learning problem can be formulated as a classification problem that classifies EDUs into a first class that is suitable for indexing (i.e., informative) and forming alternative questions for an answer and a second class that is not suitable for indexing (i.e., not informative).

In an example, a training data set is formed by using text gathered from a search engine. For example, search answer popularity, or search rank, can be learned by a search engine on the basis of a high number of searches for the same query and user selection.

To accumulate question-answer pairs with marked answers, a selection of queries against short texts can be run. The portions of these texts that are used for matching can be identified. Because longer queries are necessary to assure a corresponding match is nontrivial, public question-answer pages such as Yahoo! Answers datasets can be used. More specifically, questions from such datasets can be formed from a first sentence of the dataset. In an example, Microsoft Cognitive Services (Bing Search engine API) can execute these queries. Search results which are short texts (4-6 sentences) are selected as such texts suitable for parsing and discourse analysis.

Matched fragments of these texts are taken as elements of the training set. Such fragments from the top ten or more pages of search result forms a positive dataset, i.e. informative fragments. It includes the fragments of texts considered by the search engine to be of high relevance. For the negative dataset, fragments with matched keywords from the set of lower ranked (100-1000+) search results pages are taken, as these results are assumed to be less relevant.

FIG. 16depicts a flowchart illustrating an example of a process for training a classification model to determine informative text for indexing, in accordance with an aspect. An example classification model is machine-learning model120, which can use different models such as classifiers, Tree Kernels, or Support Vector Machine, or SVM TK.

At block1601, process1600involves accessing a set of training data comprising a set of training pairs; each training data pair includes text and an expected classification, the set of training data including both (i) a first training data pair that has an expected classification of a nucleus and (ii) a second training data pair that has an expected classification of a satellite.

At block1602, process1600involves providing one of the training data pairs to the classification model. Accordingly, the classification model receives a body of text and an expected classification.

At block1603, process1600involves receiving a determined classification from the classification model.

At block1604, process1600involves calculating a loss function by calculating a difference between the determined classification and the expected classification. Different loss functions are possible such as mean-square error, likelihood loss, log (or cross entropy) loss, etc.

At block1605, process1600involves adjusting internal parameters of the classification model to minimize the loss function. In this manner, the classification model learns to improve the accuracy of its predictions.

At block1606, process1600involves using the trained classification model. For example, the trained classification model can be used in processes1200or1300. More specifically, the trained classification model can be used in place of the rule-based scheme discussed with respect to Table4and implemented in process1200or1300.

FIG. 17depicts a simplified diagram of a distributed system1700for implementing one of the aspects. In the illustrated aspect, distributed system1700includes one or more client computing devices1702,1704,1706, and1708, which are configured to execute and operate a client application such as a web browser, proprietary client (e.g., Oracle Forms), or the like over one or more network(s)1710. Server1712may be communicatively coupled with remote client computing devices1702,1704,1706, and1708via network1710.

In various aspects, server1712may be adapted to run one or more services or software applications provided by one or more of the components of the system. The services or software applications can include non-virtual and virtual environments. Virtual environments can include those used for virtual events, tradeshows, simulators, classrooms, shopping exchanges, and enterprises, whether two- or three-dimensional (3D) representations, page-based logical environments, or otherwise. In some aspects, these services may be offered as web-based or cloud services or under a Software as a Service (SaaS) model to the users of client computing devices1702,1704,1706, and/or1708. Users operating client computing devices1702,1704,1706, and/or1708may in turn utilize one or more client applications to interact with server1712to utilize the services provided by these components.

In the configuration depicted in the figure, the software components1718,1720and1722of distributed system1700are shown as being implemented on server1712. In other aspects, one or more of the components of distributed system1700and/or the services provided by these components may also be implemented by one or more of the client computing devices1702,1704,1706, and/or1708. Users operating the client computing devices may then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or combinations thereof. It should be appreciated that various different system configurations are possible, which may be different from distributed system1700. The aspect shown in the figure is thus one example of a distributed system for implementing an aspect system and is not intended to be limiting.

Although exemplary distributed system1700is shown with four client computing devices, any number of client computing devices may be supported. Other devices, such as devices with sensors, etc., may interact with server1712.

Network(s)1710in distributed system1700may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially-available protocols, including without limitation TCP/IP (transmission control protocol/Internet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk, and the like. Merely by way of example, network(s)1710can be a local area network (LAN), such as one based on Ethernet, Token-Ring and/or the like. Network(s)1710can be a wide-area network and the Internet. It can include a virtual network, including without limitation a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infra-red network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics (IEEE) 802.17 suite of protocols, Bluetooth®, and/or any other wireless protocol); and/or any combination of these and/or other networks.

Server1712may be composed of one or more general purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX® servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other appropriate arrangement and/or combination. Server1712can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization. One or more flexible pools of logical storage devices can be virtualized to maintain virtual storage devices for the server. Virtual networks can be controlled by server1712using software defined networking. In various aspects, server1712may be adapted to run one or more services or software applications described in the foregoing disclosure. For example, server1712may correspond to a server for performing processing described above according to an aspect of the present disclosure.

Distributed system1700may also include one or more databases1714and1716. Databases1714and1716may reside in a variety of locations. By way of example, one or more of databases1714and1716may reside on a non-transitory storage medium local to (and/or resident in) server1712. Alternatively, databases1714and1716may be remote from server1712and in communication with server1712via a network-based or dedicated connection. In one set of aspects, databases1714and1716may reside in a storage-area network (SAN). Similarly, any necessary files for performing the functions attributed to server1712may be stored locally on server1712and/or remotely, as appropriate. In one set of aspects, databases1714and1716may include relational databases, such as databases provided by Oracle, that are adapted to store, update, and retrieve data in response to SQL-formatted commands.

FIG. 18is a simplified block diagram of one or more components of a system environment1800by which services provided by one or more components of an aspect system may be offered as cloud services in accordance with an aspect of the present disclosure. In the illustrated aspect, system environment1800includes one or more client computing devices1804,1806, and1808that may be used by users to interact with a cloud infrastructure system1802that provides cloud services. The client computing devices may be configured to operate a client application such as a web browser, a proprietary client application (e.g., Oracle Forms), or some other application, which may be used by a user of the client computing device to interact with cloud infrastructure system1802to use services provided by cloud infrastructure system1802.

It should be appreciated that cloud infrastructure system1802depicted in the figure may have other components than those depicted. Further, the aspect shown in the figure is only one example of a cloud infrastructure system that may incorporate an aspect of the invention. In some other aspects, cloud infrastructure system1802may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration or arrangement of components.

Client devices1804,1806, and1808may be devices similar to those described above for2802,2804,2806, and2808.

Although exemplary system environment1800is shown with three client computing devices, any number of client computing devices may be supported. Other devices such as devices with sensors, etc. may interact with cloud infrastructure system1802.

Network(s)1810may facilitate communications and exchange of data between client devices1804,1806, and1808and cloud infrastructure system1802. Each network may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially-available protocols, including those described above for network(s)1810.

Cloud infrastructure system1802may comprise one or more computers and/or servers that may include those described above for server1712.

Large volumes of data, sometimes referred to as big data, can be hosted and/or manipulated by the infrastructure system on many levels and at different scales. Such data can include data sets that are so large and complex that it can be difficult to process using typical database management tools or traditional data processing applications. For example, terabytes of data may be difficult to store, retrieve, and process using personal computers or their rack-based counterparts. Such sizes of data can be difficult to work with using most current relational database management systems and desktop statistics and visualization packages. They can require massively parallel processing software running thousands of server computers, beyond the structure of commonly used software tools, to capture, curate, manage, and process the data within a tolerable elapsed time.

Extremely large data sets can be stored and manipulated by analysts and researchers to visualize large amounts of data, detect trends, and/or otherwise interact with the data. Tens, hundreds, or thousands of processors linked in parallel can act upon such data in order to present it or simulate external forces on the data or what it represents. These data sets can involve structured data, such as that organized in a database or otherwise according to a structured model, and/or unstructured data (e.g., emails, images, data blobs (binary large objects), web pages, complex event processing). By leveraging an ability of an aspect to relatively quickly focus more (or fewer) computing resources upon an objective, the cloud infrastructure system may be better available to carry out tasks on large data sets based on demand from a business, government agency, research organization, private individual, group of like-minded individuals or organizations, or other entity.

In various aspects, cloud infrastructure system1802may be adapted to automatically provision, manage and track a customer's subscription to services offered by cloud infrastructure system1802. Cloud infrastructure system1802may provide the cloud services via different deployment models. For example, services may be provided under a public cloud model in which cloud infrastructure system1802is owned by an organization selling cloud services (e.g., owned by Oracle) and the services are made available to the general public or different industry enterprises. As another example, services may be provided under a private cloud model in which cloud infrastructure system1802is operated solely for a single organization and may provide services for one or more entities within the organization. The cloud services may also be provided under a community cloud model in which cloud infrastructure system1802and the services provided by cloud infrastructure system1802are shared by several organizations in a related community. The cloud services may also be provided under a hybrid cloud model, which is a combination of two or more different models.

In certain aspects, cloud infrastructure system1802may also include infrastructure resources1830for providing the resources used to provide various services to customers of the cloud infrastructure system. In one aspect, infrastructure resources1830may include pre-integrated and optimized combinations of hardware, such as servers, storage, and networking resources to execute the services provided by the PaaS platform and the SaaS platform.

In certain aspects, a number of internal shared services1832may be provided that are shared by different components or modules of cloud infrastructure system1802and by the services provided by cloud infrastructure system1802. These internal shared services may include, without limitation, a security and identity service, an integration service, an enterprise repository service, an enterprise manager service, a virus scanning and white list service, a high availability, backup and recovery service, service for enabling cloud support, an email service, a notification service, a file transfer service, and the like.

In certain aspects, cloud infrastructure system1802may provide comprehensive management of cloud services (e.g., SaaS, PaaS, and IaaS services) in the cloud infrastructure system. In one aspect, cloud management functionality may include capabilities for provisioning, managing and tracking a customer's subscription received by cloud infrastructure system1802, and the like.

In one aspect, as depicted in the figure, cloud management functionality may be provided by one or more modules, such as an order management module1820, an order orchestration module1822, an order provisioning module1824, an order management and monitoring module1826, and an identity management module1828. These modules may include or be provided using one or more computers and/or servers, which may be general purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and/or combination.

In exemplary operation1834, a customer using a client device, such as client device1804,1806or1808, may interact with cloud infrastructure system1802by requesting one or more services provided by cloud infrastructure system1802and placing an order for a subscription for one or more services offered by cloud infrastructure system1802. In certain aspects, the customer may access a cloud User Interface (UI), cloud UI1818, cloud UI1814and/or cloud UI1816and place a subscription order via these UIs. The order information received by cloud infrastructure system1802in response to the customer placing an order may include information identifying the customer and one or more services offered by the cloud infrastructure system1802that the customer intends to subscribe to.

After an order has been placed by the customer, the order information is received via the cloud UIs,1818,1814and/or1816.

At operation1836, the order is stored in order database1818. Order database1818can be one of several databases operated by cloud infrastructure system1802and operated in conjunction with other system elements.

At operation1838, the order information is forwarded to an order management module1820. In some instances, order management module1820may be configured to perform billing and accounting functions related to the order, such as verifying the order, and upon verification, booking the order.

At operation1840, information regarding the order is communicated to an order orchestration module1822. Order orchestration module1822may utilize the order information to orchestrate the provisioning of services and resources for the order placed by the customer. In some instances, order orchestration module1822may orchestrate the provisioning of resources to support the subscribed services using the services of order provisioning module1824.

In certain aspects, order orchestration module1822enables the management of business processes associated with each order and applies business logic to determine whether an order should proceed to provisioning. At operation1842, upon receiving an order for a new subscription, order orchestration module1822sends a request to order provisioning module1824to allocate resources and configure those resources needed to fulfill the subscription order. Order provisioning module1824enables the allocation of resources for the services ordered by the customer. Order provisioning module1824provides a level of abstraction between the cloud services provided by cloud infrastructure system1802and the physical implementation layer that is used to provision the resources for providing the requested services. Order orchestration module1822may thus be isolated from implementation details, such as whether or not services and resources are actually provisioned on the fly or pre-provisioned and only allocated/assigned upon request.

At operation1844, once the services and resources are provisioned, a notification of the provided service may be sent to customers on client devices1804,1806and/or1808by order provisioning module1824of cloud infrastructure system1802.

At operation1846, the customer's subscription order may be managed and tracked by an order management and monitoring module1826. In some instances, order management and monitoring module1826may be configured to collect usage statistics for the services in the subscription order, such as the amount of storage used, the amount data transferred, the number of users, and the amount of system up time and system down time.

In certain aspects, cloud infrastructure system1802may include an identity management module1828. Identity management module1828may be configured to provide identity services, such as access management and authorization services in cloud infrastructure system1802. In some aspects, identity management module1828may control information about customers who wish to utilize the services provided by cloud infrastructure system1802. Such information can include information that authenticates the identities of such customers and information that describes which actions those customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). Identity management module1828may also include the management of descriptive information about each customer and about how and by whom that descriptive information can be accessed and modified.

FIG. 19illustrates an exemplary computer system1900, in which various aspects of the present invention may be implemented. The system1900may be used to implement any of the computer systems described above. As shown in the figure, computer system1900includes a processing unit1904that communicates with a number of peripheral subsystems via a bus subsystem1902. These peripheral subsystems may include a processing acceleration unit1906, an I/O subsystem1908, a storage subsystem1918and a communications subsystem1924. Storage subsystem1918includes tangible computer-readable storage media1922and a system memory1910.

Bus subsystem1902provides a mechanism for letting the various components and subsystems of computer system1900communicate with each other as intended. Although bus subsystem1902is shown schematically as a single bus, alternative aspects of the bus subsystem may utilize multiple buses. Bus subsystem1902may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured to the IEEE P1986.1 standard.

Processing unit1904, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system1900. One or more processors may be included in processing unit1904. These processors may include single core or multicore processors. In certain aspects, processing unit1904may be implemented as one or more independent processing units1932and/or1934with single or multicore processors included in each processing unit. In other aspects, processing unit1904may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

In various aspects, processing unit1904can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processor(s)1904and/or in storage subsystem1918. Through suitable programming, processor(s)1904can provide various functionalities described above. Computer system1900may additionally include a processing acceleration unit1906, which can include a digital signal processor (DSP), a special-purpose processor, and/or the like.

Computer system1900may comprise a storage subsystem1918that comprises software elements, shown as being currently located within a system memory1910. System memory1910may store program instructions that are loadable and executable on processing unit1904, as well as data generated during the execution of these programs.

Depending on the configuration and type of computer system1900, system memory1910may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.) The RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated and executed by processing unit1904. In some implementations, system memory1910may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input/output system (BIOS), containing the basic routines that help to transfer information between elements within computer system1900, such as during start-up, may typically be stored in the ROM. By way of example, and not limitation, system memory1910also illustrates application programs1912, which may include client applications, Web browsers, mid-tier applications, relational database management systems (RDBMS), etc., program data1914, and an operating system1916. By way of example, operating system1916may include various versions of Microsoft Windows®, Apple Macintosh®, and/or Linux operating systems, a variety of commercially-available UNIX® or UNIX-like operating systems (including without limitation the variety of GNU/Linux operating systems, the Google Chrome® OS, and the like) and/or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS operating systems.

Storage subsystem1918may also provide a tangible computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some aspects. Software (programs, code modules, instructions) that when executed by a processor provide the functionality described above may be stored in storage subsystem1918. These software modules or instructions may be executed by processing unit1904. Storage subsystem1918may also provide a repository for storing data used in accordance with the present invention.

Storage subsystem1918may also include a computer-readable storage media reader1920that can further be connected to computer-readable storage media1922. Together and, optionally, in combination with system memory1910, computer-readable storage media1922may comprehensively represent remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing, storing, transmitting, and retrieving computer-readable information.

Communications subsystem1924provides an interface to other computer systems and networks. Communications subsystem1924serves as an interface for receiving data from and transmitting data to other systems from computer system1900. For example, communications subsystem1924may enable computer system1900to connect to one or more devices via the Internet. In some aspects, communications subsystem1924can include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.28 family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and/or other components. In some aspects, communications subsystem1924can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

In some aspects, communications subsystem1924may also receive input communication in the form of structured and/or unstructured data feeds1926, event streams1928, event updates1919, and the like on behalf of one or more users who may use computer system1900.

By way of example, communications subsystem1924may be configured to receive unstructured data feeds1926in real-time from users of social media networks and/or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and/or real-time updates from one or more third party information sources.

Additionally, communications subsystem1924may also be configured to receive data in the form of continuous data streams, which may include event streams1928of real-time events and/or event updates1919, that may be continuous or unbounded in nature with no explicit end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measuring tools (e.g. network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

Communications subsystem1924may also be configured to output the structured and/or unstructured data feeds1926, event streams1928, event updates1919, and the like to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system1900.

In the foregoing specification, aspects of the invention are described with reference to specific aspects thereof, but those skilled in the art will recognize that the invention is not limited thereto. Various features and aspects of the above-described invention may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive.