Techniques to use machine learning for risk management

Various embodiments are generally directed to techniques to build and train a machine learning model with features of which at least one feature corresponds to risk indicia and at least one other feature corresponds to a data source. These features, in general, provide data (e.g., values) indicating a degree of relevance between a particular record and a risk assessment of that particular record's subject matter. User refinement in the form of user selections and other interactions with the particular record and other records provide insights into proper risk management.

BACKGROUND

Risk management refers to the identification, assessment, and prioritization of risks in business operations and products. This is followed by coordinated and economical application of resources to minimize, monitor, and control the probability and/or impact of unfortunate events or to maximize the realization of opportunities. There are two types of events: negative events can be classified as risks while positive events are classified as opportunities. Risks manifest in various forms and may be caused by uncertainty in related markets, threats from project failures, legal liabilities, credit risk, accidents, natural causes and disasters, deliberate attack from an adversary, or events of uncertain or unpredictable root-cause. Several risk management standards have been developed but fall short of a complete solution. Complying with these standards is time-consuming and even when compliance is made, the results were not meaningful.

It is with respect to these and other considerations that the present improvements have been desired.

SUMMARY

Various embodiments are generally directed to techniques to use machine learning for risk management. In some embodiments, the machine learning model relies upon user refinement (i.e., user selections and other interactions) to ensure accuracy assessing risk in a product or process. In one embodiment, for example, an apparatus may comprise a processing circuit and logic stored in computer memory and executed on the processing circuit. The logic may be operative to cause the processing circuit to perform feature extraction on records across a plurality of datasets. Features corresponding to the feature extraction include one or more features associated with risk indicia and one or more features associated with a particular data source of at least one of the plurality of datasets. The logic is operative to cause the processing circuit to build a machine learning model based upon data generated during the feature extraction on the records. The machine learning model includes risk assessment data for the records across the plurality of datasets. The logic is operative to cause the processing circuit to identify a set of records from the records across the plurality of datasets. Each record of the set of records has risk assessment data that exceeds a baseline threshold. The logic is operative to further cause the processing circuit to return at least one record of the set of records in response to search queries. The logic is operative to further cause the processing circuit to modify the risk assessment data in the machine learning model in response to user selections corresponding to the at least one returned record. Other embodiments are described and claimed.

DETAILED DESCRIPTION

Various embodiments are directed to techniques to use machine learning for risk assessment. As mentioned above, conventional risk management strategies fall short of a meaningful solution. Using these strategies, risk managers were forced to run reports from disparate data sources and to manually connect data using spreadsheets and special applications. By implementing a machine learning model, the embodiments of the present disclosure provide a risk learning engine with fewer or no time-consuming processes and with the capability of leveraging connections between data sources. The various embodiments described herein allow risk managers to use natural language searches to quickly see all risk information related to their search query's subject matter. The risk manager can interact with the risk learning engine, for example, by selecting records having relevant risk information, to train the machine learning model to accurately represent the risk manager's thought process. In response to that user selection, the machine learning model is modified to indicate the selected record's increased relevance. The risk manager may also select or otherwise identify records having irrelevant information. Similarly, the machine learning model is modified to indicate the selected record's decreased relevance or lack of relevance altogether. Through the refinement of searches, the machine learning model as described herein identifies patterns and relationships for risk management. For instance, if a process was being improved or a product was being introduced, the machine learning model may identify previous and potential failures for the new process/new product. As a result, the embodiments can improve affordability, scalability, modularity, extendibility, or interoperability for an operator, device or network.

With general reference to notations and nomenclature used herein, the detailed descriptions which follow may be presented in terms of program processes executed on a computer or network of computers. These process descriptions and representations are used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art.

Various embodiments also relate to apparatus or systems for performing these operations. This apparatus may be specially constructed for the required purpose or it may comprise a general-purpose computer as selectively activated or reconfigured by a computer program stored in the computer. The processes presented herein are not inherently related to a particular computer or other apparatus. Various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these machines will appear from the description given.

FIG. 1illustrates a block diagram for a system100. In one embodiment, the system100may comprise a computer-implemented system100having a software application120comprising one or more components122-a. Although the system100shown inFIG. 1has a limited number of elements in a certain topology, it may be appreciated that the system100may include more or less elements in alternate topologies as desired for a given implementation.

It is worthy to note that “a” and “b” and “c” and similar designators as used herein are intended to be variables representing any positive integer. Thus, for example, if an implementation sets a value for a=5, then a complete set of components122-amay include components122-1,122-2,122-3,122-4and122-5. The embodiments are not limited in this context.

As mentioned above, the system100includes the application120as a type of software application running on an electronic device, such as a desktop application running on an operating system of a computing device, a mobile application running on a mobile operating system of a mobile device, or a web application running on a browser component of either the mobile operating system or the desktop operating system. Those skilled in the art would understand how to design, build, and deploy the software application on any type of electronic device.

The application120may be generally arranged to process input110of which some input may be provided directly to an interface component122-1via an input device, and other input may be provided to the interface component122-1via a network. For example, a user may enter data via a keyboard device attached to a computing device running the application120. The application120may be generally arranged to generate output130for the interface component122-1of which some output may be configured for display on a display device and other output may be communicated across the network to other devices. As an example, the application120may generate data that can be processed/rendered by the interface component122-1into content for a Graphical User Interface (GUI).

The application120may be generally arranged to provide a device user or users with various services; among the provided services, the application120may provide the device user with risk management services. As described herein, risk management services utilize a plurality of disparate data sources to identify potential risks or opportunities. The application120may include a search component122-2(e.g., a search engine) to scour various data sources for items or records of interest. In response to a search query, the search component122-2responds with a set of search results as the output130. The application120may accept, as the input110, user selections with respect to the set of search results and those selections are used to build and train a machine learning model.

The application120may comprise a view component122-3to build a user interface (UI) view into the plurality of data sources used for risk management. The view generated by the view component122-3may refer to a graphical user interface (GUI) through which users can access, view, and arrange records from the data sources. Similar to the search component122-2, user selections with respect to the records may be used to build and train the machine learning model to improve risk management.

FIG. 2illustrates an embodiment of an apparatus200for the system100. As shown inFIG. 2, the apparatus200includes an electronic device220communicably coupled to a plurality of datasets210(or simply datasets210). As described herein, the plurality of datasets210includes structured data (e.g., records) corresponding to various data sources.

The electronic device220(or simply device220) includes a processing circuit230and computer memory240on which logic250is stored for execution by the processing circuit230. When executed on the processing circuit230, the logic250is operative to cause the processing circuit230to perform feature extraction on records across the plurality of datasets210. A record, in general, is structured data whose content relates to certain subject matter. Each dataset of the plurality of datasets includes a collection of records whose content pertains to same or similar subject matter. Because the plurality of datasets210originate from different data sources, some records have different formats while pertaining to the same subject matter (e.g., same product or process). Furthermore, the above-mentioned feature extraction refers to a process for identifying features from the datasets210and generating feature extraction data260. It is appreciated that those skilled in the art may accomplish the feature extraction process via a function (e.g., a kernel function or a filter). The features being extracted comprise at least one feature associated with risk indicia and at least one feature associated with a particular data source of at least one of the datasets210. An example feature associated with a record's data source may indicate to what degree the record's data source is relevant to a risk assessment of the record's subject matter. Another example feature associated with a record's data source may indicate one or more other data sources that are relevant to a risk assessment of the record's subject matter. An example feature associated with a record's risk indicia may indicate to what degree the record's stored information is relevant to a risk assessment of the record's subject matter.

The logic250is operative to cause the processing circuit to build a machine learning model270based upon the feature extraction data260generated during the feature extraction on the records in the datasets210. In some embodiments, the machine learning model270includes risk assessment data for the records across the plurality of datasets210. For at least one record of the records in the datasets210, the risk assessment data includes at least one risk assessment value in accordance with a metric. It is appreciated a number of known metrics in the field of risk management are applicable. In general, a risk assessment value indicates a level of risk associated with a subject matter of a particular record or group of records.

In some embodiments, a group of records pertaining to a (new) financial product are from disparate data sources, and each record's content includes at least some risk indicia. It is appreciated that a record's content may be classified as negative indicating at least some risk or positive indicating little or no risk involved in the new financial product. To illustrate by way of example, a commercial credit card may be a new financial product for which a risk assessment is desired. Records in the form of loss events, product notes, testing issues, and/or the like provide various content to the commercia credit card; other records may be deemed irrelevant for a variety of reasons. The content in these records may indicate a considerable or high level of risk, little or no risk, and any intermediate level of risk. Features corresponding to the risk indicia in a particular record and features corresponding to the data source providing the particular record may be analyzed, quantified, and then, used to measure the particular record's level of risk or risk assessment value.

The logic250is operative to further cause the processing circuit to identify a set of records from the records across the plurality of datasets210. Each record of the identified set of records includes risk assessment data (e.g., the risk assessment value) that exceeds the baseline threshold. Depending on the machine learning model270's implementation, the risk assessment value may imply a high or significant level of risk if the value exceeds a baseline threshold. It is appreciated that any suitable implementation of a machine learning model can used as the machine learning model270. In some embodiments, the baseline threshold may be a pre-defined value established off-line arbitrarily or via some mathematical process. In other embodiments, the machine learning model270may establish the baseline threshold at a value occupying a position within twenty (20) percent of a highest possible risk assessment value or, as an alternative, greater than eighty (80) percent of the records in the datasets210.

The logic250is operative to further cause the processing circuit to return at least one record of the set of records in response to search queries280, which may be submitted into a search engine. The logic250may implement the search engine or, as an alternative, may operate the search engine as an independent module. In some embodiments, the search queries280include natural language search queries and/or keyword search queries for which the search engine identifies matching records. Most if not all matching records pertain to the same subject matter. As described herein, the user may select a record to signify that record as being more relevant or less relevant and that selection may cause an adjustment to the record's risk assessment data.

The logic250is operative to cause the processing circuit to modify the risk assessment data in the machine learning model270in response to user selections290corresponding to the at least one returned record. As described herein, the user selections290generally refers to various user interactivity data in which the user configures/organizes the returned records. In some embodiments, the user selections290form connections between records of different data sources or datasets. By way of the user selections290, a degree of relevance may be attributed to the selected records. These connections cause either an increase or a decrease in a selected record's risk assessment value. One record's content, by itself, may have a certain risk assessment value; when another record pertaining to the same subject matter becomes connected and provides context for the first record's content, the logic250modifies the certain risk assessment value to account for the context. For example, the other record may include additional risk indicia implying that the first record's certain risk assessment value is a conservative estimate. The logic250increases the certain risk assessment value to account for the additional risk indicia.

As described in the present disclosure, the user selections290may be used to accurately capture a relationship between a record's content and a risk assessment of the record's subject matter. To illustrate by way of example, the record may be a loss event attributed to a product and therefore, valuable to the risk assessment of that product. By selecting that record, the user is informing the logic250of the record's importance. In turn, the logic250modifies the risk assessment data for the record but also adjusts the machine learning model. The logic250may adjust certain feature weights such that a record of similar or equal importance can be identified automatically in the future; otherwise, a conventional risk management solution would miss that record for failing to codify the record's importance. In contrast, the record may hold benign product information having little or no impact on the product's risk assessment. The logic250may adjust certain feature weights such that a similar record can be identified and then, discarded automatically in the future.

With respect to the search queries280and corresponding search results, the user selections290includes instances where the user, after viewing records in the search results, invokes a GUI element associated with a record in order to increase or decrease that record's relevance to the subject matter's risk assessment. In response, the logic250makes a determination as to whether the record's risk assessment data is to be modified. According to some embodiments, the logic250modifies the record's risk assessment data in proportion to a number of user selections of that record by a population of users. In at least one embodiment, the logic250adds, to the feature set in the feature extraction data260, a feature related to user selections290such that the number of users selecting the record as relevant factors into the record's risk assessment value. Thus, the logic250increases the record's risk assessment value after a considerable number of selections to increase the record's relevance.

In some embodiments, the logic250(or different logic) enables different views into the plurality of datasets210where each view is an arrangement of different windows (e.g., controls). Each window displays one or more records and their content or at least a portion thereof. A control may be a window displaying one or more records that meet some criteria or otherwise filtered. By aggregating the views of the population of users, the logic250may identify patterns connecting records from disparate data sources. The logic250may adjust feature weights associated with the disparate data sources to accurately account for a data source's importance in risk assessment.

FIG. 3illustrates an embodiment of a risk learning method300for the system100. As shown inFIG. 3, the risk learning method300commences with a plurality of data sources302.

Datasets from the plurality of data sources302are feed into unstructured and structured data storage304(or simply data storage304). Each dataset may be either unstructured or structured in a known or discoverable format. It is appreciated that some unstructured datasets may be configured in an unknown format while other datasets have no format and require structure. In some embodiments, the risk learning method300provides uniformity by generating records in a generic format for storing the datasets from the plurality of data sources302.

As illustrated inFIG. 3, the data sources302refer to datasets covering issues/events; complaints; Internal Loss Data (ILD) which describe operational losses that occurred within an enterprise; External Loss Data (ELD) which describe operational losses by other financial institutions; Compliance; control programs, Monthly Industry Consent Order Summaries (MICOS) from our Legal team which provide legal opinions on industry related actions from our regulators; New Product and Marketing Initiative Governance (NPMIG) which is a process to identify risks; Product Risk Assessment (PRA) and Risk and Control Self-Assessment (RCSA) which are processes to identify and size risks.

During feature extraction3061, the risk learning method300analyzes the data in the data storage304for various features and generates feature extraction data identifying at least one of those features amongst the datasets in the plurality of data sources302. According to some embodiments of the present disclosure, the features include risk assessment features corresponding to various risk indicia possibly present in the datasets, data source features corresponding to a data source's relationship to risk assessment, and/or the like. The feature extraction data from these features corresponding to new and unknown risks in the present environment. Feature extraction3062operates to support the feature extraction3061with feature extraction data corresponding to actual realized risks affecting the present environment. When combined, both sets of the feature extraction data provide a risk assessment for various products in the present environment.

A risk learning engine308may include a machine learning model that is built from the features and the feature extraction data provided by the feature extraction3061and the feature extraction3062. The risk learning engine308may assign weights/parameters to each feature and generate risk assessment data for each record to denote a risk level for that record's subject matter (e.g., goods or services, such as a new commercial product). The risk assessment data for each record may result from combining the feature weights and corresponding feature values into a single risk assessment value. The machine learning model may include a function (e.g., a polynomial function) where the feature weights are coefficients and the feature values are the input variables such that each record has a function value indicating a particular risk level given the features present in the record. In another implementation, the machine learning model may include a probability distribution (e.g., a statistical model) where each record has a conditional probability of having a particular risk level given the features present in the record.

Outliers310having risk assessment value above a threshold value are identified from the machine learning model by the risk learning engine308and fed into an application312. It is appreciated that the risk assessment values associated with the outliers310may be heuristic values or probability values and therefore, the baseline threshold may be a heuristic value or a probability value. Each record of the outliers310provides content that is relevant to a risk assessment of the record's subject matter.

As described herein, the application312(e.g., the application120ofFIG. 1) may be a web application or a mobile application that operates a user interface between application users and the outliers310. The application312provides various mechanisms for accessing and analyzing the outliers310for insight, including a search engine and a view generator. User selections submitted through the application312for the outliers310may modify the risk assessment data in the machine learning model. In some embodiments, when the user selects a record as being relevant to the risk assessment of the record's subject matter, the risk learning engine308increases feature values, which, in turn, increases the risk assessment value. The risk learning engine308also may adjust feature weights to make the machine learning model's function better fit the actual distribution of risk assessment values.

In some embodiments, backend administrative processes are executed on the records in the data storage304including a first line assessment314and a second line assessment316. The backend administrative processes examine the record's content and modify the record's risk assessment data accordingly, providing a complete risk assessment318to the risk learning engine308. In each backend administrative process, each record's content undergoes validation to determine whether the machine learning model270accurately accounts for that record's impact on risk assessment. The complete risk assessment318, in addition to describing each record's level of risk, may instruct the risk learning engine308to modify the machine learning model, for example, by adjusting feature values, feature weights, and/or the model's (kernel) function.

FIG. 4illustrates an embodiment of a risk learning method400for the system100. As shown inFIG. 4, the risk learning method400applies to New Product and Marketing Initiative Governance (NPMIG)402, which is a process to identify risks in new products (e.g., financial products). Starting with NPMIG datasets404—a data storage device for storing NPMIG data—the risk learning method400generates performs feature extraction for a new initiative description406. In some embodiments, the new initiative description406includes a set of records with content describing the new product and marketing initiative. In addition to identifying risk indicia in those datasets, the risk learning method400, via a risk learning engine, executes the feature extraction to identify additional features for describing the new product and marketing initiative. Hence, the risk learning method400uses the features to identify new potential risks as well as to capture the content for the new initiative description406.

A risk learning engine408builds a machine learning model to identify a subset of the above-mentioned set of records having a certain risk level. In some embodiments, the machine learning model includes a metric for computing a risk assessment value associated with the new initiative description406and corresponding feature extraction data. At some previous point, feature extraction was executed on old products to identify potential risks. A set of old initiative descriptions408provides information on previous risks associated with older product and marketing initiatives. Each old initiative description408is a set of records whose content is associated with a same product (i.e., subject matter). Features corresponding to the old initiative descriptions408indicate actual risks that can be compared with the above-mentioned potential risks to determine whether there is any validity to them.

For the machine learning model, the risk learning engine410applies a clustering technique to the set of old initiative descriptions408. The clustering technique, in general, sorts the old initiative descriptions according to their risk assessment data and then, divide the sorted descriptions into partitions according percentile group. For instance, the descriptions occupying a top twenty-five (25) % are deemed to be describing significant risks and are to be compared with the new initiative description406. After performing such a comparison, the risk learning engine410, at decision block412, determines whether there is a similar risk in the new initiative description406.

FIG. 5illustrates an embodiment of a user interface500for the system100. As shown inFIG. 5, the user interface500includes a view502, an arrangement of user interface (UI) components, for an application.

Among the UI components, graphical user interface (GUI) elements known as controls504determine which records506to display. For example, a control5041may direct the view502to the records506from certain data sources, excluding other data sources, such as when the user specifically selects these data sources in a menu. As another example, a control5042may direct the view502to sort and then, display the records506from the certain data sources according to some attribute and that attribute's metric. In yet another example, a control5043may direct the view502to filter and then, display the records506from the certain data sources according to another attribute and a particular attribute value.

It is appreciated that the user selections as described herein may refer to the user's configuration of these controls504. To illustrate with respect to the control5041, when the user selects the certain data sources from the menu, those selections imply a relationship between the certain data sources such that the certain data sources are prioritized for risk management. For instance, records from one data source describing certain subject matter may provide context for records in another data source when those records describe similar or same subject matter; together, both sets of records improve upon an accuracy to each other's risk assessment and further enhance an overall risk assessment of the certain subject matter. Therefore, by connecting these sets of records, the system100may improve upon risk assessment data associated with these records. In some embodiments, by aggregating views from a number of users, the system100may learn a pattern of data sources that could be connected when executing feature extraction, improving upon the overall risk assessment.

It is further appreciated that additional user interactivity may modify the risk assessment data amongst the records506. With respect to the control5042and/or the control5043, the system100may capture a pattern amongst the sorted records and/or the filtered records in the records506display area. These patterns may provide insight into modifying the risk assessment data of at least one of these records, such as by increasing or decreasing a risk assessment value. In some embodiments, the user may select (e.g., highlight) a record in the records506display area for a risk assessment adjustment. These patterns also may provide insight into which terms to use when modifying content of at least one of these records and/or modifying search queries to return the sorted records and/or the filtered records in the records506display area.

As further depicted inFIG. 5, the view502includes a search engine including a search portal508for entering search queries and a display area for records510as the search queries' search results. User selections directed towards records in the records510display area may cause the system100to modify risk assessment data in these records. Because the records in the records510display area match the search query, these records have content pertaining to shared subject matter. In some embodiments, the user may activate a cursor, a GUI element, to increase or decrease a particular record's relevance to the risk assessment of their subject matter.FIG. 5depicts example cursors as a cursor5121, a cursor5122, and a cursor5123. A user selection of any of these example cursors may cause the system100to modify the risk assessment data in a corresponding record of the records510.

FIG. 6illustrates a block diagram of a distributed system600. The distributed system600may distribute portions of the structure and/or operations for the system100across multiple computing entities. Examples of distributed system600may include without limitation a client-server architecture, a 3-tier architecture, an N-tier architecture, a tightly-coupled or clustered architecture, a peer-to-peer architecture, a master-slave architecture, a shared database architecture, and other types of distributed systems. The embodiments are not limited in this context.

The distributed system600may comprise a client device610and a server device650. In general, the client device610and the server device650may be an electronic device and the same or similar to the apparatus200as described with reference toFIG. 2. The client system610and the server system650may each comprise a processing component630and a communications component640. The devices610,650may communicate over a communications media612using communications signals614via the communications components640.

Each of the devices610,650may comprise any electronic device capable of receiving, processing, and sending information for the system100. Examples of an electronic device may include without limitation an ultra-mobile device, a mobile device, a personal digital assistant (PDA), a mobile computing device, a smart phone, a telephone, a digital telephone, a cellular telephone, ebook readers, a handset, a one-way pager, a two-way pager, a messaging device, a computer, a personal computer (PC), a desktop computer, a laptop computer, a notebook computer, a netbook computer, a handheld computer, a tablet computer, a server, a server array or server farm, a web server, a network server, an Internet server, a work station, a mini-computer, a main frame computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, multiprocessor systems, processor-based systems, consumer electronics, programmable consumer electronics, game devices, television, digital television, set top box, wireless access point, base station, subscriber station, mobile subscriber center, radio network controller, router, hub, gateway, bridge, switch, machine, or combination thereof. The embodiments are not limited in this context.

Each of the devices610,650may execute communications operations or logic for the system100using communications component640. The communications component640may implement any well-known communications techniques and protocols, such as techniques suitable for use with packet-switched networks (e.g., public networks such as the Internet, private networks such as an enterprise intranet, and so forth), circuit-switched networks (e.g., the public switched telephone network), or a combination of packet-switched networks and circuit-switched networks (with suitable gateways and translators). The communications component640may include various types of standard communication elements, such as one or more communications interfaces, network interfaces, network interface cards (NIC), radios, wireless transmitters/receivers (transceivers), wired and/or wireless communication media, physical connectors, and so forth. By way of example, and not limitation, communication media612,642include wired communications media and wireless communications media. Examples of wired communications media may include a wire, cable, metal leads, printed circuit boards (PCB), backplanes, switch fabrics, semiconductor material, twisted-pair wire, co-axial cable, fiber optics, a propagated signal, and so forth. Examples of wireless communications media may include acoustic, radio-frequency (RF) spectrum, infrared and other wireless media.

The client device610may comprise or employ one or more client programs that operate to perform various methodologies in accordance with the described embodiments. In one embodiment, for example, the client device610may implement the application120ofFIG. 1.

The server device650may comprise or employ one or more server programs that operate to perform various methodologies in accordance with the described embodiments. In one embodiment, for example, the server device650may implement the logic250ofFIG. 2. The server device650may include backend validation data620that is generated during backend validation processes (e.g., a first line assessment and a second line assessment). Risk managers (off-line) may review various records among the plurality of datasets and adjust risk assessment values accordingly. These risk managers may verify potential risks as actual risks or false alarms. The logic250may use the backend validation data620to modify the risk assessment value of at least one record as well as the machine learning model to perform risk assessment.

FIG. 7illustrates one embodiment of a logic flow700. The logic flow700may be representative of some or all of the operations executed by one or more embodiments described herein.

In the illustrated embodiment shown inFIG. 7, the logic flow700performs feature extraction on records across a plurality of datasets at block702. For example, the logic flow700may analyze content stored in the records for features and upon identifying at least one feature, may generate feature data (e.g., feature values) for each record. The feature data may be generated during feature extraction and therefore, may be referred to herein as feature extraction data. Some of the feature data corresponds to risk assessment data for the records.

The logic flow700may build a machine learning model to include the risk assessment data at block704. For example, the logic flow700may build the machine learning model to execute a weighted risk assessment analysis of a record's feature data. The machine learning model may specify a set of feature weights and a heuristic function for computing a risk assessment value denoting a level of risk for the record content's subject matter.

The logic flow700may identify a set of records that exceed a baseline threshold at block706. For example, the logic flow700may compare each record's risk assessment value to the baseline threshold to determine whether that record includes information relevant to the risk assessment of the record's subject matter. If the record exceeds the baseline threshold by a significant margin, the logic flow700identifies the record as indicating a credible risk for the record's subject matter.

The logic flow700may return at least one record in response to a search query at block708. Because the identified set of records are relevant to the risk assessment, at least one of these records would be relevant search result to a search query in furtherance of that risk assessment. For example, if the user enters a search query with a product name, the at least one returned record includes content indicating a level of risk associated with the product or a similar product.

The logic flow700may modify the risk assessment data in the machine learning model at block710. For example, the logic flow700may respond to user selections with respect to a particular returned record by increasing or decreasing a risk assessment value of the particular returned record. As described herein, the user may select the particular returned record as being more relevant than the current risk assessment value represents. In turn, the logic flow700increases the risk assessment value such that the returned record appears at a higher position in search results for future search queries. The embodiments are not limited to this example.

FIG. 8illustrates one embodiment of a logic flow800. The logic flow800may be representative of some or all of the operations executed by one or more embodiments described herein.

In the illustrated embodiment shown inFIG. 8, the logic flow800may combine views into an aggregated view at optional block802. For example, if a number of users are share a pattern of data sources, that pattern can be used to connect datasets in these data sources. If, for example, these users are researching a new product, the pattern of data sources are further relevant towards the risk assessment of the new product.

The logic flow800may process, during a search query, a user selection of a returned record at block804. The user selection indicates a degree of relevance upon the returned record. For example, the user selection may invoke a GUI element causing an increase (or a decrease) in a risk assessment value of the returned record. In some embodiments, the user selection is risk indicia strongly implying that the returned record has important content towards understanding how risky the new product may be to introduce.

The logic flow800may update a risk assessment value of the returned record based upon the user selection at block806. As described herein, a machine learning model having features corresponding to risk assessment may be used by the logic flow800to compute the risk assessment value of the returned record. For example, the logic flow800may adjust feature values for the returned record, which in turn increases or decreases the risk assessment value. As another example, the logic flow800may adjust features weights to more accurately correlate the returned record's feature values with the increased risk assessment value. The feature weights may be coefficients in a function; instead of adjusting the weights, the logic flow800may adjust the function in a different example.

The logic flow800may modify the search query and/or modify content of the returned record at block808. For example, the logic flow800may add one or more search terms to or remove on or more search terms from the search query. The logic flow800may automatically replace the search query with the modified search query. In addition or as an alternative, the logic flow800may add content to or remove content from the returned record. By doing so, the logic flow800can provide search results with higher risk assessment values. The embodiments are not limited to this example.

FIG. 9illustrates an embodiment of an exemplary computing architecture900suitable for implementing various embodiments as previously described. In one embodiment, the computing architecture900may comprise or be implemented as part of an electronic device. Examples of an electronic device may include those described with reference toFIG. 8, among others. The embodiments are not limited in this context.

The computer902may include various types of computer-readable storage media in the form of one or more lower speed memory units, including an internal (or external) hard disk drive (HDD)914, a magnetic floppy disk drive (FDD)916to read from or write to a removable magnetic disk918, and an optical disk drive920to read from or write to a removable optical disk922(e.g., a CD-ROM or DVD). The HDD914, FDD916and optical disk drive920can be connected to the system bus908by a HDD interface924, an FDD interface926and an optical drive interface928, respectively. The HDD interface924for external drive implementations can include at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

The drives and associated computer-readable media provide volatile and/or nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For example, a number of program modules can be stored in the drives and memory units910,912, including an operating system930, one or more application programs932, other program modules934, and program data936. In one embodiment, the one or more application programs932, other program modules934, and program data936can include, for example, the various applications and/or components of the system100.

A monitor944or other type of display device is also connected to the system bus908via an interface, such as a video adaptor946. The monitor944may be internal or external to the computer902. In addition to the monitor944, a computer typically includes other peripheral output devices, such as speakers, printers, and so forth.

When used in a LAN networking environment, the computer902is connected to the LAN952through a wire and/or wireless communication network interface or adaptor956. The adaptor956can facilitate wire and/or wireless communications to the LAN952, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of the adaptor956.

When used in a WAN networking environment, the computer902can include a modem958, or is connected to a communications server on the WAN954, or has other means for establishing communications over the WAN954, such as by way of the Internet. The modem958, which can be internal or external and a wire and/or wireless device, connects to the system bus908via the input device interface942. In a networked environment, program modules depicted relative to the computer902, or portions thereof, can be stored in the remote memory/storage device950. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.

FIG. 10illustrates a block diagram of an exemplary communications architecture1000suitable for implementing various embodiments as previously described. The communications architecture1000includes various common communications elements, such as a transmitter, receiver, transceiver, radio, network interface, baseband processor, antenna, amplifiers, filters, power supplies, and so forth. The embodiments, however, are not limited to implementation by the communications architecture1000.

As shown inFIG. 10, the communications architecture1000comprises includes one or more clients1002and servers1004. The clients1002may implement the client device610. The servers1004may implement the server device950. The clients1002and the servers1004are operatively connected to one or more respective client data stores1008and server data stores1010that can be employed to store information local to the respective clients1002and servers1004, such as cookies and/or associated contextual information.

The clients1002and the servers1004may communicate information between each other using a communication framework1006. The communications framework1006may implement any well-known communications techniques and protocols. The communications framework1006may be implemented as a packet-switched network (e.g., public networks such as the Internet, private networks such as an enterprise intranet, and so forth), a circuit-switched network (e.g., the public switched telephone network), or a combination of a packet-switched network and a circuit-switched network (with suitable gateways and translators).