MACHINE LEARNING FEATURE RECOMMENDER

The described technology is generally directed towards a machine learning feature recommender, for use in connection with a feature store. By collecting data and recommending machine learning features to users based on collected data, embodiments can facilitate data scientists' discovery of features that have been used by their colleagues and that are likely to make their machine learning models more performant. The disclosed machine learning feature recommender can reduce the effort involved in developing machine learning models.

TECHNICAL FIELD

The subject application is related to machine learning and artificial intelligence, and in particular to search and recommendation of features for use in connection with machine learning models.

BACKGROUND

When a data scientist sets out to solve a business problem with a machine learning model, they are faced with the daunting task of feature engineering: developing variables that are understandable to a machine learning model and that will help the model learn what the data scientist intends. Feature cleaning and engineering is estimated to account for around 80% of a data scientist's time. Feature engineering is even more difficult when a machine learning model is intended for real-time production, in which case the machine learning model may be designed to gather its features and return scores in fractions of a second. Feature reuse can streamline feature engineering, and so feature reuse can be crucial to the business success of a mature data science organization.

In the past several years, feature stores, such as the HOPSWORKS® feature store, have become popular among companies that rely heavily on machine learning and artificial intelligence. We anticipate feature stores will increase in popularity, with more companies releasing commercially available feature stores.

In today's feature stores, data scientists share their features in a central repository with a searchable metadata layer over the top. This allows data scientists to search for features that already exist. However, today's feature stores fall short by requiring the data scientist to understand the features they want. If the data scientist is not aware of an existing feature that might help them to build a machine learning model, the data scientist is unlikely to find that feature.

DETAILED DESCRIPTION

One or more aspects of the technology described herein are generally directed towards a machine learning feature recommender, for use in connection with a feature store. By collecting data and recommending machine learning features to users based on collected data, embodiments can facilitate data scientists' discovery of features that have been used by their colleagues and that are likely to make their machine learning models more performant. The disclosed machine learning feature recommender can reduce the effort involved in developing machine learning models.

The term “facilitate” as used herein is in the context of a system, device or component “facilitating” one or more actions or operations, in respect of the nature of complex computing environments in which multiple components and/or multiple devices can be involved in some computing operations. Non-limiting examples of actions that may or may not involve multiple components and/or multiple devices comprise transmitting or receiving data, establishing a connection between devices, determining intermediate results toward obtaining a result, etc. In this regard, a computing device or component can facilitate an operation by playing any part in accomplishing the operation. When operations of a component are described herein, it is thus to be understood that where the operations are described as facilitated by the component, the operations can be optionally completed with the cooperation of one or more other computing devices or components, such as, but not limited to, sensors, antennae, audio and/or visual output devices, other devices, etc.

FIG. 1illustrates an example feature store comprising a recommendation engine, in accordance with various aspects and embodiments of the subject disclosure.FIG. 1includes example feature store100, and illustrates interactions of the feature store100with a user110(also referred to herein as a data scientist110) via a terminal111, wherein the user110is interested in finding features for a machine learning (ML) model112. The feature store100includes a recommendation engine102, a machine learning feature store104, and recommendation engine data106.

InFIG. 1, the user110can supply an input113to the recommendation engine102via terminal111. Input113can comprise any of a variety of data that can be used to determine a feature recommendation, as described further herein. The recommendation engine102can optionally be configured to store the input113in the recommendation engine data106. The recommendation engine data106can also include feature data105, which can identify and describe the various machine learning features included in the machine learning feature store104. At operation107, the recommendation engine102can be configured to use the input113, optionally along with other data stored in the recommendation engine data106, to determine a feature recommendation114. The feature recommendation114can comprise identification(s) of one or more features stored in the machine learning feature store104, for delivery to the user110. The recommendation engine102can send the feature recommendation114to the terminal111or to another device associated with the user110, or the recommendation engine102can store the feature recommendation114in a user110account, accessible by the user110when the user logs into the feature store100.

In some embodiments, the illustrated recommendation engine102can be configured to recommend machine learning features. The recommendation engine102can be added to a feature store100, and the feature store100can include numerous other elements not illustrated herein for the sake of brevity. In particular, the feature store100can comprise feature selection and delivery mechanisms (not illustrated inFIG. 1) to deliver a selected feature to a user110.

A data scientist user110that comes to a feature store100to find machine learning features that will be relevant to his or her machine learning model112can have features recommended via feature recommendation114, wherein the recommended features can comprise features likely to be incorporated into the machine learning model112. The recommended features can comprise features likely to lead to increases in ML model112performance. For example, when a data scientist user110provides an input113comprising a search for existing features, namely features stored in the machine learning feature store104, the recommendation engine102can include feature recommendations114in results of that search, and the feature recommendations114can optionally be ordered by their likely relevance to the data scientist's110interest.

In some embodiments, the feature store100can be configured to provide several phases of user experience for a data scientist user110. A first time the user110signs on to the feature store100, the user110can be asked a few questions such as, what kind of model are you building, who is the model for, etc. Answers to these questions can constitute an input113. Such survey questions acquire input113to make some initial feature recommendations114and avoid the cold-start problem. A few feature recommendations114can initially be made by the recommendation engine102and the user110can choose which features they wish to explore further. User selected features from among feature recommendations114can be used as a further input113to feed back into the recommendation engine102, e.g., by storing feature selection information in the recommendation engine data106. In an embodiment, the recommendation engine102can be configured to learn which recommended features are selected for use by which users, and recommendation engine102can recommend features having a high probability of use by a given user110.

In a second user experience phase, the data scientist user110can search for a feature they think will be useful. The entered search terms from user110can constitute and input113. The recommendation engine102can be configured to include feature recommendations114in search results, and/or to rank search results by likelihood of use or selection by the user110. The recommendation engine102can furthermore provide feature recommendations114comprising features that other users with similar interests (e.g., similar to the entered search terms) have found useful.

In an example third user experience phase, another user (other than user110) has uploaded a feature that the first user110might find useful, according to the recommendation engine102and based on prior input113received from the first user110. The recommendation engine102can be configured to provide a feature recommendation114comprising a push alert to the first user110, notifying the first user110of the new feature they might like to use.

In an example fourth user experience phase, the user110has previously used features from the feature store100to build a model. Such previously used features can constitute input113. Furthermore, the user110can upload feature importance data from their model, and the feature importance data can constitute further input113. The recommendation engine102can be configured to account for the feature importance data in its future feature recommendations114to user110as well as to other users.

In summary, feature recommendation engine102can be configured to recommend different machine learning features to a data scientist or engineer user110to include in their models, such as ML model112, to improve model accuracy and performance. In order to avoid the cold-start problem (where the recommendation engine102doesn't know what features to recommend with no information about a user110), the recommendation engine102can be configured ask the user110questions the first time the user110signs onto the feature store100. After answering the onboarding questions, the recommendation engine102can recommend some features for the data scientist110to include in their model112. When the data scientist110searches for a feature, the recommendation engine102can surface features that may be useful and relevant to the data scientist110. When a user adds a new feature to the store, other users for whom the feature is deemed relevant by the recommendation engine102can receive a notification suggesting they add the feature to their models. When a data scientist110builds a model using features from the feature store100, the user110can upload their feature importance information to the feature store100so that the recommendation engine102can identify the strongest features for future users.

For the purpose of this disclosure, “feature” is a term or art in machine learning, and refers to an individual measurable property or characteristic of a phenomenon being observed, e.g., a variable. As a practical matter, a feature can be associated with a mechanism for a machine model to retrieve the feature, e.g., a uniform resource locator (URL) or other address that links to feature data. Machine learning is applied in many different domains, e.g., weather prediction, spam detection, computer vision, speech recognition, etc., each domain having different features that are expected to be of importance. Furthermore, sometimes unexpected features can be important in any given domain.

For example, for a model that predicts the weather in Dallas, the temperature measured on the nearby coast, e.g., in Houston, may be important, while the inland temperature in Amarillo might not be particularly important. Alternatively, the opposite could be true—the temperature in Amarillo may be quite useful for Dallas weather predictions. By building up recommendation engine data106, embodiments of this disclosure can better understand the features that are likely to be important across a variety of different domains. Furthermore, embodiments of this disclosure can usefully suggest features that might be unexpected by a particular data scientist. For example, while features such as hospital visits are expected to be important in modeling seasonal flu outbreaks, it might not be expected that internet searches for flu symptoms made from devices located in a particular region can also strongly correlate to the likelihood of a flu outbreak in that region. Embodiments of this disclosure are therefore useful to recommend both powerful features and unexpected features.

In some embodiments, the recommendation engine102can be “domain agnostic” in that it can be configured to make feature recommendations in any of multiple different domains. The recommendation engine102can determine a domain of ML model212based on input113, and the recommendation engine102can make feature recommendations114based in part on the determined domain of ML model212. Different domains can have different useful features, and by building recommendation engine data106, the recommendation engine102can build up specialized knowledge about important features across multiple different domains.

In some embodiments, the recommendation engine102can itself be configured as a machine learning model. The recommendation engine102can be trained to use recommendation engine data106, including, e.g., inputs113and feature data105, to determine recommendations114that have relatively higher probability of use by the user110. When a feature is selected for use in a ML model112by the user110, such selection can reinforce future recommendations of the selected feature to similar users. Alternatively, features that are not selected for use can be assigned reduced likelihoods of recommendation to similar users.

FIG. 2illustrates another example feature store comprising a recommendation engine, in accordance with various aspects and embodiments of the subject disclosure.FIG. 2includes example feature store200, and illustrates interactions of the feature store200with multiple users210,220,230, wherein the user210is interested in finding features for a machine learning (ML) model212, and the users220,230have previously built machine learning models222,232, respectively, using features from the feature store200.FIG. 2does not illustrate user terminals, such as111inFIG. 1, however it is understood that the users210,220,230interact with feature store200via user computing devices. The feature store200includes a feature store interface240, a recommendation engine202, a machine learning feature store204, and recommendation engine data206. The feature store interface240comprises search241, user profile242, user/project surveys243, importance information244, feature selections245, messaging246, social networking247, and feature contributions248.

InFIG. 2, the various components241-248of the feature store interface240can comprise, e.g., components that interact with users210,220,230. The components241-248can each collect different types of input data, e.g., data included in inputs213,223,224,233, and234. The inputs213,223,224,233, and234can be stored as inputs250in the recommendation engine data206for use by the recommendation engine202in configuring feature recommendations, such as feature recommendations214, to any of the users, e.g., to user210. Feature data205from the machine learning feature store204can also be included in the recommendation engine data206. The recommendation engine202can be configured to use an input213, optionally along with other input data collected from a user210, to perform an operation207on recommendation engine data206, in order to configure the feature recommendation214.

The search241component can comprise a search interface into which the users210,220,230can type search terms. Entered search terms can be stored along with user identification/profile information in the recommendation engine data206. User profile242can comprise one or more interfaces into which users210,220,230can provide user profile information, e.g., name, employment information, interests, ML projects, favorite features, or other information. Entered user profile information can be stored in the recommendation engine data206. User/project surveys243can comprise interfaces that request information from users210,220,230regarding their interest areas, e.g., their ML models212,222,232. User/project survey data can be stored in the recommendation engine data206.

Importance information244can comprise an interface to receive uploaded feature importance files, e.g., inputs224and234, generated by completed ML models222,232.FIG. 6provides an example feature importance information interface. ML models222,232can track the relative importance, or weight, of the features they use, and can provide importance information to importance information244. Received importance information can be stored in the recommendation engine data206, along with corresponding information about the ML models222,232and users222,232that provided the importance information.

Feature selections245can comprise features selected for use by the users210,220,230. Feature selections245can be stored in the recommendation engine data206. Messaging246can provide a mechanism to communicate feature recommendations214to the users210,220,230. In some cases, feature recommendations214can be included in other interface components, e.g., with search results provided via search214, or in a user profile page supported by user profile242. In other embodiments, messaging246can be used to surface feature recommendations214to a user210.

Social networking247can comprise social networking connections between users210,220,230. Social networking data can be stored in the recommendation engine data206. Feature contributions248can comprise features added to the machine learning feature store204by the users210,220,230. Identifications of the features contributed by a user can be stored in the recommendation engine data206.

The recommendation engine202can be configured to use the data stored in recommendation engine data206to determine feature recommendations214, which recommend features stored in the machine learning feature store204to a particular user210, optionally in connection with a particular ML model212under development by the user210. The recommendation engine202can be configured to use probability of use of a feature, e.g., the probability that the user210will select and retrieve a recommended feature from the machine learning feature store204, as a proxy for utility of the feature to the user210, and so in some embodiments, the recommendation engine202can be configured to formulate recommendations by maximizing a function indicative of probability of selection and/or use by a user210associated with known searches, user profile, surveys, and other collected data as described herein. In another embodiments, the recommendation engine202can be configured to recommend features having relatively high utility and/or importance scores in connection with a give type of ML model212to be built by a user210. In a still further embodiment, the recommendation engine202can be configured to recommend features based on a combination of probability of use scores and feature utility/importance scores.

FIG. 3illustrates example user survey interfaces to gather input data for use in recommending features, in accordance with various aspects and embodiments of the subject disclosure.FIG. 3comprises a series of user/project survey interfaces310,320,330,340which can be presented to a user, e.g., by a user/project surveys component243such as illustrated inFIG. 2.

An example first user/project survey interface310can collect a machine learning domain input, indicative of the domain of a user's ML project. For example, first user/project survey interface310asks, “What kind of model are you interested in building? Select all that apply.” A few example selectable domains are illustrated, including, “Fraud,” “Credit and Collections,” and “Customer Churn.” A wide variety of other domain selections can be included in particular embodiments. In some cases, generalized domain selections can be followed by more specific selections.

An example second user/project survey interface320can collect an input indicating whether the features for a user's ML project include real-time features. For example, second user/project survey interface320asks, “Do you need your feature set to be available in real-time?” The user can select “Yes” or “No”. Some features are available in real-time and others are not, and so the user's selection can be used to narrow the features recommended to the user210to include only features available in real-time, when appropriate.

An example third user/project survey interface330can collect an input indicative of a target of a user's ML project. For example, third user/project survey interface330asks, “Which persona or entity will your model be targeting?” A few example selectable targets are illustrated, including, “Customers,” “Purchasing Transactions,” and “Billing Events.” A wide variety of other target selections can be included in particular embodiments.

An example fourth user/project survey interface340can collect an input indicative of a user's features of interest. For example, fourth user/project survey interface340asks, “Please select all features you're interested in using and we'll included them in a dataset for you to explore.” A few example selectable features of interest are illustrated, including, “Number of consecutive months in which the customer has paid in full on time,” “Number of visits to website in past 24 hours,” and “Total minutes on phone with customer service last month.” A wide variety of other features of interest can be included in particular embodiments.

FIG. 4illustrates an example search interface which can be used to gather input data and to deliver feature recommendations, in accordance with various aspects and embodiments of the subject disclosure. The example search interface410includes a text entry field into which a user can enter search terms, search results based on the search terms entered in the text entry field, and additional recommendations based on both the search terms and other user information. The search interface410can be supported, e.g., by a search241component of a features store200, illustrated inFIG. 2.

InFIG. 4, the search results are illustrated in a left column, and additional recommendations are illustrated in a right column. A user has entered “Phone ca . . . ” in the text entry field, and example search results have been displayed in the search interface410based on the user's entry in the text entry field. The search results include features available in a machine learning feature store such as204, illustrated inFIG. 2. The search results can include, inter alia, feature recommendations214generated by recommendation engine202, or the search results can optionally be ranked by recommendation engine202. Example search results include a first selectable feature, “Total minutes on phone with customer service last month,” a second selectable feature, “Number of marketing calls received in last 24 hours,” and a third selectable feature, “Average number of dropped calls per day last week.”

Example additional recommendations have also been displayed in the search interface410based on the user's entry in the text entry field. The additional recommendations also include features available in a machine learning feature store such as204, illustrated inFIG. 2. The additional recommendations can include feature recommendations214generated by recommendation engine202, or the additional recommendations can optionally be ranked by recommendation engine202. Example additional recommendations include a first selectable feature, “Number of incoming phone calls last 24 hours,” a second selectable feature, “Churn propensity,” and a third selectable feature, “Average daily data use last 30 days.”

FIG. 5illustrates an example feature recommendation, in accordance with various aspects and embodiments of the subject disclosure. The feature recommendation510can be sent to a user as a push type notification, to notify the user of a recently stored feature in a feature store. The feature recommendation510can be sent to a user for whom the recently stored feature matches user data such as user searches, user profile data, etc. The feature recommendation510includes example notification text, “Another feature, “average number of phone calls per day last six months” has been added to the feature store. The feature recommendation510further includes a selectable function button, “Go to the feature store and check it out” which can link to a feature store website. The feature recommendation510further includes a selectable function button, “Stop receiving these notifications” which can send an unsubscribe message to unsubscribe a user from push type feature recommendations.

FIG. 6illustrates an example interface to receive feature importance information, in accordance with various aspects and embodiments of the subject disclosure. The example interface610includes a text entry/file drag and drop field into which a user210can identify or place a feature importance file. The interface610can be supported, e.g., by an importance information244component of a feature store200, illustrated inFIG. 2. The feature importance file can be received at the feature store200via interface610and stored in the recommendation engine data206.

FIG. 7illustrates an example machine learning feature store comprising features associated with multiple different domains, in accordance with various aspects and embodiments of the subject disclosure. The illustrated machine learning feature store704can implement a machine learning feature store204or a machine learning feature store104. The machine learning feature store704can include multiple stored features, for example, features711,712,713, and714. Each feature711,712,713, and714can be associated with metadata such as title, source, description, etc., along with metadata indicating feature domains, as shown. Feature711is illustrated as associated with a weather domain, feature712is illustrated as associated with a weather domain and an agriculture domain, feature713is illustrated as associated with a traffic domain, and feature714is illustrated as associated with a cellular communications domain. Because machine learning model engineering can span a wide range of applicable domains, storing feature domain information along with features can assist with identifying feature recommendations214by a recommendation engine202.

FIG. 8is a flow diagram representing example operations of a system that provides a feature store, in accordance with various aspects and embodiments of the subject disclosure. The illustrated blocks can represent actions performed in a method, functional components of a computing device, or instructions implemented in a machine-readable storage medium executable by a processor. While the operations are illustrated in an example sequence, the operations can be eliminated, combined, or re-ordered in some embodiments.

The operations illustrated inFIG. 8can be performed, for example, by one or more computing devices such as illustrated inFIG. 11, which can implement a feature store200such as illustrated inFIG. 2. Example operation802comprises receiving, by a device comprising a processor, a first input213, wherein the first input213comprises data descriptive of a machine learning model212associated with a machine learning model domain. The data descriptive of the machine learning model212can comprise any of the various inputs described herein. For example, as illustrated inFIG. 3, the data descriptive of the machine learning model212can include a first indication of a machine learning model domain received via an interface such as310, a second indication of whether the group of features used by the machine learning model212comprises real-time features available in real-time received via an interface such as320, a third indication of a target associated with the machine learning model212received via an interface such as330, or a fourth indication of which features are of interest in connection with the machine learning model212, received via an interface such as340.

Example operation804comprises using, by the device, the first input213to identify a feature among features stored in a machine learning feature store204. The features stored in the machine learning feature store204can be associated with multiple different machine learning model domains, as illustrated inFIG. 6. Identifying a feature in the machine learning feature store204can be based on a probability of use of the feature by the machine learning model212, e.g., of selection of the feature by the user210, being higher than other probabilities of use of other features stored in the machine learning feature store204. The first input213can be used, along with other inputs received from user210, to determine the probability of use of the feature. Probabilities of use can be determined by the recommendation engine202in view of recommendation engine data206. Example operation806comprises recommending, by the device, e.g., by the recommendation engine202, the feature, e.g., by sending feature recommendation214to the user210, for inclusion in a group of features used by the machine learning model212.

Operations808-832illustrate additional optional approaches to collect and use inputs to improve feature recommendations such as214. Operations808-810are directed to collecting and storing user feature selections to improve feature recommendations. In an example, using the first input213to identify the feature can comprise using the first input213to identify multiple features among the features stored in the machine learning feature store204. The feature recommendation214can recommend the multiple features. The multiple probabilities of use of the multiple recommended features214by the machine learning model212can be higher than other probabilities of use of the other features stored in the machine learning feature store204. The recommending operation806can comprise recommending the multiple features214for inclusion in the group of features used by the machine learning model212. Example operation808comprises receiving, by the device, feature selections, e.g., selections made by user210, from among the multiple features214. Example operation810comprises storing, by the device, the feature selections, e.g., in recommendation engine data206, for subsequent probability of use determinations associated with the multiple features214.

Operations812-814are directed to collecting and storing feature importance information. Example operation812comprises receiving, by the device, feature importance information, e.g., via an interface such as610illustrated inFIG. 6, indicating an importance of a feature determined by a machine learning model. Example operation814comprises storing, by the device, e.g., in recommendation engine data206, the feature importance information for subsequent probability of use determinations. In some scenarios, after the ML model212is built, the ML model212can also report back feature importance information regarding the feature that was recommended via feature recommendation214.

Operations816-822are directed to search information. Example operation816comprises receiving, by the device, a second input, e.g., a second input213, wherein the second input213comprises a feature search input. Example operation818comprises searching, by the device, the features stored in the machine learning feature store204to identify search results, the search results comprising result features associated with the feature search input213. Example operation820comprises sorting, by the device, e.g., by the recommendation engine202, the search results based on respective probabilities of use of the search results by the machine learning model212. Example operation822comprises storing, by the device, the second input213, e.g., in the recommendation engine data206, for subsequent probability of use determinations.

Operations824-832are directed to user profile information. Example operation824comprises storing, by the device, a user profile comprising a user identifier, e.g., of user210, and the first input213. Example operation826comprises using, by the device, the user profile to identify a second feature among the features stored in the machine learning feature store204. The second feature can be identified based on a second probability of use of the second feature in connection with the user profile being higher than other probabilities of use of other features stored in the machine learning feature store204. Example operation828comprises recommending, by the device, e.g., by a second feature recommendation214, the second feature to in connection with the user profile.

Example operation830comprises determining, by the device, e.g., by the recommendation engine202, the second probability of use of the second feature at least in part by evaluating a similarity of the user profile, e.g. of user210, and a second user profile, e.g., of user220, wherein the second feature is associated with the second user profile of user220. The recommendation engine202can recommend features based on similarity between user profiles by identifying a degree of similarity and recommending features used by one user220to another user210. Example operation830comprises recommending, by the device, a recently stored feature, e.g., a feature stored in machine learning feature store204by user220in the past week, in connection with the user profile, e.g., of user210. Operation830can also be based in part on similarity between user profiles of user220and user210.

FIG. 9is a flow diagram representing another set of example operations of a system that provides a feature store, in accordance with various aspects and embodiments of the subject disclosure. The illustrated blocks can represent actions performed in a method, functional components of a computing device, or instructions implemented in a machine-readable storage medium executable by a processor. While the operations are illustrated in an example sequence, the operations can be eliminated, combined, or re-ordered in some embodiments.

The operations illustrated inFIG. 9can be performed, for example, by one or more computing devices such as illustrated inFIG. 11, which can implement a feature store200such as illustrated inFIG. 2. The illustrated operations pertain to search. Example operation902comprises receiving a feature search input, e.g., input213, wherein the feature search input213is associated with a user profile, e.g., of user210. The data associated with the user profile can comprise, inter alia, model data descriptive of a machine learning model212associated with a machine learning model domain.

Example operation904comprises, based on the feature search input213, searching a machine learning feature store204in order to identify search results, the search results comprising features associated with the feature search input213. Example operation906comprises, based on data associated with the user profile of user210, determining respective probabilities of use of the search results. Determining the respective probabilities of use of the search results can be based on data associated with the user profile of user210as well as data associated with other users, e.g., data associated with a second user profile, of a second user220. For example, user220's prior search history and feature selections can be used to determine probabilities of use of certain features by user210, especially when user210has a similar profile, indicating similar interests and projects, as user220. Feature importance information determined by a machine learning model222associated with the second user profile, of user220, can also be used to determine probabilities of use of features by user210in connection with ML model212. Example operation908comprises sorting the search results based on the respective probabilities of use of the search results. The sorted search results can be provided, e.g., as recommendation214, to the user210.

FIG. 10is a flow diagram representing another set of example operations of a system that provides a feature store, in accordance with various aspects and embodiments of the subject disclosure. The illustrated blocks can represent actions performed in a method, functional components of a computing device, or instructions implemented in a machine-readable storage medium executable by a processor. While the operations are illustrated in an example sequence, the operations can be eliminated, combined, or re-ordered in some embodiments.

The operations illustrated inFIG. 10can be performed, for example, by one or more computing devices such as illustrated inFIG. 11, which can implement a feature store200such as illustrated inFIG. 2. The illustrated operations pertain to feature importance information. Example operation1002comprises receiving feature importance information, e.g., as inputs224and234, determined by machine learning models222,232, wherein the feature importance information224,234comprises feature importance information associated with multiple features machine learning models222,232, and wherein the multiple features are from multiple machine learning model domains. For example, the machine learning models222,232can be from multiple different machine learning models domains, e.g., weather, cellular networking, disease modeling, etc. The received feature importance information224,234can be stored in the recommendation engine data206.

Example operation1004comprises receiving data, e.g., input213, descriptive of a machine learning model212associated with a machine learning model domain of the multiple machine learning model domains. For example, inputs such as those associated withFIG. 3can be received, including a first indication of the machine learning model212domain, a second indication of whether a group of features used by the machine learning model212comprises real-time features that consume real-time data, a third indication of a target associated with the machine learning model212, and/or a fourth indication of ones of the features that are of interest for inclusion in the machine learning model212.

Example operation1006comprises using the data213descriptive of the machine learning model212and the feature importance information224and/or234to identify a recommended feature214among features stored in a machine learning feature store204, wherein the recommended feature214is represented in the machine learning model domain. Operation1006can be performed, for example, by the recommendation engine202.

In an embodiment, using the data213and the feature importance information224and/or234to identify the recommended feature214comprises determining a probability of use of the recommended feature214in the machine learning model212based on the feature importance information224and/or234. Determining the probability of use of the recommended feature214in the machine learning model212can be further based on user profile data of user210, associated with the data213descriptive of the machine learning model212. The user profile data can be associated with a user identity of user210, and the user profile data can comprise various data discussed herein in connection withFIG. 2, e.g., search history data associated with the user identity.

FIG. 11is a block diagram of an example computer that can be operable to execute processes and methods in accordance with various aspects and embodiments of the subject disclosure. The example computer can be adapted to implement, for example, any of the various network equipment described herein.

The system bus1108can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory1106includes ROM1110and RAM1112. A basic input/output system (BIOS) can be stored in a non-volatile memory such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer1102, such as during startup. The RAM1112can also include a high-speed RAM such as static RAM for caching data.

The computer1102further includes an internal hard disk drive (HDD)1114(e.g., EIDE, SATA), one or more external storage devices1116(e.g., a magnetic floppy disk drive (FDD)1116, a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive1120(e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDD1114is illustrated as located within the computer1102, the internal HDD1114can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment1100, a solid state drive (SSD) could be used in addition to, or in place of, an HDD1114. The HDD1114, external storage device(s)1116and optical disk drive1120can be connected to the system bus1108by an HDD interface1124, an external storage interface1126and an optical drive interface1128, respectively. The interface1124for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

A number of program modules can be stored in the drives and RAM1112, including an operating system1130, one or more application programs1132, other program modules1134and program data1136. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM1112. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer1102can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system1130, and the emulated hardware can optionally be different from the hardware illustrated inFIG. 11. In such an embodiment, operating system1130can comprise one virtual machine (VM) of multiple VMs hosted at computer1102. Furthermore, operating system1130can provide runtime environments, such as the Java runtime environment or the .NET framework, for applications1132. Runtime environments are consistent execution environments that allow applications1132to run on any operating system that includes the runtime environment. Similarly, operating system1130can support containers, and applications1132can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

A monitor1146or other type of display device can be also connected to the system bus1108via an interface, such as a video adapter1148. In addition to the monitor1146, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

The computer1102can operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s)1150. The remote computer(s)1150can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer1102, although, for purposes of brevity, only a memory/storage device1152is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN)1154and/or larger networks, e.g., a wide area network (WAN)1156. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the internet.

When used in a LAN networking environment, the computer1102can be connected to the local network1154through a wired and/or wireless communication network interface or adapter1158. The adapter1158can facilitate wired or wireless communication to the LAN1154, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter1158in a wireless mode.

When used in a WAN networking environment, the computer1102can include a modem1160or can be connected to a communications server on the WAN1156via other means for establishing communications over the WAN1156, such as by way of the internet. The modem1160, which can be internal or external and a wired or wireless device, can be connected to the system bus1108via the input device interface1144. In a networked environment, program modules depicted relative to the computer1102or portions thereof, can be stored in the remote memory/storage device1152. It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer1102can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices1116as described above. Generally, a connection between the computer1102and a cloud storage system can be established over a LAN1154or WAN1156e.g., by the adapter1158or modem1160, respectively. Upon connecting the computer1102to an associated cloud storage system, the external storage interface1126can, with the aid of the adapter1158and/or modem1160, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface1126can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer1102.

The above description includes non-limiting examples of the various embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the disclosed subject matter, and one skilled in the art can recognize that further combinations and permutations of the various embodiments are possible. The disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.