Patent Publication Number: US-2022214780-A1

Title: User interface for machine learning feature engineering studio

Description:
BACKGROUND 
     Machine learning algorithms build a mathematical model based on sample data, known as “training data,” in order to make predictions or decisions without being explicitly programmed to perform the task. In machine learning, a feature is an observable property of an object in a dataset. A feature vector is a list of features of an object in a dataset. The feature vector is generated from information about the object and events related to the object. The generation of feature vectors requires considerable technical knowledge and labor by a user, such as a data scientist. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The following drawings show generally, by way of example, but not by way of limitation, various examples discussed in the present disclosure. In the drawings: 
         FIG. 1  shows example feature engineering system ingesting data and outputting query results. 
         FIG. 2A  shows an example feature engineering system in training stage and application stage. 
         FIG. 2B  shows the components an example feature engine. 
         FIG. 2C  shows a feature engine in a data ingestion configuration process. 
         FIG. 2D  shows a feature engine in a feature creation process. 
         FIG. 3  shows example event data being persisted in related event stores. 
         FIG. 4  shows example event data over time. 
         FIG. 5  shows example event data, anchor times, feature values, and transformations used to compute features. 
         FIG. 6  shows an example feature engineering method. 
         FIG. 7  shows an example computing node. 
         FIG. 8  shows an example cloud computing environment. 
         FIGS. 9A-C  show an example of a user interface of a feature studio to create a new project. 
         FIG. 10  shows an example confirm new project page of the user interface of the feature studio. 
         FIG. 11A  shows an example of a user interface give the user the option of importing an existing feature or creating a new feature by way of the feature studio. 
         FIG. 11B  shows an example New Feature button of the user interface of the feature studio. 
         FIGS. 12A-B  show an example new feature card of the user interface of the feature studio. 
         FIG. 13  show an example graph and a table in the new feature window of the user interface of the feature studio. 
         FIGS. 14A-G  show an example feature visualization and values of a feature in the new feature window of the user interface of the feature studio. 
         FIG. 15A-C  shows an example list of events in the user interface of the feature studio. 
         FIG. 16A  shows an example data schema of the feature studio. 
         FIG. 16B  shows a process for relating entities using the data schema. 
         FIG. 17  shows an example user interface of an all projects page. 
         FIGS. 18A-B  show example feature cards in the user interface of the feature studio. 
         FIGS. 19A-C  show an example comparison of two features in the user interface of the feature studio. 
         FIGS. 20A-C  show example versioning of features of a project. 
         FIG. 21  shows an example history of features committed to a project. 
         FIG. 22  shows an example select approver window in the user interface of the feature studio. 
         FIG. 23  shows an example notification indicating commitment of a project. 
         FIG. 24A  shows an example export process of a project. 
         FIG. 24B  shows an example confirm information page in an export window in the user interface of the feature studio. 
         FIG. 25  shows an example select time window page in the export window in the user interface of the feature studio. 
         FIG. 26  shows an example select target page in the export window in the user interface of the feature studio. 
         FIG. 27  shows an example summary page in the export window in the user interface of the feature studio. 
         FIG. 28  shows an example notification indicating exportation of a project. 
         FIG. 29A-F  shows an example user interface of a feature store. 
         FIG. 30  shows an example process for a feature studio. 
     
    
    
     DETAILED DESCRIPTION 
     Recently in machine learning and artificial intelligence there has been a significant focus on tools that are directed to bringing machine learning models to production. Many of these tools incorporate so-called notebooks into production pipelines. Notebooks are interfaces (on-line and/or local) that allow a user such as a data scientist (for clarity and to distinguish from other types of users, the term data scientist is used herein interchangeably with the user of the system; however users of the system are not limited to data scientists) to create documents containing code, visualizations and text. The notebooks are used for data analysis and are used to manipulate raw data and visualizations and are used to try to understand the data that will be used in a machine learning model. 
     As disclosed herein, a feature studio aids in the development of machine learning models and addresses many of the shortcomings of notebooks. In the feature studio disclosed herein features are grouped into a project that corresponds to a specific model. Each project has specified primary and secondary entities that all features are built upon. For example, in the context of a machine learning model for a video on demand system, specific entity examples are users, i.e., users of the video on demand system that control the selection, playback, and control over a selected video, and content, e.g., movies and other shows. 
     Features are individual independent variables that are a foundational aspect of a machine learning model and serve as input to the machine learning model. Models use a set of features to make predictions. An aspect of this disclosure is a feature studio which provides tools to take data gathered from various external systems, e.g., a video on demand system, and use that data to calculate features, e.g., over specified time ranges. A project in the feature studio as described herein may name a feature or event as a target. The target is the feature that the machine learning model is designed to predict. 
     The feature studio described herein allows for features to be updated iteratively by the data scientist. The iterative nature of the feature studio allows a data scientist to adjust the manner in which data is processed such as in cleaning and/or calculating features for a given set of input data. In other aspects, the feature studio allows multiple users to collaborate and work on features concurrently in a single project. 
     In an embodiment, the feature studio provides for viewing data schemas in a user interface. Data schemas for data that is ingested into the feature studio systems are shown. Users can explore data schemas by visualizing raw events and creating features. For example, as explained more fully below, a data scientist selects a button to initiate feature creation via a “feature card.” Thereafter, the data scientist will have the ability to see statiscal information for data selected for the feature and to allow users to describe features with an expression (Fx) using a set of predefined language (DSL) that performs calculations on the data to create the feature. The features may be automatically visualized once the expression is executed. Descriptive statistics (count, mean, min, max, etc.) are automatically calculated and represented visually once the Fx is run. This allows the data scientist to understand aspects of the data and to consider changes to the DSL that change aspects of the displayed statistics. Additionally, the system may provide chart interactivity. Chart interactivity allows users to customize their charts that show the feature data set. In that regard, the system may automatically select chart types based on data type (histogram, bar chart, etc.). In embodiments, users can choose to see other chart types, select and zoom into data via charts, customize colors, axes, etc. for charts, and drill down into data to see raw events for a specific entity. 
     As explained below, the system provides pre-defined feature transformations. According to that aspect, users apply feature transforms visually using tabs/menus. Such feature transforms include operations such as scaling data, cleaning/filtering data, encoding data, windowing data, and binning. When transforms are applied, the data visualization and statistics automatically update with transformed values. Users also view and reorder transformations on the data. Additionally, users can select specific time windows to compute features over, including rolling window, fixed date/time anchors, and so on. 
       FIG. 1  shows an example feature engineering system  100 . Feature engineering system  100  ingests data from data sources  101 ,  102 , stores the data, and uses the data for computation of features. Ingestion and/or storing of the data continuously and/or as new data becomes available allows for up-to-date feature computations. A user can query feature engineering system  100  at any time to receive features based on the most current ingested data or data from a particular time. In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a phenomenon, object, or entity being observed. Choosing informative, discriminating, and independent features is an important step for effective algorithms in pattern recognition, classification and regression. Features can be numeric, such as values or counts. Features can be structural, such as strings and graphs, like those used in syntactic pattern recognition. 
     In an embodiment, feature engineering system  100  is configured to efficiently provide and/or generate features for the user for use in the training or application stage of machine learning. In the training stage, a model is produced by providing a machine learning algorithm with training data, such as several training examples. Each training example includes properties, such as features. The properties may include a desired target, such as in supervised machine learning. A set of features for a specific instance or entity is known as a feature vector. The training example may include several feature vectors, which may be organized in columns with the same properties described for each instance or entity. In supervised machine learning, a model may be produced that generates results or predictions for an entity based on a feature vector that is input and associated with that entity. The algorithm produces a model that is configured to minimize the error of results or predictions made using the training data. 
     In the application stage, a model may be used to generate results or make predictions and may be tested or evaluated based on the generated results or predictions. Applying the model may involve computing a feature vector using the same computations that were used in training of the model, but for an instance that was not present in the training example. The model may be evaluated based on the accuracy or error of the data in the generated feature vector. 
     System  100  is configured to ingest event data from one or more sources  101 ,  102  of data. In some configurations, a data source includes historical data, e.g., from historical data source  101 . In that case, the data includes data that was received and/or stored within a historic time period, i.e. not real-time. The historical data is typically indicative of events that occurred within a previous time period. For example, the historic time period may be a prior year or a prior two years, e.g., relative to a current time, etc. Historical data source  101  may be stored in and/or retrieved from one or more files, one or more databases, an offline source, and the like or may be streamed from an external source. 
     In another aspect of example feature engineering system  100 , the data source includes a stream of data  102 , e.g., indicative of events that occur in real-time. For example, stream of data  102  may be sent and/or received contemporaneous with and/or in response to events occurring. In an embodiment, data stream  102  includes an online source, for example, an event stream that is transmitted over a network such as the Internet. Data stream  102  may come from a server and/or another computing device that collects, processes, and transmits the data and which may be external to the feature engineering system. 
     The data from sources  101 ,  102  may be raw data. The raw data may be unprocessed and/or arbitrarily structured. In an embodiment, the data from sources  101 ,  102  may be organized in fields and/or tables, such as by the system  100 . If source  101 ,  102  is a database, e.g., a relational database, it may have a schema. The schema is a system that defines the fields, the tables, relationships, and/or sequences of the data in the database. The schema can be provided to feature engineering system  100  to provide a definition of the data. The fields can have one or more user-defined labels. The labels can be provided to feature engineering system  100  to provide a definition of the data. 
     In an embodiment, the ingested data is indicative of one or more events. The ingested data is indicative of one or more entities associated with one or more of the events. An example of an event may include a browsing event or a watch event, e.g., a click stream. An example of the entity may include a user or a product, etc. 
     In an embodiment, system  100  includes a feature engine  103 . Feature engine  103  is operable on one or more computing nodes which may be servers, virtual machines, or other computing devices. The computing devices may be a distributed computing network, such as a cloud computing system or provider network. 
     According to an embodiment, feature engine  103  includes an event ingestion module  104 . Event ingestion module  104  is configured to ingest the data from one or more of sources of data  101 ,  102 . For example, event ingestion module  104  may import data from historical data source  101 , such as to perform a set-up and/or bootstrap process, and also may be configured to receive data from stream of data  102  continuously or in real-time. 
     According to another aspect of the disclosed subject matter, event ingestion module  104  is configured to assign events arrival timestamps, such as based on ingesting the data indicating the events. Additionally, event ingestion module  104  may be configured to assign the arrival timestamps using a distributed timestamp assignment algorithm. In an embodiment, the distributed timestamp algorithm assigns timestamps comprising a plurality of parts. For example, a part of a timestamp may have a time component. According to an aspect, the time component indicates an approximate comparison between machines, such as an approximate comparison between a time that data source  101 ,  102  sent the data and a time that feature engine  103  ingested the data. According to another aspect, the timestamp may have a unique machine identification (ID) that prevents duplicate timestamps among other things. According to yet another aspect, the timestamp has a sequence number. An aspect of the sequence number allows multiple timestamps to be generated. The timestamps may be used to indicate a total order across all events. If events from data stream  102  are a partitioned stream, e.g., a Kafka stream, a Kinesis stream, etc., the timestamps indicate a total order across all events and indicate an order of the events within each partition. The timestamps facilitate approximate comparisons between events from different partitions. 
     In some embodiments, the ingested data includes an indication of an occurrence time associated with an event. The occurrence time is a time that the event occurred. The occurrence time may be different than the time component and/or an arrival time associated with the event and/or the ingested data. 
     According to an aspect, feature engine  103  is configured to determine an entity associated with an event in the ingested data. For example, feature engine  103  may determine the entity associated with the event using the schema, the fields, and/or the labels of the data. As another example, the ingested data may indicate the entity, such as by a name, number, or other identifier. Feature engine  103  may also be configured to group events in the ingested data by entity. 
     In embodiments, feature engine  103  is configured to de-duplicate events. If a duplicate of same events are received, ingesting the data may include de-duplicating the events. Techniques for de-duplicating the events may include using unique identifiers associated with events to track unique identifiers that have been ingested. If an event arrives having a unique identifier that is a duplicate of a unique identifier of an event that has already been ingested, the arriving event may be ignored. 
     In embodiments, feature engine  103  is configured to de-normalize events. In particular, events may be associated with more than one entity. De-normalizing an event includes storing a copy of an event for each entity associated with the event. Notably, this is different from de-duplicating events in that de-duplicating recognizes and removes duplicates from the same set of data so that the feature engine does not double count events, for example. 
     In embodiments, feature engine  103  is configured to filter the data. Filtering the data includes such actions as determining optimal events and/or events that may be used to determine a feature. Feature engine  103  may be configured to continuously group, de-normalize, and/or filter data as it is received, such as from data stream  102 . 
     In embodiments, feature engine  103  includes a related event store  105 . In that instance, feature engine  103  is configured to store an indication of an entity associated with an event in related event store  105 . Feature engine  103  is configured to store groupings of events associated with common entities in related event store  105 . Feature engine  103  is configured to continuously store and/or update associated data stored to related event store  105  as data is ingested, such as from data stream  102 . Related event store  105  facilitates efficient, on-demand access to results  113  to a user query. Query results  113  may include events associated with specific entities. Query results  113  may include statistics across a plurality of entities. 
     Feature engine  103  includes a feature computation layer  106 . Feature computation layer  106  is configured to determine one or more features associated with an entity. In embodiments, the features to be determined are defined by a user. In embodiments, feature computation layer  106  is configured to determine a feature using a feature configuration for the feature. In embodiments, the feature configuration is received from the user, such as from a feature studio as described more fully herein. 
     In embodiments, feature computation layer  106  is configured to determine the features using the raw data and/or events stored to related event store  105 . The feature computation layer  106  may be configured to determine the features by applying a variety of numerical processes to the data, such as arithmetic operations, aggregations, and various other techniques. Determination of the features may be an experimental process. For example, the feature computation layer  106  may determine which features would be useful for a model. A useful feature may be one that is informative. A feature may be informative if it is useful to the task that a model is being trained for and that correlates with the goal of the model. A feature may be useful if it is discriminating. A discriminating feature may have different values for different goals. A useful feature may be a feature that is independent. An independent feature may not be related to or depend on other features. A useful feature may be a feature that does not suffer from leakage. A feature that does not suffer from leakage is one that does not depend on information that is only available from (or after) a target event. 
     In an embodiment, a user of the system  100  may determine useful features for a model by evaluating the features using both numerical methods and attempts to train a model using the features. By attempting to train the model using the features, the user may see if the model trained using the features of interest has less error, such as by testing the model using a validation set, as compared to the model trained with different features. 
     Selection of useful values for a model may reduce a number of training examples needed to train the model. When more features are used to train and/or use a model, exponentially more training examples are needed to train the model. Determining a good combination of features for a model involves balancing the usefulness of the information captured by each feature with the additional need for training data that the feature imposes. Therefore, determining useful features enables production of a good model with a minimal number of training examples needed to produce the model. 
     According to an aspect, feature computation layer  106  is configured to compute features by performing aggregations across events associated with an entity. Computing features from large amounts of raw data is a technically complicated process, as it may involve computing aggregate properties across all of the raw data. 
     According to an aspect, feature computation layer  106  is configured to continuously determine features, such as when feature engine  103  ingests new data from data stream  102 . Determining features may include updating features and/or feature vectors, such as based on ingesting new data from data stream  102 . The feature computation layer  106  may be configured to compute the features and/or update the features at a speed that supports iteration and exploration of potential features to determine good features for a model. The continuous computation of features again highlights the importance of determining good features. As events continue to be produced and/or ingested the size of the raw data set, e.g., saved to the event store  105 , increases over time. As a result of the system&#39;s  100  feature determination and updating function, the work needed to compute features does not increase over time and/or as the size of the raw data set increases. 
     Determining features may include accessing information outside related event store  105 , e.g., by performing lookups from external databases that haven&#39;t been ingested by feature engineering system  100 . According to another aspect, feature computation layer  106  is configured to determine and/or update features in response to user queries. 
     The feature engineering system  100  may simplify collaboration in feature generation and/or selection. Features are often defined by users, such as data scientists. A company may have multiple data scientists producing features for one or more models. The data scientists may need to use different tools to access different kinds of raw data and/or events, further complicating the process of producing features. Collaboration on features produced in ad-hoc and varied ways makes it difficult to share features between users and/or projects. In addition, the techniques for producing features may vary based on the data size and the need for producing the feature vectors “in a production environment.” This may lead to the need to implement features multiple times for different situations. However, the feature engineering system  100  may address these shortcomings by ingesting and/or saving raw data and/or events from a variety of sources and making the features available to users in different locations and/or using different devices, such as via the feature studio described further herein. 
     In an embodiment, feature computation layer  106  is configured to compute feature vectors. A feature vector is a list of features of an entity. The feature computation layer  106  may be configured to compute and/or update feature vectors as events are ingested by the feature engine  103 . The feature computation layer  106  may be configured to compute and/or update feature vectors in response to user queries. 
     In an embodiment, feature engine  103  includes a feature store  107 . Feature computation layer  106  may store the determined features and/or generated feature vectors to feature store  107 . Feature store  107  makes deployed features available for users. According to an aspect, feature computation layer  106  keeps feature store  107  up-to-date, such as by computing and updating values of features when new events are received and/or when a request is received from a user. Based on the features stored to feature store  107 , feature computation layer  106  may avoid recomputing features using the same events. For example, if feature computation layer  106  has determined features using events up to arrival time x, feature computation layer  106  determines features using events up to arrival time x+n by only considering events that arrived after arrival time x and before arrival time x+n. 
     According to an aspect, feature computation layer  106  updates the features and/or save the new features to feature store  107 . As a result, feature store  107  is configured to make up-to-date query results  113  available on-demand. Query results  113  may include features and/or feature vectors, such as across a plurality of entities and/or associated with a specific entity. Query results  113  may include aggregate statistics across a plurality of entities. 
       FIG. 2A  shows an example feature engineering system  200 . System  200  includes one or more data sources  201 . Data sources  201  may be similar to data sources  101 ,  102  in  FIG. 1 . Data sources  201  may include sources of historical data, data streams, or a combination thereof. 
     System  200  includes a feature engine  203 . Feature engine  203  may be similar to feature engine  103  in  FIG. 1 . Feature engine  203  may receive data associated with a plurality of entities from data sources  201  and/or a user, such as from a feature studio  215  via an API  212 . Feature studio  215  allows users to define features that feature engine  203  will determine using the ingested data. A feature can be defined using one or more formulas. For example, if “Purchases.amount” is amounts of purchases of a user, a user may define a feature “total purchases of a user” with the formula “SUM(Purchases.amount),” which adds up the amounts of the user&#39;s purchases. 
     Feature engine  203  has functionalities for both the training stage and the application stage of a machine learning process. For the training stage, feature engine  203  is configured to generate training examples  208  to produce a machine learning model  210 . Training examples  208  are generated using the ingested data. In an embodiment, training examples  208  are feature vectors. Training examples  208  are output to the user, such as via API  212  and/or feature studio  215 . The user can feed training examples  208  to a model training algorithm  209  to produce a machine learning model  210 . Model  210  may be used to make predictions using new and/or different data, e.g., data different from the data of training examples  208 . 
     For the application stage, feature engine  203  is configured to generate feature vectors  211 , which may be fed to machine learning model  210 . In an embodiment, a user requests a feature vector  211  for a specific entity via feature studio  215  and/or via API  212 . In response to receiving the request for feature vector  211 , feature engine  203  generates and/or outputs feature vector  211 , such as via feature studio  215  and/or via API  212 . 
     Generating feature vector  211  may include determining one or more features associated with the entity that make up the feature vector using the ingested data. If the features have already been determined, e.g., before receiving the request, and have been stored, such as to feature store  107  in  FIG. 1 , feature engine  203  retrieves the stored features associated with the entity and uses the previously determined features and the newly arriving events to generate updated values of the features. According to an aspect, feature engine  203  determines features using a configuration  214 . Configuration  214  may be an algorithm. Configuration  214  may be received from the user, such as via feature studio  215  and/or API  212 . 
     After receiving feature vector  211  from feature engine  203 , the user may feed feature vector  211  to machine learning model  210 . Machine learning model  210  is configured to use feature vector  211  to make predictions and/or determine information associated with the entity. Machine learning model  210  is configured to output the predictions and/or information via feature studio  215  and/or API  212 . 
     As an illustrative example, during the training stage, feature engine  203  receives data associated with a plurality of entities comprising one thousand people. The data indicates movies that the people have watched and movies that the people want to watch. Feature engine  203  may generate training examples  208 . The user feeds training examples  208  to a training algorithm  209  to produce a machine learning model  210 . Machine learning model  210  represents what movies people should watch based on their movie-viewing history. 
     During the application stage, the user requests a feature vector  211  for an entity, such as a particular person via API  212  and/or feature studio  215 . Feature engine  203  generates a feature vector  211  comprising a list of movies that the person has watched. Feature engine  203  outputs the feature vector  211  to the user via API  212  and/or feature studio  215 . The user feeds feature vector  211  to machine learning model  210 . Machine learning model  210  predicts one or more movies that the person should watch. The user may use the prediction to provide the person with movie suggestions or for targeted advertising. 
     In addition to feature vector  211 , feature engine  203  is configured to output other query results  213  in response to a user query. For example, other query results  213  may include feature values, statistics, descriptive information, a graph, e.g., a histogram, and/or events associated with one or more entities. According to an aspect, query results  213  are associated with a time specified by the user. According to another aspect, query results  213  are computed using all feature values, a sample of feature values, or aggregated feature values. 
     In an embodiment, the user interacts with feature engine  203  to update the feature value and/or feature vector  211  computations, such as via feature studio  215 . For example, the user may indicate a new configuration  214  that should be applied to compute feature values and/or feature vectors  211 . As another example, the user may indicate that particular features are no longer necessary, e.g., should not be computed and/or should not be included in feature vectors or computations of query results  213 . 
       FIG. 2B  shows the feature studio  215 . According to an embodiment, the feature studio  215  is a system of components. The components interact to enable a user to create, visualize, and use features via a feature studio interface  216 , e.g., the user interface shown in  FIGS. 9A-30 , without writing code. The components include a data engine  217 . The data engine  217  receives data from a data source, e.g., data stream  102  and/or historical data source  101  in  FIG. 1 , and stores the data in an event store  218 , e.g., related event store  105  in  FIG. 1 . 
     The components include a feature studio service  219  that receives data from the data engine  217 . The feature studio service  219  computes features based on the data and stores the features in a studio data repository  220 , e.g., feature store  107  in  FIG. 1 . The feature studio service  219  provides data and/or features to the feature studio UI  216 , such as in response to a user request and/or query via the feature studio UI  216 . The feature studio UI  216  enables data scientists to author features and visualize the computed values of the features. The feature studio UI  216  and/or the feature studio service  219  can store computed features, visualizations, projects, and or data in the studio data repository  220 . According to aspects, parts of the feature studio  215  may run in a user&#39;s browser and/or on the user&#39;s machine, while other parts may run on servers hosted elsewhere. 
       FIG. 2C  shows a data ingestion configuration process  200 C within the feature studio  215 . The feature studio  215  gives the user several options for configuring how data is ingested and made available to the feature engine  203  and/or the feature studio  215 . For example, the user can configure how data is retrieved by connecting new data sources (step  225 ). As another example, the user can define the types of entities within the data set (step  226 ). As yet another example, the user can configure how raw data is mapped to event views and related to entities ( 227 ). 
       FIG. 2D  shows a feature creation process  200 D within the feature studio  215 . The feature creation process  200 C includes various actions  221 ,  222 ,  223 ,  224  performed by a user in the feature studio  215 , which are represented in the feature studio UI  216 , which provides a central view showing defined features and visualizations of computed data. For example, the user may choose to add a new feature (action  223 ), e.g., by selecting the New Feature button  1105  in  FIG. 11B  and/or by entering a function that describes the feature, e.g., as described in relation to  FIGS. 12A-12B . The new feature is defined by the feature studio  215  and displayed in the feature studio UI  216 . 
     The user may choose to apply a predefined transform (action  222 ) to one or more features, e.g., as described with respect to  FIG. 5 . The feature studio  215  may apply the transforms to the features and display the updated features and/or visualizations of the updated features in the feature studio UI  216 . The feature studio  215  may generate updated graphs (action  224 ) based on the transformed features and display the graphs, e.g.,  1308  in  FIG. 13 , in the feature studio UI  216 . 
     The user may choose to input and/or change the formula for a feature (action  221 ), as described in relation to  FIGS. 12A-12B . The feature studio  215  may calculate the feature using the formula and display the calculated feature and/or a visualization of the feature in the feature studio UI  216 . The visualization may be interacted with to better understand various characteristics, such as the behavior at the tail ends of the distribution. The feature definition may be further adjusted, updating the data within the visualization. Feature edits include applying custom or predefined transforms, changing the configuration of transforms, reordering or removing steps within the formula, and/or other changes. The feature studio  215  may generate updated graphs (action  224 ) based on the calculated features and display the graphs in the features studio UI  216 . 
       FIG. 3  shows example event data  300 . In an embodiment, event data  300  is stored in a plurality of related event stores  303 ,  304 ,  305 . Related event stores  303 ,  304 ,  305  may be similar to related event store  105  in  FIG. 1 . One or more computing devices, e.g., feature engine  103  in  FIG. 1 , event ingestion module  104  in  FIG. 1 , and/or feature engine  203  in  FIG. 2A  may persist, e.g., store, event data  300  to related event stores  303 ,  304 ,  305 . 
     According to an aspect, event data  300  is persisted to related event stores  303 ,  304 ,  305  at different rates, such as based on network latency and/or processing of the computing devices. As shown in  FIG. 3 , the rate of event data  300  that has fully persisted, partly persisted, and is being received (“future events”) may vary across related event stores  303 ,  304 ,  305 . Fully persisted events are events that have been persisted to event stores  303 ,  304 ,  305 . Partly persisted events are events that have been sent to event stores  303 ,  304 ,  305 , but have not been received, data that is still being ingested by a computing device, and/or data that has been received by related event stores  303 ,  304 ,  305  but is not yet persisted. Future events are events that have not been sent to related event stores  303 ,  304 ,  305 . 
     In an embodiment, in order to reach consensus on timing of events from event data  300 , despite network and/or processing delays, the computing devices store the events to related event stores  303 ,  304 ,  305  with associated timestamps. According to an aspect, the timestamps are multi-part timestamps, such as the timestamps described in reference to  FIG. 2 . According to another aspect, the timestamps include arrival timestamps that indicate times that the events were received by the computing devices. The timestamps may be assigned after events are received and before they are persisted. Timestamps may be assigned as soon as possible after arrival of events to ensure that the timestamps accurately indicate the arrival order of events at the computing devices. The timestamps may be similar to the Twitter Snowflake ID and/or the Sonyflake. 
     In an embodiment, based on the arrival timestamps, the system can avoid recomputing feature values. A feature computation layer, such as feature computation layer  106  in  FIG. 1 , determines that a feature value with a known arrival time will not change by determining that no events with earlier arrival times will be persisted. Determining that no events with earlier arrival times will be persisted may be performed by causing related event stores  303 ,  304 ,  305  to report minimum local arrival times  315 ,  316 ,  317  of any not-yet-persisted events and remembering previously reported values of minimum local arrival time  315 ,  316 ,  317  of any not-yet-persisted event. The minimum time of minimum local arrival times  315 ,  316 ,  327  marks the complete point  318 , a time prior to which new data affecting the computed feature values will not be received. The computation layer remembers features that are computed using events with timestamps at and/or prior to complete point  318 . Avoiding recomputing of feature values increases the efficiency of feature computation. 
     According to an aspect, computed features may be stored with an indication of the times at which they were computed. When new events are received, new feature values are computed using a feature value with the latest computation time and/or a feature value with the latest events and the new events. 
     New events may be received in an order that does not correspond to their occurrence times. In this case, in order to update feature values, the occurrence times of events that arrived after the latest feature value computation time are determined. The minimum occurrence time of the determined occurrence times represents an oldest event of the newly received events. The computed feature value with the largest computation time that is less than or equal to the minimum occurrence time is identified and represents the real point at which to start feature computation. All of the events that occurred after the real point are re-processed. 
     According to an aspect, ordered aggregations are performed using this method applied across feature values and events associated with a specific entity. The feature studio  215  may allow the user to define features using a DSL. The feature engine  203  may support the DSL to compute values associated with the features, which can be defined by aggregating ingested events related to each entity. Alternatively or additionally, the features can be defined by aggregating past and current values of other features and aggregations related to each entity. 
     According to an aspect of the disclosed subject matter, the arrival timestamps facilitate deploying configuration updates without causing a shut-down of the system. Once a configuration update is deployed, events that persisted after the configuration update was deployed, e.g., have a timestamp later than the deployment time, will be processed using the latest configuration. Events that persisted when and/or prior to the configuration update being deployed, e.g., have a timestamp at or earlier than the deployment time, may have been ingested using an older configuration. Therefore, the events that persisted when and/or prior to the configuration update being deployed are re-processed using the latest configuration. 
     To determine which events should be re-processed, related event stores  303 ,  304 ,  305  reports the arrival time that the latest configuration went into effect. The maximum time of the arrival times serves as a cutoff arrival time. Events having timestamps after the cutoff arrival time are processed with the new configuration. Events having timestamps before this time are not re-processed. Not re-processing events having timestamps before the cutoff arrival time saves time and improves system efficiency. 
       FIG. 4  shows example events  400  for two entities  420 ,  421  over time. Events  400  may be events in a dataset ingested by a feature engine, e.g., feature engine  103  in  FIG. 1 , feature engine  203  in  FIG. 2 , from a data source, e.g., data sources  101 ,  102  in  FIG. 1 , data sources  201  in  FIG. 2 . According to an aspect, values of features may be determined and/or sampled at arbitrary points in times, anchor times  422 , over a continuous domain. The feature values may be determined using events  400  associated with the entity having arrival or occurrence times before anchor time  422 , at anchor time  422 , or after anchor time  422 . The feature values may be determined using events  400  having arrival or occurrence times that are some “gap”  423  before or after the anchor time. Gap  423  may be determined by the user, by a feature computation layer, e.g., feature computation layer  106  in  FIG. 1 , or based on a feature configuration. 
     As an illustrative example, events  400  are user activity on a subscription-based service. A user wants to develop a model that predicts a likelihood of users cancelling their subscription based on their activity. To generate training examples, anchor times  422  are set as times at which users cancelled their subscriptions for the service. Feature values are determined using events  400  within a gap  423  of 7-days from anchor events  422 . The feature values may be used to generate the training examples. 
     Anchors time  422  may vary depending on whether the feature to be determined is a target feature or a predictor feature. A target feature is a past or present event. For a target feature, feature values are determined using events  400  after a selected anchor time  422 . A predictor feature is a future event. For a predictor feature, feature values are determined using events  400  prior to selected anchor time  422 . Determining predictor features using events  400  prior to selected anchor time  422  prevents using data to train a model that includes information about the future, e.g., “leakage”. Leakage occurs when information that is only available after the event to be predicted has happened are used as the prediction. 
     As an illustrative example, there is a website that has functionalities that are only available to paid users. A model is developed to determine which users are likely to become paid users. However, if the model is trained using information about paid users using the paid functionalities, leakage will result. As a consequence of the leakage, the model can determine that users using the paid functionalities are likely to be paid users, but cannot predict which users are likely to become paid users. 
     To avoid leakage, an anchor time T is selected at a time at which a user becomes a paid user. By computing feature values using events prior to the anchor time T, leakage is prevented. Computation of feature values, such as a number of times that a user used paid functionalities before they became a paid user, returns a value of 0 because the user cannot use paid functionalities, yet. 
     Also, leakage may happen when events occurring within a relatively small timeframe before a target event are used. An example target event is a user making a purchase on a website. Users who are likely to buy items on the website may be likely to go to a “check-out” page of the website. Users may often visit the “check-out” page shortly before making a purchase. Therefore, a time of the visiting of the “check-out” page is selected as the anchor time, e.g., instead of using a time of the purchase as the anchor time. Next, a gap  423  of one hour is determined. Only events outside an hour from the anchor time are used to compute features. As such, gap  423  prevents events that commonly lead up to the visiting of the “check out” page from being used in the computation of predictor features, thus preventing leakage. 
     Anchor time  422  may be determined in any of several ways. For example, anchor time  422  may be input by a user, such as via API  212  and/or feature studio  215  in  FIG. 2 . As another example, anchor time  422  may be determined based on a maximum number of anchor times  422 . The maximum number of anchor times  422  may be input by a user or determined based on a desired limited number of training examples in a dataset. As another example, anchor times  422  may be determined based on a minimum time interval between anchor times  422  for an entity or input by a user. Anchor times  422  may be defined relative to the occurrence time of events  400  associated with an entity. To illustrate, if events  400  in a dataset are patient LDL cholesterol levels, anchor times  422  may be defined as two months prior to events comprising cholesterol levels over a threshold level, such as 160 md/dL. As another example, the user may define anchor times  422  as conditioned on properties of events  400  or feature values. To illustrate, if events  400  in a dataset are purchases, an anchor time  422  can be conditioned on a cost of a purchase being above a threshold amount, such as $2,000. 
     Additionally, anchor times  422  may be randomly selected. The likelihood of selecting an anchor time  422  over a particular time interval may depend on feature values over the interval. Anchor times  422  may be selected to yield desired statistical properties in the resulting feature values. For example, anchor times  422  corresponding to the occurrence of an event  400  may be balanced with anchor times  422  corresponding to non-occurrence of the event  400 . 
     As an illustrative example, a model is developed to predict whether customers will sign-up for a service. If all of the training data includes anchor times  422  with a target feature value indicating that a customer signed-up for the service, the model may predict that everyone signs-up, while still being accurate based on the training data. Instead, customers and anchor times are selected such that 50% of the examples include a customer signing up and 50% of the examples include a customer not signing up. The examples of a customer not signing up are data from customers who have never signed up. The examples of a customer signing up are data from customers who have signed up and an anchor time is a time being before their signing up. A rule is created that each customer may only be used for training once. 
       FIG. 5  shows example events  500  for an entity over time. Anchor times  530  are determined based on whether the features to be computed are target features  535  or predictor features  536 . Also, time gaps from anchor times  530  are determined based on whether the features to be computed are target features  535  or predictor features  536 . If the features are target features  535 , gap times prior to anchor times  530  are determined. If the features are predictor features  536 , gap times after anchor times  530  are determined. 
     Based on selected anchor times, a set of feature values  537  is computed for an associated entity or subset of all possible entities. Feature values  537  may be exported to generate training examples and to train models. A final transformation  538 , such as a log transform, a statistical transformation, and/or a Box-Cox transformation is performed on feature values  537 . For example, maximum  539 , minimum  540 , and mean  541  values of a feature  537  are used to scale feature values  537  to a fixed range. 
     In an embodiment, information computed to apply final transformation  538  is stored, such as by feature engine  103  in  FIG. 1  or feature engine  203  in  FIG. 2 . A user may retrieve the stored information to apply the same transformation  538  in the future, such as when making predictions with a trained model. As a result, the system ensures that the values used for training are computed and transformed in the same way as values that are used for application. 
     In an embodiment, feature values  542  with the final transformations applied are stored, such as by feature engine  103  in  FIG. 1  or feature engine  203  in  FIG. 2 , to a feature store, such as feature store  107  in  FIG. 1 . As a result, computed features  542  are readily available for quick model application. A user who wants to use a model trained on a particular exported dataset may efficiently retrieve stored pre-computed values  542 . 
       FIG. 6  shows an example feature engineering method  600 . At step  610 , data is received, e.g., ingested, from a data source, e.g., data stream  102  and/or historical data source  101  in  FIG. 1  and/or data sources  201  in  FIG. 2 , by an event ingestion module of a feature engineering system, e.g., event ingestion module  104  in  FIG. 1  and/or feature engine  203  in  FIG. 2 . The ingested data indicates a plurality of events, e.g., live events, historical events, historical events republished to a stream, etc. The events are associated with one or more entities, e.g., users, products, etc. The data is filtered and/or denormalized. 
     At step  620 , an event dataset is generated. The event dataset includes groups of the events associated with entities. The event dataset is stored to a related event store, e.g., related event store  105  in  FIG. 1 . 
     In an embodiment, the event dataset includes a plurality of timestamps associated with the events. The timestamps each include a time component. The time component may be a time that the event occurred or a time that the data was ingested and/or received, such as by the feature engineering system. The timestamps each include a unique machine identifier. The unique machine identifier is an identifier associated with a machine that sent the data, on which the event occurred, and/or that ingested/received the data. Each of the timestamps includes a sequence number. The sequence number may be associated with an order of packet in which the data was received or an order in which the events occurred. 
     At step  630 , an indication of one or more features is received. The one or more features are associated with the plurality of entities. An indication of the one or more features is received via a user interface, e.g., feature studio  215  in  FIG. 2A  and/or feature studio UI  216  in  FIG. 2B . A configuration is received, such as via the user interface. The configuration is a formula for computing the one or more features. 
     A value of a feature is determined for an entity using event data associated with the entity. The event data is retrieved from the related event store. The value is determined by using the configuration. 
     In an embodiment, the value of the feature is determined using events before or after an anchor time and/or the timestamps. The anchor time may be determined in any of a variety of ways. The anchor time may be indicated by a user, such as via the user interface. The anchor time may be determined by the feature engine. The anchor time may be randomly determined. The anchor time may be determined based on whether the features are predictor features or target features. The anchor time may be determined based on receiving an indication of a number of feature vectors associated with an entity, a number of anchor times per entity, a minimum time interval between anchor times associated with the entity, a time in relation to an occurrence time of an event, or a condition associated with a property of an event or a feature value. The anchor time may be determined to ensure a statistical property associated with the values of the one or more features. 
     At step  640 , a feature vector dataset, e.g., feature vector  211  and/or training examples  202  in  FIG. 2 , is generated. The feature vector dataset includes a plurality of feature vectors associated with the plurality of entities. The plurality of feature vectors may include lists of values of the one or more features. The feature values and/or the feature vectors are stored to a feature store, e.g., feature store  107  in  FIG. 1 . The feature values and/or the feature vectors may be exported. According to an aspect, a transformation, e.g., transformation  538  in  FIG. 5 , is applied to one or more feature values and/or feature vectors. 
     In an embodiment, the vector dataset includes a plurality of timestamps associated with the feature vectors. Each of the timestamps indicates a time that a feature vector was generated. Each of the timestamps includes a time that a feature value was computed. 
     At step  650 , an indication of at least one entity of the plurality of entities is received. The indication of the at least one entity may be received via the user interface. The indication of the at least one entity may be a query. 
     At step  660 , at least one feature vector, e.g., feature vector  211  in  FIG. 2 , from the feature vector dataset is output. The feature vector is associated with the indicated entity. The feature vector is retrieved from the feature store. 
     According to an aspect, additional event data is received. The additional event data is stored to the related event store. Based on the timestamps of the events, the events stored to the related event store is replaced or updated. The additional event data is used to update feature values and/or update feature vectors. Based on the timestamps associated with the feature vectors, it may be determined that one or more of the feature values should be computed at new timestamps. Based on the timestamps associated with the feature vectors, it may be determined that some features and not others should be used to compute updated feature values. 
     According to another aspect, a new configuration is received. The new configuration defines how to compute feature values. The new configuration is used instead of a previously received and/or used configuration. Based on the timestamps associated with the feature vectors, it is determined that new feature values need to be computed for at least a portion of the feature values. For example, new feature values must be computed for feature values having timestamps earlier than the time that the new configuration was received. Based on the timestamps, it may be determined that new feature values do not need to be computed for a portion of the feature values. For example, new feature values are not needed for feature values having timestamps later than the time that the new configuration was received. 
     The new configuration may ask for features that have not yet been computed. These new features are computed using the events in the related event store. The new features may be computed using events in the related event store having timestamps earlier than the time that the new configuration was received. 
       FIG. 9A  shows a page  900  of an example user interface of a machine learning feature engineering studio (“feature studio”), e.g., feature studio  215  in  FIG. 2A  and/or feature studio UI  216  in  FIG. 2B . The feature studio operates in conjunction with system  200  and, in particular, feature engine  203 . The feature studio allows a user to automatically perform machine learning feature engineering using any of the methods and systems disclosed herein. The feature studio may be software, a webpage, or an application. Whereas traditional feature engineering requires programming by a user, such as in a Jupyter notebook, the feature studio allows users to define and select features to be used to build a model intuitively within the feature studio. Computations are performed automatically by the feature engine described above. 
     The user interface includes a starting page  900  for initiation of a project. A project is a specific related set of features that a data scientist creates to organize features related to a data model. Within the project the data scientist build or stores a machine learning model using grouped or committed, e.g., selected, features. The feature studio allows data scientists working concurrently on the project to define and select the features. The features are based on entities, which the users may define in the feature studio. The features are calculated over time ranges selected by the users. The features are updated iteratively by the users. 
     The project may have a target, such as a feature or event, selected by the users. The target is a dependent or predictive variable. The target may be defined by predictive features or events. The target can be calculated at a current time, a past time, or an event time. The user interface may be configured to manage time/data leakage in calculating the target at a selected time. 
     The user interface includes a plurality of fields  901 ,  902 ,  903  for a user to define attributes of the project. According to an aspect, the user interface includes a “Select your entity” field  901 . In the entity field  901 , the user selects a type of an entity of the project. The entity is a long-lived, uniquely identifiable thing. The entity may represent a business object relevant to a project of the data scientist. Entities may participate in multiple events. Example entities include a user, a group of users, a product, or a group of products. The entity may be identified by name, number, or another identifier. For example,  FIG. 9A  shows the user having selected “Content” as the entity of the project because this project is about content, such as movies or other video programming that a user has watched or interacted with. As shown in  FIG. 9B , the entity field  901  may be a drop-down menu with pre-defined entity values. 
     According to an aspect, the user interface includes a “Name your project here” field  902 . In the project name field  902 , the user selects a name of the project. For example,  FIG. 9A  shows the user having entered in the name “Feature Content—Tiger King” for the project. 
     According to an aspect, the user interface includes an “Add your project comments here” field  903 . In the comments field  903 , the user enters notes about the project. For example,  FIG. 9A  shows the user having entered the comments “Prediction for the click view rate—Tiger King.” The user may add other users to work on the project to give the other users access to the project via the user interface. 
     The user interface includes a “Continue” button  904 . The user may select the Continue button  904  after entering the attributes in the fields  901 ,  902 ,  903 . Selection of the Continue button  904  causes another page of the user interface to be displayed. 
       FIG. 9C  further illustrates use of the starting page  900  to initiate a project. For example, the data scientist may want to build a model to predict the turnover rate of new users of a video on demand service from the time of sign-up. In this example, the data scientist selects “User-group” as the project entity  901 . The User-group entity may be selected to prepare a model based on user group interactions. The user selects “User Turnover Rate” as the project name  902 . The user adds comments  903 : “Prediction for the turnover rate of a new user from sign up.” 
       FIG. 10  shows the next page  1000  of the user interface. The page  1000  allows a user to confirm the attributes of the project that were selected in the page  900  shown in  FIGS. 9A-9C . The page  1000  displays the selected attributes. The page  1000  includes an “Edit” button  1005 . Selection of the Edit button  1005  allows the user to re-select the attributes of the project. The page includes a “Continue” button  1006 . Selection of the Continue button  1006  causes another page of the user interface to be displayed. Selection of the Continue button  1006  causes the selected attributes of the project to be stored. 
       FIG. 11A  illustrates how the system gives a data scientist the ability to create new features, import features from the feature store, or both in creating a new project or editing an existing project. For example, by selecting Feature Store  1103 , the data scientist is given the option to import preexisting features as is explained in more detail with respect to  FIGS. 29A-F . Another option is for the data scientist to select Feature Studio  1107  to create new features as explained more fully hereinafter. 
       FIG. 11B  shows the next page  1100  of the user interface. The page  1100  includes a “New Feature” button  1105 . Selection of the New Feature button  1105  causes the system to bring up subsequent UI&#39;s to begin defining the feature based on the ingested data. 
       FIG. 12A  shows the next page  1200  of the new feature user interface. The page  1200  includes a window  1206  that allows the user to define a feature, e.g., Feature_01. The window  1206  includes a field  1208  for the user to enter a formula, e.g., function, expression, for the feature. For example, the user may input the formula “MEAN(customer_id if !isNull(PageView.contentID)” in the field  1208 . This is an example of functions performed on identified ingested data structures to define a feature, here Feature_01. The specific syntax used in the function is predefined by the system. 
     The window  1206  includes a “Run” button  1207 . Selection of the Run button  1207  causes the formula to be applied to the data and the user interface to display a visualization of the defined feature. The user may define a plurality of features via the user interface. Each of the features will be displayed, e.g., as a “card,” on a page. The defined features may be stored in a feature store, e.g., feature store  107  in  FIG. 1 , and reused in other projects as described more fully below with respect to  FIGS. 29A-F .  FIG. 12B  shows the window  1206  with a formula entered in the formula field  1208 . 
     More aspects of the feature studio tools provided to the data scientist are illustrated with respect to  FIGS. 13 and 14A -G.  FIG. 13  shows the window  1206  with a graph  1308  and a table  1309 . Based on selection of the Run button  1207  in  FIG. 12 , the graph  1308  will show a visualization of the defined feature. According to an aspect, the table  1309  shows values of the feature for each selected entity. The user may also select statistics, e.g., “STATS,”  1310  of the feature, which will populate the table  1309 . The statistics  1310  help the user understand and engineer features. The statistics  1310  include mean, standard deviation, e.g., std, minimum, e.g., min, maximum, e.g., max, and percentages of the values, e.g., 25%, 50%, 75%, etc. 
     The user may select a transformation  1311  to be applied to the visualization and values of the feature from a menu of transformations  1311 . The transformations  1311  include clean, e.g., filter, scale, bin, and order. Based on selection of a transformation, a visualization and values of the feature are automatically updated. 
       FIG. 14A  shows the window  1206  with the graph  1308  showing a visualization  1411  along with the populated stats table  1309 .  FIG. 14A  shows the visualization  1411  as a histogram. However, the visualization  1411  may be a scatter plot, a line graph, heat map, pair plot, or another type of graphic representation of the values of the feature. The user interface may be configured to display the visualization  1411  as a type of graphic representation based on a data type. The data scientist may select to have the visualization  1411  displayed as a different type of graphic representation. The user may zoom-in or zoom-out of the visualization  1411 . The user can customize the visualization  1411 , such as by changing a color or axis of the visualization. 
     In an embodiment, a user may configure and/or interact with the visualization  1411  in various ways without affecting the computed values. Configuring the visualization  1411  can include changing the number of bins shown on a histogram or changing whether the X-axis uses percentage or absolute values, as examples. Interacting with the visualization  1411  can include hovering over parts for more information. According to an aspect, the user may visualize the values of features that would be exported with a specific configuration for selecting entities and times to export. The user may visualize the latest values of each feature for all entities. 
       FIG. 14B  shows the windows  1206  with options to clean the data included in the feature to, e.g., remove outliers, including options for the “if-then” statement. The user selects a condition for application of the modification, such as equal to, greater than, less than, greater than or equal to or less than or equal to. The condition may be selectable from a drop-down menu  1412  of conditions. The user selects a value for application of the modification. The value may be input in a field or may be selectable from a drop-down menu of values. In this example, the data scientist may put in a value such as  30 , 000 . The data scientist selects a result of satisfaction of the “if-then” statement, such as to cap, drop, or set the values of the feature. The result may be selectable from a drop-down menu  1413  of results.  FIG. 14C  shows the window  1206  with the fields of the clean transformation filled. 
     According to an aspect, the user can make other changes to the feature. As shown in  FIG. 14D , selection of the order function shows changes  1414  that were made to the feature and the order in which the changes were made. Changes  1414  include creating the feature, e.g., inputting the formula for the feature, and any transformations applied to the feature. The order transformation allows the user to remove, e.g., undo, any of the changes applied to the feature. 
       FIG. 14E  shows selection of the scale function. For the scale function, the user selects a scale type. The scale type may be selected from a drop-down menu  1415 . The scale types include a log scale, a variance scale, and a minimum/maximum scale. Selection of the log scale allows the user to select a type of the log scale. Example log scales include log base, natural log (e.g., Ln), log (1+a number) (e.g, Log 1p), or base 10 logarithm (e.g., Log 10). The user may select a minimum/maximum scale, a minimum input, a maximum input, a minimum range, and a maximum range for the visualization and values of the feature. 
       FIG. 14F  shows selectable options  1416  of the window  1206 . The selectable options  1416  may be displayed in a menu that may be expanded based on the user selecting an icon. The selectable options  1416  allow the user to modify the features. The selectable options  1416  include a “Set as target” option, a Maximize option, a Rename option, a Duplicate option, and a Delete option. Based on the user selecting the Set as target option, an icon  1417  in the window  1216  may change color to indicate that the feature has been set as a target for the model of the project. 
       FIG. 14G  shows windows  1206  with visualizations of features “Account_Activation” and “Account_Reactivate.” This demonstrates that the name of Feature_01 has been changed to “Account_Activation.” 
       FIG. 15A  shows a list  1512  of data including events. The data may be received or ingested using any of the systems or methods disclosed herein. For example, the data may be received from an event store, e.g., related event store  105  in  FIG. 1 . The events in the list  1512  may be used in defining a feature. As shown in  FIG. 15A  events may be complex data structures. For example, the event “VideoWatchEvent” contains time and customer_id. 
     As shown in  FIG. 15B ,a list of data  1312  may be shown alongside the window  1206 . The list  1312  may show descriptions of data which includes events, such as based on the user putting a cursor over a data item of an event in the list  1312 . The list  1312  may be similar to the list  1512  in  FIG. 15 . As shown in  FIG. 15C , samples of the events in the list  1312  may be displayed. Sample values (from the incoming data) may be displayed when the cursor is over the incoming data schema. If you move the cursor over the event field, the system displays samples from the actual data. For example, if you put a cursor on “customer_id” field samples can be 34, 565, 12331, 4324, 45432 representing various samples that are actually present in the event data. 
     Selection of an event in the list  1512  causes display of a visualization or values of the defined feature applied to the event. Selection of an event in the list  1512  shows a list of entities for which the feature is defined. For example, selection of the “Video Watch” event shows the underlying data “time(long)” and “customer_id (long).” The underlying data “customer_id (long)” may be copied from the list  1512  and pasted in the formula field in  FIG. 12 , as in the formula shown in the formula field in  FIG. 14A-G . 
       FIG. 16A  shows more details of an example data schema  1613  for storing the events, such as on a backend. The events may be imported into the system according to data schema  1613  from historical and real-time (e.g., streamed) sources, e.g., data stream  102  and/or historical data source  101  in  FIG. 1 . The data schema  1613  includes a plurality of attributes  1614  of the events, such as a name, a type, a documentation type (e.g., doc), and one or more fields. The data schema may be viewed by the user via the user interface, allowing the user to explore data schemas by visualizing raw events. 
       FIG. 16B  shows a process  1600  for relating entities, such as using the data schema  1613 . The user may define relationships between entities via the user interface. Defining a related entity can involve specifying the type of related entity, how the related instances are determined, and/or a name for the relationship. For example, a project focused on flight information may have relationships to a departure airport and an arrival airport. 
     The user may access the data schema  1613  via the feature studio (step  1601 ). From the data schema  1613 , the user may create a new relation (step  1602 ). Creating the new relation (step  1602 ) can include selecting a related entity type (step  1603 ), defining the relation (step  1604 ), and/or naming the relation (step  1605 ). After the relation is created, the relation may be represented in the data schema  1613  (step  1606 ). 
     If the project has related entities, the feature studio may allow defining features computed over events of the related entity. Defining such features may behave similarly to defining features for the primary entity of the project, with an additional indication that the events being aggregated are those of the related entity, rather than the primary entity. The feature studio and the data engine may compute the values of features computed from related entities at the same time as the features computed for the primary entity. 
     According to an aspect, an overview of the data schema  1613  of different event types available within the project may be displayed in the user interface. The data schema  1613  can include the specific fields available and the associated types of data in each field. The data schema  1613  can include how the events are related to the primary and related entities. The data schema  1613  can also allow the user to create formulas using the given data. Creating formulas can include copying and pasting the expression fragment for data from the data schema  1613 . 
       FIG. 17  shows an “All Projects” page  1700  of the user interface. The All Projects page  1700  lists all of the projects available to data scientist by attributes including name  1718 , entity  1719 , owner  1720 , and modified date  1721 . The data scientist may sort the projects in the list by any of the attributes. The All Projects page  1700  includes a search bar  1722  in which the data scientist may search within their projects, such as by search term or attribute. 
       FIG. 18A  shows feature cards  1821   a  displayed in a page  1800  of the user interface. The feature cards  1821   a  include thumbnail views of the features defined by the data scientist, including thumbnail views of the visualizations and names of the features. In  FIG. 18A , the thumbnail views show the visualizations as histograms. However, the thumbnail views may show the visualizations as bar charts, scatter plots, line graphs, heat maps, pair plots, or another type of graphic representation. According to an aspect, the user interface shows a history of changes performed to the feature, such as transformations, cleaning, name changes, formula changes, and target changes. The user interface shows an order of the changes. According to another aspect, the feature cards  1821   a  include selectable options  1822 . The selectable options  1822  include compare, duplicate, and delete. A selection of the delete option causes the feature to be removed from the user interface and deleted from storage. A selection of the duplicate option causes the feature to be copied. The duplicate feature may be shown in the user interface. A selection of the compare option allows the data scientist to select one or more features to compare.  FIG. 18B  shows feature cards  1821   b  with the underlying formulas (e.g., functions)  1823  that are applied to the data to create the features displayed.  FIG. 18B  shows feature cards  1821   b  corresponding to the features selected in  FIGS. 9C, 29C, and 29D . 
       FIG. 18B  shows the page  1800  with two of the feature being selected for comparison. The selected features may have a border indicating that they have been selected. After selecting the features, the data scientist may run the comparison by selecting a “Compare” button  1814 . 
     In an embodiment, as shown in  FIG. 19A , values of two or more features for a specific entity can be compared. The process  1900  includes a user using the feature studio  215  to select features to compare (step  1901 ). After selecting the features to compare, the user selects the type of comparison, such as with a scatter plot or a histogram (step  1902 ). A visualization of the selected comparison of the selected features is displayed in the user interface, as shown in  FIGS. 19B-19C . The user can interact with the comparison (step  1903 ), such as by changing the type of visualization of the comparison, changing the scale of the comparison, applying dimensionality reduction, or saving the comparison. Alternatively, the user can dismiss the comparison (step  1904 ). After dismissing the comparison, the user is returned to the original user interface display (step  1905 ). 
       FIG. 19B  shows a comparison  1906  of two features, e.g., Feature_01 and Feature_02, that were selected via the user interface described with respect to  FIGS. 18A-18B . The comparison  1906  includes visualizations of the features shown on a same graph. The data scientist may select to view the visualizations as bar graphs, scatter plots, heat maps, pair plots, or other graphical representations.  FIG. 19B  shows the comparison in bar chart form.  FIG. 19C  shows the comparison  1906  as a scatter plot. The user interface may apply dimensionality reduction, such as principal component analysis (PCA), to the features being compared. 
       FIG. 20A  shows a versioning process  2000  of the feature studio. Features in a project may be created and/or edited (step  2001 ). After the features are created and/or edited in step  2001 , the features may be committed (step  2002 ) and/or visualized/exported (step  2003 ). Committing the features includes accepting the created features or the changes to the features. Visualizing the features includes reviewing the features. Exporting the features includes publishing the features, such as for use in other projects. The features may be visualized/exported (step  2003 ) after the features are committed (step  2002 ). 
     Committed features may be reviewed for approval (step  2004 ). For example, a user with approval permissions may review the committed features. Committed features may be recorded in a history log associated with the project (step  2005 ). According to an aspect, the changes and commits are recorded in the history log along with times that they were made The committed features may be recorded (step  2005 ) in the history log once they are approved in step  2004 . The versioning process  2000  is useful when multiple users are collaborating on a project, so that a user can see what changes other users made. 
       FIG. 20B  shows the versioning process as shown in the user interface. According to an embodiment, changes to defined features are committed to a project by the data scientist. Multiple data scientists working on a project may have access to the project via the user interface and may have permissions to commit the changes to the features to the project. The changes may be shown in a log  2021 . As shown in  FIG. 20C , the data scientist may select a feature in the log  2021  to view a log  2023  of changes to the selected feature. 
     As shown in  FIG. 20B , the changes shown in the log  2021  may be committed by a selecting a “Commit now” button  2022 . One or more of the data scientists may be designated approvers and may be assigned permissions for approving the commitment of the changes to the features to the project. Committing features to the project may cause the changed features to be persisted in the feature store. 
     As shown in  FIG. 21 , according to an aspect, the user interface displays a history  2121  for a project. The history  2121  includes an indication of features committed for a project. The history  2121  includes names of data scientists that committed the features. The history  2121  includes times that the features were committed. The history  2121  includes comments from the data scientists that committed the features. 
     As shown in  FIG. 22 , according to embodiments, the user interface includes a “Select Approver” feature  2223 . The Select Approver feature  2223  allows the data scientist to select a second data scientist as an approver of a project. The Select Approver feature  2223  includes an approver field  2224 . The data scientist may enter the name of the data scientist to designate as the approver in the approver field  2224 . The approver field  2224  may include a drop-down menu that is pre-filled with the names of the data scientists that are working on the project. The data scientist designated as the approver is granted approving permissions. The Select Approver feature  2223  may include a comments field  2225 . The data scientist may enter notes in the comments field  2225 , such as for the approving data scientist or for other data scientists that have access to the project. The Select Approver feature  2223  may include a Cancel button  2226  that allows the data scientist to cancel the selection of the approver. The Select Approver feature  2223  may include a Finish button  2227  that allows the data scientist to save the selection of the approver. 
     As shown in  FIG. 23 , the project may be committed. The data scientist may commit the project after committing features or changes to the features to the project. Based on the project being committed, a notification  2310  is displayed. Based on the project being committed, the project may be run, such as by selecting a “Run” button  2221 . 
     As shown in  FIG. 24A , the data scientist may select to export the project from the feature studio  215 , such as to a production environment. Exporting the project includes producing a dataset containing one or more computed features. Exporting the project includes generating API endpoints to deliver feature vectors to the production environment. The API endpoints deliver a set of feature vectors for a specific entity. The features may be exported as a bulk dataset, such as of all features or of feature “examples” or as a single feature vector for a specific entity. Bulk export of features can be used to export data for training of validating a model. Exporting may produce the dataset in a variety of formats, such as CSV or Apache Parquet. The resulting dataset may be downloaded by the user in a location of their choice. The exported features may be committed or uncommitted features. The feature studio  215  may allow the user to configure how examples are selected for export. For example, the user may configure how to choose the entities and times exported. An entity may be exported at multiple times. 
     As shown in  FIG. 24B , an exporting window  2427  is displayed via the user interface. A “Confirm Your Info” page  2400  is displayed via the window  2427 . The Confirm Your Info page  2400  includes fields for exporting the model of the project. The fields include a “Select your entity” field  2428 , a “Name your project here” field  2429 , an “Add your project comments here” field  2430 , and a “Number of examples to train your model” field  2431 . The Confirm Your Info page  2400  includes a selectable Save button  2432  that allows the data scientist to save the data input in the fields for exporting the project. This feature could be used to export the model for production. 
     As shown in  FIG. 25 , selection of the Save button  2432  causes the window  2427  to show a “Select Time Window” page  2500 . The Select Time Window page  2500  includes fields for the data scientist to select the time interval over which features are computed. The fields includes a “Select the time range here” fields  2533 , a “Start Time” field  2534 , and an “End Time” field  2535 . The fields may include drop-down menus. The Start Time  2534  field and the End Time field  2535  may include calendars with selectable dates. The Select Time Window page includes a selectable Continue button  2536  that the data scientist may select to confirm the selected time window. The features for export are calculated based on the selected times. 
     As shown in  FIG. 26 , selection of the Continue button  2536  causes the window  2427  to show a “select target” page  2600 . The select target page  2600  includes fields for the data scientist to select a target feature, the feature that the data scientist is trying to predict with the model. The fields include a “Select your target from the list of features” field  2637  that allows the data scientist to select one of the features defined by the data scientist as the target feature. The fields include a “Select what format you want to export as” field  2638  and a “Select your export destination” field  2639 . The export format field  2638  may include formats such as CSV, Numpy, Petastorm, RecordIO, and TFRecords. The export destination field  2639  may include destinations such as a desktop, a local folder, and cloud storage. The fields may include drop-down menus. The select target page  2600  may include a selectable Continue button  2640 . 
     As shown in  FIG. 27 , selection of the Continue button  2640  causes the window  2427  to show a summary page  2700 . The summary page  2700  shows the data input in the fields by the data scientist in the confirm information  2400 , select time window  2500 , and select target pages  2600 . The data scientist may review the data input in the fields. The summary page  2700  includes a selectable Back button  2740  that allows the data scientist to go back to one of the previous pages and change the data input in the fields. The summary page  2700  includes a selectable Export button  2741  that allows the data scientist to export the project. As shown in  FIG. 28 , a notification  2842  may be displayed indicating that the project has been exported. 
     As shown in  FIG. 29A , according to an embodiment, the features defined by the data scientist are displayed in a feature store page  2900 . In the feature store page  2900 , the data scientist may browse the features in a list. The feature store page  2900  may allow the data scientist to have a better understanding of the features that are available for importation into a new project. The feature store page  2900  includes a list of the features by attributes including name, entity, formula, or other metadata. The data scientist may sort the features by any of the attributes. The features may be organized in the feature store page  2900  by common formula. The feature store page  2900  includes a search bar  2943  that allows the data scientist to search the features, such as by search term or attribute. In the feature store page  2900 , the data scientist may copy features to projects and archive features. The feature store page  2900  may support roles and permissions, such as restricting read and write capabilities. 
     As shown in  FIG. 29B , the feature store page  2900  may display a thumbnail view of one of the features. For example, the feature store page  2900  may display the thumbnail view in response to the data scientist moving a cursor over the feature. The thumbnail view may show the visualization as histograms. However, the thumbnail view may show the visualization as a bar chart, a scatter plot, a line graph, a heat map, a pair plot, or another type of graphic representation. 
     As shown in  FIG. 29C , in the feature store page  2900 , the data scientist can select a feature, such as by checking a box  2944  in the list of features. For example,  FIG. 29D  shows the data scientist selecting a feature called “Account_Activation.” Selection of the feature causes importation of the feature into a project. The selected features may be imported to the project based on the data scientist selecting an “Add to new project” button  2945 . Selection of the button  2945  may cause the new project page  900  in  FIGS. 9A-9C  to be displayed for the data scientist to define attributes of the project and to confirm selection of the features. 
     According to an aspect, the system automatically selects features that are related to a feature selected by the data scientist. As shown in  FIG. 29D , the data scientist has selected the “Account_Activation” feature. In response, the system automatically selects features that are related to the “Account_Activation” feature: the “Account_Reactivation” feature, the “Search_trigger” feature, and the “Time_on_page” feature. 
       FIG. 30  shows an example process  3000  for a feature studio. At step  3010 , an indication to start a project is received via a user interface of the feature studio. The indication to start the project may be received from a data scientist in the page  900  shown in  FIGS. 9A-C . The indication to start the project may include attributes of the project, such an entity of the project, a name of the project, and comments. A confirmation of the new project may be received from the data scientist, such as via a selection of the continue button  1006  in the page  1000  in  FIG. 10 . 
     At  3020 , fields for input of a feature is displayed. The fields may be displayed based on receiving an indication to define a new feature from the data scientist, such as via a selection of the New Feature button  1105  in  FIG. 11 . The fields may include the formula field  1207  in  FIG. 12 . At  3030 , inputs indicating the feature are received via the user interface. The inputs may include a formula. The feature is associated with an entity. 
     At  3040 , data in an event store associated with the entity is determined. The event store may include historic data and live data, e.g., data from a data stream. The event store may comprise the related event store  105  in  FIG. 1 . 
     At  3050 , the feature is calculated. The feature may be calculated by a backend of the feature studio. At step  3060 , a visualization of the features is displayed. The visualization may be similar to the visualization  1411  in  FIG. 14 . The visualization may include a bar chart, a scatter plot, a line graph, a histogram, heat map, pair plot, or another type of graphic representation. Values of the feature may be displayed. The visualization may be shown as a feature card along with feature cards of other features of the project, such as in page  1800  of  FIGS. 18A-18B . The data scientist may interact with the features, such as by changing them or comparing them. 
     An indication of a transformation, such as the transformations  1311  in  FIG. 13 , may be received. The feature may be recalculated based on the transformation. An updated visualization of the transformed feature may be displayed. The data scientist may commit the feature to the project. The data scientist may commit other features to the project, as well. 
     At  3070 , an indication to export the project is received via the user interface. The indication to export the project may be received via the window  2127  in  FIG. 21 . The indication may include a confirmation of data associated with the project (e.g., as shown in  FIG. 21 ), a time window of the project (e.g., as shown in  FIG. 22 ), and a target of the project (e.g., as shown in  FIG. 23 ). 
     At  3080 , a feature vector is calculated using the feature. The feature vector is calculated using any features committed to the project. The feature vector may be calculated using a backend of the feature studio. At  3090 , the calculated feature vector is exported to a production environment. 
       FIG. 7  shows an example computing node  700 . Computing node  700  may be a component of feature engineering system  100  in  FIG. 1  and/or feature engineering system  200  in  FIG. 2 . Computing node  700  may include feature engine  103  in  FIG. 1  and/or feature engine  203  in  FIG. 2A  or a component thereof. 
     Computing node  700  may be a general-purpose computing device. Computing node  700  may be a node in a cloud computing environment. Computing node  700  may be an on-premises device, such as a node of a distributed system running in a data scientist&#39;s data center. The components of computing node  700  may include, but are not limited to, one or more processors or processing units  716 , a system memory  728 , and a bus  718  that couples various system components including system memory  728  to processor  716 . 
     The bus  718  in the example of  FIG. 7  represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (‘ISA’) bus, Micro Channel Architecture (‘MCA’) bus, Enhanced ISA (‘EISA’) bus, Video Electronics Standards Association (‘VESA’) local bus, and Peripheral Component Interconnects (‘PCI’) bus. 
     Computing node  700  may include a variety of computer system readable media. Such media may be any available media that is accessible by computing node  700 , and it includes both volatile and non-volatile media, removable and non-removable media. 
     The system memory  728  in  FIG. 7  may include computer system readable media in the form of volatile memory, such as random access memory (‘RAM’)  730  and/or cache memory  732 . Computing node  700  in ay further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, a storage system  734  may be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk, e.g., a “floppy disk,” and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In such instances, each may be connected to bus  718  by one or more data media interfaces. As will be further depicted and described below, memory  728  may include at least one program product having a set, e.g., at least one, of program modules that are configured to carry out the functions of embodiments of the invention. 
     Computing node  700  may include a program/utility  740  having a set (at least one) of program modules  742  that may be stored in memory  728 . Computing node  700  of  FIG. 7  may also include an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules  742  generally carry out the functions and/or methodologies of embodiments of the invention as described herein. 
     Computing node  700  of  FIG. 7  may also communicate with one or more external devices  714  such as a keyboard, a pointing device, a display  724 , and so on that enable a data scientist to interact with computing node  710 . Computing node  700  may also include any, devices, e.g., network card, modem, etc., that enable computing node  700  to communicate with one or more other computing devices. Such communication may occur, for example, via I/O interfaces  722 . Still yet, computing node  700  in ay communicate with one or more networks such as a local area network (‘LAN’), a general wide area network (‘WAN’), and/or a public network, e.g., the Internet, via network adapter  720 . As depicted, network adapter  720  communicates with the other components of computing node  700  via bus  718 . It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computing node  700 . Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, and so on. 
       FIG. 8  shows example components of a cloud computing system  800 . Cloud computing system  800  may include feature engineering system  100  in  FIG. 1 , feature engineering system  200  in  FIG. 2 , feature engine  103  in  FIG. 1 , and/or feature engine  203  in  FIG. 2 . Cloud computing system  800  may be used to perform any of the disclosed methods, such as method  600  in  FIG. 6 . Cloud-based computing generally refers to networked computer architectures where application execution, service provision, and data storage may be divided, to some extent, between clients and cloud computing devices. The “cloud” may refer to a service or a group of services accessible over a network, e.g., the Internet, by clients, server devices, and cloud computing systems, for example. 
     In one example, multiple computing devices connected to the cloud may access and use a common pool of computing power, services, applications, storage, and files. Thus, cloud computing enables a shared pool of configurable computing resources, e.g., networks, servers, storage, applications, and services, that may be provisioned and released with minimal management effort or interaction by the cloud service provider. 
     As an example, in contrast to a predominately client-based or server-based application, a cloud-based application may store copies of data and/or executable program code in the cloud computing system, while allowing client devices to download at least some of this data and program code as needed for execution at the client devices. In some examples, downloaded data and program code may be tailored to the capabilities of specific client devices, e.g., a personal computer, tablet computer, mobile phone, smartphone, and/or robot, accessing the cloud-based application. Additionally, dividing application execution and storage between client devices and the cloud computing system allows more processing to be performed by the cloud computing system, thereby taking advantage of the cloud computing system&#39;s processing power and capability, for example. 
     Cloud-based computing can also refer to distributed computing architectures where data and program code for cloud-based applications are shared between one or more client devices and/or cloud computing devices on a near real-time basis. Portions of this data and program code may be dynamically delivered, as needed or otherwise, to various clients accessing the cloud-based application. Details of the cloud-based computing architecture may be largely transparent to data scientists of client devices. Thus, a PC user or a robot client device accessing a cloud-based application may not be aware that the PC or robot downloads program logic and/or data from the cloud computing system, or that the PC or robot offloads processing or storage functions to the cloud computing system, for example. 
     In  FIG. 8 , cloud computing system  800  includes one or more cloud services  804 , one or more cloud platforms  806 , cloud infrastructure  808  components, and cloud knowledge bases  810 . Cloud computing system  800  may include more of fewer components, and each of cloud services  804 , cloud platforms  806 , cloud infrastructure components  808 , and cloud knowledge bases  810  may include multiple computing and storage elements as well. Thus, one or more of the described functions of cloud computing system  800  may be divided into additional functional or physical components, or combined into fewer functional or physical components. In some further examples, additional functional and/or physical components may be added to the examples shown in  FIG. 8 . Delivery of cloud computing based services may involve multiple cloud components communicating with each other over application programming interfaces, such as web services and multi-tier architectures, for example. 
     Example cloud computing system  800  shown in  FIG. 8  is a networked computing architecture. Cloud services  804  may represent queues for handling requests from client devices. Cloud platforms  806  may include client-interface frontends for cloud computing system  800 . Cloud platforms  806  may be coupled to cloud services  804  to perform functions for interacting with client devices. Cloud platforms  806  may include applications for accessing cloud computing system  800  via user interfaces, such as a web browser and/or feature studio  215  in  FIG. 2 . Cloud platforms  806  may also include robot interfaces configured to exchange data with robot clients. Cloud infrastructure  808  may include service, billing, and other operational and infrastructure components of cloud computing system  800 . Cloud knowledge bases  810  are configured to store data for use by cloud computing system  800 , and thus, cloud knowledge bases  810  may be accessed by any of cloud services  804 , cloud platforms  806 , and/or cloud infrastructure components  808 . 
     Many different types of client devices may be configured to communicate with components of cloud computing system  800  for the purpose of accessing data and executing applications provided by cloud computing system  800 . For example, a computer  812 , a mobile device  814 , a host  816 , and a robot client  818  are shown as examples of the types of client devices that may be configured to communicate with cloud computing system  800 . Of course, more or fewer client devices may communicate with cloud computing system  800 . In addition, other types of client devices may also be configured to communicate with cloud computing system  800  as well. 
     Computer  812  shown in  FIG. 8  may be any type of computing device, e.g., PC, laptop computer, tablet computer, etc., and mobile device  814  may be any type of mobile computing device, e.g., laptop, smartphone, mobile telephone, cellular telephone, tablet computer, etc., configured to transmit and/or receive data to and/or from cloud computing system  800 . Similarly, host  816  may be any type of computing device with a transmitter/receiver including a laptop computer, a mobile telephone, a smartphone, a tablet computer etc., which is configured to transmit/receive data to/from cloud computing system  800 . 
     Any of the client devices used with cloud computing system  800  may include additional components. For example, the client devices one or more sensors, such as a digital camera or other type of image sensor. Other sensors may further include a gyroscope, accelerometer, Global Positioning System (GPS) receivers, infrared sensors, sonar, optical sensors, biosensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chip sensors, wireless sensors, and/or compasses, among others, for example. 
     Any of the client devices may also include a user interface (UI) configured to allow a data scientist to interact with the client device. The UI may be various buttons and/or a touchscreen interface configured to receive commands from a human or provide output information to a human. The UI may be a microphone configured to receive voice commands from a human. 
     In  FIG. 8 , communication links between client devices and cloud  800  may include wired connections, such as a serial or parallel bus, Ethernet, optical connections, or other type of wired connection. Communication links may also be wireless links, such as Bluetooth, IEEE 802.11 (IEEE 802.11 may refer to IEEE 802.11-2007, IEEE 802.11n-2009, or any other IEEE 802.11 revision), CDMA, 3G, GSM, WiMAX, or other wireless based data communication links. 
     In other examples, the client devices may be configured to communicate with cloud computing system  800  via wireless access points. Access points may take various forms. For example, an access point may take the form of a wireless access point (WAP) or wireless router. As another example, if a client device connects using a cellular air-interface protocol, such as CDMA, GSM, 3G, or 4G, an access point may be a base station in a cellular network that provides Internet connectivity via the cellular network. 
     As such, the client devices may include a wired or wireless network interface through which the client devices may connect to cloud computing system  800  directly or via access points. As an example, the client devices may be configured to use one or more protocols such as 802.11, 802.16 (WiMAX), LTE, GSM, GPRS, CDMA, EV-DO, and/or HSPDA, among others. Furthermore, the client devices may be configured to use multiple wired and/or wireless protocols, such as “3G” or “4G” data connectivity using a cellular communication protocol, e.g., CDMA, GSM, or WiMAX, as well as for “WiFi” connectivity using 802.11. Other types of communications interfaces and protocols could be used as well.