Abstract:
A system and method enables users to selectively expose and optionally monetize their data resources, for example on a web site. Data assets such as datasets and models can be exposed by the proprietor on a public gallery for use by others. Fees may be charged, for example, per new model, or per prediction using a model. Users may selectively expose public datasets or public models while keeping their raw data private.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims priority to and is a continuation of U.S. patent application Ser. No. 14/025,063, filed Sep. 12, 2013, which is a non-provisional of U.S. Provisional Application No. 61/710,175 filed Oct. 5, 2012, both of which are incorporated herein in their entirety by this reference. 
     
    
     COPYRIGHT NOTICE 
       [0002]    ©2012-2013 BigML, Inc. A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. 37 CFR §1.71(d). 
       TECHNICAL FIELD 
       [0003]    This invention pertains to computer-implemented methods and apparatus for machine learning or modeling digital datasets, and utilizing data models to make predictions about new data in a secure, private or shared system. 
       BACKGROUND OF THE INVENTION 
       [0004]    Machine Learning uses a number of statistical methods and techniques to create predictive models for classification, regression, clustering, manifold learning, density estimation and many other tasks. A machine-learned model summarizes the statistical relationships found in raw data and is capable of generalizing them to make predictions for new data points. Machine-learned models have been and are used for an extraordinarily wide variety of problems in science, engineering, banking, finance, marketing, and many other disciplines. Uses are truly limited only by the availability and quality of datasets. Building a model on a large dataset can take a long time. Further, the time and resources necessary to build a model increases as the required quality or depth of the model increases. In view of these investments, some models are valuable to other users. Datasets themselves also may have value in view of the investment to acquire, check or scrub, and store the dataset. In fact, many interesting datasets are built after laborious processes that merge and clean multiple sources of data. Often datasets are based on proprietary or private data that owners do not want to share in their raw format. 
       SUMMARY OF PREFERRED EMBODIMENTS 
       [0005]    The following is a summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not intended to identify key/critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later. 
         [0006]    In one aspect, a system in accordance with the present disclosure may enable an owner of a dataset to control permissions that would enable another party or second user to use the dataset to make a new model derived from the dataset. The owner or the system may charge the second user a fee to use the dataset. The fee may be shared with the owner. 
         [0007]    In another aspect, a system in accordance with the present disclosure may enable an owner of a model to control permissions that would enable another party or second user to use the model to make a predictions based on a new input dataset provided by the second user. The owner or the system may charge the second user a fee to use the model. The fee may be shared with the owner of the model. The model may be, for example, a decision tree model. 
         [0008]    According to another aspect, a dataset or a model may be displayed or advertised in a public gallery. The display may contain a summary, thumbnail or metadata describing the dataset or the model, as the case may be. 
         [0009]    According to yet another aspect, a model may be published to a public gallery in a black-box form that enables use of the model to make predictions without disclosing its internal operation. In another aspect, a model may be published to a public gallery in a white-box form that enables use of the model to make predictions and also discloses a descriptive and actionable version of the model. The actionable version enables the user to understand operation and if desired modify the model. Different fees may be charged for black-box and white-box models. Fees may be flat rate, per prediction, or based on other criteria. 
         [0010]    Additional aspects and advantages of this invention will be apparent from the following detailed description of preferred embodiments, which proceeds with reference to the accompanying drawings. The invention is not intended to be limited by the drawings. Rather, the drawings merely illustrate examples of some embodiments of some aspects of this disclosure. 
         [0011]    Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing, steps, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, logic block, process, etc., is generally conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared and otherwise manipulated in a computer system. 
         [0012]    In short, the invention is intended to be implemented in software; i.e., in one or more computer programs, routines, functions or the like. Thus it may best be utilized on a machine such as a computer or other device that has at least one processor and access to memory, as further described later. In a preferred embodiment, a system is hosted on a server to provide the features described herein to remote users. The server may comprise one or more processors. The server may be remotely hosted “in the cloud.” Accordingly, in this description, we will sometimes use terms like “component,” “subsystem,” “model server,” “prediction server,” or the like, each of which preferably would be implemented in software. 
         [0013]    It should be born in mind that all of the above and similar terms are to be associated with the appropriate physical quantities they represent and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as ‘processing,’ ‘computing,’ ‘calculating,’ ‘determining,’ ‘displaying’ or the like, refer to the action and processes of a computer system, or any other digital electronic computing device having a processor, that manipulates and transforms data represented as physical (electronic) quantities within the system&#39;s registers and memories into other data similarly represented as physical quantities within the system memories or registers or other such information storage, transmission or display devices. The term processor encompasses multiple processors acting in concert. A system may be located in one or more physical locations, e.g. distributed or housed “in the cloud” (a centralized location where plural processors and related equipment may be housed and remotely accessed). Further, we use the term “user” in a broad sense relative to a processor or other computing platform, program or service. A user may be a natural person or another processor or other computing platform, program or service. For example, an API may be used to enable another machine or processor to utilize a system of the type described herein. 
         [0014]    Note that the invention can take the form of an entirely hardware embodiment, an entirely software/firmware embodiment or an embodiment containing both hardware and software/firmware elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc. Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0015]      FIG. 1  is a simplified process and system hybrid diagram, illustrating one example of an implementation consistent with the present disclosure, in which a first user interacts with a system to create a new data source, a new dataset derived from the data source, a new model of the dataset, and a new prediction that utilizes the model of the dataset. 
           [0016]      FIG. 2  shows the diagram of  FIG. 1  amended to illustrate the first user publishing the dataset to a public gallery. 
           [0017]      FIG. 3  shows the diagram of  FIG. 2  amended to illustrate a second user acquiring the dataset, and the dataset cloned and copied to the second user&#39;s workspace or “dashboard.” 
           [0018]      FIG. 4  shows the diagram of  FIG. 3  amended to illustrate the second user utilizing the acquired clone dataset in the second user&#39;s private dashboard. 
           [0019]      FIG. 5  shows the diagram of  FIG. 1  amended to illustrate User #1 utilizing the system to make a model public in a black-box way. 
           [0020]      FIG. 6  shows the diagram of  FIG. 5  amended to illustrate a second User #2 utilizing the User #1 black-box model to make predictions in the user #2 private dashboard. 
           [0021]      FIG. 7  shows the diagram of  FIG. 1  amended to illustrate the first user publishing a white-box type of model to the public gallery. 
           [0022]      FIG. 8  shows the diagram of  FIG. 7  amended to illustrate the second user purchasing access to the white-box public model, and showing the white-box model cloned and copied to the user #2 private dashboard. 
           [0023]      FIG. 9  shows the diagram of  FIG. 8  amended to illustrate the second user utilizing the cloned model to make a new prediction in the user #2 private dashboard. 
           [0024]      FIG. 10  shows an example of a source summary screen display. 
           [0025]      FIG. 11  shows an example of a dataset summary screen display. 
           [0026]      FIG. 12  shows an example of a graphical model screen display. 
           [0027]      FIG. 13  shows an example of an interactive prediction user interface. 
           [0028]      FIG. 14  shows a second example of a dataset summary screen display. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0029]      FIG. 1  is a simplified process and system hybrid diagram, illustrating one example of an implementation consistent with the present disclosure. In the figure, a user #1 (reference  102 ) has access to her own private workspace or “dashboard”  100 . In a preferred embodiment, a user may interact with the system using a website. Programs, data, etc, in the user #1 dashboard  100  are not visible to other users. Users may be registered with the system, and may have login names, and passwords, or utilize other known security measures. A registered user  102  can create a new data source  110  by uploading local data or using a URL that a source server  112  may use to retrieve the data. The data may be stored in a data store accessible to the user  102  and to the source server  112 . In one example, the user may provide access credentials to enable the source server to access selected data stored at  120 . A data source or source is the raw data that that is used to create a predictive model. A dataset can be a single dataset or a set of datasets. A dataset can be created using a number of transformations over one or more datasets. 
         [0030]    A source is usually a (large) file, and may be in a comma-separated values (CSV) format, although many other formats may be used. Each column in the data file may represent a feature or field. A column, often the last column, usually represents the class or objective field. The file might have a first row named header with a name for each field. 
         [0031]    A registered user  102  may transform her new source data  110  into a dataset  134  as follows. An illustrative system further includes a dataset server  130 . The new data source  110  may provide raw data to the dataset server  130  as indicated by arrow  132 . A dataset may comprise a binary and structured version of a source where each field has been processed and serialized according to its type. A field may be numeric, categorical, a date, time, or text. Other filed types may be created. In an embodiment, for each field, descriptive statistics including histograms that depict the distribution may be computed automatically, depending on the type of the field. In general the new dataset  134  is derived from the data source  110 . Note the new dataset  134  is associated with user  102 &#39;s private dashboard  100  so that the dataset is accessible to user  102  but not available to anyone else at this juncture. 
         [0032]    The user  102  may transform the new dataset  134  into a model. A model is a structured representation of the dataset with predictive power. A model may be created using a subset of fields of the dataset as input fields, and another subset of the dataset fields as objective fields (the ones that the user wants the model to predict). A model can be a single model or an ensemble of models. An ensemble can be created using the same or different datasets or techniques. Details of methods for creating models are known. In the illustration of  FIG. 1 , the dataset  134  is provided to a model server  140  as indicated by arrow  142 . The model server creates a new model  144  based on the dataset and selected parameters. For example, a model may use all the instances in the dataset to build the model, or just a subset as specified by the user through a number of sampling options. 
         [0033]    Datasets and models may be enriched with various types of metadata, for example:
       Descriptive metadata about the origin or purpose of the data like descriptions, tags, categories, or pictures.   Technical metadata. For example, number of rows, number of fields, types of fields, type of algorithm, parameters of the algorithm.   Privacy metadata that defines who is authorized to get access, under what conditions and what restrictions apply.       
 
         [0037]    Note in the present example the new model  144  is associated with user  102 &#39;s private dashboard  100  so that the model is accessible to user  102  but not available to anyone else at this juncture. The model can be used to make predictions on new input data that were not used to build the model. The model uses the new input data to predict values of the objective fields. Some examples are described later. 
         [0038]    The user  102  may make predictions using a prediction server  150 . The model  144  is accessible to the prediction server  150  as indicated by arrow  152 . Further, the user provides new input data  154  to the prediction server for the purpose of predicting objective field values, based on applying the model to that new input data. The results form a new prediction  160 . Note again that the model, the new input data, and the new prediction  160  are all associated exclusively to the user  102  secure dashboard  100 . In an embodiment, each registered user may have a number of sources, datasets, models, and predictions that are kept private in her dashboard. In some embodiments, interactions between a user and the system can be done through a user interface or programmatically through an API. 
         [0039]    A user  102  may make her dataset  134  public. Referring now to  FIG. 2 , it shows a public gallery  200 . The public gallery may be implemented, for example, as a webpage on a website. Examples are shown below. Once a data set is made public, other users can have access to the summary statistics for each individual field in the data set, but cannot have access to the raw data that was used to build the data set. In some embodiments, the system can be exposed to the general public via a public gallery where heterogeneous users can exchange datasets and/or models. In some embodiments, the system may be accessible only to the members or a selected subset of members of the same organization. If the data set is free or once a user has paid the requested fee, the data set can be used within the system by that paying user ( 208 ) to create new models. 
         [0040]    Preferably, unique identifiers are assigned to each resource in the system. For example, unique identifiers may be assigned to each source, dataset, model, ensemble, prediction, and evaluation. By way of illustration, identifiers may look like this:
       source/5202c67b035d072c00006974   dataset/5202ce99035d072bf9002476   model/521e8c01035d0750c6000a2a   ensemble/521e8ce1035d0750c6000a3f   prediction/521e8b05035d0750cd00079d   evaluation/521e8d3e035d0750c6000a46       
 
         [0047]    Any identifiers that are at least unique within the system may be used. In use, for example, a software call to create a new model may include an identifier of the specific dataset to be used to create the new model. Resource identifiers can be used by users to refer to various assets in their private dashboards or in a public gallery. 
         [0048]    Referring again  FIG. 2 , a data set  202  has been made public by its owner, here user  102 , and placed in the public gallery  200 . The dataset may be placed in the public gallery by the dataset server  130 . A second user  208  (or any user) now has access to the public data set  202 . The owner, in this example user  102 , can decide whether other users can have access to the data set for free or need to pay a fee before using the data set. Such fees may be collected, for example in the context of a web site, by way of bank transfers, credit cards, payment services such as PayPal, etc. 
         [0049]    Referring now to  FIG. 3 , this figure illustrates a data set, namely public data set  204 , placed in the public gallery  200  by the data set server  130 . A 2 nd  user  202  pays consideration to access or purchase the data set  204 . In this case, the data set is cloned to create a new private data set to  10 . Private data set to  10  is placed in user  202 &#39;s private dashboard  212 . The private dataset can be used by user  202  to create a new model using the model server  140 .  FIG. 4  illustrates this scenario, in which the user  202  as acquired the private data set to  10 . User  202  then utilizes the model server  140  to create a new model of her private data set to  10 . The new model  220  is stored in the user&#39;s private dashboard  212 . Further, in  FIG. 4 , the 2 nd  user  202  may apply the new model  222  the prediction server  150  in order to create a new prediction  240 . The new prediction may be based on new input data  230 , provided by the user  202  to the prediction server. 
         [0050]    Evaluations provide a convenient way to measure the performance of a predictive model. A user can evaluate a model using a dataset of her own. In some embodiments, the server may provide various performance metrics that provide an estimate of how well a given model predicts an outcome as compared to its performance faced with data similar to the dataset tested. 
         [0051]    In an embodiment, a user also can make a model public that she has built using a system of the type illustrated herein. A model may be uploaded, for example, into a user&#39;s private dashboard from an external source. Or, the user may choose to make public a model that she created using the system, as describe above (new model  144 ). Once a model is made public, other users may have access to a thumbnail picture, for example, that represents the model, and or other meta information about the model. Meta data may include, for example, the number of fields and number of instances of the dataset used to build the model. 
         [0052]    The owner of a model can control whether other users can have access to the internal structure of the model, for example using a selected one of at least two methods: black-box or white-box. If the owner so lacks a black box method, other users will not be able to see how the model works internally, but they will still be able to use the model to make predictions using their own input data. If the owner uses the white box method, other users will have access to a descriptive and actionable version of the model that explains how it works, in addition to the ability to make predictions as conferred by black-box models. For example, an actionable version of a model may be provided in various formats. These may include JSON, PMML, Microsoft Excel, a sequence of if—then rules and various programming languages such as Python, Ruby, Java, etc. 
         [0053]      FIG. 5  illustrates a scenario in which the owner as published a black-box public model  300  into a public gallery  200 . As shown, the model  300  may be placed in the public gallery by the model server  140 . The gallery display may include information about the model such as the number of fields and instances in the dataset. 
         [0054]      FIG. 6  illustrates the use of the black-box public model  300  by a user  208 . In this illustration, the user  208  a pays consideration requested for use of the model, and then uses the black-box model  300  and the prediction server  150  in order to create a new prediction  302 . The new prediction may be based on new input data  304  provided by user  208 . The new prediction  302  is proprietary to the user  208  and thus is placed in the user&#39;s private dashboard  212 . 
         [0055]      FIG. 7  illustrates a case where a white box public model  400  has been published to the public gallery  200 . Referring now to  FIG. 8 , it illustrates the user  208  paying consideration to purchase the white box public model  400 . In that case, the model  400  is cloned and a new model  500  is formed in the user&#39;s private dashboard  212 . 
         [0056]    Turning now to  FIG. 9 , continuing the prior example, the user  208  can now use her new model  500  and create a new prediction by using the prediction server  150  to apply new input data (not shown) to the model  500 . The resulting new prediction  502  is written to the user&#39;s private dashboard  212 . 
         [0057]    Once a user makes a data set or model public, the corresponding data set or model may be exposed in both an individual gallery or dashboard, and a public gallery, such as the public gallery  200 . In the public gallery, data sets and models can be filtered by an interested user, according to variety of variables such as popularity, recency, white box, black-box, price and other attributes including but not limited to names, descriptions, tags, etc. In a preferred embodiment of the public gallery may be implemented on a webpage. 
         [0058]      FIG. 10  shows an example of a source summary screen display. For example, a display of this general type may be used in the context of a public gallery on a web site to describe a data source. In this case, the data relates to the iris flowers. The raw data comprises various instances of input fields (sepal length, sepal width, etc.) and a species objective or output field. Values for some instances may be displayed. The field types may be indicated as well (for example, “123” indicating a numeric type, “ABC” indicating a category type). The design of this display is merely illustrative; the layout is not critical. 
         [0059]      FIG. 11  shows an example of a dataset summary screen display. In this example, the dataset is derived from the source of  FIG. 10 . A display of this type may include counts for each field, and a histogram of values for each field. A display of this type, for example, may be used in the context of a public gallery on a web site to describe a dataset. The design of this display is merely illustrative; the layout is not critical. 
         [0060]      FIG. 12  shows an example of decision tree model visualization. This graphic illustrates a model of the dataset of  FIG. 11 . For example, branch line widths may be used to indicate number of instances of sample data. Each end leaf may correspond to a species of iris; the predictions based on the input variables indicated in the dataset. Other types of visualizations of models may be implemented. 
         [0061]      FIG. 13  shows an example of an interactive prediction user interface. In this illustration, sliders are used for a user to input values of the numeric variables, in order to predict a species of iris using the model of  FIG. 12 . 
         [0062]      FIG. 14  is another example of a dataset summary screen display. In this case the dataset reflects survivors of the famous sinking of the RMS Titanic (1912). The field names include name, age, class/department, fare etc. The second column indicates the type of each field (for example, types of data may include text, numeric, binary, etc.). For each field, subsequent columns may indicate the respective count of items, number missing, and number of errors. These columns are merely illustrative and not critical. On the right, a histogram indicates the corresponding distribution of values for each field. For example, the first one is an age distribution. The next histogram suggests there were seven different classes of passengers. 
         [0063]    A person of ordinary skill in the art will recognize that they may make many changes to the details of the above-described exemplary systems and methods without departing from the underlying principles. Only the following claims, therefore, define the scope of the exemplary systems and methods.