Patent Description:
Machine learning is used in a variety of applications and domains such as healthcare, Internet of Things (IOT), finance, and security. Decades of research have created a huge assortment of algorithms and techniques that can be applied to these applications. Selecting the best algorithm for an application may be difficult and resource intensive. For example, a classification task can be done by several algorithms such as support vector machines (SVMs), random forests, decision trees, artificial neural networks, and more. Each of these algorithms has many variations and configurations and performs differently for different datasets. Choosing the best algorithm is typically a manual task performed by a data scientist or a machine learning expert having years of experience.

Some automatic algorithm selection techniques incur significant computational overhead, such as during product research and development (R&D), which can prolong a time to market. There are hundreds of machine learning algorithms. Training and testing each one to find the best performing might not be feasible. Automatic approaches to selective training typically ultimately use a single regressor/classifier for predicting algorithm performance, which causes different algorithms to interfere with each other in the selection model, thereby lowering accuracy. These approaches also do not consider algorithm hyperparameters, which can significantly affect algorithm performance and behavior.

The article "<NPL>, discloses a technique for building an ensemble model which averages the output of several different models. In order to select models for the final ensemble, the models are initialized with different model parameters, and cross validation is used to find proper values for these parameters and to select the best performing models for the final ensemble. For a k-fold cross validation, k training data sets and k validation data sets are determined. This leads to k training-validation rounds. In every round, several different model classes are trained with a variety of model parameters. Only one model is selected in each round to become a member of the final ensemble, namely the best model with respect to the validation set. Certain data preprocessing operations may be performed in order to select the most discriminating features and reduce the dimensionality of the learning problem.

Embodiments are described herein according to the following outline:.

Techniques are provided herein for optimal selection of machine learning algorithms based on performance predictions by trained algorithm-specific regressors. In an embodiment, a computer derives meta-feature values from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For each trainable algorithm and each regression meta-model that is respectively associated with the algorithm, a respective score is calculated by invoking the meta-model based on at least one of: a) a respective subset of meta-feature values, and/or b) hyperparameter values of a respective subset of hyperparameters of the algorithm. One or more of the algorithms are selected based on the respective scores. Based on the inference dataset, the one or more algorithms may be invoked to obtain a result.

In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles. Techniques are also provided herein for optimal training of regressors and/or ensembles.

An example computer C1 may, in an embodiment, optimally select trainable algorithms based on performance predictions by trained regressors. Computer C1 may be one or more computers such as an embedded computer, a personal computer, a rack server such as a blade, a mainframe, a virtual machine, or any computing device that uses scratch memory during numeric and symbolic processing.

Computer C1 contains or accesses specifications of multiple distinct trainable machine learning algorithms, such as A1-A3, each of which may perform analysis such as classification, regression, clustering, or anomaly detection. For example, algorithm A1 may be a support vector machine (SVM) or an artificial neural network (ANN), and algorithm A2 may be a decision tree or a random forest.

Each of algorithms A1-A3 is trainable and perhaps due for tuning (retraining) or not yet trained. Each of algorithms A1-A3 may or may not be ready (trained) for immediate use on inference dataset D. Inference dataset D may be empirical data, either exhaustive or representative, that any of algorithms A1-A3 may eventually use for training or inference such as data mining.

Training any one algorithm of A1-A3 is computationally very expensive, which may be aggravated by the amount of raw data in inference dataset D. Computational feasibility may require that computer C1 (or another computer) train only one or a small subset of algorithms A1-A3.

Ideally, computer C1 would select (for training and/or inference) a few of algorithms A1-A3 that could produce the best (most accurate, least error) results with inference dataset D. However, because some or all of algorithms A1-A3 may still need training or retraining, accuracy prediction for algorithms A1-A3 may be difficult or impossible.

Accuracy prediction may be further aggravated by the amount of available algorithms such as A1-A3. Machine learning has hundreds of algorithms and is still rapidly growing. Many of these algorithms are readily available in reusable libraries such as TensorFlow and scikit-learn.

Computer C1 creates or obtains meta-models for each of algorithms A1-A3 to quickly and accurately predict the performance of each algorithm. For example, computer C1 may create meta-models MM1-MM3 as performance predictors of algorithm A1.

Each of meta-models MM1-MM3 is itself an instance of trainable regression algorithm, although not the same algorithm for which the meta-models are trained for. For example, meta-models MM1-MM3 may each be a distinct neural network that is already trained to predict the performance of algorithm A1, which may be support vector machine instead of a neural network. Training of meta-models is discussed later herein.

In operation, computer C1 obtains inference dataset D and should use meta-models, such as MM1-MM3, to select a more or less optimal subset of algorithms A1-A3 to eventually be tuned with inference dataset D. When predicting performance of an algorithm, a meta-model should consider features of the algorithm and features of inference dataset D.

Features of an algorithm are referred to as hyperparameters. For example, algorithm A1 has hyperparameters HP1-HP4.

If algorithm A1 is a support vector machine, then hyperparameters typically include C and gamma. If algorithm A1 is a neural network, then hyperparameters may include features such as a count of layers and/or a count of neurons per layer.

Each of algorithms A1-A3 may have many configuration alternatives based on hyperparameter values. For example, each distinct configuration of algorithm A1 is based on a distinct set of values for hyperparameters HP1-HP4.

Each of hyperparameters HP1-HP4 may logically be a separate axis in a multidimensional hyperspace. Each distinct configuration of algorithm A1 corresponds to a distinct point in that hyperspace.

Some of hyperparameters HP1-HP4 may be continuous variables, meaning that even a tiny subrange of such a hyperparameter may contain an infinite amount of points. Due to such intractable combinatorics, computer C1 should not consider many or most of the points in the hyperspace.

Instead, computer C1 may intelligently or randomly sample the hyperspace to limit which configuration alternatives of algorithm A1 does computer C1 actually predict performance for. For each actual configuration alternative or set of related configuration alternatives, computer C1 has a separate meta-model, such as MM1-MM3.

Each of meta-models MM1-MM3 was trained to predict how a particular configuration (or set of related configurations) of algorithm A1 will perform for a variety of datasets that are similar or dissimilar to inference dataset D. Related configurations are those that have identical or similar values for a subset of hyperparameters HP1-HP4.

For example, meta-model MM1 was trained by observing the performance of instances of algorithm A1 that had configurations that had identical or similar values for hyperparameters HP2-HP3. Thus, values V2-V3 may be associated with meta-model MM1.

Values V3 and V5 are dissimilar values of hyperparameter HP3. For example, hyperparameter HP3 may be a count of layers in a neural network, and inference dataset D may be a collection of photographs, such that analysis of monochrome photos needs fewer layers than for full-color photos.

Values V3 and V5 may be associated with respective meta-models MM1 and MM3 in that meta-model MM1 was trained by observing the performance of algorithm A1 configured with fewer layers for monochromatic photography and meta-model MM3 was trained by observing the performance of algorithm A1 configured with more layers for full-color photography.

Features of a dataset itself as a whole are referred to as meta-features. For example, inference dataset D has meta-features MF1-MF4.

For example if inference dataset D is a collection of photographs, then meta-feature MF1 may be a count of photographs or an arithmetic mean of pixels per photo, and meta-feature MF2 may be a statistical variance of all pixel luminosities of all of the photos or median count of edges of all photos, which may be somewhat rigorous to calculate.

Unlike hyperparameters that may have many values, such as values V4 and V6 for hyperparameter HP4, each meta-feature has at most one value. For example, meta-feature MF1 has value MFV1.

Some meta-features may be applicable to some but not all datasets. For example, some meta-features may naturally lack values for inference dataset D. For example, a meta-feature for a statistically modal haircut style may lack a value if none of the photographs of inference dataset D contain people.

Meta-feature values MFV1-MFV4 may characterize inference dataset D, such that somewhat similar datasets (such as monochrome photos) should have somewhat similar meta-feature values (such as color count). Likewise, different configuration alternatives of algorithm A1 may be more suited or less suited for analyzing different categories of datasets.

For example, meta-model MM1 may correspond to one set of hyperparameter values that performed well for monochrome photos, and meta-model MM3 may correspond to another set of hyperparameter values that performed well for full-color photos. If inference dataset D mostly contains monochrome photos, then meta-model MM1 should indicate better suitability of its hyperparameter values.

Whereas with mostly photos of full-color, meta-model MM3 should indicate better suitability of its hyperparameter values. Thus, by stimulating already-trained meta-models with respective subsets of hyperparameter values and meta-feature values of a new (unfamiliar) inference dataset such as D, computer C1 may detect how suitable are various hyperparameter configurations of various algorithms A1-A3.

Thus, computer C1 can spontaneously match trainable algorithms and their alternate configurations to particular unfamiliar datasets. Thus, computer C1 (or a downstream computer) can efficiently limit training to an optimal subset of contextually promising algorithms (and configurations) based on the dataset.

For example, meta-models MM1-MM3 may each be an already trained neural network that takes a subset of hyperparameter values and a subset of meta-feature values as stimulus inputs. Training of meta-models is discussed later herein.

In an embodiment, the actual values of hyperparameter values (not meta-feature values) are used to train meta-models MM1-MM3. In an embodiment, these actual values are used to stimulate meta-models MM1-MM3 after training during inferencing for algorithm selection. In an embodiment, these actual values are used for both training and inferencing. In an embodiment, default hyperparameter values (not shown) are instead used during inferencing. Thus in many embodiments, inferencing need not reuse the same hyperparameter values that were used for training a same meta-model.

Meta-models MM1-MM3 are already trained regressors that process inputs to emit a comparative suitability score. For example, meta-model MM1 emits score SC1.

Scores SC1-SC3 share a performance measurement scale. For example, a score may predictively measure how proficient (accuracy such as error rate) would a particular configuration of a particular algorithm become after training for a fixed duration with a particular training dataset, for which inference dataset D is representative (e.g. small sample) of the training dataset.

Likewise, a score may instead predictively measure how much time does a particular configuration of a particular algorithm need to achieve a fixed proficiency for a particular training data set. Instead, a score may simply be a comparative measure of abstract suitability.

Regardless of score semantics, each meta-model of each algorithm emits a score. Computer C1 may select the best one or few algorithms (perhaps also best hyperparameter values), such as A2 as shown, as ranked based on sorted scores.

Computer C1 (or a downstream computer) may then use selected algorithm A2 to achieve a result, such as result R. For example, computer C1 may use inference dataset D (or a larger dataset that includes D) to actually train one or a few alternate configurations of algorithm A2. For example, result R may be a well configured and well trained instance of algorithm A2 that is ready for production use.

The techniques herein improve the performance of computer C1 itself in various ways. By pruning the hyperparameter hyperspace, training of an excessive count of hyperparameter configurations is avoided. By selecting well suited algorithms and/or their configurations, training of an excessive count of different algorithms is avoided. By scoring based on fitness for actual dataset meta-feature values, contextual suitability of selection is increased.

Thus, subsequent training (e.g. by computer C1) occurs faster. Likewise, the trained selected algorithm(s) achieve higher accuracy in production use (e.g. by computer C1). Thus, computer C1 is accelerated as an algorithm training computer and is more reliable (accurate) as a production inference computer. By reducing the computational burden of these activities, the techniques herein are accelerated (save time) and save energy.

Computer C1 may optimally select trainable algorithms based on performance predictions by trained regressors, in an embodiment.

Step S1 is preparatory. In step S1, meta-feature values are derived from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For example, meta-features MF1-MF4 may be predefined by human experts as aspects that are generally obtainable from many or all datasets of some application.

For example, most application datasets consist of data units such as pixels, photographs, or tuples (e.g. database table rows). To the extent that a machine learning algorithm may have some configurations that adequately learn with little training data and other configurations that more accurately learn with much training data, one useful meta-feature may be the size of the dataset, such as a count of rows, pixels, or photographs.

Meta-feature values MFV1-MFV4 may be extracted or synthesized from inference dataset D. Thus, the character of inference dataset D may be known.

Step S2 is repeated for each trainable algorithm that is available to computer C1. For each meta-model associated with a same algorithm, step S2 calculates a respective score by invoking the meta-model based on at least one of: a) a respective subset of meta-feature values, or b) hyperparameter values of a respective subset of hyperparameters of algorithm.

For example, already-trained distinct meta-models MM1-MM3 may be individually stimulated with a respective subset of meta-feature values MFV1-MFV4 and a respective subset of hyperparameter values V1-V6 as inference inputs. For example, meta-model MM1 calculates score SC1 based on meta-feature values MFV1-MFV2 and hyperparameter values V2-V3.

After step S2 is sufficiently repeated, all meta-models of all algorithms A1-A3 have scores. Based on those scores, at least one promising algorithm is selected for training. For example, computer C1 selects algorithm A2 that has the highest scoring meta-model of all algorithms or the highest mean, median, or modal score of all algorithms.

Based on the inference dataset, step S4 invoked the selected algorithm(s) to obtain a result. This may or may not entail training at least one model (distinctly configured instance) of the selected algorithm.

Step S4 may finish by invoking the trained model(s) of the selected algorithm to obtain a result. For example, the result may be a classification/recognition of an object within inference dataset D or a larger dataset.

An explanation that distinguishes a model from a meta-model of an algorithm is discussed later. Training models and metamodels is also discussed later.

An example computer C2, in an embodiment, trains an algorithm-specific machine learning ensemble for contextually accurate performance prediction. Computer C2 may be an implementation of computer C1.

Computer C2 contains configurable trainable algorithms ALG1-ALG3. Each algorithm is associated with its own machine learning ensemble, such as machine learning ensemble MLE.

Ensemble MLE is trainable because it contains trainable meta-models MMOD1-MMOD2. Each of meta-models MMOD1-MMOD2 corresponds to at least a subset of hyperparameters and/or hyperparameter values of algorithm ALG1.

When meta-model MMOD1-MMOD2 is stimulated with same input values, respective performance prediction scores PPSC1-PPSC2 is generated. Ensemble MLE integrates scores PPSC1-PPSC2 to synthesize ensemble score ESC, which is ensemble MLE's performance prediction score, which statistically is accurate more often than either of scores PPSC1-PPSC2 individually.

In an embodiment, scores PPSC1-PPSC2 are averaged to synthesize ensemble score ESC. In an embodiment, a (normalized exponential) softmax function integrates scores PPSC1-PPSC2 to synthesize ensemble score ESC. Unlike a statistical mean, a softmax function may use sigmoidal normalization to reduce distortion caused by outlying (deviant) scores, which may suppress meta-model(s) when they are noisy.

Although not shown, ensemble MLE may itself be a composite neural network that contains the neural networks of meta-models MMOD1-MMOD2. Ensemble MLE may have an additional final layer that applies softmax to scores PPSC1-PPSC2 to calculate ensemble score ESC.

Ensemble MLE is trained with performance tuples PT1-PT3 in sequence as inputs. Each of tuples PT1-PT3 is a historical record of the performance of algorithm ALG1 when configured with particular hyperparameter values, trained, and then tested with a particular test dataset.

For example, tuple PT2 has test score TSC2 that occurred when algorithm ALG1 was configured with hyperparameters HPARV2, trained with a training dataset (not shown), and then tested with test dataset TD1 to generate test score TSC2. Each machine learning model, such as MLM1-MLM2, is a trained instance of algorithm ALG1 that was configured with respective hyperparameter values, such as HPARV1-HPARV2 respectively.

All of models MLM1-MLM2 and meta-models MMOD1-MMOD2 are trained and/or trainable. However, models MLM1-MLM2 are instances of algorithm ALG1 and trained with actual application training data. Whereas, meta-models MMOD1-MMOD2 are instances of a different algorithm.

For example, algorithm ALG1 and models MLM1-MLM2 may be support vector machines (SVMs), and meta-models MMOD1-MMOD2 may be neural networks that predict the performance of those SVMs. After configuring and training models MLM1-MLM2, they are tested with test datasets TD1-TD2 to generate scores TSC1-TSC3.

A test dataset is reusable across multiple tests to generate multiple test scores. For example, models MLM1-MLM2 may both be tested with test dataset TD1 to produce test scores TSC1-TSC2 respectively.

A model is reusable across multiple tests to generate multiple test scores. For example, model MLM2 may be tested with test datasets TD1-TD2 to produce test scores TSC3-TSC3 respectively.

For each test, a tuple is recorded that references the variables of the test, which may include the identity of the test dataset, the test score, the hyperparameter values, and the identity of the algorithm or the model. For example, tuple PT2 records that algorithm ALG1 was configured with hyperparameter values HPARV2 to create and train machine learning model MLM2 that produced test score TSC2 when tested with test dataset TD1.

Tuples PT1-PT3 may be stored in volatile memory or durably, such as in a relational database or disk file. After model training and testing that generates tuples PT1-PT3, the tuples are used as inputs for training ensemble MLE.

For each tuple during ensemble training, hyperparameter values may be obtained from the tuple, and meta-feature values may be obtained directly or indirectly from the training set of the tuple. In an embodiment, those values obtained for the tuple may be used to select one or some of meta-models MMOD1-MMOD2 for training based on those values or subranges of values that included those values. In an embodiment, those values are used to train all of meta-models MMOD1-MMOD2.

Those values are injected as stimulus input into some or all meta-models for training. Thus, meta-models MMOD1-MMOD2 and ensemble MLE may learn to predict the performance of algorithm ALG1 and/or models MLM1-MLM2.

Algorithms ALG2-ALG3 each also has its own trainable ensemble. Because each ensemble (and its meta-models) is trained for a separate algorithm, each ensemble learns to predict the performance of its own algorithm very well and without cross-training interference that would otherwise be caused by having to learn the contradictory performance quirks of multiple distinct algorithms.

Thus during inferencing after ensemble training, computer C2 may invoke multiple ensembles to detect how well each algorithm would perform for a same unfamiliar inference dataset. For example, computer C2 may select one or a few algorithms with the best ensemble score(s) for subsequent expensive training with an unfamiliar training dataset for which the inference dataset is representative. Thus, expensive training of algorithms with unpromising ensemble scores for that inference dataset may be avoided without loss of optimality.

In an embodiment, cross validation such as k-fold cross validation is used to create many pairs of training dataset and test dataset from one original training dataset, such that each pair contains the original training dataset but partitioned between training dataset and test dataset in different ways.

Likewise, some meta-models may be reserved (dedicated) for training with only a particular subset of hyperparameters, meta-features, values, and/or value subranges. Tuples whose values do not correlate with those expectations of a particular meta-model may be skipped while training that meta-model. A tuple may be skipped while training that meta-model if the tuple is missing a value that the meta-model expects.

In an embodiment, meta-features that are missing a value in a percentage of the distinct tuples (all tuples or one meta-model's bagged tuples as explained later) that exceeds a threshold are excluded. In an embodiment, individual tuples (all tuples or one meta-model's bagged tuples) that are missing a value are excluded.

For example with tuples PT1-PT3, only PT2 is suitable for training both meta-models MMOD1-MMOD2. Multiple meta-models that accept a same shared tuple, such as PT2, may take different subsets of (hyperparameter and meta-feature) values that are associated with the shared tuple.

Because meta-models MMOD1-MMOD2 are trained with somewhat dissimilar subsets of tuples and somewhat dissimilar subsets of values from shared tuples, meta-models MMOD1-MMOD2 are actually trained with different data. Thus, meta-models MMOD1-MMOD2 actually are distinct after training, even if originally configured identically (e.g. same layer count). Thus, ensemble MLE integrates an actual diversity of predictions. Training dataset partitioning is further discussed below.

Training multiple models and meta-models of multiple algorithms is computationally intensive and amenable to horizontal scaling for acceleration. For example, each computer of a cluster may train one or a few models or meta-models in tandem. Because training and especially testing of each model may be concurrent, tuples may be concurrently generated.

Computer C2 may train an algorithm-specific machine learning ensemble for contextually accurate performance prediction, in an embodiment.

Step ST1 is preparatory. In step ST1, testing datasets are received, stored, generated, or otherwise obtained. For example, a human expert or a data warehouse provides test datasets TD1-TD2 as files, query result sets, or streams.

Steps ST2-ST5 are repeated for each available machine learning algorithm. In particular, steps ST2-ST4 are repeated for each model of each algorithm. In particular, steps ST3-ST4 are repeated for each testing dataset with each model of each algorithm.

Step ST2 configures a model of an algorithm based on respective particular values for hyperparameters of the algorithm. For example, each of models MLM1-MLM2 is separately configured with a distinct set of values, such as HPARV1-HPARV2 respectively, for a shared set of hyperparameters that are common to algorithm ALG1.

After sufficient repetitions of step ST2, all models of all algorithms ALG1-ALG3 are distinctly configured and ready for training. Although not shown, model training may occur between steps ST2-ST3 based on training datasets.

Each occurrence of step ST3 performs a distinct test. Step ST3 tests a model of an algorithm with one of the testing datasets to calculate a respective test score. For example during one test, model MLM2 may generate test score TSC2 when stimulated by test dataset TD1.

Step ST4 records a distinct tuple for each test performed in step ST3. Each tuple references, identifies, or contains each of: respective particular values for hyperparameters of the algorithm, the testing dataset, the respective test score, and the algorithm.

For example, tuple PT2 is recorded when model MLM2 is tested with test dataset TD1. Tuple PT2 may indicate test dataset TD1, test score TSC2, hyperparameters HPARV2, algorithm ALG1, and/or model MLM2.

After sufficient repetitions of step ST4, all models of all algorithms ALG1-ALG3 have been individually tested with each testing dataset TD1-TD2, and all tuples PT1-PT3 were recorded. Thus, all ensembles are ready for training.

Step ST5 independently trains each algorithm ALG1-ALG3's ensemble, such as MLE. For each meta-model that is associated with an algorithm, step ST5 trains all of the algorithm's meta-models based on at least one distinct tuple recorded for that algorithm.

For example, tuples PT1-PT2 are used to train meta-model MMOD2. After sufficient repetitions of step ST5, all meta-models of all algorithms ALG1-ALG3 have been trained, which means that all ensembles, such as MLE, have been trained.

In an embodiment, ensemble training is implemented with some combination of Keras Python library, TensorFlow, and/or Apache MXNet, which horizontally scale such as with multiple graphical processing units (GPUs).

After step ST5, each of algorithm ALG1-ALG3's ensemble is ready for inferencing, such as during exposure to a particular inferencing dataset (likely larger than any test dataset TD1-TD2).

An example computer C3 may, in an embodiment, adjust feature (hyperparameter and meta-feature) values for uniformity, portability, and genericity. Computer C3 may be an implementation of computer C1.

Computer C3 contains adjusted data DT1 and DT2 that was transformed from raw (e.g. wild) data. Raw data may naturally be numeric or categoric.

Categoric data is non-numeric data whose domain consists of discrete (symbolic) values. For uniformity, portability, and genericity, non-numeric data may be converted to numeric.

For example, calendar months January thru December may convert to integers from one to twelve. Thus, a natural relative ordering of months is preserved, and a natural distance between months (e.g. between March=<NUM> and July=<NUM>) is meaningful (e.g. <NUM> - <NUM> = <NUM>).

However, categoric data may have values (categories) that are naturally unordered, in which case a dense numeric encoding may cause neighboring numbers to be falsely treated as semantically similar (adjacent). To prevent such false associations from distorting training, a geometric scale of numbers may be assigned to the categories, such that each number is at least an order of magnitude away from all other values.

For example, table one-hot DT1 shows terrestrial continents as categories, which are naturally unordered. With one-hot encoding, each bit in a string of seven or eight bits may be reserved for a continent, somewhat like a bitmap.

For example, the least significant bit is reserved for Africa, which is one-hot encoded as <NUM>. Alternatively, categories may be encoded as one-cold, which is the bitwise inverse of one-hot, such that only one bit is clear (zero) in the bitmap.

Numeric data may be normalized for uniformity, portability, and genericity. Normalization may entail imposing a zero-mean and/or unit-variance.

Zero-mean shifts the range of numbers until the arithmetic mean becomes zero, which may or may not also be the arithmetic median and/or mode. For example, gaussian distribution DT2 of raw data has a raw mean of <NUM>, which is then substracted from all values to achieve a normalized mean of zero, with lesser values going negative.

Unit-variance scales (e.g. compresses) the range of numbers until the statistical variance is one. For example, adjacent raw values <NUM> and <NUM> may be <NUM> - <NUM> = <NUM> units apart. Those two numbers are compressed to be one unit apart, which in this example are -<NUM> and -<NUM> respectively.

When both zero-mean and unit-variance adjustments occur (e.g., for gaussian distribution DT2), then values, subranges, and distances become uniform regardless of discrepant original raw scales. For example, a compass direction may range from zero to <NUM>, and a Fahrenheit temperature may seasonally range from -<NUM> to <NUM>.

After normalization as described, a delta of a given magnitude between two normalized compass directions may represent a same amount of change as the same given magnitude of change between to normalized temperatures. Thus, exploration of a multidimensional solution space may occur along either (direction or temperature) axis using similar increments.

Therefore, all axes are likely to be explored and scored more or less evenly, such that optimization of one axis does not accidentally dominate over consideration of the other axes of the solution space. Such normalization also makes scores readily comparable, such that scores from sibling meta-models can be reliably ranked (relatively ordered by value), as well as scores emitted from different algorithms.

An example computer C4 may, in an embodiment, improve ensemble meta-learning with optimizations such as boosting and bagging. Computer C4 may be an implementation of computer system C1.

Computer C4 trains an ensemble that includes meta-models MMDL1-MMDL6 that learn by using tuples TPL as a training dataset. Tuples TPL recorded performance for various test datasets from various configurations of a particular machine learning algorithm X after training. Meta-models MMDL1-MMDL6 may be instances of another machine learning algorithm Y, which may or may not be different from algorithm X.

Tuples TPL contain hyperparameters HYPAR that configured distinct instances of algorithm X. Tuples TPL also contains meta-features MFT as extracted from various testing datasets for algorithm X.

Although not shown, tuples TPL may also contain other data such as test scores of algorithm X. Each tuple (row) of tuples TPL represents one test involving one model (instance) of algorithm X that inferenced one testing dataset.

A tradeoff between bias and variance is endemic to machine learning such as with training of meta-models MMDL1-MMDL6. Variance (a. overfitting) is oversensitivity during training that causes noise to be mistaken for meaningful patterns that make lasting false impressions.

underfitting) is under-sensitivity that causes meaningful patterns to be ignored as noise that prolongs training or prevents complete training.

Distortions such as variance can be decreased by refactoring an ensemble training dataset, such as tuples TPL, by decomposing the dataset into partially overlapping subsets of data with a technique known as bootstrap aggregating (a.

Feature bagging entails training each of meta-models MMDL1-MMDL6 with a distinct partially overlapping subset of training dataset features such as meta-features MFT and hyperparameters HYPAR. For example, only meta-features F-G and hyperparameters I-J of tuples TPL may be used to train meta-model MMDL5.

Sample bagging entails training each of meta-models MMDL1-MMDL6 with a distinct partially overlapping subset of training dataset rows of tuples TPL. For example, only rows Q-T of tuples TPL may be used to train meta-model MMDL1.

Sample bagging and feature bagging may both be used to train a same meta-model. For example, meta-models MMDL1-MMDL6 may each be trained with a respective subset of columns of a respective subset of rows of tuples TPL.

The subsets of tuples TPL may partially overlap. For example although not shown, sample bagging may train meta-model MMDL2 with the bottom four rows of tuples TPL, which includes row T that is also used to train meta-model MMDL1 as shown. Ideally, at most one-third of a meta-model's training subset should overlap with other subset(s), whether sample or feature bagging.

Another ensemble meta-learning optimization is hypothesis boosting that can decrease variance and especially bias. Boosting assigns weights to rows of tuples TPL.

Each row of tuples TPL depending on how easy (readily) is training meta-models MMDL1-MMDL6 to learn from that row. Initially all rows are weighted equally.

During training, a row that is processed accurately (low error) has its weight reduced. Conversely, a row with high error gets increased weight.

Training can be repeated with some or all rows of tuples TPL as their weights evolve to achieve a higher accuracy than training with each row only once. In an embodiment, training is repeated only with rows having at least a threshold weight.

In an embodiment, the threshold is progressively increased. In an embodiment, training ceases when few or no row weights exceed the threshold.

Boosting may also assign weights to meta-models MMDL1-MMDL6 of an ensemble based on their accuracy. Thus, a more reliable meta-model may have more influence over an ensemble score than a less reliable meta-model.

Meta-model weights can be adjusted based on observed accuracy at various times during training. In an embodiment, training ceases when most or all meta-model weights cease to change by at least a threshold amount.

Ensemble meta-learning is amenable to additional meta-optimization before training. An ensemble itself has hyperparameters such as percentage of features (hyperparameters, meta-features) per meta-model for feature bagging, percentage of rows for sample bagging, and count of meta-models.

Likewise, meta-models MMDL1-MMDL6 themselves have hyperparameters such as count of layers if an ensemble is composed of neural networks or other hyperparameters if composed of another machine learning algorithm. Meta-optimization may use gradient descent, Bayesian optimization, SVM, or a decision tree. Computer C4 may use a hyperparameter optimization tool such as the hyperopt Python library to optimize the configuration of an ensemble and/or its constituent meta-models MMDL1-MMDL6. By design, hyperopt is horizontally scalable.

<FIG> is a block diagram of a basic software system <NUM> that may be employed for controlling the operation of computing system <NUM>. Software system <NUM> and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

Software system <NUM> is provided for directing the operation of computing system <NUM>. Software system <NUM>, which may be stored in system memory (RAM) <NUM> and on fixed storage (e.g., hard disk or flash memory) <NUM>, includes a kernel or operating system (OS) <NUM>.

The OS <NUM> manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as 802A, 802B, 802C. 802N, may be "loaded" (e.g., transferred from fixed storage <NUM> into memory <NUM>) for execution by the system <NUM>. The applications or other software intended for use on computer system <NUM> may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, an app store, or other online service).

Software system <NUM> includes a graphical user interface (GUI) <NUM>, for receiving user commands and data in a graphical (e.g., "point-and-click" or "touch gesture") fashion. These inputs, in turn, may be acted upon by the system <NUM> in accordance with instructions from operating system <NUM> and/or application(s) <NUM>. The GUI <NUM> also serves to display the results of operation from the OS <NUM> and application(s) <NUM>, whereupon the user may supply additional inputs or terminate the session (e.g., log off).

OS <NUM> can execute directly on the bare hardware <NUM> (e.g., processor(s) <NUM>) of computer system <NUM>. Alternatively, a hypervisor or virtual machine monitor (VMM) <NUM> may be interposed between the bare hardware <NUM> and the OS <NUM>. In this configuration, VMM <NUM> acts as a software "cushion" or virtualization layer between the OS <NUM> and the bare hardware <NUM> of the computer system <NUM>.

VMM <NUM> instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine comprises a "guest" operating system, such as OS <NUM>, and one or more applications, such as application(s) <NUM>, designed to execute on the guest operating system. The VMM <NUM> presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.

In some instances, the VMM <NUM> may allow a guest operating system to run as if it is running on the bare hardware <NUM> of computer system <NUM> directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware <NUM> directly may also execute on VMM <NUM> without modification or reconfiguration. In other words, VMM <NUM> may provide full hardware and CPU virtualization to a guest operating system in some instances.

In other instances, a guest operating system may be specially designed or configured to execute on VMM <NUM> for efficiency. In these instances, the guest operating system is "aware" that it executes on a virtual machine monitor. In other words, VMM <NUM> may provide para-virtualization to a guest operating system in some instances.

A computer system process comprises an allotment of hardware processor time, and an allotment of memory (physical and/or virtual), the allotment of memory being for storing instructions executed by the hardware processor, for storing data generated by the hardware processor executing the instructions, and/or for storing the hardware processor state (e.g. content of registers) between allotments of the hardware processor time when the computer system process is not running. Computer system processes run under the control of an operating system, and may run under the control of other programs being executed on the computer system.

The term "cloud computing" is generally used herein to describe a computing model which enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and which allows for rapid provisioning and release of resources with minimal management effort or service provider interaction.

A cloud computing environment (sometimes referred to as a cloud environment, or a cloud) can be implemented in a variety of different ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or to the general public. In contrast, a private cloud environment is generally intended solely for use by, or within, a single organization. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud comprise two or more types of cloud (e.g., private, community, or public) that are bound together by data and application portability.

Generally, a cloud computing model enables some of those responsibilities which previously may have been provided by an organization's own information technology department, to instead be delivered as service layers within a cloud environment, for use by consumers (either within or external to the organization, according to the cloud's public/private nature). Depending on the particular implementation, the precise definition of components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), in which consumers use software applications that are running upon a cloud infrastructure, while a SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), in which consumers can use software programming languages and development tools supported by a PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything below the run-time execution environment). Infrastructure as a Service (IaaS), in which consumers can deploy and run arbitrary software applications, and/or provision processing, storage, networks, and other fundamental computing resources, while an IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) in which consumers use a database server or Database Management System that is running upon a cloud infrastructure, while a DbaaS provider manages or controls the underlying cloud infrastructure and applications.

The above-described basic computer hardware and software and cloud computing environment presented for purpose of illustrating the basic underlying computer components that may be employed for implementing the example embodiment(s). The example embodiment(s), however, are not necessarily limited to any particular computing environment or computing device configuration. Instead, the example embodiment(s) may be implemented in any type of system architecture or processing environment that one skilled in the art, in light of this disclosure, would understand as capable of supporting the features and functions of the example embodiment(s) presented herein.

Claim 1:
A computer-implemented method comprising:
deriving a plurality of meta-feature values from an inference dataset that contains a plurality of data units by, for each meta-feature of a plurality of meta-features, deriving a respective meta-feature value for the inference dataset from multiple data units of the plurality of data units of the inference dataset, wherein each meta-feature characterizes the inference dataset as a whole, and wherein the inference dataset comprises mono-chromatic photographs and/or full-color photographs having pixels;
for each algorithm of a plurality of trainable algorithms:
for each already trained algorithm-specific meta-model of a respective plurality of algorithm-specific regression meta-models that can predict performance of the algorithm, calculating a respective score emitted by the meta-model, of a plurality of meta-model scores, by stimulating the meta-model with (i) a respective subset of meta-feature values of the plurality of meta-feature values and (ii) a respective subset of hyperparameter values of a plurality of hyperparameters of the algorithm as inference inputs;
selecting, based on the plurality of meta-model scores, and training a subset of the plurality of trainable algorithms with the inference dataset or with a training set for which the inference dataset is representative to classify or recognize an object within the inference dataset or a larger data set;
wherein training of the plurality of trainable algorithms is limited to said training said subset of the plurality of trainable algorithms.