Source: https://patents.google.com/patent/US10607159B2/en
Timestamp: 2020-07-13 03:01:26
Document Index: 69161230

Matched Legal Cases: ['Application No. 62', 'Application No. 62', 'Application No. 62', 'Application No. 62', 'Application No. 62', 'Application No. 62']

US10607159B2 - Systems and methods implementing an intelligent optimization platform - Google Patents
US10607159B2
US10607159B2 US16/243,361 US201916243361A US10607159B2 US 10607159 B2 US10607159 B2 US 10607159B2 US 201916243361 A US201916243361 A US 201916243361A US 10607159 B2 US10607159 B2 US 10607159B2
US16/243,361
US20190147362A1 (en
2019-01-09 Priority to US16/243,361 priority patent/US10607159B2/en
2019-01-09 Assigned to SigOpt, Inc. reassignment SigOpt, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KE, George, CLARK, SCOTT, JOHNSON, ALEXANDRA, HAYES, PATRICK, MCCOURT, MICHAEL
2019-01-09 Application filed by Sigopt Inc filed Critical Sigopt Inc
2019-05-16 Publication of US20190147362A1 publication Critical patent/US20190147362A1/en
2020-03-31 Publication of US10607159B2 publication Critical patent/US10607159B2/en
238000005457 optimization Methods 0.000 title claims description 396
This application is a continuation of U.S. patent application Ser. No. 15/977,168, filed 11 May 2018, which claims the benefit of U.S. Provisional Application No. 62/507,503, filed 17 May 2017, U.S. Provisional Application No. 62/540,367, filed 2 Aug. 2017, U.S. Provisional Application No. 62/593,785, filed 1 Dec. 2017, U.S. Provisional Application No. 62/608,090, filed 20 Dec. 2017, U.S. Provisional Application No. 62/608,076, filed 20 Dec. 2017, and U.S. Provisional Application No. 62/578,788, 30 Oct. 2017, all of which are incorporated in their entireties by this reference.
The system 100 preferably implements an intelligent model optimization platform no including an ensemble of Bayesian optimization processes and machine learning techniques that function to automate an optimization or tuning of features (including weights or coefficients of features) of a model, architecture of a model, and hyperparameters of a model using an ensemble of Bayesian optimization techniques.
The intelligent optimization platform 110 includes the plurality of queue worker machines 120 (which may also be referred to herein as optimization worker machines or API workers), the platform database 130, the shared work queue 135 and the ensemble of optimization models (tuning sources) 140. The intelligent optimization platform 110 generally functions to interact with the API server implementing the intelligent API 105 to receive API requests for implementing new optimization work requests and returning responses or suggestions to the API 105. Using the plurality of intelligent queue worker machines 120, the intelligent optimization platform 110 functions to asynchronously execute a plurality of optimization work requests in real-time and in parallel. This asynchronous execution and parallel processes of the intelligent optimization system 110 provide speed in computing efficiencies in the exploration and exploitation processes (generally, optimization) of features, hyperparameters, models and system architectures. Additionally, or alternatively, the asynchronous processing of work requests enables generation of response to a work request with partial computation states.
The intelligent optimization platform 110 may be implemented using a combination of networked hardware computing servers. Preferably, the intelligent optimization platform is implemented via a distributed networked computing system, such as cloud computing systems, that allows the many processes implemented by the intelligent optimization platform 110 to be implemented in parallel and among disparate computers thereby, in some embodiments, mitigating the possibility of failure or bottlenecking in the optimization pipeline of the intelligent optimization platform 110. However, it shall be noted that the intelligent optimization platform 110 may be implemented with any private and/or public computing servers including combinations thereof. Accordingly, the intelligent optimization platform no may be implemented as a remote web service accessible by multiple clients over the Internet, the Web, or any suitable communication network (e.g., a global area network, a wide area network, a local area network, etc.) that may function to place disparate computing resources in operable connection and communication.
The plurality of intelligent queue worker machines 120 preferably relate to disparate services operating on the intelligent optimization platform 110 that executes code asynchronously with respect to other services or queue working machines of the platform 110. In some embodiments, each of the plurality of intelligent queue worker machines 120 include disparate modules that may function to operate and/or process tasks independently of other intelligent queue worker machines. In some embodiments, each of the plurality of intelligent queue worker machines may be implemented across multiple In some embodiments, each of the plurality of intelligent queue worker machines 120 functions to selectively trigger one or more optimization work requests to one or more optimization models (sources) engines of the ensemble of optimization models 140. And, once the work on the optimization work request is completed by the selected optimization model(s), the queue working machine returns the responses or results to the platform database 130 and stores them therein.
The ensemble of optimization models 140 preferably includes a plurality of disparate optimization sources (e.g., Bayesian optimization algorithms) or machine learning (tools) models that operate to optimize or tune hyperparameters, features, models, system architectures and the like using varying optimization algorithms. In a preferred embodiment, the ensemble of optimization models 140 define a core optimization engine of the intelligent optimization platform no. The features and the parameters of the core optimization engine comprising the ensemble of optimization models 140 may also be optimized continually by one or more of the intelligent queue worker machines 120 (e.g., using Hyperopt, etc.).
Additionally, S220 may function to implement or operate the plurality of disparate work machines asynchronously for performing a plurality of disparate tuning operations or the like. That is, S220 may implement each of the plurality disparate, optimization work machines such that they may operate in parallel and independent of other optimization work machines operating to process an optimization work request. In this regard, the interdependence of the plurality of disparate, optimization work machines may be very limited and preferably, non-existent, such that each of the plurality of disparate, optimization worker machines may complete work on an optimization work request at different times. However, it shall be noted that some of the disparate optimization worker machines may function to pass resultant data of a tuning operation to another optimization worker machine for performing an additional and/or different tuning operation. Accordingly, in some embodiments, the plurality of disparate, optimization machines may function to work at different paces and on different optimization work request at any given time.
1. A machine learning-based system for tuning a model for improved effectiveness including accuracy and/or computational performance, the system comprising:
a remote machine learning-based tuning service implemented via a distributed network computing system that:
implements an ensemble of distinct optimization sources that generate suggested hyperparameter values for optimizing an objective function of an external model;
implements a plurality of distinct tuning worker machines that each operate to set a tuning operation for tuning hyperparameters of the external model according to one or more tuning parameters of a tuning work request by selectively operating the ensemble of distinct optimization sources, wherein the plurality of distinct tuning worker machines include:
(i) a first distinct tuning worker machine that optimizes hyperparameters of one or more of the ensemble of distinct optimization sources;
(ii) a second distinct tuning worker machine that generates suggestions for new hyperparameter values for each of the hyperparameters of the external model; and
(iii) a third distinct tuning worker machine that suggests an importance or a weight of the hyperparameters of the external model;
wherein the remote machine learning-based tuning service:
executes the tuning work request for tuning the external model, wherein executing the tuning work request includes:
generating, using the ensemble of distinct optimization sources, a plurality of suggested hyperparameters values for the hyperparameters of the external model;
generating a surrogate model of the external model based on the objective function for the external model, wherein the surrogate model comprises a hypothesis of and a proxy of the external model when a structure of the external model is not known;
testing a performance of each of the plurality of suggested hyperparameter values for the hyperparameters of the external model by providing each of the plurality of suggested hyperparameters values as input into the surrogate model;
identifies a subset of the plurality of suggested hyperparameter values for the hyperparameters of the external model based on a likelihood that the subset of suggested hyperparameter values would perform well in the external model as determined by the testing; and
returns the subset of the plurality of suggested hyperparameter values as a response to the tuning work request.
each of the distinct optimization sources of the ensemble of distinct optimization sources includes one of a machine learning-based optimization source and a Bayesian-based optimization source.
each of the plurality of distinct tuning worker machines is implemented by a distinct computer or computing server that enables each of the plurality of distinct tuning worker machine to operate independently and/or in parallel.
the remote machine learning-based tuning service generates the surrogate model using at least one of the plurality of distinct tuning source and based at least on the objective function of the external model.
an application programming interface that is in operable communication with the remote machine learning-based tuning service and that enables a configuration of the tuning work request.
an asynchronous work queue comprising a plurality of distinct tuning work request accessible by each of the plurality of distinct worker machines for asynchronous processing.
the model comprises a machine learning model; and
the tuning work request identifies hyperparameters of the machine learning model that are desired for tuning and one or more tuning constraints defining one or more tuning parameters that constrain one or more tuning sessions performed by the remote machine learning-based tuning service when tuning the hyperparameters.
the one or more tuning constraints include (i) a tuning budget that limits a number of tuning sessions run by the intelligent optimization platform against the hyperparameters of the external model and (ii) bounds data that identify a maximum value and minimum value for each of the hyperparameters that limit discoverable values for each of the hyperparameters during a tuning session.
the remote machine learning-based tuning service:
executes the tuning work request for tuning the external model according to an estimated time of completion;
receives an intervening tuning work request relating to a tuning work request that is received at a time during the execution of the tuning work request before the estimated time of completion of the tuning work request; and
upon receipt of the intervening tuning work request, automatically switching between the plurality of distinct optimization sources to expedite an evaluation of suggested hyperparameter values.
in response to returning the subset of the plurality of suggested hyperparameter values, improving an effectiveness of the external model by tuning the external model using one or more of the plurality of suggested hyperparameter values of the subset.
implementing an asynchronous work queue comprising a plurality of distinct tuning work request accessible by each of the plurality of distinct worker machines for asynchronous processing.
if one or more of the plurality of distinct tuning worker machines fail, generating a plurality of suggested hyperparameter values for each of the hyperparameters of the external model based on partial information from remaining operable distinct tuning worker machines of the plurality of distinct tuning worker machines.
13. A method for tuning a machine learning model, the method comprising:
implementing a remote machine learning-based tuning service implemented via a distributed network computing system that includes:
an ensemble of distinct optimization sources that generate suggested hyperparameter values for optimizing an objective function of an external model;
a plurality of distinct tuning worker machines that each operate to set a tuning operation for tuning hyperparameters of the external model according to one or more tuning parameters of a tuning work request by selectively operating the ensemble of distinct optimization sources, wherein the plurality of distinct tuning worker machines include:
executing, by the remote machine learning-based tuning service, the tuning work request for tuning the external model, wherein executing the tuning work request includes:
identifying a subset of the plurality of suggested hyperparameter values for the hyperparameters of the external model based on a likelihood that the subset of suggested hyperparameter values would perform well in the external model as determined by the testing; and
returning the subset of the plurality of suggested hyperparameter values as a response to the tuning work request.
selecting a first tuning source of the plurality of distinct tuning sources;
providing hyperparameter data relating to the hyperparameters of the external model and tuning constraint data derived from the tuning work request to the selected first tuning source to enable a first tuning operation by the first tuning source; and
selecting a second tuning source of the plurality of distinct tuning sources that is different than the first tuning source;
providing hyperparameter data relating to the hyperparameters of the external model and tuning constraint data derived from the tuning work request to the selected second tuning source to enable a second tuning operation by the second tuning source; and
providing an input of the first tuning data or the second tuning data, generated as a result of the first tuning operation or the second tuning operation, to the selected third tuning source to enable a third tuning operation by the third tuning source,
wherein generating the suggested hyperparameter values for the hyperparameters of the external model is based on the third tuning operation.
implementing an application programming interface that is in operable communication with the remote machine learning-based tuning service and that enables a configuration of the tuning work request.
US16/243,361 2017-05-17 2019-01-09 Systems and methods implementing an intelligent optimization platform Active US10607159B2 (en)
US16/243,361 US10607159B2 (en) 2017-05-17 2019-01-09 Systems and methods implementing an intelligent optimization platform
US15/977,168 Continuation US10217061B2 (en) 2017-05-17 2018-05-11 Systems and methods implementing an intelligent optimization platform
US16/796,489 Continuation US20200202254A1 (en) 2020-02-20 Systems and methods implementing an intelligent optimization platform
US20190147362A1 US20190147362A1 (en) 2019-05-16
US10607159B2 true US10607159B2 (en) 2020-03-31
US16/243,361 Active US10607159B2 (en) 2017-05-17 2019-01-09 Systems and methods implementing an intelligent optimization platform
US (2) US10217061B2 (en)
EP (1) EP3625689A1 (en)
US20190354496A1 (en) * 2018-05-18 2019-11-21 International Business Machines Corporation Selecting a priority queue from which to process an input/output (i/o) request using a machine learning module
US10417059B1 (en) * 2018-08-03 2019-09-17 Intuit Inc. Staged application programming interface
US20160110657A1 (en) 2014-10-14 2016-04-21 Skytree, Inc. Configurable Machine Learning Method Selection and Parameter Optimization System and Method
2018-05-11 EP EP18801697.6A patent/EP3625689A1/en active Pending
2019-01-09 US US16/243,361 patent/US10607159B2/en active Active
Bergstra , et al., "Hyperopt: A Python Library for Model Selection and Hyperparameter Optimization", Computational Science & Discovery, Aug. 2015.
Zhou et al ("Combining Global and Local Surrogate Models to Accelerate Evolutionary Optimization") (Year: 2007). *
US20190147362A1 (en) 2019-05-16
EP3625689A1 (en) 2020-03-25
US10217061B2 (en) 2019-02-26
Bragg et al. 2013 Crowdsourcing multi-label classification for taxonomy creation
Liaw et al. 2018 Tune: A research platform for distributed model selection and training
Zhou et al. 2017 Hybrid teaching–learning-based optimization of correlation-aware service composition in cloud manufacturing
Kim et al. 2013 Optimal job scheduling in grid computing using efficient binary artificial bee colony optimization
Corazza et al. 2010 How effective is tabu search to configure support vector regression for effort estimation?
Gibert et al. 2010 Choosing the right data mining technique: classification of methods and intelligent recommendation
Barreno et al. 2008 Optimal ROC curve for a combination of classifiers
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HAYES, PATRICK;MCCOURT, MICHAEL;JOHNSON, ALEXANDRA;AND OTHERS;SIGNING DATES FROM 20180711 TO 20180713;REEL/FRAME:047940/0952