Black-box optimization using neural networks

Methods and systems for determining an optimized setting for one or more process parameters of a machine learning training process are described. One of the methods includes processing a current network input using a recurrent neural network in accordance with first values of the network parameters to obtain a current network output, obtaining a measure of the performance of the machine learning training process with an updated setting defined by the current network output, and generating a new network input that includes (i) the updated setting defined by the current network output and (ii) the measure of the performance of the training process with the updated setting defined by the current network output.

BACKGROUND

This specification relates to determining optimized settings for process parameters of a machine learning training process using neural networks.

Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a long short term (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allow the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.

SUMMARY

This specification describes how a process parameter optimization system can determine optimized values for one or more process parameters of a machine learning training process, e.g., values for one or more hyper-parameters of the machine learning training process.

In one innovative aspect of the present disclosure, a system for determining an optimized setting for one or more process parameters of a machine learning training process includes one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement: a recurrent neural network and a subsystem.

The recurrent neural network has a plurality of network parameters and is configured to: receive a sequence of network inputs, each network input in the sequence comprising (i) a respective input setting for the one or more process parameters and (ii) a measure of a performance of the training process with the input setting, and process the sequence of network inputs in accordance with the network parameters to generate a respective network output for each network input that defines an updated setting for the one or more process parameters.

The subsystem is configured to: determine, for each of a plurality of candidate settings for the one or more process parameters, a respective measure of the performance of the machine learning training process with the candidate setting, wherein the determining comprises repeatedly performing the following: processing a current network input using the recurrent neural network in accordance with first values of the network parameters to obtain a current network output, obtaining a measure of the performance of the machine learning training process with the updated setting defined by the current network output, and generating a new network input that comprises (i) the updated setting defined by the current network output and (ii) the measure of the performance of the training process with the updated setting defined by the current network output; and select a candidate setting from the plurality of candidate settings as the optimal setting for the one or more process parameters using the measures of the performance for the candidate settings.

In some implementations, the process parameters comprise one or more hyper-parameters for the training process.

In some implementations, the process parameters comprise one or more architecture settings of the machine learning model being trained using the machine learning training process.

In some implementations, the system further comprises a plurality of worker computing units, wherein each worker computing unit is configured to: receive an input setting for the one or more process parameters; execute the training process with the input setting; and measure the performance of the training process with the input setting, and wherein obtaining a measure of the performance of the training process with the updated setting defined by the current network output comprises: providing the updated setting defined by the current network output to one of the plurality of worker computing units and, obtaining, from the one of the plurality of worker computing units, the measure of the performance of the training process.

In some implementations, each worker computing unit operates asynchronously from each other worker computing unit.

In some implementations, determining, for each of a plurality of candidate settings for the one or more process parameters, a respective measure of the performance of the training process with the candidate setting further comprises: generating a plurality of initial network inputs, each initial network input comprising (i) a placeholder setting and (ii) a placeholder measure of performance; processing each of the plurality of initial network inputs using the recurrent neural network to generate a respective initial network output for each initial network input; and providing the updated settings defined by the initial network outputs to respective worker computing units in the plurality of worker computing units.

In some implementations, each network input further includes a binary variable that indicates whether or not the network input includes placeholder values, and wherein the binary variable in each initial network input indicates that the initial network input includes placeholder values.

In some implementations, the recurrent neural network is a differentiable neural computer (DNC) or is a long short-term memory (LSTM) neural network.

In some implementations, the subsystem is further configured to: determine the first values of the network parameters from initial values of the network parameters by training the recurrent neural network to, at each iteration of the training, optimize a training function.

In some implementations, the training function for each iteration is sampled from a training distribution.

In some implementations, training the recurrent neural network comprises training the recurrent neural network to minimize a summed loss function.

In some implementations, training the recurrent neural network comprises training the recurrent neural network to minimize an expected posterior improvement loss function.

In some implementations, training the recurrent neural network comprises training the recurrent neural network to minimize an observed improvement loss function.

In some implementations, the subsystem is further configured to: train the machine learning model using the machine learning training process with the optimized setting for the process parameters; and output the trained machine learning model.

Machine learning training processes, e.g., stochastic gradient descent-based training processes, require values of several hyper-parameters, e.g., learning rate, batch size, batch sampling strategy, and so on, to be appropriately tuned before the machine learning training process can be successfully executed to train a machine learning model. When executed with hyper-parameter values that have not been appropriately selected, the machine learning training process can fail to train the machine learning model to attain acceptable performance or can take an excessive amount of time for the model to attain acceptable performance. In other words, a low quality setting for one or more of the hyper-parameters can result in the machine learning training process not performing well.

Conventional solutions for tuning hyper-parameters can be resource intensive, i.e., can require large amounts of computational resources before appropriate values are found, can require significant manual fine-tuning or hand-engineering from users before they can applied to tune the hyper-parameters of a particular training process for a particular machine learning model, or both.

For example, random hyper-parameter search and grid hyper-parameter search both require large amounts of computational resources and often do not even consider the hyper-parameter setting that would have been optimal for training. As another example, optimization packages that use Bayesian optimization to tune hyper-parameters are resource intensive and are heavily hand-engineered.

In contrast, the techniques described in this specification for determining optimized process parameter values can quickly determine a high-quality setting for process parameters of the training process. In other words, the system determines the high-quality setting in a manner that minimizes the amount of computational resources required to discover the high-quality setting. In particular, by using a recurrent neural network as described in this specification, the described system effectively chooses candidate parameter settings to be evaluated while taking into consideration how well previously evaluated settings have performed. Additionally, once the recurrent neural network has been trained, the described techniques do not require any hand-engineering before being applied to a new hyper-parameter tuning scenario, i.e., they are fully automatic.

By parallelizing the evaluation of the performance of the training process, the described system can more quickly determine a high-quality setting for the process parameters even if the computing units performing the evaluation are operating asynchronously from one another.

The trained recurrent neural network can effectively transfer to optimizing different black box functions. That is, the system can train the recurrent neural network to optimize one function and then use the trained recurrent neural network to optimize one or more different black box functions. Thus, once trained, the described techniques are fully automatic and can be applied in multiple different hyper-parameter tuning scenarios. Additionally, the recurrent neural network can be trained using simple training functions that are differentiable and computationally-efficient to evaluate to minimize the amount of computational resources used by the training process. That is, even though the recurrent neural network will later be applied to optimize complex, non-differentiable functions, e.g., loss functions for machine learning model training, the recurrent neural network can be trained using simple training functions to ensure that training the recurrent neural network does not require an excessive amount of computational resources.

By using a recurrent neural network as described in this specification to select which candidate settings are evaluated, the system can determine an optimized setting for the process parameters that results in a high-performing machine learning training process, i.e., that results in the training of the machine learning model converging quicker and using fewer computational resources.

By parallelizing the evaluation of the candidate settings, the system can determine a higher-quality optimized setting quickly.

Because the recurrent neural network can be trained on one function and then used to optimize a different function, the recurrent neural network can be trained in a computationally efficient manner.

DETAILED DESCRIPTION

This specification describes a process parameter optimization system implemented as one or more computer programs on one or more computers in one or more locations that determines an input that optimizes, i.e., minimizes or maximizes, a black box function.

A black box function is a function for which the closed form is not known, but that can be evaluated at a query point in the domain of the function. That is, while the closed form of the function is not known, the output generated by the function for a given input can be evaluated.

In particular, the black box function is a function that measures the performance of a training process for a machine learning model, i.e., a process that trains a machine learning model to determine trained values of the parameters of the machine learning model, e.g., stochastic gradient descent or another gradient-descent based training process. More specifically, the training process is a process that trains the machine learning model to determine values for the weights of the machine learning model that optimize a training objective function, e.g., a loss function, that depends on the outputs generated by the model.

The machine learning model can be any of a variety of machine learning models, e.g., a deep neural network, a generalized linear model, or a support vector machine, configured to perform any of a variety of machine learning tasks, e.g., image processing tasks, speech recognition tasks, sequence transduction tasks, machine translation tasks, and so on.

The training objective function is generally a function that measures an error between the outputs generated by the model and outputs that should have been generated by the model. In this example, the black box function can be the training objective function, e.g., the loss function, as evaluated on a set of test inputs after the machine learning model has been trained using the training objective function for some number of iterations or for some period of time.

In these examples, the system optimizes the black box function with respect to one or more process parameters of the training process, i.e., determines the setting for the process parameters that minimizes or maximizes the measure of performance of the training process.

The process parameters can include one or more hyper-parameters of the training process. A hyper-parameter is a value that affects the performance of the training process but that is not learned as part of the training process. Examples of hyper-parameters can include, e.g., learning rate, momentum rate, batch size, and so on. More generally, instead of or in addition to the hyper-parameters, the process parameters can include architecture settings for the machine learning model, e.g., number of hidden layers, types of hidden layers, number of nodes per hidden layer, number of convolutional layers, number and properties of filters of the convolutional layers, and so on.

As described above, the process parameter optimization system100is a system that determines optimized values for one or more process parameters of a machine learning training process.

Because the optimized values for the process parameters will generally be different for different machine learning models, even if the two machine learning models are trained on the same training data and using a machine learning training process that is otherwise the same, the process parameter optimization system100generates respective optimized process parameter values for each machine learning model that is to be trained using the machine learning training process.

Once the process parameter optimization system100has determined the optimized values for training a given machine learning model, the process parameter optimization system100can train the machine learning model using the machine learning training process and in accordance with the optimized values of the process parameters. Once trained, the process parameter optimization system100can then output the trained machine learning model, i.e., output data specifying the trained values of the weights of the machine learning model to a user device or to another system, use the trained machine learning model to perform inference in accordance with the trained values of the weights of the model, or both.

Alternatively or in addition to training the machine learning model, the process parameter optimization system100can provide the optimized values of the process parameters to another system for use in training the machine learning model using the machine learning training process.

The recurrent neural network110is configured to receive a sequence of network inputs and process the sequence of network inputs in accordance with the network parameters to generate a respective network output for each network input. The recurrent neural network100can be implemented as any appropriate type of recurrent neural network that maintains an internal state or other data between network inputs in the sequence and uses and updates the internal state or other data when processing a given network input in the sequence. For example, the recurrent neural network110can be a long short-term memory (LSTM) neural network or a neural network that is augmented with an external memory, e.g., a differentiable neural computer (DNC).

In particular, the network input at a given time step t in the sequence includes at least (i) a respective input setting {tilde over (x)}t-1for each of the one or more process parameters that are being optimized and (ii) a measure {tilde over (y)}t-1of a performance of the training process with the input setting {tilde over (x)}t-1according to the measure that is being optimized by the system100.

The network output generated by the recurrent neural network110for the network input at the given time step t defines an updated setting xtfor the one or more process parameters, i.e., a setting that is predicted to, in the case where the measure of performance is being minimized, increase, or, in the case where the measure is being maximized, decrease, the measure of performance of the process relative to the measure in the network input. As described above, the recurrent neural network110uses an internal state ht-1when generating the network output at time step t and also updates the internal state ht-1to generate an updated internal state ht. For example, the network output can be a probability distribution over possible combinations of settings for the one or more process parameters. As another example, the network can directly predict the updated settings.

In order for the network outputs generated by the recurrent neural network110to predict effective process parameter settings, i.e., to predict settings that are helpful in determining high-quality optimized process parameter settings, the system100trains the recurrent neural network on training data. Training the recurrent neural network110is described in more detail below with reference toFIG. 2.

The system100determines the optimized settings for the process parameters by determining measures of performance for various candidate settings for the process parameters. After termination criteria are satisfied, e.g., after measures for a predetermined number of candidate settings have been determined or after a predetermined amount of time has elapsed since beginning to determine measures for candidate settings, the system100can select the candidate settings that have the best measure of performance as the optimized settings. That is, the system100can select the candidate settings that have the highest measure if the measure is being maximized or the lowest measure if the measure is being minimized.

The system100can select the candidate settings to be evaluated by repeatedly providing inputs to the recurrent neural network110and using the outputs generated by the recurrent neural network110to select new settings for evaluation.

In particular, as part of selecting and evaluating candidate settings, the system100can process a current network input using the recurrent neural network in accordance with the trained values of the network parameters to obtain a current network output that defines an updated setting for the process parameters. For example, at the time step t, the system100can process the network input that includes at least (i) a respective input setting {tilde over (x)}t-1for each of the one or more process parameters that are being optimized and (ii) a measure {tilde over (y)}t-1of the performance of the training process with the input setting {tilde over (x)}t-1to obtain the network output that defines the updated setting xt.

The system100then obtains a measure of the performance of the machine learning training process with the updated setting defined by the current network output and generates a new network input that includes (i) the updated setting defined by the current network output and (ii) the measure of the performance of the training process with the updated setting defined by the current network output.

In particular, to obtain the measure of performance, the system100executes the training process with the process parameters set to the updated setting, e.g., for a fixed number of iterations or for a fixed amount of time, and then determines the measure by computing the value of the blackbox function as a result of the training. For example, the system100can measure the loss on a test set after the training process has been executed.

Because the recurrent neural network110maintains an internal state and continues to update the internal state while processing each network input, the updated settings generated by the recurrent neural network110take into account the input settings that have already been evaluated and the measures for those input settings. Thus, by repeatedly providing inputs to the recurrent neural network110as described herein, the system100causes the recurrent neural network110to predict output settings that effectively explore the space of possible process settings and that result in optimized output settings that cause the training process to have a high performance quality.

In some implementations, the system100evaluates settings and generates new inputs serially, i.e., one after the other. That is, the system100waits until a given candidate setting has been evaluated before selecting and evaluating a new candidate setting.

In other implementations, to decrease the time required to determine the optimized settings, to improve the effectiveness of determining the optimized settings and allow more candidate settings to be evaluated, or both, the system100evaluates multiple candidate settings in parallel. In particular, in these implementations, the system100includes multiple worker computing units120. In the example ofFIG. 1, the system100includes N different worker computing units, where N is an integer greater than one.

Each of the worker computing units is configured to receive an input setting for the process parameters and evaluate the input setting to generate the measure of performance of the machine learning training process with the updated setting. More specifically, the worker computing unit executes the training process with the input setting and measures the performance of the training process with the input setting, e.g., by determining the loss on the test set after the training has process has been completed.

The worker computing units are configured so that they can operate independently and asynchronously from each other. In some implementations, only partial independence of operation is achieved, for example, because workers share some resources. A computing unit may be, e.g., a computer, a core within a computer having multiple cores, or other hardware or software within a computer capable of independently performing the computations to evaluate an input setting.

Because the worker computing units operate asynchronously, the network input to the recurrent neural network110at time step t+1 may not correspond to the updated setting generated at time step t. For example, in the example ofFIG. 1, the system100provides the updated setting xtto the i-th worker computing unit for evaluation. Before the i-th worker computing unit has finished the evaluation, the j-th worker computing unit finishes evaluating a setting {tilde over (x)}tto generate a measure {tilde over (y)}t. The system100can then provide a network input at time step t that includes the setting {tilde over (x)}tand the measure {tilde over (y)}trather than waiting for i-th worker computing unit to finish evaluation the setting xt. Thus, having the worker computing units120operate asynchronously can accelerate the determination of the optimized setting for the process parameters.

To initiate the process of selecting candidate architectures, the system100generates one or more initial network inputs that include placeholder initial values for the setting and corresponding measure to cause the recurrent neural network110to generate initial settings to be evaluated. In some implementations, to assist the recurrent neural network110in generating diverse queries that will be effective in initiating the search for the optimized parameters, each network input provided to the recurrent neural network110also includes a binary variable o that indicates whether the network input includes placeholder values. That is, the binary variable can take one value, e.g., zero, that indicates that the network input is a placeholder input and another value, e.g., one, that indicates that the network input has been generated as a result of an evaluation of a candidate setting.

FIG. 2shows an example training iteration200during the training of the recurrent neural network110.

During a given training iteration200, the system100adjusts the current values of the network parameters of the recurrent network110as of the beginning of the training iteration200.

In particular, the system100adjusts the current values of the network parameters to minimize a loss function170that depends on network outputs generated by the recurrent neural network over a number of time steps. For ease of description, the example iteration200includes three time steps: t−1, t, and t+1, which is the last time step in the iteration200. However, in practice, each training iteration will generally include a large, fixed number of time steps.

More particularly, during training, the network output x generated by the recurrent neural network at each of the time steps defines a query to a training function150and the loss function170depends on values of the training function150for the queries.

Advantageously, the training function150does not need to be the same as the function that the recurrent neural network110is used to optimize after the recurrent neural network110has been trained, which will be referred to as the test time function.

In particular, the training function150can be a function that is differentiable and computationally efficient to evaluate. Thus, even when the test time function is a computationally intensive and non-differentiable function, i.e., a function that measures performance of a training process, the training function150can be a different function that allows the recurrent neural network110to be trained in a computationally intensive manner.

Generally, the training function150for a given training iteration is sampled from a prior distribution that permits efficient sampling and function differentiation. An example of such a distribution from which the training function can be sampled is a Gaussian Process.

The loss function170can be any appropriate loss that measures how well the query points generated by the recurrent neural network110during the training iteration optimize the training function150.

For example, the loss function170can be a summed loss that is a sum of the values of the training function150for the query points generated by the recurrent neural network110over the time steps in the iteration.

As another example, to encourage the recurrent neural network110to explore, the loss function170can be an expected posterior improvement loss function. The expected posterior improvement loss function can be a sum of the expected posterior improvements of the query points for the time steps in the iteration. The expected posterior improvement of a query point at a given time step is the expected posterior improvement of querying the query point at the time step given the observed values of the training function at earlier time steps.

As another example of a loss that encourages the recurrent neural network110to explore, the loss function170can be an observed improvement loss function. The observed posterior improvement loss function can be a sum of the observed posterior improvements of the query points for the time steps in the iteration. The observed posterior improvement of a query point at a given time step is the minimum of (i) zero and (ii) the difference between the value of the training function for the query point and the smallest observed value of the training function at any earlier time step.

As shown inFIG. 2, to train the neural network110on the loss function170, at each time step the system100provides an input to the recurrent neural network110that includes the query point from the preceding time step and the observed value of the training function150for that query point.

Once the observed value for the last time step for the iteration has been determined, the system100updates the values of the network parameters by determining the gradient of the loss function170with respect to the network parameters and performing stochastic gradient descent (SGD).

Because the training function150has been selected to be computationally efficient and differentiable, evaluating the training function150at the query points and backpropagating through the training function150to determine the gradients are both performed in a computationally-efficient manner. Thus, the recurrent neural network110can be trained in a computationally-efficient manner that minimizes the amount of computational resources consumed by the training.

In implementations where the network inputs after training include the binary variable o, the network inputs during training also include the binary variable and the system generates placeholder initial inputs for the first several time steps of each training iteration.

Additionally, in implementations where after training the system100uses multiple worker computing units operating asynchronously to evaluate candidate settings, during training the system forces the recurrent neural network not to rely on a specific ordering of network inputs by at times processing network outputs out of order during a training iteration, i.e., by ensuring that for at least some of the time steps in the iteration, the network input at a given time step does not correspond to the network output generated at the preceding time step.

FIG. 3is a flow chart of an example process300for determining an optimized setting for process parameters of a machine learning training process. For convenience, the process300will be described as being performed by a system of one or more computers located in one or more locations. For example, a process parameter optimization system, e.g., the process parameter optimization system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process300.

The system can repeatedly perform the process300to generate and evaluate new candidate settings for the process parameters. Once termination criteria have been satisfied, the system can select one of the candidate settings as the optimized setting for the process parameters.

The system process a current network input using the recurrent neural network and in accordance with trained values of the network parameters to obtain a current network output (step302). As described above, if any candidate settings have already been evaluated, the network input includes the mostly-recently evaluated candidate setting and the performance measure for the candidate setting. If no candidate settings have been evaluated, the current network input is a dummy initial input that includes placeholder values and encourages the recurrent neural network to effectively predict initial candidate settings. That is, if the system evaluates N settings in parallel, for the first N iterations of the process300, the current network input is an initial input that includes placeholder values for the setting and for the performance measure. In some cases, the initial inputs also each include a binary variable set to a value that indicates that the initial input includes placeholder values.

The system obtains a measure of performance of the model training process with the updated setting defined by the current network output (step304). In some cases, the system directly executes the training process to determine the measure of performance. In other cases, the system sends the updated setting to a worker computing unit from a set of multiple worker computing units that are each configured to execute the training process with received input settings to generate the measure of performance for the input settings.

The system generates a new network input (step306), i.e., to be used in the next iteration of the process300. The new network input includes the updated setting and the measure of performance for the updated setting. As described above, in some cases, the new network input also includes a binary variable set to a value that indicates that the new network input does not include placeholder values.

Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TENSORFLOW® framework, a MICROSOFT® COGNITIVE TOOLKIT framework, an APACHE® SINGA framework, or an APACHE® MXNET framework.