Patent Description:
"<NPL>) describes efficient neural architecture search methods in which an algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification.

The architecture search techniques described in this specification can determine a high-performing architecture for a dense prediction task in a computationally efficient manner. In particular, because dense prediction tasks require generating predictions for a large number of the pixels in the input image, such tasks require the network to operate on high resolution imagery. This makes existing architecture search techniques, e.g., techniques geared for image classification or other, non-dense image processing tasks, ill-suited for use for these tasks. This is because many of these tasks rely on low-resolution proxy tasks that would not be representative of the final dense image prediction task or require a search space to be searched that is so large as to make such searching computationally infeasible when operating on high-resolution images.

The described techniques, on the other hand, effectively limit the search space to identifying the best architecture for a dense prediction cell, resulting in architectures that have performance that exceeds the previous state-of-the-art in multiple dense image prediction tasks.

Moreover, by making use of the described techniques, the resulting architecture can be more computationally efficient than the previously state-of-the-art models while exceeding their performance. As examples of the kinds of results that the described techniques can achieve, the resulting architectures can achieve state-of-the-art performance on several dense prediction tasks, including achieving <NUM>\% mIOU accuracy on the Cityscapes data set (street scene parsing), <NUM>% mIOU accuracy on the PASCAL-Person-Part data set (person-part segmentation), and <NUM>% mIOU accuracy on the PASCAL VOC <NUM> data set (semantic image segmentation). At the same time, the resulting architectures are more computationally efficient, requiring approximately half the parameters and half the computational cost as previous state of the art systems for these data sets.

Additionally, by making use of a smaller backbone during the search than will be included in the final architecture, the amount of resources consumed by the search process is reduced. Additionally, by pre-training the backbone, e.g., on an object segmentation task, and then holding the backbone fixed during the search, the amount of resources consumed by the search process is reduced. Additionally, by pre-computing and then caching the feature maps generated by the pre-trained backbone, the amount of resources consumed by the search is reduced. As a particular example, when the backbone is smaller and the feature maps generated by the pre-trained backbone are pre-computed and cached, the system can perform the search with much less latency and much greater data efficiency (i.e., using much less memory), than techniques that rely on training candidate neural networks without pre-caching inputs and without decreasing the size of the.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines an architecture for a neural network that is configured to perform a dense image prediction task.

A dense image prediction task requires assigning a label or otherwise making a prediction for each pixel of the image. Thus, dense image prediction tasks generally require the neural network to operate on high-resolution images.

One example of a dense image prediction task is an image segmentation task. In an segmentation task, the input is an image and the output is a respective label for every pixel in the image that classifies the content depicted at that pixel in the image.

One example of an image segmentation task is a person-part segmentation task, where the inputs are images of one or more people and the output is a respective label for every pixel in the image such that the labels classify which pixels correspond to which person parts (e.g., head, torso, legs, and so on) and which correspond to the background (i.e., do not depict any people).

Another example of an image segmentation task is a semantic image segmentation task. In a semantic image segmentation task, the input is an image and the output is a respective label for every pixel in the image that identifies which object class the pixel belongs to, e.g., from a set of multiple foreground object classes and one or more background object classes.

Another example of an image segmentation task is a scene parsing task. In a scene parsing task, the input is an image and the output is a respective label for every pixel in the image that identifies which portion of a scene depicted in the image the pixel belongs to.

Another example of a dense image prediction task is an object detection task. In an object detection task, the input is an image and the output is data that specifies which pixels of the image are parts of an image of an object. For example, the output may be a label for each pixel in the image that identifies whether the pixel is part of an image of an object. As another example, the output may be a score for each of a large number of bounding boxes in the image that indicates whether the bounding box is part of an image of an object.

<FIG> shows an example neural architecture search system <NUM>. The neural architecture search system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The neural architecture search system <NUM> is a system that obtains training data <NUM> for training a neural network to perform a dense image prediction task and a validation set <NUM> for evaluating the performance of the neural network on the dense image prediction task and uses the training data <NUM> and the validation set <NUM> to determine an architecture for a neural network that is configured to perform the dense image prediction task i.e., to receive inputs and generate outputs that conform to the requirements of the dense prediction task. The architecture defines the number of layers in the neural network, the operations performed by each of the layers, and the connectivity between the layers in the neural network, i.e., which layers receive inputs from which other layers in the neural network.

Generally, the training data <NUM> and the validation set <NUM> both include a set of neural network inputs and, for each network input, a respective target output that should be generated by the neural network to perform the dense image prediction task. For example, a larger set of training data may have been randomly partitioned to generate the training data <NUM> and the validation set <NUM>. In the dense image prediction task setting, each neural network input is an image and the neural network output for the neural network input identifies the labels that should be assigned to some or all of the pixels in the neural network input.

The system <NUM> can receive the training data <NUM> and the validation set <NUM> in any of a variety of ways. For example, the system <NUM> can receive training data as an upload from a remote user of the system over a data communication network, e.g., using an application programming interface (API) made available by the system <NUM>, and randomly divide the uploaded data into the training data <NUM> and the validation set <NUM>. As another example, the system <NUM> can receive an input from a user specifying which data that is already maintained by the system <NUM> should be used for training the neural network, and then divide the specified data into the training data <NUM> and the validation set <NUM>.

Generally, the system <NUM> determines the architecture for the neural network by searching a space of candidate architectures to identify one or more best performing architectures.

Each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task.

Thus, each candidate architecture includes the same neural network backbone as each other candidate architecture but has a different dense prediction cell from each other candidate architecture.

<FIG> shows a candidate neural network architecture <NUM>. As described above, the architecture <NUM> receives a neural network input <NUM>, i.e., one of the three images shown in <FIG>, and generates a neural network output <NUM> that assigns labels to the neural network input. As shown in <FIG>, the neural network output <NUM> is represented as an overlay over the corresponding input image, with pixels assigned the same label being in the same shade in the overlay.

The candidate neural network architecture includes a backbone <NUM> that is made up of multiple neural networks layers, e.g., convolutional layers optionally combined with other types of layers (batch normalization, pooling, and so on) and receives the neural network input <NUM> and generates feature maps that are provided as input to a dense prediction cell (DPC) <NUM>. The dense prediction cell <NUM> processes the feature maps to generate the neural network output <NUM>.

As indicated above, each candidate neural network in the search space has the same backbone <NUM> but all of the candidate neural networks have different DPCs <NUM>. Thus, the search space for the architecture search is the space of possible architectures for the DPC <NUM>.

An example of a space of possible architectures for the DPC <NUM> that can be searched by the system <NUM> follows.

For example, the DPC can have B branches, where B is a fixed integer greater than one. Each of the B branches of the DPC maps an input tensor to the branch to an output tensor by applying an operation to the input tensor. The DPC can then combine, e.g., concatenate, the output tensors generated by all of the B branches to generate a combined tensor output or use the output tensor generated by a designated one of the B branches, e.g., the last of the B branches in a processing order, as the combined tensor output. B can be, e.g., an integer in the range of <NUM> to <NUM>, inclusive, with larger values of B allowing more flexibility and a larger search space but increasing the computational cost of the search process.

In some cases, the combined tensor output can be used as the output of the DPC <NUM> for the dense prediction task. In other cases, the DPC processes the combined tensor output through one or more output layers, e.g., a sigmoid output layer, to generate a final output.

Thus, the different architectures in the search space each specify (i) an input tensor from a set of input tensors to be provided as input to each of the B branches, and (ii) an operation from a set of operations to be performed by each of the B branches.

The input tensor for a given branch can be selected from a set that includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any branches before the branch in the processing order of the B branches. Thus, for the first branch in the processing order, the set includes only the feature maps generated by the backbone while for the last of the B branches in the processing order, the set includes (i) the feature maps generated by the backbone and (ii) output tensors generated by any of the B-<NUM> other branches.

The operator space, i.e., the space of possible operations from which the operation performed by each of the B blocks is selected, can include one or more of the following: (<NUM>) a convolution with a <NUM> x <NUM> kernel, (<NUM>) one or more atrous separable convolutions, each having a different sampling rate, and (<NUM>) one or more spatial pyramid pooling operations, each having a respective grid size.

When the operator space includes multiple atrous separable convolutions, each of the atrous separable convolutions will have a different sampling rate. For example, the sampling rate can be defined as r_h x r_w, where each of r_h and r_w is selected from the set of {<NUM>, <NUM>, <NUM>, <NUM>,.

When the operator space includes multiple spatial pyramid pooling operations, each of the spatial pyramid pooling operations will have a different grid size. For example, the grid size can be defined as g_h x g_w, where each of g_h and g_w is selected from the set of {<NUM>, <NUM>, <NUM>, <NUM>}.

Other examples of search spaces can include different operator spaces and different possible values for the sampling rates and grid sizes of the operators in the space.

Returning to the description of <FIG>, the system <NUM> includes a candidate selection engine <NUM>, a training engine <NUM>, and a quality evaluation engine <NUM>.

To search the space of candidate architectures, the system repeatedly performs a set of architecture search operations.

At each iteration of the operations, the candidate selection engine <NUM> selects one or more candidate architectures from the space of possible candidate architectures using quality measures for candidate architectures that have already been evaluated.

The training engine <NUM> then trains the selected candidate architectures on at least some of the training data <NUM> and the quality evaluation engine <NUM> evaluates the trained candidate architectures using the validation set <NUM>.

Selecting candidate architectures, training the candidate architectures, and evaluating the quality of the architectures is described in more detail below with reference to <FIG> and <FIG>.

After the architecture search operations have been repeatedly performed, the system <NUM> can identify as the best performing candidate architectures a threshold, fixed number of candidate architectures evaluated during the search that had the best quality measures.

Once the system <NUM> has identified the best performing candidate architectures, the system <NUM> determines the final architecture for the neural network based on the best performing candidate architectures.

For example, the system <NUM> can identify a threshold number of best performing candidate architectures, generate, from each of the best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task. The system then selects the best performing, e.g., as determined based on accuracy on the validation set <NUM>, trained architecture as the architecture of the neural network.

The system <NUM> replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.

In other words, the system uses a smaller backbone for the architecture search than is employed by the final architecture.

For example, the system can employ an Xception architecture for the backbone in the final architecture while employing the MobileNet-v2 architecture for the candidate architectures, i.e., during the search. The MobileNet-v2 architecture requires roughly one twentieth the computational cost of the Xception architecture and cuts down the number of channels in the backbone feature maps from <NUM> to <NUM> dimensions. The Xception architecture is described in more detail in F. Xception: Deep learning with depthwise separable convolutions. In CVPR, <NUM> while the MobileNet-v2 architecture is described in more detail in <NPL>.

Using the smaller backbone can allow the system to make architecture search more computationally efficient while still providing a quality signal for how well the final architecture having the larger backbone will perform on the dense image prediction task.

Additional techniques that can be employed by the system to improve the computational efficiency of the search are described below with reference to <FIG>.

The neural network search system <NUM> can then output architecture data <NUM> that specifies the final architecture of the neural network, i.e., data specifying the layers that are part of the neural network, the connectivity between the layers, and the operations performed by the layers. For example, the neural network search system <NUM> can output the architecture data <NUM> to the user that submitted the training data.

In some implementations, instead of or in addition to outputting the architecture data <NUM>, the system <NUM> uses the trained neural network having the final architecture to process requests received by users, e.g., through the API provided by the system. That is, the system <NUM> can receive inputs to be processed, use the trained neural network to process the inputs, and provide the outputs generated by the trained neural network or data derived from the generated outputs in response to the received inputs.

<FIG> is a flow diagram of an example process <NUM> for determining a final architecture for a dense image prediction task. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, e.g., the neural architecture search system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system receives training data for the dense image prediction task (step <NUM>).

The system searches a space of candidate architectures to identify two or more best performing architectures using the training data (step <NUM>). Searching the space of candidate architectures is described in more detail below with reference to <FIG>. Once the search has completed, the system can select, as the best performing architectures, a threshold number of candidate architectures that had the best quality evaluation during the search.

The system determines the architecture for the neural network based on the one or more best performing candidate architectures (step <NUM>). The system can generate, from each of the identified best performing candidates, a final architecture, and then train the final architectures to convergence on the dense prediction task. The system then selects the best performing, e.g., as determined based on accuracy on the validation set, trained architecture as the architecture of the neural network.

The the system replaces the neural network backbone with a different, larger neural network backbone that has more parameters and that allows the final neural network to perform better on the dense prediction task.

<FIG> is a flow diagram of an example process <NUM> for searching the space of candidate architectures. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, e.g., the neural architecture search system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system obtains data specifying a pre-trained backbone for use in searching the space of candidate architectures (step <NUM>). In some cases, the system pre-trains the backbone along with a placeholder DPC on a placeholder dense image prediction task to determine trained values of the parameters of the backbone, i.e., so that the system only needs train the backbone once for the entire search process. In some other cases, the system obtains data specifying the trained values of the backbone parameters from another system that has already pre-trained the backbone.

Optionally, the system processes at least some of the training inputs in the training data using the pre-trained backbone to generate feature maps for the training inputs (step <NUM>). In other words, the system processes each of the training inputs using the pre-trained backbone, i.e., in accordance with the trained values of the parameters of the backbone, to generate feature maps and then stores the generated feature maps for use during the search process. When the system is using the smaller backbone for the search, because the feature maps generated by the smaller backbone have many fewer channels than the feature maps generated by the larger backbone (that will be used in the final architecture), storing the feature maps requires much less storage space than would be required to store feature maps generated by the larger backbone.

The system then repeatedly performs steps <NUM>-<NUM> until termination criteria for the search are satisfied, e.g., until a threshold number of candidate architectures have been evaluated, until the highest performing candidate architecture reaches a threshold accuracy, or until a threshold amount of time has elapsed.

The system selects two or more candidate architectures from the space of candidate architectures (step <NUM>). The system can use any of a variety of techniques for searching the space to select the candidate architectures.

For example, the system can use a random search strategy. In a random search strategy, at each iteration of the process <NUM>, the system selects one or more architectures from the space of candidate architectures uniformly at random while also selecting one or more architectures that are close to, i.e., are similar to, the currently best observed architectures, i.e., the architectures already evaluated as part of the search that have been found to perform best on the dense prediction task. Random search strategies that can be employed by the system are described in more detail in <NPL> and in <NPL>.

As another example, the system can use a reinforcement learning guided search strategy to select the candidate actions. In particular, at each iteration of the process <NUM>, the system can select architectures that are output by a recurrent neural network that is being trained through reinforcement learning to output candidate architectures that perform well on the task. Examples of reinforcement learning guided search strategies that can be employed by the system are described in more detail in <NPL>.

The system trains the selected one or more candidate architectures on at least a portion of the training data (step <NUM>). That is, for each selected candidate, the system trains a neural network having the architecture until criteria for stopping the training are satisfied. During the training, the system holds the trained values of the backbone parameters fixed while updating the values of the parameters of the DPC in the neural network.

In particular, in implementations where step <NUM> was performed to pre-process the training data using the pre-trained backbone, the system does not process the training inputs using the backbone during step <NUM> and instead accesses the pre-generated feature maps from memory. This greatly decreases the processing power and time consumed by the training, since training inputs do not need to be processed through backbone to generate the feature maps before the feature maps can processed by the DPC.

The system can perform this training using an early stopping criterion, i.e., can perform the training for a fixed number of iterations instead of to convergence.

The system evaluates the performance of each of the trained candidate architectures (step <NUM>). The system uses an evaluation metric that measures the performance of the trained neural network having the candidate architecture on the validation set to evaluate the performance.

Any evaluation metric that is appropriate for dense prediction tasks can be used. For example, the system can use the pixel-wise mean intersection-over-union (mIOU) over the validation data set as the evaluation metric. As another example, the system can use the mean pixel accuracy over the validation data set as the evaluation metric.

Claim 1:
A computer-implemented method comprising:
obtaining (<NUM>) training data for a dense image prediction task to generate a prediction for each pixel in an input image; and
determining an architecture for a neural network configured to perform the dense image prediction task, comprising:
pre-training a first neural network backbone to determine pre-trained values of the parameters of the first neural network backbone;
searching (<NUM>) a space of candidate architectures to identify one or more best performing architectures using the training data, by repeatedly performing the following:
selecting one or more candidate architectures,
training the selected one or more candidate architectures on at least a portion of the training data, and
for each of the trained candidate architectures, evaluating the performance of the trained candidate architecture on the dense image prediction task;
wherein each candidate architecture in the space of candidate architectures comprises (i) the same first neural network backbone that is configured to receive an input image and to process the input image to generate a plurality of feature maps and (ii) a different dense prediction cell configured to process the plurality of feature maps and to generate an output for the dense image prediction task;
wherein training the selected one or more candidate architectures on at least a portion of the training data comprises holding the values of the parameters of the first neural network backbone fixed during the training while adjusting values of the parameters of the different dense prediction cell in the candidate architecture; and
determining (<NUM>) the architecture for the neural network based on the best performing candidate architectures, comprising:
for each of the two or more best performing candidate architectures, generating a final architecture and further training the final architecture to convergence on the dense prediction task using the training data, wherein generating the final architecture comprises replacing the first neural network backbone with a larger neural network backbone having more parameters; and
selecting the best performing final architecture as the architecture for the neural network.