Patent Description:
The present disclosure relates generally to neural network architecture. More particularly, the present disclosure relates to systems and methods for producing an architecture of a pyramid layer.

Current state-of-the-art convolutional neural network architectures (e.g., architectures used to perform object detection) are manually designed. Although this approach has been successful and delivered strong performances on many benchmarks, these architectures are generally not optimized. For instance, while the backbone models of RetinaNet, Mask RCNN, Fast RCNN, and Faster RCNN inherit highly optimized architectures from years of research in state-of-art classification networks, their feature pyramid networks, which combine features at multiple scales, are generally overlooked, and thus under-optimized. However, the architecture of a feature pyramid network represents a large associated search space. The possible connections and operations to combine feature representations from different scales grow exponentially with the number of layers in the architecture.

In a paper by <NPL>), the authors begin by investigating current feature pyramids solutions, and then reformulate the feature pyramid construction as the feature reconfiguration process. Finally, they propose a reconfiguration architecture to combine low-level representations with high-level semantic features in a highly-nonlinear way.

In a paper by <NPL>), the authors study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, they propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability.

In a paper by <NPL>), the authors present BranchyNet, a novel deep network architecture that is augmented with additional side branch classifiers. The architecture allows prediction results for a large portion of test samples to exit the network early via these branches when samples can already be inferred with high confidence.

According to an aspect of the present invention, there is provided a computing system, as set out in claim <NUM>. According to another aspect of the present invention, there is provided a computer-implemented method for producing an architecture of a pyramid layer that processes a set of pyramidal feature representations output by a backbone model, the method set out in claim <NUM>. According to another aspect of the present invention, there is provided a computing system, as set out in claim <NUM>.

Generally, the present disclosure is directed to systems and methods that perform an iterative search to optimize an architecture for a pyramid layer of a feature pyramid network that combines feature representations at multiple scales. For example, reinforcement learning and/or evolutionary techniques can be used to perform the iterative search. A search space proposed by the present disclosure is designed to cover possible cross-scale connections to combine feature representations at different scales. The search may also be constrained to find an architecture that can be applied repeatedly. As a result, the resulting architecture can be stackable and/or can be used for anytime object detection ("early exit").

More particularly, in some implementations, different architectures of a pyramid layer can be iteratively generated through an evolutionary method or learned, for example, in a reinforcement learning context. For example, a controller model can be trained to select or otherwise generate new model architectures in a given search space using reinforcement learning. The controller model can be trained based on a reward that is based on one or more performance characteristics of a machine-learned model that includes one or more pyramid layer having the most recently proposed architecture. Thus, through trial and error, pyramid layer architectures can be designed or optimized for specific performance characteristics.

The resulting architecture can be light-weight and flexible. First, the resulting architecture can be scalable in that it can be applied repeatedly to improve performance. Second, the resulting architecture can work well with a variety of backbone models, such as ResNet-<NUM>, ResNet-<NUM>, ResNet-<NUM>, and AmoebaNet. Third, the resulting architecture, when combined with various backbone models, can achieve excellent performance characteristics related to speed, accuracy, or other characteristics. Additionally, the learned architecture may be configured for a variety of tasks, including object detection, object recognition, image classification, other visual processing tasks, or other, non-visual machine-learning tasks.

Thus, according to one aspect of the present disclosure, a computing system can include a controller model configured to iteratively generate new architectures for the pyramid layer. The pyramid layer can be configured to receive a plurality of input feature representations (e.g., feature maps) output by the backbone model (or a previous sequential pyramid layer in the event there are multiple stacked pyramid layers). The plurality of input feature representations can have a plurality of different input resolutions. The pyramid layer can be configured to perform convolutions and/or other operations with respect to the input feature representations. In particular, input feature representations can be combined with (e.g., summed with, globally pooled with, etc.) other feature representations (e.g., input feature representations, internal feature representations, and/or output feature representations), including combinations of two feature representations that have the same or different resolutions. The operations performed by the pyramid layer on the feature representations can result in the pyramid layer producing and outputting a plurality of output feature representations. The plurality of output feature representations can have a plurality of different output resolutions.

In some implementations, new feature representations generated by the pyramid layer can include one or more internal feature representations that are internal to the pyramid layer architecture. The internal feature representation(s) can be distinct from the input feature representations and the output feature representations. The internal feature representation(s) can be connected with other internal feature representation(s), input feature representations, and/or output feature representations. At least one of the plurality of output feature representations can be generated based on one or more of the internal feature representations.

In some implementations, for each of a plurality of search iterations, the controller model can be configured to generate the new pyramid layer architecture by constructing and appending a plurality of merging cells to produce the new pyramid architecture. Each merging cell can have two input feature representations and one output feature representation. This can generate cross-scale connections between various feature representations of the pyramid layer, which can result in semantically strong feature representations.

More specifically, to construct one of the merging cells, the controller model can be configured to select a first input feature representation from a set of available feature representations that includes the plurality of input feature representations. The controller model can select a second, different input feature representation of the set of available feature representations. The controller model can select a first resolution of the plurality of different output resolutions and select an operation that combines the first input feature representation with the second, different input feature representation to produce a new feature representation with the first resolution. For example, the operation can include a variety of suitable binary operations, such as a sum operation and a global pooling operation. A plurality of merging cells can be independently and/or sequentially constructed in this fashion. The plurality of merging cells can then be appended to generate the pyramid layer architecture. Thus, the controller model can use merging cells to generate the new pyramid layer architecture.

In some implementations, for at least some of the merging cells, the controller model can add the new feature representation to the set of available feature representations for potential selection at a next merging cell. Thus, in some instances, a newly created feature representation can be merged with another feature representation in the next merging cell. Such operations can facilitate discovery or learning of various cross-scale connections.

In some implementations, the first input feature representation and the second, different input feature representation can be constrained to have different respective resolutions. For example, the first input feature representation and the second, different input feature representation can have different respective resolutions that are non-adjacent within a pyramidal structuring of the plurality of input resolutions. Thus, the new architecture of the pyramid layer can be constrained to have cross-scale connections that combine feature representations at different scales.

In some implementations, for at least some of the plurality of merging cells, the controller model can be constrained to select one of the output resolutions such that the new feature representation generated by the merging cell can form one of the plurality of output representations. Thus, the architecture of the pyramid layer can be constrained to have pre-determined output resolutions. Furthermore, in some implementations, two or more of the input resolutions can be constrained to be identical to at least two of the output resolutions. Such features can facilitate stacking of the pyramid layers.

The number of the merging cells can affect the complexity and size of the resulting pyramid architecture. The number of merging cells can be a user-defined hyperparameter, which can provide the user with increased control over the resulting pyramid architecture. However, in other embodiments, the number of merging cells can be a learnable parameter such that the size and/or complexity of the resulting pyramid architecture can optimize the desired performance characteristics (e.g., fast solving time, high accuracy, etc.) of the resulting pyramid architecture.

According to another aspect, without intervention, merging cells as described herein can result in feature representations that lack output connections with other feature representations. Such a configuration is generally not desirable as such feature representations consume resources without contributing the output of the pyramid layer. To prevent this configuration, in some implementations, the controller model can be configured to sum each feature representation that does not connect to any of the plurality of output feature representations with the output feature representation that has a corresponding resolution. Thus, the controller model can be configured to constrain or otherwise modify the pyramid layer architecture to prevent formation of feature representations that lack output connections with subsequent feature representations.

The computing system can be configured to perform a series of iterations in which the pyramid layer architecture is iteratively modified and evaluated to improve the pyramid layer architecture. For example, the computing system can receive a new pyramid layer architecture as an output of the controller model and evaluate one or more performance characteristics of a machine-learned pyramidal feature model that includes the backbone model and one or more pyramid layers that have the new pyramid layer architecture. Example performance characteristics can include accuracy, precision, solving time, number of iterations or flops, and/or combinations thereof.

The computing system can determine an outcome for the architecture based on evaluated performance of the machine-learned pyramidal feature model. As one example, in some implementations, the controller model can include a reinforcement learning agent. For each of the plurality of iterations, the computing system can be configured to determine a reward based, at least in part, on the one or more evaluated performance characteristics associated with a machine-learned pyramidal feature model that includes the backbone model and one or more pyramid layers that have the new pyramid layer architecture. The computing system can modify one or more parameters of a policy implemented by the controller model based on the reward. The controller model may include a neural network (e.g., a recurrent neural network). Thus, the controller model can be trained to design the pyramid architecture in a manner that maximizes, optimizes, or otherwise adjusts a performance characteristic associated with the resulting machine-learned pyramidal feature model.

As another example, in an evolutionary scheme, the performance of the most recently proposed architecture can be compared to a best previously observed performance to determine, for example, whether to retain the most recently proposed architecture or to discard the most recently proposed architecture and instead return to a best previously observed architecture. To generate the next iterative architecture, the controller model can perform evolutionary mutations on the model selected based on the comparison described above.

In some implementations, evaluating the one or more performance characteristics of the machine-learned pyramidal feature model can include evaluating the one or more performance characteristics of the machine-learned pyramidal feature model that includes the backbone model and a plurality of stacked pyramid layers that each have the new pyramid layer architecture. For example, during evaluation of the machine-learned model, the backbone model can take an image as input. The machine-learned model can perform object detection, object classification, and/or semantic segmentation for the image based on the plurality of output feature representations output by a final pyramid layer of the one or more pyramid layers. Thus, the performance characteristic(s) of the architecture of the pyramid layer can be evaluated and iteratively improved.

In some embodiments, the performance characteristic may be evaluated using the actual task, (e.g., the "real task") for which the pyramid architecture is being optimized or designed. For instance, the performance characteristic may be evaluated using set of images that will be used to train to the resulting model that includes the pyramid layer. However, in other embodiments, the performance characteristic may be evaluated using a proxy task that has a relatively shorter training time and also correlates with the real task. For instance, evaluating the performance characteristics using the proxy task may include using lower resolution images than the real task (e.g., down-sampled versions of the images), using a smaller version of the backbone model, and/or evaluating the real task for fewer epochs than would generally be used to train the model using the real task.

In some implementations, during evaluation of the machine-learned model, the machine-learned model can generate predictions based on the respective plurality of output feature representations output by any one of the one or more pyramid layers to perform an early exit, for example, for "anytime" object detection. In other words, during inference, the model can generate solutions (e.g., object detection information) from a pyramid layer that is internal to the backbone model. For instance, output from multiple pyramid layers within the model can contain solutions (e.g., object detection information). This property can be desirable when computation resource or latency during inference is a concern. Additionally, "anytime" object detection can be used to dynamically adjust the amount of resources to use at inference time.

The systems and methods of the present disclosure provide a number of technical effects and benefits. For example, implementations described herein can generate model architectures which have improved accuracy/latency tradeoff compared with other, manually generated, model architectures. Furthermore, stackable model architectures generated in accordance with implementations described herein can be used to dynamically adjust computational resources to use at inference time, allowing "anytime" object detection. Various example implementations described herein generate model architectures which are specifically adapted for object detection, or other visual processing tasks such as image classification or semantic segmentation.

As one example, the systems and methods of the present disclosure can be included or otherwise employed within the context of an application, a browser plug-in, or in other contexts. Thus, in some implementations, the models of the present disclosure can be included in or otherwise stored and implemented by a user computing device such as a laptop, tablet, or smartphone. As yet another example, the models can be included in or otherwise stored and implemented by a server computing device that communicates with the user computing device according to a client-server relationship. For example, the models can be implemented by the server computing device as a portion of a web service (e.g., a web email service).

<FIG> depicts a block diagram of an example computing system for producing an architecture of a pyramid layer according to example embodiments of the present disclosure. The system <NUM> includes a user computing device <NUM>, a server computing system <NUM>, and a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The user computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the user computing device <NUM> to perform operations.

The user computing device <NUM> can store or include one or more controller models <NUM>. For example, the controller model <NUM> can be or can otherwise include a machine-learned models such as neural networks (e.g., a recurrent neural network) or other multi-layer non-linear models. Neural networks can include recurrent neural networks (e.g., long short-term memory recurrent neural networks), feed-forward neural networks, or other forms of neural networks. Example controller models <NUM> are discussed with reference to Figures 3A and 3B.

In some implementations, the one or more controller models <NUM> can be received from the server computing system <NUM> over network <NUM>, stored in the user computing device memory <NUM>, and the used or otherwise implemented by the one or more processors <NUM>. In some implementations, the user computing device <NUM> can implement multiple parallel instances of a single reinforcement learning agent models.

Additionally or alternatively, one or more controller models <NUM> can be included in or otherwise stored and implemented by the server computing system <NUM> that communicates with the user computing device <NUM> according to a client-server relationship. For example, the controller model <NUM> can be implemented by the server computing system <NUM> as a portion of a web service (e.g., a reinforcement learning simulation service). Thus, one or more controller models <NUM> can be stored and implemented at the user computing device <NUM> and/or one or more controller models <NUM> can be stored and implemented at the server computing system <NUM>.

The user computing device <NUM> can also include one or more user input component <NUM> that receives user input. For example, the user input component <NUM> can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can enter a communication.

The server computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the server computing system <NUM> to perform operations.

As described above, the server computing system <NUM> can store or otherwise includes one or more controller models <NUM>. For example, the controller model <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep recurrent neural networks) or other multi-layer non-linear models. Example controller models <NUM> are discussed with reference to Figures 3A and 3B.

In some implementations, the systems and methods can be provided as a cloud-based service (e.g., by the server computing system <NUM>). Users can provide a pre-trained or pre-configured reinforcement learning agent model. The users can set or adjust inputs and/or setting to customize the simulated environment, for example to simulate a real-world environment in which the user intends to deploy the reinforcement learning agent model. The user can then simulate performance of the reinforcement learning agent model over time in the simulated environment to predict and/or optimize performance of the agent model or multiple different variants thereof in the real-world environment.

The server computing system <NUM> can train the controller models <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing system <NUM> or can be a portion of the server computing system <NUM>.

The training computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the training computing system <NUM> to perform operations. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The training computing system <NUM> can include a model trainer <NUM> that trains the controller models <NUM> stored at the server computing system <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer <NUM> can train or pre-train controller models <NUM> based on training data <NUM>. The training data <NUM> can include labeled and/or unlabeled data. For instance, the training data <NUM> can include training pyramid layer architect structures.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device <NUM> (e.g., based on communications previously provided by the user of the user computing device <NUM>). Thus, in such implementations, the controller model <NUM> provided to the user computing device <NUM> can be trained by the training computing system <NUM> on user-specific communication data received from the user computing device <NUM>. In some instances, this process can be referred to as personalizing the model.

The model trainer <NUM> includes computer logic utilized to provide desired functionality. The model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer <NUM> includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device <NUM> can include the model trainer <NUM> and the training dataset <NUM>. In such implementations, the models <NUM> can be both trained and used locally at the user computing device <NUM>. In some of such implementations, the user computing device <NUM> can implement the model trainer <NUM> to personalize the models <NUM> based on user-specific data.

<FIG> depicts a block diagram of an example computing device <NUM> that performs according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

The central intelligence layer includes a number of machine-learned models. For example, as illustrated in <FIG>, a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device <NUM>.

<FIG> depicts an example system <NUM> that is configured to perform a series of iterations in which a pyramid layer architecture is iteratively modified and evaluated to improve the pyramid layer architecture according to example embodiments of the present disclosure. The system <NUM> may include a controller model <NUM> according to example embodiments of the present disclosure. The controller model <NUM> may be configured to generate a new pyramid architecture <NUM> based on a reward <NUM>. The reward <NUM> may be based, at least in part, on one or more performance characteristics associated with a machine-learned pyramidal feature model that includes the backbone model and one or more pyramid layers that have the new pyramid layer architecture. The computing system can perform a performance evaluation <NUM> of the new pyramid architecture <NUM>, for example, by evaluating one or more performance characteristics of a machine-learned pyramidal feature model that includes the backbone model and one or more pyramid layers that have the new pyramid layer architecture <NUM>. The computing system can modify one or more parameters of the controller model <NUM> based on the reward <NUM>. Thus, the computing system can determine an outcome for the new pyramid architecture <NUM> based on the performance evaluation <NUM> of the machine-learned pyramidal feature model. As such, the controller model can be trained to design the pyramid architecture in a manner that maximizes, optimizes, or otherwise adjusts a performance characteristic associated with the resulting machine-learned pyramidal feature model.

In some implementations, evaluating the one or more performance characteristics of the machine-learned pyramidal feature model (represented by block <NUM>) can include evaluating the one or more performance characteristics of the machine-learned pyramidal feature model that includes the backbone model and a plurality of stacked pyramid layers that each have the new pyramid layer architecture. For example, during evaluation of the machine-learned model, the backbone model can take an image as input. The machine-learned model can perform object detection, object classification, and/or semantic segmentation for the image based on the plurality of output feature representations output by a final pyramid layer of the one or more pyramid layers. Thus, the performance characteristic(s) of the architecture of the pyramid layer can be evaluated and iteratively improved.

In some embodiments, the performance characteristic may be evaluated using the actual task, (e.g., the "real task") for which the pyramid architecture is being optimized or designed. For instance, the performance characteristic may be evaluated using a group of images that will be used to train to the resulting model that includes the pyramid layer. However, in other embodiments, the performance characteristic may be evaluated using a proxy task that has a relatively shorter training time and also correlates with the real task. For instance, evaluating the performance characteristics using the proxy task may include using lower resolution images than the real task (e.g., down-sampled versions of the images), using a smaller version of the backbone model, and/or evaluating the real task for fewer epochs than would generally be used to train the model using the real task.

<FIG> depicts an example system <NUM> that is configured to perform a series of iterations in which a pyramid layer architecture is iteratively modified and evaluated to improve the pyramid layer architecture according to example embodiments of the present disclosure. A controller model <NUM> may include a reinforcement learning agent <NUM> that is configured to generate the new pyramid architecture <NUM> based on the reward <NUM>. Referring to <FIG>, the controller models <NUM>, <NUM> may be configured to perform additional operations, for example as described below with reference to <FIG>.

<FIG> depicts a backbone model <NUM> and a pyramid layer <NUM> according to example embodiments of the present disclosure. The pyramid layer <NUM> can be configured to receive a plurality of input feature representations <NUM> (e.g., feature maps) output by the backbone model <NUM> (or a previous sequential pyramid layer in the event there are multiple stacked pyramid layers). The plurality of input feature representations can have a plurality of different input resolutions. The pyramid layer can be configured to perform convolutions and/or other operations with respect to the input feature representations. In particular, input feature representations can be combined with (e.g., summed, globally pooled, etc.) other feature representations (e.g., input feature representations, internal feature representations, and/or output feature representations), including combinations of two feature representations that have the same or different resolutions. The operations performed by the pyramid layer on the feature representations can result in the pyramid layer producing and outputting a plurality of output feature representations. The plurality of output feature representations can have a plurality of different output resolutions.

As described elsewhere herein and indicated by the "× N" in <FIG>, a sequence of the pyramid layers <NUM> can be stacked one after another (e.g., the input representations for a particular pyramid layer can be the output representations from a previous sequential pyramid layer and the output representations provided by the particular pyramid layer can be the input representations for a next sequential pyramid layer). The inputs to the first pyramid layer can be the features directly taken from the feature hierarchy in the backbone model <NUM>.

In some implementations, the output feature representations for a final pyramid layer (and/or an intermediate pyramid layer if anytime exit is enabled) can be provided as inputs into further networks <NUM> which may, for example, perform classification and/or regression based on the received output representations. Thus, in some implementations, additional classifier and/or box regression heads <NUM> can be attached after all the pyramid layers during training of the model. During inference, the heads <NUM> can generate detections based on the feature representations generated by the final pyramid layer (and/or an intermediate pyramid layer if anytime exit is enabled).

In particular, as regards anytime exit, one advantage of scaling with repeated pyramid layers is that the feature pyramid representations can be obtained at any given of layer. This enables anytime detection which can generate detection results at any given pyramid layer. Thus, in some implementations, classifier and box regression heads <NUM> can be attached after all pyramid layers during training. During inference, the model can generate detections from any pyramid layer. This property can be desirable when computation resource or latency is a concern during inference and provides a solution that can dynamically decide how much resource to use for generating detections.

In one example, the input features <NUM> are in <NUM> scales {C3, C4, C5, C6, C7} with corresponding stride of {<NUM>, <NUM>, <NUM>, <NUM>, <NUM>} pixels. The C6 and C7 were created by applying stride <NUM> and stride <NUM> max pooling to C5. The input features were then passed to a series of merging cells in a pyramid layer to combine features with cross-scale connections and generate augmented feature representations {P3, P4, P5, P6, P7}. Since both inputs and outputs of the pyramid layer <NUM> can have the same scales, the architecture of pyramid layer <NUM> can be replicated and concatenated repeatedly to generate the scalable model architecture. The number of pyramid layers can be controlled to trade-off speed and accuracy. More pyramid layers generally produces greater accuracy but slower performance. Fewer pyramid layers generally produces faster performance but less accuracy.

<FIG> depicts a plurality of feature representations <NUM> and a merging cell <NUM> according to example embodiments of the present disclosure. For each of a plurality of search iterations, the controller model can be configured to generate the new pyramid layer architecture by constructing and appending a plurality of merging cells <NUM> to produce the new pyramid architecture. Each merging cell <NUM> can have two input feature representations and one output feature representation. This can generate cross-scale connections between various feature representations of the pyramid layer, which can result in semantically strong feature representations.

More specifically, to construct one of the merging cells <NUM>, the controller model can be configured to select a first input feature representation <NUM> from a set of available feature representations that includes the plurality of input feature representations <NUM>. The controller model can select a second, different input feature representation <NUM> of the set of available feature representations. The controller model can select a first resolution of the plurality of different output resolutions and select an operation <NUM> that combines the first input feature representation <NUM> with the second, different input feature representation <NUM> to produce a new feature representation <NUM> with the first resolution. For example, the operation <NUM> can include a variety of suitable binary operations, such as a sum operation and a global pooling operation. The merging cell <NUM> can be configured to perform a convolution <NUM> (e.g., a 3x3 convolution). A plurality of merging cells <NUM> can be independently and/or sequentially constructed in this fashion. The plurality of merging cells <NUM> can then be appended to generate the pyramid layer architecture. Thus, the controller model can use merging cells <NUM> to generate the new pyramid layer architecture, for example as described below with reference to <FIG>.

In some implementations, for at least some of the merging cells <NUM>, the controller model can add the new feature representation to the set of available feature representations for potential selection at a next merging cell (illustrated by arrow <NUM>). Thus, in some instances, a newly created feature representation can be merged with another feature representation in the next merging cell. Such operations can facilitate discovery or learning of various cross-scale connections.

In some implementations, the first input feature representation <NUM> and the second, different input feature representation <NUM> can be constrained to have different respective resolutions. For example, the first input feature representation <NUM> and the second, different input feature representation <NUM> can have different respective resolutions that are non-adjacent within a pyramidal structuring of the plurality of input resolutions, for example as described below with reference to <FIG>. Thus, the new architecture of the pyramid layer can be constrained to have cross-scale connections that combine feature representations at different scales.

<FIG> depicts architecture graphs of successive pyramid layers generated by a controller model according to example embodiments of the present disclosure. The dots represent feature representations, and the arrows represent connections between the feature representations. Input feature layers are circled and located on the left side of each graph. Feature representations in the same row can have the same resolution. Resolution decreases in the upwards direction. For example, referring to <FIG>, the lowest row of dots <NUM> represent feature representations and potential features representations having the same resolution. The next row of dots <NUM> represent feature representations and potential features representations having the same resolution, which is lower than the resolution of dots in the lowest row <NUM>. In this example, the output feature representations may only be allowed to form connections with other output feature representations that have a greater resolution.

<FIG> illustrates a baseline, or initial pyramid structure architecture according to aspects of the present disclosure. <FIG> illustrate architectures discovered using various search iterations according to aspects of the present disclosure. In this example, a proxy task was used to evaluate the pyramid architectures. The discovered architectures converged as the reward of the proxy task progressively improves. <FIG> illustrates the final architecture used in subsequent experiments with other backbone models. The final architecture illustrated in <FIG> is also illustrated in <FIG>.

In some implementations, new feature representations generated by the pyramid layer can include one or more internal feature representations that are internal to the pyramid layer architecture. For example, referring to <FIG>, the pyramid layer architecture can include a first internal feature representation <NUM> and a second internal feature representation <NUM>. The internal feature representation(s) <NUM>, <NUM> can be distinct from the input feature representations <NUM>, <NUM>, <NUM> and the output feature representations <NUM>, <NUM>, <NUM>, <NUM>.

The internal feature representation(s) can be connected with other internal feature representation(s), input feature representations, and/or output feature representations. For example, referring to <FIG>, the first internal feature representation <NUM> can be connected with two input feature representations <NUM>, <NUM> as inputs and be connected with the second internal feature representation <NUM> as an output. The second feature representation <NUM> can be connected with each of the first internal feature representation <NUM> and one of the input feature representations <NUM> as inputs and with one of the output feature representations <NUM> as an output. At least one of the plurality of output feature representations <NUM>, <NUM>, <NUM>, <NUM> can be generated based on one or more of the internal feature representations <NUM>, <NUM>.

As indicated above, the plurality of merging cells can appended to generate the pyramid layer architecture. Referring to <FIG>, connected feature representations can represent respective appended merging cells. For example, the in one merging cell the two of the input feature representations <NUM>, <NUM> can be selected, for example as described above with reference to <FIG>. The input feature representations <NUM>, <NUM> can be combined to produce a new feature representation (e.g., the first internal feature representation <NUM>). As illustrated, the first internal feature representation <NUM> can have a different resolution than one or both of the input feature representations <NUM>, <NUM> such that cross-scale connections are created.

In some implementations, new feature representation can be available for potential selection at a next merging cell. For example, referring to <FIG>, the first internal feature representation <NUM> and one of the input feature representations <NUM> were combined to produce the second internal feature representation <NUM>. Thus, in some instances, a newly created feature representation (e.g., the first internal feature representation <NUM>) can be merged with another feature representation (e.g., input feature representations <NUM>) in the next merging cell. Such operations can facilitate discovery or learning of various cross-scale connections.

In some implementations, the first input feature representation and the second, different input feature representation can be constrained to have different respective resolutions. For example, the first input feature representation and the second, different input feature representation can have different respective resolutions that are non-adjacent within a pyramidal structuring of the plurality of input resolutions. For example, as illustrated in <FIG>, the first internal feature representation <NUM> and input feature representation <NUM> have different resolutions. Additionally, the second internal feature representation <NUM> has a different resolution than each of the first internal feature representation <NUM> and the input feature representation <NUM>. Thus, the new architecture of the pyramid layer can be constrained to have cross-scale connections that combine feature representations at different scales.

In some implementations, for at least some of the plurality of merging cells, the controller model can be constrained to select one of the output resolutions such that the new feature representation generated by the merging cell can form one of the plurality of output representations. For example, referring to <FIG>, the controller model can select one of the output feature representations <NUM> as an output for the first internal feature representation <NUM> and one of the input feature representations <NUM>. Thus, the architecture of the pyramid layer can be constrained to have pre-determined output resolutions. Furthermore, in some implementations, two or more of the input resolutions (e.g., the bottom three rows containing input feature representations <NUM>, <NUM>, <NUM> respectively) can be constrained to be identical to at least two of the output resolutions (e.g., the rows including output feature representations <NUM>, <NUM>, <NUM>). Such constraints can facilitate stacking of the pyramid layers.

For example, referring to <FIG>, if an internal feature representation <NUM> lacks connections to any of the plurality of output feature representations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, the controller model can be configured to connect the internal feature representation <NUM> with the output feature representation <NUM> that has the same output resolution.

<FIG> illustrates the pyramid layer architecture <NUM> of <FIG>. The pyramid layer architecture <NUM> was iteratively generated as described above with reference to <FIG>. Input feature representations <NUM>, <NUM>, <NUM> may be input into the pyramid layer architecture <NUM>. The input feature representations <NUM>, <NUM>, <NUM> may be combined as shown to generate the internal feature representations <NUM>, <NUM>. For example, the input feature representations <NUM>, <NUM>, <NUM> can be adjusted to the output resolution by nearest neighbor upsampling or max pooling, if needed or helpful. The merged feature map can be followed by a Rectified Linear Unit (ReLU), a 3x3 convolution, and/or a batch normalization layer, for example as illustrated. The pyramid layer architecture <NUM> may include output feature representations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>.

<FIG> depicts architecture graphs of successive pyramid layers generated by a controller model according to example embodiments of the present disclosure. In this example, in contrast to the example described above with reference to <FIG>, each output feature representations may be free to form connections with other output feature representations that have greater or lower resolutions. <FIG> illustrates the pyramid layer architecture of <FIG> in the same manner that <FIG> illustrates the pyramid layer architecture of <FIG>.

<FIG> depicts a flow chart diagram of an example method to perform according to example embodiments of the present disclosure. Although <FIG> depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement.

The computing system can perform a series of iterations in which a pyramid layer architecture is iteratively modified and evaluated to improve the pyramid layer architecture according to example embodiments of the present disclosure. For example, at <NUM>, the computing system can receive a new pyramid layer architecture as an output of a controller model that is configured to generate new architectures for a pyramid layer, for example as described above with reference to <FIG>.

At <NUM>, the computing system can evaluate one or more performance characteristics of a machine-learned pyramidal feature model that includes a backbone model and one or more pyramid layers that have the new pyramid layer architecture, for example as described above with reference to <FIG>.

At <NUM>, the computing system can determine an outcome for the new pyramid layer architecture based on the evaluated performance characteristics, for example as described above with reference to <FIG>. The computing system can then return to step <NUM>.

Claim 1:
A computing system comprising:
one or more processors;
a controller model (<NUM>, <NUM>) configured to generate new architectures (<NUM>) for a pyramid layer (<NUM>) that receives a plurality of input feature representations (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) output by a backbone model (<NUM>) and, in response, outputs a plurality of output feature representations (<NUM>, <NUM>, <NUM>, <NUM>), wherein the plurality of input feature representations have a plurality of different input resolutions and wherein the plurality of output feature representations have a plurality of different output resolutions; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
for each of a plurality of iterations:
receiving a new pyramid layer architecture as an output of the controller model;
and evaluating one or more performance characteristics of a machine-learned pyramidal feature model that comprises the backbone model and one or more pyramid layers that have the new pyramid layer architecture, wherein during evaluation of the machine-learned model, the backbone model takes an image as input and the machine-learned model performs one of object detection, object classification, or semantic segmentation for the image based on the plurality of output feature representations output by a final pyramid layer of the one or more pyramid layers;
wherein, for each of the plurality of iterations the controller model is configured to generate the new pyramid layer architecture by performing controller operations, the controller operations to construct and append a plurality of merging cells (<NUM>) to produce the new pyramid layer architecture, wherein each merging cell has at least two input feature representations (<NUM>, <NUM>) and at least one output feature representation (<NUM>), the merging cell combining the at least two input feature representations by forming cross-scale connections between the input feature representations to produce the at least one output feature representation.