NEURAL NETWORK MODELS USING PEER-ATTENTION

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing a network input using a neural network to generate a network output. In one aspect, a method comprises processing a network input sing a neural network to generate a network output, where the neural network has multiple blocks, wherein each block is configured to process a block input to generate a block output, the method comprising, for each target block of the neural network: generating attention-weighted representations of multiple first block outputs, comprising, for each first block output: processing multiple second block outputs to generate attention factors; and generating the attention-weighted representation of each first block output by applying the respective attention factors to the corresponding first block output; and generating the target block input from the attention-weighted representations; and processing the target block input using the target block to generate a target block output.

BACKGROUND

This specification relates to processing data using machine learning models.

Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

SUMMARY

This specification generally describes a system implemented as computer programs on one or more computers in one or more locations for processing a network input using a neural network to generate a network output. The neural network implements a “peer-attention” mechanism, i.e., where the outputs of one or more blocks in the neural network are processed to generate a set of attention factors that are applied to the channels of an input to another block in the neural network. A “block” refers to a group of one or more neural network layers.

According to a first aspect there is provided method performed by one or more data processing apparatus for processing a network input using a neural network to generate a network output, wherein the neural network comprises a plurality of blocks that each include one or more respective neural network layers, wherein each block is configured to process a respective block input to generate a respective block output, the method comprising, for each of one or more target blocks of the neural network: generating a target block input to the target block, comprising: receiving a respective first block output of each of one or more respective first blocks, wherein each first block output comprises a plurality of channels, wherein the first block outputs are generated by the first blocks during processing of the network input by the neural network; generating a respective attention-weighted representation of each first block output, comprising, for each first block output: receiving a respective second block output of each of one or more second blocks, wherein at least one of the second block outputs is different than the first block output, wherein the second block outputs are generated by the second blocks during processing of the network input by the neural network; processing the second block outputs to generate a respective attention factor corresponding to each channel of the first block output; and generating the attention-weighted representation of the first block output by applying each attention factor to the corresponding channel of the first block output; and generating the target block input from at least the attention-weighted representations of the first block outputs; and processing the target block input using the target block to generate a target block output.

In some implementations, processing the second block outputs to generate a respective attention factor corresponding to each channel of the first block output comprises: generating a combined representation by combining the second block outputs using a set of attention weights, wherein each attention weight corresponds to a respective second block output; processing the combined representation using one or more neural network layers to generate the respective attention factor corresponding to each channel of the first block output.

In some implementations, generating the combined representation by combining the second block outputs using the set of attention weights comprises: scaling each second block output by a function of the corresponding attention weight; and determining the combined representation based on a sum of the scaled second block outputs.

In some implementations, processing the combined representation using one or more neural network layers to generate the respective attention factor corresponding to each channel of the first block output comprises: processing the combined representation using a pooling layer that performs global average pooling over spatial dimensions of the combined representation; and processing an output of the pooling layer using a fully connected neural network layer.

In some implementations, values of the attention weights are learned during training of the neural network.

In some implementations, generating the attention-weighted representation of the first block output by applying each attention factor to the corresponding channel of the first block output comprises: scaling each channel of the first block output by the corresponding attention factor.

In some implementations, generating the target block input from at least the attention-weighted representations of the first block outputs comprises: combining the attention-weighted representations of the first block outputs using a set of connection weights, wherein each connection weight corresponds to a respective attention-weighted representation of a first block output.

In some implementations, combining the attention-weighted representations of the first block outputs using the set of connection weights comprises: scaling each attention-weighted representation of a first block output by a function of the corresponding connection weight.

In some implementations, values of the connection weights are learned during training of the neural network.

In some implementations, each block in the neural network is associated with a respective level in a sequence of levels; and for each given block that is associated with a given level that follows a first level in the sequence of levels, the given block only receives block outputs from other blocks that are associated with levels that precede the given level.

In some implementations, the target block is associated with a target level, and the target block receives: (i) a respective first block output of each first block that is associated with a level that precedes the target level, and (ii) a respective second block output of each second block that is associated with a level that precedes the target level.

In some implementations, the neural network performs a video processing task.

In some implementations, the network input comprises a plurality of video frames.

In some implementations, the network input further comprises data defining one or more segmentation maps, wherein each segmentation map corresponds to a respective video frame and defines a segmentation of the video frame into one or more object classes.

In some implementations, the network input further comprises a plurality of optical flow frames corresponding to the plurality of video frames.

In some implementations, the neural network comprises a plurality of input blocks, wherein each input block includes one or more respective neural network layers, wherein the plurality of input blocks comprise: (i) a first input block that processes the plurality of video frames, and (ii) a second input block that processes the one or more segmentation maps.

In some implementations, each block of the plurality of blocks is configured to process a block input at a respective temporal resolution.

In some implementations, each block comprises one or more dilated temporal convolutional layers having a temporal dilation rate corresponding to the temporal resolution of the block.

In some implementations, each block of the plurality of blocks is a space-time convolutional block that comprises one or more convolutional neural network layers.

In some implementations, the neural network generates the network output by processing the target block outputs.

This specification describes a neural network that implements a “peer-attention” mechanism, i.e., where the outputs of one or more blocks in the neural network are processed to generate a set of attention factors that are applied to the channels of an input to another block in the neural network. Generally, the outputs of different blocks in the neural network can encode different information at various levels of abstraction. Using peer-attention enables the neural network to focus on relevant features of the network input by integrating different information across various levels of abstraction, and can thereby improve the performance (e.g., prediction accuracy) of the neural network. Moreover, using peer-attention can enable the neural network to achieve an acceptable level of performance over fewer training iterations, thereby reducing consumption of computational resources (e.g., memory and computing power) during training.

The peer-attention mechanism can be flexible and data-driven, e.g., because the attention weights (i.e., that govern the influence that each block exerts on the attention factors applied to the input channels of each other block) are learned, and because the attention factors are dynamically conditioned on the network input. The peer-attention mechanism can therefore improve the performance of the neural network more than a conventional attention mechanism, e.g., that can be hand-engineered or hard-coded.

The neural network can perform a video processing task by processing a multi-modal input that includes: (i) a set of video frames, (ii) optical flow frames that each correspond to an apparent movement of objects between two consecutive video frames, and (iii) segmentation maps that each correspond to a respective video frame and that define a segmentation of the video frame into one or more object classes. Processing the video frames, optical flow frames, and the segmentation maps enables the neural network to learn interactions between semantic object information and raw appearance and motion features, which can improve the performance (e.g., prediction accuracy) of the neural network compared to neural networks that do not process segmentation maps.

DETAILED DESCRIPTION

FIG.1shows an example neural network system100. The neural network system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The neural network system100processes a network input102using one or more blocks arranged in levels to generate a network output104that characterizes the network input. The one or more blocks are arranged in an ordered sequence of levels such that each block belongs to only one of the levels. Each block of the one or more blocks is configured to process a block input using one or more neural network layers to generate a block output.

The neural network system100can be configured to process any appropriate network input, e.g., network input102. The network input102can have space and time dimensions. For example, the network input can include a sequence of video frames, a sequence of optical flow frames corresponding to the sequence of video frames, a sequence of object segmentation maps corresponding to the sequence of video frames, or a combination thereof. In other examples, the network input can include representations of an image (e.g., represented by an intensity value or RGB values for each pixel in the image), an audio waveform, a point cloud (e.g., generated by a lidar or radar sensor), a protein, a sequence of words (e.g., that form one or more sentences or paragraphs), a video (e.g., represented in a sequence of video frames), one or more optical flow images (e.g., generated from a sequence of video frames), a segmentation map (e.g., represented by a one-hot encoding of an integer class value per pixel in an image, or per pixel in a video frame in a sequence of video frames, where each integer class value represents a different class of object), or any combination thereof.

The neural network system100can be configured to generate any appropriate network output, e.g., network output104, that characterizes the network input. For example, the neural network output can be a classification output, a regression output, a sequence output (i.e., that includes a sequence of output elements), a segmentation output, or a combination thereof. Each level in the neural network system can include any appropriate number of blocks. The number of the blocks in each level and architectures of the blocks in each level can be selected in any appropriate way, e.g., can be received as input from a user of the system100or can be determined by an architecture search system. An example of an architecture search system for determining the respective number and architecture of blocks in each level is described in more detail with reference to PCT Application No. PCT/US2020/34267, which is incorporated by reference herein.

The neural network system100can be configured to have a variety of block types. That is, each block can have a respective combination of neural network layers, and respective neural network parameter values corresponding to the respective combination of neural network layers. A block can have any appropriate neural network architecture that enables it to perform its described function, i.e., processing a block input to generate a block output that characterizes the block input. In particular, a block can include any appropriate types of neural network layers (e.g., fully-connected layers, attention-layers, convolutional layers, etc.) in any appropriate numbers (e.g., 1 layer, 5 layers, or 25 layers), and connected in any appropriate configuration (e.g., as a linear sequence of layers).

For example, the system can have a variety of input blocks for level1(e.g., to process a variety of corresponding network input types), a variety of intermediate blocks, and one or more output blocks for the final level (e.g., to generate a variety of network outputs).

Each block can be a space-time convolutional block, i.e., a block that includes one or more convolutional neural network layers and that is configured to process a space-time input to generate a space-time output. Space-time data refers to an ordered collection of numerical values, e.g., a tensor of numerical values, which includes multiple spatial dimensions, a temporal dimension, and, optionally, a channel dimension. Each block can generate an output having a respective number of channels. Each channel can be represented as an ordered collection of numerical values, e.g., a 2D array of numerical values, and can correspond, e.g., to one of multiple filters in an output convolutional layer in the block.

Each block can include, e.g., spatial convolutional layers (i.e., having convolutional kernels that are defined in the spatial dimensions), space-time convolutional layers (i.e., having convolutional kernels that are defined across the spatial and temporal dimensions), and temporal convolutional layers (i.e., having convolutional kernels that are defined in the temporal dimension). Each block of the plurality of blocks can be, e.g., configured to process a block input at a respective temporal resolution.

Each block can include, e.g., one or more dilated temporal convolutional layers (i.e., having convolutional kernels that are defined in the temporal dimension, with a dilation factor equal to one for normal temporal convolutional layers, or with a dilation factor greater than one for dilated temporal convolutional layers). Each block's temporal dilation rate can correspond to the temporal resolution of the block.

The system described herein is widely applicable and is not limited to one specific implementation. However, for illustrative purposes, a small number of example implementations are described below.

In some implementations, the neural network can be configured to perform a video processing task. In these implementations, the neural network can process a network input that includes a sequence of multiple video frames, and optionally other data as well, e.g., a sequence of optical flow frames corresponding to the sequence of video frames, a respective segmentation map (e.g., including a class value for each pixel in the video frame) generated from each of the one or more video frames, or both.

In one example, the video processing task is an action classification task where the neural network generates an action classification output that includes a respective score for each action in a set of possible actions. The score for an action can characterize a likelihood that the video frames depict an agent, e.g., a person, an animal, or a robot, performing the action, e.g., running, walking, etc. In some cases, the action classification output includes a respective score for each action in a respective set of possible actions related to each of multiple classes of objects. The score for an action related to a particular object can characterize a likelihood that the video frames depict an agent, e.g., a person, an animal, or robot, performing the action with the object, e.g., that the agent is reading a book, speaking on a phone, riding a bicycle, driving a car, etc.

In another example, the video processing task is a super resolution task, e.g., where the neural network generates an output sequence of video frames having a higher spatial and/or temporal resolution than the input sequence of video frames.

In another example, the video processing task is an artefact removal task, e.g., where the neural network generates an output sequence of video frames that are an enhanced version of the input sequence of video frames that exclude one or more artefacts present in the input sequence of video frames.

In some implementations, the neural network can be configured to process an image to generate an object recognition output that includes a respective score for each object class in a set of possible object classes. The score for an object class can characterize a likelihood that the image depicts an object in the object class, e.g., a road sign, a vehicle, a bicycle, etc.

In some implementations, the neural network can be configured to process one or more medical images (e.g., magnetic resonance images (MRIs), computed tomography (CT) images, ultrasound (US) images, or optical coherence tomography (OCT) images) of a patient, to generate a network output characterizing the medical images. The network output can include, e.g.: (i) a respective referral score for each of a plurality of referral decisions that represents a predicted likelihood that the referral decision is the most appropriate referral decision for the patient, (ii) a respective condition score for each of one or more medical conditions that represents a predicted likelihood that the patient has the medical condition, (iii) a respective progression score for each of one or more condition states that represents a predicted likelihood that a state of a corresponding medical condition will progress to the condition state at a particular future time, and/or (iv) a respective treatment score for each of a plurality of treatments that represents a predicted likelihood that the treatment is the best treatment for the patient.

In some implementations, the neural network can be configured to process an observation (e.g., including one or more of an image, a sequence of video frames, a sequence of optical flow frames, etc.) characterizing a state of an environment to generate an action selection output that includes a respective score for each action in a set of possible actions that can be performed by the agent. The action to be performed by the agent can be selected using the action selection output, e.g., by selecting the action having the highest score. The agent can be, e.g., a mechanical or robotic agent interacting with a real-world environment, or a simulated agent interacting with a simulated environment.

Generally, the neural network system100has more than one block level. Each block level can have one or more blocks, and each block can include different neural network layer types. The neural network system100can include a variety of input blocks in level1(e.g., block110a, block110b, block110c, and so on) to process network input102, a variety of blocks in intermediate levels2through N-1(e.g., blocks120a,120b,120c, . . . in level2, blocks130a,130b,130c, . . . in level3, and so on) to further process the block outputs from the input blocks, and an output block (e.g., block140) in a final level N to generate network output104. For example, the neural network system100can have a level1which includes a variety of input blocks to process a variety of input types, such as an input block to process raw RGB video input, an input block to process optical flow data characterizing the RGB video input, and an input block to process a segmentation map (e.g., generated for each of the video frames in the raw RGB video input). Each block input modality can be fed to multiple input blocks, e.g., a single raw RGB video input can go to multiple input blocks configured to process raw RBG video input.

The neural network can perform a machine learning task by processing a multi-modal input. For example, the neural network can perform a video processing task by processing (i) a set of video frames, and (ii) a respective segmentation map for each of the video frames that define a segmentation of the video frame into one or more object classes. The video processing task can include, e.g., an action classification task, e.g., identifying that an agent in the scene (e.g., a person, an animal, or a robot) is performing an action related to one of the object classes, e.g., reading a book, driving a car, riding a bicycle, or speaking on a phone. Processing both the video frames and the segmentation maps can enable the neural network to learn interactions between semantic object information and raw appearance and motion features, which can improve the performance (e.g., prediction accuracy) of the neural network compared with neural networks that do not process segmentation maps.

The neural network system100processes the network input using input blocks in the first level, and generates the block input for each block in each level after the first level by processing the block output of one or more respective blocks from preceding levels. Generally, for each given block that is associated with a given level that follows the first level in the sequence of levels, the given block only receives block outputs from other blocks that are associated with levels that precede the given level. The connections between blocks are shown using arrows inFIG.1. That is, the arrows shown represent that the output of one block is provided to another block. For example, to generate the block input for target block130b, the system can process the block output from block110a, block110b, and block120c. The connections between blocks can skip levels, such as block output from block110ccontributing to the block input for target block140.

Each block output includes a set of channels. A channel can be represented by an ordered collection of numerical values, e.g., a vector or matrix of numerical values. For example, a block output can have multiple output channels, each output channel in the block output corresponding to a different convolutional filter in the block.

The system100can generate the respective block input for some or all of the blocks after the first level using “peer-attention.” The system implements peer-attention using an attention factor engine106, as will be discussed in more detail below with reference toFIG.2.

The neural network system100has a set of neural network parameters. The system can update the neural network parameters using the training engine108.

The training engine108can train the neural network system100using a set of training data. The set of training data can include multiple training examples, where each training example specifies: (i) a training input to the neural network, and (ii) a target output that should be generated by the neural network by processing the training input. For example, each training example can include a training input that specifies a sequence of video frames and/or a corresponding sequence of optical flow frames, and a target classification output, e.g., that indicates an action being performed by a person depicted in the video frames. The training engine108can train the neural network system100using any appropriate machine learning training technique, e.g., stochastic gradient descent, where gradients of an objective function are backpropagated through the neural network at each of one or more training iterations. The objective function can be, e.g., a cross-entropy objective function, or any other appropriate objective function.

It will be appreciated that the neural network system100can be trained for video processing tasks other than classification tasks by a suitable selection of training data and/or loss function. For example, the neural network system100can be trained for super resolution (in the spatial and/or temporal domain) using a training set comprising down-sampled videos and corresponding higher-resolution ground-truth videos, with a loss function that compares output of the neural network to a higher-resolution ground-truth video corresponding to the down-sampled video input to the neural network, e.g. an L1 or L2 loss. As a further example, the neural network system100can be trained to remove one or more types of image/video artefact from videos, such as blocking artefacts that can be introduced during video encoding. In this example, the training dataset can include a set of ground truth videos, each with one or more corresponding “degraded” videos (i.e. with one or more types of artefact introduced), with a loss function that compares output of the neural network system100to a ground-truth video corresponding to the degraded video input to the neural network system100, e.g. an L1 or L2 loss.

FIG.2shows a diagram of an example data flow200illustrating the operations performed by a neural network system implementing peer-attention to generate the block input for a block, referred to for convenience as a “target” block, in any level after the first level. That is, a target block can refer to any block after the first level of blocks. An example of a neural network system, e.g., neural network system100, that can perform the operations of data flow200is described in more detail above with reference toFIG.1.

The system generates the target block input for a target block by processing a respective block output of each of one or more other blocks to generate a combined representation of the respective block outputs. The target block can then process the combined representation as the target block input to generate a target block output.

The system receives a respective block output of each of one or more “first” blocks (e.g., first block outputs204a,204b, and204cfrom blocks202a,202b, and202c, respectively), where each first block can come from any level preceding the target level of the target block. (For convenience, each block that provides a block output to the target block will be referred to as a first block.) The first block outputs each include multiple channels, and each is generated by a respective first block during processing of a network input, e.g., the network input102ofFIG.1. For example, each channel in a respective first block output can correspond to a filter in a convolutional layer in the respective first block.

For each first block output, the system generates a respective attention factor for each channel of the first block output by processing a respective block output of each of one or more “second” blocks, where at least one of the respective second blocks is different from the first block. (For convenience, each block that generates a block output that is used for generating attention factors to be applied to the channels of a first block output will be referred to as a “second” block). Each second block output comes from a block in a level preceding the target level of the target block. Generally, the set of second block outputs processed to generate the attention factors for one first block output can be different from the set of second blocks processed to generate the attention factors for another first block output.

The system can generate the respective attention factors from the one or more second block outputs using an attention factor engine106. For example, the attention factor engine can generate a combined representation of the respective second block outputs, and process the combined representation to generate the respective attention factors, as is discussed in further detail with reference toFIG.4. With reference toFIG.2, the respective second block outputs processed to generate respective attention factors208afor first block output204aare shown (i.e., second block outputs206a,206b, and206c). For convenience, the respective second block outputs processed to generate attention factors208b(i.e., for first block output204b) and the respective second block outputs processed to generate attention factors208c (i.e., for first block output204c) are omitted from the diagram. An attention factor can be represented by a numerical value, e.g., a floating point numerical value. A set of attention factors for a block output can be represented by a collection of ordered numerical values (e.g., a vector of floating point numerical values), where each value corresponds to a channel of the block output.

For each first block output, the system generates an attention-weighted representation of the first block output. The system can generate the attention-weighted representation of the first block output by applying each attention factor to the corresponding channel of the first block input. For example, the system can generate the attention-weighted representation by scaling each channel of the first block output by the corresponding attention factor. With reference toFIG.2, the system applies attention factors208ato first block output204ato generate attention-weighted representation210a, attention factors208bto first block output204bto generate attention-weighted representation210b, and attention factors208cto first block output204cto generate attention-weighted representation210c.

The system generates the target block input214by processing the attention-weighted representations210a,210b, and210c. The system can generate the target block input214by generating a combined representation of the attention-weighted representations. For example, the system can generate a weighted sum of the attention-weighted representations using a set of connection weights212, as is discussed in further detail with reference toFIG.3. With reference toFIG.2, the system generates the target block input214by scaling each attention-weighted representation by a function of the corresponding weight in the connection weights212, then summing the scaled attention-weighted representations.

The target block216processes the target block input214to generate a target block output218that characterizes the target block input214. Generally, the target block output218has multiple channels. In some cases, the target block output218can be processed as either a respective first block output, a respective second block output, or both, for one or more target blocks in subsequent levels. In another case, the target block216can process the target block214such that the target block output218is the network output.

FIG.3is a flow diagram of an example process for generating the target block input for a target block. For convenience, the process300will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system100ofFIG.1, appropriately programmed in accordance with this specification, can perform the process300.

The system receives a respective first block output of one or more first blocks (302). Each first block can be from any level preceding the target level of the target block. For example, for a target block in level5of a neural network system, the system can receive respective first block outputs from first blocks in levels1,2,3,4, or any combination thereof

For each first block output, the system implements a “peer-attention” mechanism, i.e., where the outputs of one or more second blocks (where at least one of the second blocks is different from the first block) in the neural network are processed to generate a set of attention factors that are applied to the channels of the first block output, as is described in steps304-306. For convenience, a second block providing output to generate the attention factors for a first block output will be referred to as an “attention connection.”

For each first block output, the system receives a respective second block output of each of one or more second blocks (304), where at least one of the second blocks is different than the first block. Each second block can be in any level preceding the target level of the target block. For example, for a target block in level5and a first block in level2, a second block can be in levels1,2,3, or4.

For each first block output, the system generates respective attention factors (306). The system can generate an attention factor for each channel of the first block output by processing the one or more second block outputs. For example, the system can generate a combined representation of the one or more second block outputs, and process the combined representation using one or more neural network layers to generate the attention factors for the first block output, as is discussed in further detail with reference toFIG.4.

Generally, the outputs of different blocks in the neural network can encode different information at various levels of abstraction. Using peer-attention enables the neural network to focus on relevant features of the network input by integrating different information across various levels of abstraction, and can thereby improve the performance (e.g., prediction accuracy) of the neural network. Moreover, using peer-attention can enable the neural network to achieve an acceptable level of performance over fewer training iterations, thereby reducing the consumption of computational resources (e.g., memory and computing power) during training.

For each first block output, the system generates an attention-weighted representation of the first block output (308). The system can generate an attention-weighted representation of the first block output by applying each attention factor to the corresponding channel of the first block output. For example, the system can scale each channel of the first block output by the corresponding attention factor using an elementwise multiplication, as

where j indexes the first blocks, Xjattrepresents the attention-weighted representation of the first block output, Ajrepresents the attention factors corresponding to the first block output, and Xjoutrepresents the respective first block output of the first block j.

The system generates the target block input for a target block based at least in part on the attention-weighted representations of the first block outputs (310). For example, the system can generate the target block input based on a weighted sum of the attention-weighted representations of the first block outputs using a set of connection weights, e.g., connection weights212inFIG.2, as

where i indexes the target block, j indexes the first blocks, Xiinrepresents the target block input, Xjattrepresents the attention-weighted representation of the first block output of first block j, σ(.) represents the sigmoid function, wjirepresents the connection weight from block j to block i, and P(i) returns all j for first blocks contributing to the target block i. The connection weights are learnable parameters which can be trained, e.g., by training engine108ofFIG.1.

Generally, any block can receive a block output from any block in a preceding level, and the blocks can be connected in any appropriate way. In some implementations, the blocks can be initially fully connected, i.e., such that each block in each level provides its block output to each block in each subsequent level. During training of the neural network, the respective connection weight associated with each block connection is trained, and optionally, some of the block connections can be removed (“pruned”) during or after training. For example, the system can optionally remove any connections having a connection weight that is less than a predefined value, or the system can remove a predefined number of connections having connection weights with the lowest values.

FIG.4is a flow diagram of an example process for generating the attention factors for a respective first block output. For convenience, the process400will be described as being performed by a system of one or more computers located in one or more locations. For example, an attention factor engine, e.g., the attention factor engine106ofFIG.1, appropriately programmed in accordance with this specification, can perform the process400.

The system receives a respective second block output of each of one or more second blocks (402). Each second block can be from any level preceding the target level of the target block. For example, if the target block is from level3, the second blocks can be from level1, level2, or a combination of the two.

The system scales each second block output by a function of a corresponding attention weight (404). The corresponding attention weights are learnable parameters which can be trained, e.g., by training engine108ofFIG.1, and each attention weight corresponds to a second block output. In one example, the system can apply a softmax function to the attention weights corresponding to each second block output, then scale each second block output by the corresponding attention weight output by the softmax function. Using a softmax function can emphasize the contribution of the most impactful second block or blocks.

The system generates a combined representation of the scaled second block outputs (406). For example, the system can represent the combined representation as,

where i indexes the target block, k indexes the second blocks, Xcomrepresents the combined representation of second block outputs, Xkoutrepresents a respective second block output of a second block k, H represents a vector including a respective attention weight for each second block output, softmaxk(H) represents the k-th component of the softmax of the vector H, and Q(i) return all k for second blocks contributing to the combined representation. The attention weights are learnable parameters which can be trained, e.g., by training system108ofFIG.1.

Generally, any block can receive a second block output from any number of second blocks in preceding levels, i.e., by respective attention connections, for use in generating an attention-weighted representation of a first block output. In some implementations, the system can initialize the blocks as fully connected with attention connections, i.e., such that for any block that processes a block input generated by peer-attention, the attention-weighted representation of each first block output is generated using every feasible second block output. During training of the neural network, the respective attention weights associated with each attention connection are trained, and optionally, some of the attention connections can be removed (“pruned”) during or after training. For example, the system can optionally remove any attention connections having an attention weight that is less than a predefined value, or the system can remove a predefined number of attention connections with the lowest attention weight values.

The peer-attention mechanism can be flexible and data-driven, e.g., because the attention weights are learned, and because each the attention factors are dynamically conditioned on the network input. The peer-attention mechanism can therefore improve the performance of the neural network more than a conventional attention mechanism, e.g., that can be hand-engineered or hard-coded.

The system generates the attention factors by processing the combined representation using one or more neural network layers (408). For example, the system can process the combined representation using a global average pooling layer over the spatial dimensions of each channel, followed by a fully-connected layer, and an elementwise sigmoid function, as

where j indexes the first blocks, Ajrepresents the attention factors for the first block j, σ(.) represents the elementwise sigmoid function, f represents the fully connected neural network layer, GAP(.) represents the global average pooling, and Xcomrepresents the combined representation of the second block outputs. The fully connected layer outputs a vector with a number of elements equal to the number of channels of the corresponding first block output.