Patent Description:
Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output.

<NPL>" describes a Dynamic Filter Network, where filters are generated dynamically conditioned on an input. A dynamic filter module consists of a filter generating network that produces filters conditioned on an input, and a dynamic filtering layer that applies the generated filters to another input.

This specification describes a neural network system implemented as computer programs on one or more computers in one or more locations that includes one or more conditional neural network layers.

By including one or more conditional neural network layers as described in this specification, a neural network can generate network outputs with an accuracy comparable to (or higher than) some conventional neural networks while consuming fewer computational resources (e.g., memory and computing power). For example, the neural network can dynamically determine the values of the conditional layer weights used to process a conditional layer input, unlike some conventional neural network layers where the values of the layer weights are fixed during inference. This can enable the conditional neural network layer to effectively increase the model complexity of the neural network to achieve higher accuracy levels while, in some cases, minimally affecting computational resource consumption by the neural network.

Moreover, a conditional neural network layer as described in this specification can dynamically determine the values of the conditional layer weights used to process a conditional layer input from amongst an infinite set of possible conditional layer weights. In contrast, for some conventional neural network layers, even when the values of the layer weights can be dynamically determined from the layer inputs, they can only be selected from a finite set of possible layer weights. Compared to these conventional neural network layers, the conditional neural network layer as described in this specification allows a greater increase in model complexity while, in some cases, minimally affecting computational resource consumption by the neural network.

The operations performed by the conditional neural network layer described in this specification to dynamically determine the conditional layer weights from the conditional layer input are differentiable. Therefore, the neural network can be trained from end-to-end using gradients of an objective function with respect to the neural network parameters. In particular, the differentiability of the operations performed by conditional layers can enable the neural network to be trained more effectively than some conventional neural network layers that dynamically select layer weights from a finite set of possible layer weights using non-differentiable operations.

This specification describes a neural network having a set of neural network layers, where one or more of the neural network layers are "conditional" neural network layers.

A conditional layer is configured to receive a layer input, and to process the layer input in accordance with the values of a set of conditional layer weights to generate a layer output. The conditional layer is "dynamic", that is, the conditional layer dynamically determines the values of the conditional layer weights based on the layer input. More specifically, to determine the values of the conditional layer weights, the conditional layer uses a differentiable mapping parameterized by a set of "decision parameters" to project the layer input onto one or more "latent parameters". The latent parameters collectively specify the conditional layer weights, by parametrizing the conditional layer weights by one or more B-splines. Dynamically selecting the conditional layer weights based on the layer input can increase the representational capacity of the conditional layer and enable the conditional layer to generate richer layer outputs. Moreover, the conditional layer determines the conditional layer weights using an end-to-end differentiable procedure, which facilitates training the conditional neural network (e.g., using backpropagation techniques) to generate accurate prediction outputs.

In some implementations, the conditional layers of the neural network may be "hierarchical" as well as dynamic, that is, for one or more of the conditional layers, the neural network may condition the decision parameters for the conditional layer on the latent parameters of a preceding conditional layer. Being hierarchical may further increase the representational capacity of the conditional layers, thereby enabling the conditional layers to generate richer layer outputs that can result in the neural network generating more accurate prediction outputs.

The methods and systems described herein may be applied to classifying (or otherwise characterizing) image and/or audio data. Accordingly, the neural network may be an image classifier or an audio classifier, for instance, for use in speech recognition. These features and other features are described in more detail below.

<FIG> shows an example neural network system <NUM>. The neural network system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The neural network system <NUM> includes a neural network <NUM>, which can be a feed-forward neural network, a recurrent neural network, or any other appropriate type of neural network. The neural network <NUM> is configured to receive a network input <NUM> and to generate a network output <NUM> from the network input <NUM>. The network input <NUM> can be any kind of digital data input, and the network output <NUM> can be any kind of score, classification, or regression output based on the input.

The system <NUM> described herein is widely applicable and is not limited to one specific implementation. However, for illustrative purposes, a small number of example implementations are described below.

In one example, the input to the neural network <NUM> may be an image or features that have been extracted from an image. In this example, the output generated by the neural network <NUM> for the image may be a respective score for each of a set of object categories, with the score for each category representing an estimated likelihood that the image depicts an object belonging to the category.

In another example, the input to the neural network <NUM> may be a sequence of text in one (natural) language, and the output generated by the neural network <NUM> may be a score for each of a set of pieces of text in another language. The score for each piece of text in the other language may represent an estimated likelihood that the piece of text is a proper translation of the input text into the other language.

In another example, the input to the neural network <NUM> may be a sequence representing a spoken utterance, and the output generated by the neural network <NUM> may be a respective score for each of a set of pieces of text. The score for each piece of text may represent an estimated likelihood that the piece of text is the correct transcription of the utterance.

In another example, the input to the neural network <NUM> may be a sequence of physiological measurements of a user, and the output generated by the neural network <NUM> may be a respective score for each of a set of possible diagnoses for the condition of the user. The score for each diagnosis may represent an estimated likelihood that the diagnosis is accurate.

In another example, the input to the neural network <NUM> may be a sequence of text from a communication received from a user, and the output generated by the neural network <NUM> may be a respective score for each of a set of possible responses to the received communication. The score for each response may represent an estimated likelihood that the response matches the intention of the user.

The neural network <NUM> includes one or more conditional neural network layers (e.g., the conditional layer <NUM>), and may optionally include one or more other neural network layers (i.e., that differ from the conditional layers described in this document).

As will be described further with reference to <FIG>, the conditional layer <NUM> is configured to receive a layer input <NUM>, and to process the layer input <NUM> in accordance with the values of the conditional layer weights to generate a layer output <NUM>. Generally, the layer input <NUM> may be the network input <NUM> (i.e., if the conditional layer <NUM> is an input layer in the neural network <NUM>) or the output of another layer of the neural network <NUM> (e.g., another conditional layer). The layer input <NUM> and the layer output <NUM> may be represented as ordered collections of numerical values, e.g., vectors or matrices of numerical values.

The system <NUM> can be implemented in a resource-constrained environment (e.g., a mobile device) more readily than conventional neural network systems. For example, by including conditional layers (e.g., the conditional layer <NUM>), data defining the parameters of the system <NUM> can occupy much less storage capacity than data defining the parameters of a conventional neural network system.

<FIG> illustrates a block diagram of an example conditional layer <NUM>. The conditional layer <NUM> is configured to process the layer input <NUM> in accordance the values of a set of conditional layer weights <NUM> to generate the layer output <NUM>. In one example, the conditional layer weights may be fully-connected layer weights, e.g., represented by a point in <MAT>, where M is the dimensionality of the layer output <NUM> and N is the dimensionality of the layer input. In this example, the conditional layer <NUM> may generate the layer output <NUM> by applying an M × N weight matrix defined by the conditional layer weights to the layer input <NUM>. In another example, the conditional layer weights may be convolutional filter weights, e.g., two-dimensional (<NUM>-D) convolutional filter weights that are represented by a point in <MAT>, where h is the height and w is the width of the 2D convolutional filters, c is the number of input channels, and f is the number of filters. In this example, the conditional layer <NUM> may generate the layer output <NUM> by applying each of the h × w × c convolutional filters to the layer input <NUM>.

Prior to processing the layer input <NUM> using the conditional layer weights <NUM>, the conditional layer <NUM> is configured to dynamically determine the conditional layer weights <NUM> based on the layer input <NUM>. To determine the conditional layer weights <NUM>, the conditional layer <NUM> uses a set of decision parameters <NUM> to project (i.e., map) the conditional layer input <NUM> onto a set of one or more latent parameters <NUM> that implicitly specify the values of the conditional layer weights <NUM>. The latent parameters <NUM> parametrize the values of the conditional layer weights as a B-spline (or a sum of B-splines), as will be described in more detail below.

Each of the latent parameters <NUM> is a continuous variable, i.e., can assume any value in a continuous range of possible latent parameter values, e.g., the continuous interval [<NUM>,<NUM>]. In some cases, different latent parameters may have different continuous ranges of possible values, e.g., one latent parameter may have [<NUM>,<NUM>] as its continuous range of possible values, while another latent parameter may have [<NUM>,<NUM>] as its continuous range of possible values. Projecting the layer input <NUM> onto latent parameters <NUM> having a continuous range of possible values enables the conditional layer <NUM> to select a particular set of conditional layer weights from a set of infinitely many possible conditional layer weights. This enables the conditional layer to have a higher representational capacity than, e.g., a neural network layer with predetermined layer weights, or a "finite" conditional neural network layer that selects the conditional layer weights from a finite set of possible layer weights.

In some cases, the decision parameters <NUM> of the conditional layer may have static values that are determined during training of the neural network, e.g., by iterative optimization using stochastic gradient descent. In other cases, rather than having static values, the values of the decision parameters <NUM> may vary depending on the network input being processed by the neural network. In one example, the conditional layer <NUM> may determine the values of the decision parameters <NUM> based on the latent parameter values of a preceding conditional neural network layer, as will be described in more detail with reference to <FIG>.

The conditional layer <NUM> may project the layer input <NUM> onto the latent parameters <NUM> using any appropriate differentiable mapping. A few examples follow.

In one example, the conditional layer <NUM> may have a single latent parameter φ that is the result of an inner product (i.e., dot product) between the layer input x and the decision parameters θ, i.e.,: <MAT> where <·,·> refers to the inner product operation, σ is a sigmoid activation function that maps the values of the latent parameter into the range [<NUM>,<NUM>], and the layer input x and the decision parameters θ are both flattened into vectors before being processed using the inner product operation.

In another example, the conditional layer <NUM> may have multiple latent parameters, each of which are determined as the result of an inner product between the layer input and a respective subset of the decision parameters, e.g., as described with reference to equation (<NUM>).

In another example, the conditional layer <NUM> may have a single latent parameter that is the result of processing the layer input <NUM> using a <NUM> × <NUM> convolutional filter, followed by a sigmoid activation function and a global averaging operation, i.e., that averages every component of the output of the convolutional layer. In this example, the decision parameters define the values of the components of the <NUM> × <NUM> convolutional filter of the convolutional layer, e.g., by a vector having a dimensionality equal to the number of channels of the layer input.

In another example, the conditional layer <NUM> may have multiple latent parameters, each of which are determined as the result of processing the layer input <NUM> using a respective <NUM> × <NUM> convolutional filter followed by a sigmoid activation function and a global averaging operation.

After determining the latent parameter values <NUM>, the conditional layer <NUM> uses the latent parameter values <NUM> to determine the values of the conditional layer weights <NUM>. Generally, the latent parameters <NUM> parameterize the conditional layer weights by one or more differentiable functions. That is, the conditional layer <NUM> determines the values of the conditional layer weights as the result of applying one or more differential functions to the latent parameter values <NUM>. The differentiable functions may be, e.g., polynomial functions or piecewise polynomial functions of any appropriate degree. A piecewise polynomial function refers to a function having a domain that is partitioned into multiple sub-domains, such that the function is specified by a respective (possibly different) polynomial on each sub-domain.

The latent parameters <NUM> parameterize the conditional layer weights by one or more B-splines. A B-spline (or basis spline) is a piecewise polynomial parametric function with bounded support and a specified level of smoothness up to Cd-<NUM>, where d is the degree of the B-spline, that approximately interpolates a set of control points ("knots"). More specifically, a B-spline S in <MAT> parameterized by a latent parameter φ can be represented as: <MAT> where <MAT> are the control points in <MAT>, and each Bk(·) is a piecewise polynomial function of the form: <MAT> where <MAT> are coefficients in <IMG> that can be determined from continuity and differentiability constraints on the B-spline. A B-spline having a specified level of smoothness (i.e., that can be differentiated a certain number of times) may be uniquely specified by the control points <MAT>. B-splines have the property that changes to each control point only changes the B-spline locally. This makes it easier to optimize the B-spline through adapting the position of each control point.

The latent parameters may parameterize the conditional layer weights in any of a variety of ways. A few examples follow.

In one example, the conditional layer may have one latent parameter that parametrizes a B-spline having control points in <MAT>, where d is the number of conditional layer weights. In this example, the conditional layer may determine the values of the conditional layer weights as the position on the B-spline specified by the value of the latent parameter.

In another example, the conditional layer may have one latent parameter that parameterizes multiple B-splines having control points in <MAT>, where d is the number of conditional layer weights. In this example, the conditional layer determines the values of the conditional layer weights as the sum of the respective positions specified on each B-spline by the value of the latent parameter.

In another example, the conditional layer weights may include multiple convolutional filters, and the conditional layer may have a respective latent parameter corresponding to each convolutional filter. In this example, the respective latent parameter corresponding to each convolutional filter may parametrize a respective B-spline having control points in <MAT>, where b is the number of conditional layer weights specifying the convolutional filter. The conditional layer may determine the values of the conditional layer weights specifying each convolutional filter as the position on the corresponding B-spline specified by the value of the corresponding latent parameter. That is, the conditional layer may determine the convolutional filters ω as: <MAT> where f is the number of convolutional filters, <MAT> are the B-splines, <MAT> are the latent parameters, and ⊕ is a stacking operator that stacks the convolutional filters.

In another example, the conditional layer weights may include multiple convolutional filters, and the conditional layer may have a latent parameter that parametrizes a B-spline in <MAT>, where a is the number of convolutional filters. In this example, the conditional layer determines the values of a "nested" latent variables, one for each convolutional filter, as the respective position on the B-spline that is specified by the value of the latent parameter. Each nested latent parameter may parameterize a respective B-spline in <MAT>, where b is the number of conditional layer weights specifying each convolutional filter. The conditional layer may determine the values of the conditional layer weights specifying each convolutional filter as the position on the corresponding B-spline specified by the value of the corresponding nested latent parameter.

For illustrative purposes, in the conditional layer <NUM> depicted in <FIG>, the latent parameter <NUM> specifies the position <NUM> on the B-spline <NUM>. For illustrative purposes only, the B-spline is depicted as being two-dimensional, i.e., in <MAT>.

The conditional layer <NUM> uses the values of the conditional layer weights <NUM> to process the layer input <NUM> to generate the layer output <NUM>. The layer output <NUM> may subsequently be provided to a subsequent layer of the neural network, or the layer output <NUM> may be an output of the neural network.

<FIG> is an illustration of an example B-spline <NUM> in <MAT> that is specified by control points indicated as circles, e.g., the circle <NUM>. A B-spline (or another appropriate differentiable mapping) can represent an embedding of possible values of the conditional layer weights of a conditional layer on a low-dimensional (e.g., one-dimensional) manifold, as described with reference to <FIG>. Each point on the manifold (in the case of <FIG>, each point on the B-spline <NUM>) represents a possible set of values of the conditional layer weights.

<FIG> is a block diagram that illustrates the conditional layers <NUM>-A and <NUM>-B, where the neural network determines the decision parameters <NUM>-B of the conditional layer <NUM>-B based on the latent parameters <NUM>-A of the preceding conditional layer <NUM>-A.

The conditional layer <NUM>-A processes the layer input <NUM>-A in accordance with values of the conditional layer weights <NUM>-A to generate the layer output <NUM>-A, in the same manner as described with reference to <FIG>. That is, the conditional layer <NUM>-A uses the decision parameters <NUM>-A to project the layer input <NUM>-A onto one or more latent parameters <NUM>-A that parameterize the conditional layer weights <NUM>-A by one or more B-spline <NUM>-A. The conditional layer <NUM>-A determines the values of the conditional layer weights <NUM>-A as the result of applying the B-splines <NUM>-A to the latent parameters <NUM>-A, and then uses the conditional layer weights <NUM>-A to generate the layer output <NUM>-A.

In addition to parameterizing the conditional layer weights <NUM>-A, the latent parameters <NUM>-A of the conditional layer <NUM>-A also parametrize the decision parameters <NUM>-B of the subsequent conditional layer <NUM>-B by one or more differentiable functions. The neural network determines the values of the decision parameters <NUM>-B of the subsequent conditional layer <NUM>-B as the result of applying the differentiable functions to the latent parameter values <NUM>-A of the conditional layer <NUM>-A. As before, the differentiable functions may be, e.g., polynomial functions or piecewise polynomial functions, e.g., the B-spline <NUM>-A, and the latent parameters <NUM>-A may parametrize the decision parameters <NUM>-B similarly to how they parametrize the conditional layer weights <NUM>-A.

The conditional layer <NUM>-B uses the decision parameters <NUM>-B to project the layer input <NUM>-A onto one or more latent parameters <NUM>-B that parametrize the conditional layer weights <NUM>-B of the conditional layer <NUM>-B by one or more B-spline <NUM>-B. The conditional layer <NUM>-B determines the values of the conditional layer weights <NUM>-B as the result of applying the B-splines <NUM>-B to the latent parameters <NUM>-B, and then uses the conditional layer weights <NUM>-B to generate the layer output <NUM>-B.

In some implementations, the neural network may directly condition the latent parameter values <NUM>-B of the conditional layer <NUM>-B on the latent parameter values <NUM>-A of the preceding conditional layer <NUM>-A. In one example, the neural network may determine the latent parameter values φi+<NUM> of the conditional layer <NUM>-B as: <MAT> where α is a hyper-parameter in the interval [<NUM>,<NUM>], φi are the latent parameter values of the conditional layer <NUM>-A, and D(xi+<NUM>; θi+<NUM>) is the projection of the layer input <NUM>-A of the conditional layer <NUM>-B using the decision parameters <NUM>-B of the condition layer <NUM>-B, e.g., as described with reference to equation (<NUM>). Conditioning the latent parameters of subsequent conditional layers on the latent parameters of preceding conditional layers may enforce a semantic relationship between sections of splines (or other parametrizing function) of consecutive conditional layers.

<FIG> illustrates an example data flow for training the neural network <NUM> using a training system <NUM> based on a set of training data <NUM>. Training the neural network <NUM> refers to iteratively adjusting the model parameters <NUM> of the neural network <NUM> to (approximately) optimize an objective function <NUM>, which will be described in more detail below.

Generally, training the neural network <NUM> includes, for each conditional layer, iteratively adjusting the parameter values of the differentiable functions that parametrize the conditional layer weights of the conditional layer. The differentiable functions parameterizing the conditional layer weights are B-splines, and the control points specifying the B-splines are adjusted at each training iteration. Adjusting a particular control point of a B-spline may have the effect of only changing the B-spline locally, i.e., in the vicinity of the particular control point, which can make it easier to optimize the B-spline through adapting the position of each control point.

For each conditional layer having a fixed set of decision parameters, the values of the decision parameters are iteratively adjusted over the training iterations. In some cases, the decision parameters of certain conditional layers are parameterized by one or more differentiable functions (e.g., polynomial or piecewise polynomial functions) of the latent parameters of a preceding conditional layer, as described with reference to <FIG>. In these cases, the parameter values of the differentiable functions parametrizing the decision parameters are iteratively adjusted over the course of the training iterations. The differentiable functions parameterizing the decision parameters of a conditional layer are B-splines, and the control points specifying the B-splines may be adjusted at each training iteration.

The training system <NUM> trains the neural network <NUM> based on a set of training data <NUM> composed of training examples, where each training example specifies: (i) a network input, and (ii) a target output that should be generated by the neural network <NUM> by processing the network input. At each training iteration, the training system <NUM> samples a "batch" of training examples <NUM> from the training data <NUM>, and processes the network inputs of the training examples in accordance with the current values of the model parameters <NUM> of the neural network <NUM> to generate corresponding network outputs. The training system <NUM> then adjusts the current values of the model parameters <NUM> of the neural network <NUM> using gradients of an objective function <NUM> that depends on: (i) the network outputs generated by the neural network <NUM>, and (ii) the corresponding target outputs specified by the training examples. The training system <NUM> can determine the gradients using, e.g., backpropagation techniques, and can use the gradients to adjust the current values of the model parameters using any appropriate gradient descent optimization technique, e.g., Adam or RMSprop.

Generally, the objective function <NUM> characterizes the accuracy of the network outputs generated by the neural network <NUM> by measuring a similarity between the network outputs and the corresponding target outputs specified by the training examples, e.g., using a cross-entropy loss term or a squared-error loss term.

The objective function <NUM> may also include additional terms that encourage the neural network <NUM> to fully utilize the representational capacity of the conditional layers, and to "specialize" respective portions of the continuous range of possible latent parameter values to handle network inputs corresponding to certain target outputs. To this end, the objective function <NUM> may measure the mutual information between the values of the latent parameters of the conditional neural network layers and the target outputs specified by the training examples. In one example, the objective function <IMG> may be given by: <MAT> <MAT> where λ is a hyper-parameter between <NUM> and <NUM>, <IMG> measures the similarity between the network outputs and the target outputs, I is the total number of conditional layers, i indexes the conditional layers, <MAT> characterizes the mutual information between the latent parameter of conditional layer i and the target outputs in the current batch of training examples, H(φi) is the entropy of the distribution P(φi) of the latent parameter of conditional layer i, H(φi|Y) is the entropy of the distribution P(φi|Y) of the latent parameter of conditional layer i conditioned on the target outputs, and wu and ws are hyper-parameters.

Referring to equation (<NUM>), by maximizing the entropies <MAT> of the distributions <MAT> of the latent parameters of the conditional layers, the objective function may encourage the neural network to generate latent parameters that are more evenly distributed throughout the continuous range of possible latent parameter values. This can result in the neural network <NUM> more fully utilizing the representational capacity of the conditional layers. By minimizing the entropies <MAT> of the distributions <MAT> of the latent parameters of the conditional layers conditioned on the target outputs, the objective function may encourage the neural network to specialize respective portions of the continuous range of possible latent parameter values to handle network inputs having certain target outputs.

The training system <NUM> can approximate the distribution of the latent parameter P(φi) and the distribution of the latent parameter conditioned on the target output P(φi|Y) based on the sampled latent parameter - target outputs pairs <MAT> of the current batch of N training examples, where yn is the target output specified by training example n. To this end, the training system <NUM> may quantize the continuous range of possible latent parameter values (e.g., [<NUM>,<NUM>]) into B bins, and count the samples that fall into each bin using a soft (i.e., differentiable) quantization function, e.g., the soft quantization function U(φ; cb, wb, v) given by: <MAT> where U(·) returns almost <NUM> when the latent parameter φ is inside the bin described by the center cb and the width wb, and almost <NUM> otherwise. The parameter v controls the sharpness (slope) of the soft quantization. The illustration <NUM> shows examples of the soft quantization function U(·) with respective bin centers and slopes. It can be appreciated that a higher value of v results in a sharper quantization.

Using the soft quantization function U(·), e.g., as described with reference to equation (<NUM>), the training system <NUM> can discretize the continuous latent parameter φi with B bins, which approximates φi as a discrete latent parameter Λi. The training system <NUM> can approximate the entropy H(φi) of the distribution P(φi) of the latent parameter φi as: <MAT> <MAT> where b indexes the bins and n indexes the training examples of the current batch. Similarly, the training system <NUM> can approximate the entropy H(φi|Y) of the distribution P(φi|Y) of the latent parameter φi conditioned on the target outputs as: <MAT> <MAT> where c indexes the C possible target outputs, and <MAT> returns <NUM> if target output for training example n is c, and <NUM> otherwise.

Characterizing the mutual information between the values of the latent parameters and the target outputs using a soft quantization function causes the objective function to be differentiable, which facilitates the training of the neural network <NUM>.

The training system <NUM> may continue training the neural network <NUM> until a training termination criterion is met, e.g., until a predetermined number of training iterations have been performed, or until the accuracy of the neural network <NUM> (e.g., evaluated on a held out validation set) satisfies a predetermined threshold.

<FIG> is a flow diagram of an example process <NUM> for processing a layer input using a conditional layer to generate a layer output. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system obtains the values of the decision parameters of the conditional layer <NUM>. In some cases, the values of the decision parameters of the conditional layer are determined during training and are held fixed thereafter. In some other cases, the system determines the values of the decision parameters of the conditional layer from values of one or more latent parameters of a preceding conditional layer. More specifically, the system may determine the values of the decision parameters of the conditional layer in accordance with a parametrization of the decision parameters by the latent parameters of the preceding conditional layer, as a B-spline or as a hypersurface defined as a sum of B-splines.

The system determines the values of the latent parameters of the conditional layer from a continuous set of possible latent parameter values by processing the layer input and the decision parameters <NUM>. The system may determine the values of the latent parameters by processing the layer input and the decision parameters using a differentiable mapping, e.g., by determining an inner product between the layer input and the decision parameters, and processing the result of the inner product by a sigmoid function. In some cases, the system determines the latent parameter values for the conditional layer from: (i) the result of processing the layer input and the decision parameters, and (ii) the latent parameter values of a preceding conditional layer. In some cases, the number of latent parameters may be substantially less than the dimensionality of the layer input and the dimensionality of the conditional layer weights, e.g., by multiple orders of magnitude.

The system determines the values of the conditional layer weights from the values of the latent parameters <NUM>. The system determines the values of the conditional layer weights in accordance with a parametrization of the conditional layer weights by the latent parameters, as a B-spline or a hypersurface defined by a sum of B-splines. Each B-spline is defined by a set of knots (control points) that determined during training.

In one implementation, the conditional layer weights include weights of multiple convolutional filters, and the latent parameters parametrize multiple nested latent parameters, each of which parameterizes the weights of a corresponding convolutional filter. In these implementations, the system determines the values of the nested latent parameters from the values of the latent parameters in accordance with the parameterization of the nested latent parameters by the latent parameters. Then, for each convolutional filter, the system determines the values of the weights of the convolutional filter from the value of the corresponding nested latent variable in accordance with the parameterization of the weights of the convolutional filter by the nested latent parameter.

The system processes the layer input in accordance with the values of the conditional layer weights to generate the layer output <NUM>. For example, the conditional layer may be a fully-connected layer, where the conditional layer weights specify a weight matrix and the conditional layer generates the layer output by multiplying the weight matrix by the layer input. As another example, the conditional layer may be a convolutional layer, where the conditional layer weights specify multiple convolutional filters and the conditional layer generates the layer output by convolving the convolutional filters with the layer input.

Claim 1:
A method implemented by a data processing apparatus, the method comprising:
processing a network input (<NUM>) comprising an image or audio signal using a neural network (<NUM>) comprising a plurality of neural network layers to generate a network output (<NUM>) characterizing the image or audio signal, wherein each neural network layer is configured to process a respective layer input (<NUM>) in accordance with respective values of a plurality of layer weights to generate a respective layer output (<NUM>), wherein one or more of the neural network layers is a conditional neural network layer (<NUM>), and wherein processing a layer input using a conditional neural network layer (<NUM>) to generate a layer output (<NUM>) comprises:
obtaining values of one or more decision parameters of the conditional neural network layer (<NUM>) which are determined when the neural network (<NUM>) is trained;
processing (i) the layer input (<NUM>), and (ii) the decision parameters of the conditional neural network layer (<NUM>), to determine values of one or more latent parameters of the conditional neural network layer (<NUM>) from a continuous set of possible latent parameter values, wherein the values of the one or more latent parameters of the conditional neural network layer (<NUM>) parameterize the conditional layer weights as a B-spline or as a hypersurface defined as a sum of multiple B-splines, wherein each B-spline is defined by a plurality of knots, wherein values of the knots defining the B-spline are determined during training; and
determining the values of the conditional layer weights from the values of the one or more latent parameters, comprising:
determining the values of the conditional layer weights from the values of the one or more latent parameters in accordance with the parametrization of the conditional layer weights by the one or more latent parameters; and
processing the layer input (<NUM>) in accordance with the values of the
conditional layer weights to generate the layer output (<NUM>),
wherein:
the network output characterizes a classification of the image or audio signal into a predetermined set of categories; or
the network input (<NUM>) comprises an image and the network output characterizes a natural language caption directed to contents of the image; or
the network input (<NUM>) comprises an audio signal and the network output characterizes words spoken in the audio signal.