COMPRESSED MATRIX REPRESENTATIONS OF NEURAL NETWORK ARCHITECTURES BASED ON SYNAPTIC CONNECTIVITY

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing brain emulation neural networks using compressed matrix representations. One of the methods includes obtaining a network input; and processing the network input using a neural network to generate a network output, comprising: processing the network input using an input subnetwork of the neural network to generate an embedding of the network input; and processing the embedding of the network input using a brain emulation subnetwork of the neural network, wherein the brain emulation subnetwork has a brain emulation neural network architecture that represents synaptic connectivity between a plurality of biological neurons in a brain of a biological organism, the processing comprising: obtaining a compressed matrix representation of a sparse matrix of brain emulation parameters; and applying the compressed matrix representation to the embedding of the network input to generate a brain emulation subnetwork output.

BACKGROUND

This specification relates to processing data using machine learning models.

Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of computational units to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

SUMMARY

This specification describes systems implemented as computer programs on one or more computers in one or more locations for implementing neural networks that include one or more brain emulation neural network layers whose parameters have been determined according to the synaptic connectivity between neurons in the brain of a biological organism, e.g., a fly.

For example, the parameters of a brain emulation neural network layer can be determined using a synaptic connectivity graph. A synaptic connectivity graph refers to a graph representing the structure of synaptic connections between neurons in the brain of a biological organism, e.g., a fly. For example, the synaptic connectivity graph can be generated by processing a synaptic resolution image of the brain of a biological organism.

For convenience, throughout this specification, an artificial neural network layer whose parameters have been determined using synaptic connectivity is called a “brain emulation” neural network layer. An artificial neural network having at least one brain emulation neural network layer is called a “brain emulation” neural network. Identifying an artificial neural network as a “brain emulation” neural network is intended only to conveniently distinguish such neural networks from other neural networks (e.g., with hand-engineered architectures), and should not be interpreted as limiting the nature of the operations that can be performed by the neural network or otherwise implicitly characterizing the neural network.

The architecture of a brain emulation neural network layer of a brain emulation neural network can be represented using a weight matrix that includes the parameters of the brain emulation neural network layer. When the brain emulation neural network processes a network input, the brain emulation neural network layer is configured to process a layer input generated from the network input by multiplying the weight matrix against the layer input to generate a layer output.

The weight matrix of a brain emulation neural network layer can have a high sparsity; that is, a high proportion of the elements of the weight matrix can have a value of zero. In these cases, the weight matrix can be represented using a compressed matrix representation.

In this specification, a compressed matrix representation of a matrix is any representation of the matrix that leverages the sparsity of the matrix to reduce the number of bits required to represent the matrix. For example, a compressed matrix representation can explicitly identify only a proper subset of the elements of the matrix, and not explicitly identify the other elements of the matrix. In some implementations, the elements of the sparse matrix that are not explicitly identified in the compressed matrix representation all have value zero, and some or all of the elements of the sparse matrix that are explicitly identified in the compressed matrix representation have respective non-zero values. In some implementations, the compressed matrix representation only explicitly identifies the elements of the matrix that have a non-zero value. In some other implementations, the compressed matrix representation explicitly identifies the elements of the matrix that have a non-zero value, while also explicitly identifying a proper subset of the elements of the matrix that have a value of zero. As a particular example, during training of a neural network that includes a weight matrix of parameters represented using a compressed matrix representation, some of the parameters of the weight matrix that are explicitly identified in the compressed matrix representation may temporarily have values of zero, e.g., if a parameter update causes the parameter to change from a non-zero value to a value of zero. Specific examples are described in more detail below.

For example, the compressed matrix representation can be a compressed list (often called “COO”) representation, a compressed sparse row (CSR) representation, or a compressed sparse column (CSC) representation.

The weight matrix of a brain emulation neural network layer can be stored using a compressed matrix representation, applied using a compressed matrix representation, or both.

This specification also describes systems for training a neural network that includes one or more such brain emulation neural network layers, including updating the values of the parameters that are represented used the compressed matrix representation.

The systems described in this specification can train and implement a brain emulation neural network using a compressed matrix representation. As described in this specification, brain emulation neural networks can achieve a higher performance (e.g., in terms of prediction accuracy), than other neural networks of an equivalent size (e.g., in terms of number of parameters). Put another way, brain emulation neural networks that have a relatively small size (e.g., 100 or 1000 parameters) can achieve comparable performance with other neural networks that are much larger (e.g., thousands or millions of parameters). Therefore, using techniques described in this specification, a system can implement a highly efficient, low-latency, and low-power-consuming neural network. That is, a system that implements a brain emulation neural network can reduce the use of computational resources, e.g., memory and computational power, relative to systems that implement other neural networks.

Leveraging compressed matrix representations can further improve the efficiency of a system configured to execute a brain emulation neural network. In some implementations, brain emulation neural networks can have a very high sparsity, representing the fact that most pairs of neurons in the brain of a biological organism do not share a synaptic connection. Thus, brain emulation neural networks in particular can enjoy significant efficiency improvements from compressed matrix representations. The reduced size of the compressed matrix representation of a weight matrix of a brain emulation neural network layer improves the memory efficiency, computational efficiency, and time efficiency of processing layer inputs using the brain emulation neural network layer. Furthermore, this improved efficiency can further reduce the amount of power necessary for the system to execute the brain emulation neural network.

These efficiency gains can be especially important in low-resource or low-memory environments, e.g., on mobile devices or other edge devices. Additionally, these efficiency gains can be especially important in situations in which the brain emulation neural network is continuously processing network inputs, e.g., in an application that continuously processes input audio data to determine whether a “wakeup” phrase has been spoken by a user.

The systems described in this specification can implement a brain emulation neural network having an architecture specified by a synaptic connectivity graph derived from a synaptic resolution image of the brain of a biological organism. The brains of biological organisms may be adapted by evolutionary pressures to be effective at solving certain tasks, e.g., classifying objects or generating robust object representations, and brain emulation neural networks can share this capacity to effectively solve tasks. In particular, compared to other neural networks, e.g., with manually specified neural network architectures, brain emulation neural networks can require less training data, fewer training iterations, or both, to effectively solve certain tasks.

DETAILED DESCRIPTION

FIG.1andFIG.2show two examples of neural network computing systems for implementing neural networks that include at least one brain emulation neural network layers.

FIG.1shows an example neural network computing system100. The neural network computing system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The neural network computing system100includes a brain emulation neural network102. The brain emulation neural network102includes one or more brain emulation neural network layers (e.g., the brain emulation neural network layer108). Optionally, the brain emulation neural network102can also include one or more other neural network layers, e.g., one or more feed-forward neural network layers, recurrent neural network layers, convolutional neural network layers, or any other appropriate type of neural network layer.

The brain emulation neural network102is configured to process a network input to generate a network output for a particular machine learning task. The network input for the brain emulation neural network102can be any kind of digital data input, and the network output for the brain emulation neural network102can be any kind of score, classification, or regression output based on the input. That is, the brain emulation neural network102can be configured for any appropriate machine learning tasks; example tasks are discussed below.

The brain emulation neural network layer108includes multiple brain emulation parameters, and is configured to process a brain emulation layer input110, generated from the network input of the brain emulation neural network102, and to generate a brain emulation layer output112, which can be processed by subsequent neural network layers in the brain emulation neural network102. In general, the brain emulation layer input110can be the network input to the brain emulation neural network102(i.e., if the brain emulation neural network layer108is the first layer in the brain emulation neural network102) or the output of another layer of the brain emulation neural network102. Similarly, the brain emulation layer output112can be the network output of the brain emulation neural network102(i.e., if the brain emulation neural network layer108is the final layer in the brain emulation neural network102). The brain emulation layer input110and the brain emulation layer output112may be represented in any appropriate numerical format, for example, as vectors or as matrices.

The brain emulation neural network layer108can have a brain emulation architecture that is based on a synaptic connectivity graph representing synaptic connectivity between neurons in the brain of a biological organism. An example process for determining a network architecture using a synaptic connectivity graph is described below with respect toFIG.4B. In some implementations, the architecture of the brain emulation neural network layer108can be specified by the synaptic connectivity between neurons of a particular type in the brain, e.g., neurons from the visual system or the olfactory system. This process is described in more detail below with reference toFIG.6.

In particular, the architecture of the brain emulation neural network layer108can be represented by a weight matrix that the brain emulation neural network layer108applies to the brain emulation layer input110to generate the brain emulation layer output112. Each element of the weight matrix can be a respective brain emulation parameter of the brain emulation neural network layer108. As a particular example, the brain emulation layer input110can be an N×1 vector of elements, the weight matrix of the brain emulation neural network layer108can be an M×N matrix of elements, and the brain emulation layer output112can be an M×1 vector of elements.

Each brain emulation parameter of the weight matrix can correspond to a pair of neurons in the brain of the biological organism, where the value of the brain emulation parameter characterizes a strength of a neuronal connection between the pair of neurons. In other words, each row and column of the weight matrix can correspond to a respective neuron in the brain of the biological organism, and the value of each brain emulation parameter can characterize a strength of a neuronal connection between (i) the neuron corresponding to the row of the brain emulation parameter and (ii) the neuron corresponding to the column of the brain emulation parameter.

In particular, the weight matrix can be an M×N matrix, where each of the M rows corresponds to a neuron in a first set of neurons and each of the N columns corresponds to a neuron in a second set of neurons in the brain of the biological organism. The first set of neurons and the second set of neurons can be overlapping (i.e., one or more neurons in the brain of the biological organism is in both sets) or disjoint (i.e., there does not exist a neuron in the brain of the biological organism that is in both sets). As a particular example, the first set and the second set can be the same. That is, the weight matrix can be an N×N matrix where the same neurons in the brain of the biological organism are represented by both the rows and the columns of the weight matrix. The process of generating the weight matrix is described in more detail below.

Because many pairs of neurons in the brain of the biological organism may not share a synaptic connection at all, many brain emulation parameters of the weight matrix of the brain emulation neural network layer may have values of zero. In other words, the sparsity of the weight matrix may be high. In this specification, the sparsity of a matrix is a measure of the number or proportion of zero elements in the matrix. In this specification, a matrix may be referred to as a “sparse matrix” if the sparsity of the matrix satisfies a certain threshold. For example, in some implementations the weight matrix of a brain emulation neural network layer has a sparsity of 50% (i.e., where 50% of the brain emulation parameters of the weight matrix have a value of zero), 60%, 70%, 80%, 90%, 95%, or 99%.

Therefore, the neural network computing system100can store a compressed matrix representation122of the weight matrix of the brain emulation neural network layer108. As described above, the compressed matrix representation122has a smaller memory footprint than the weight matrix when represented fully (i.e., as an array with all zero values represented) because only a proper subset of the brain emulation parameters of the weight matrix are explicitly identified (i.e., represented) by the compressed matrix representation.

As described above, in some implementations, the compressed matrix representation122only explicitly identifies the brain emulation parameters that have a non-zero value. That is, the compressed matrix representation122does not explicitly represent the brain emulation parameters that have values of zero.

In some other implementations, the compressed matrix representation122explicitly identifies both (i) each brain emulation parameter that has a non-zero value, and (ii) a proper subset of the brain emulation parameters that have a value of zero. For example, as described in more detail below, during training of the neural network102, the compressed matrix representation may temporarily explicitly represent brain emulation parameters that have values of zero, e.g., if a parameter update causes the parameter to change from a non-zero value to a value of zero. As another example, the compressed matrix representation122may represent brain emulation parameters that have values of zero during an evolutionary process of updating the compressed matrix representation. Example evolutionary processes are discussed in more detail below with reference toFIG.2.

In this specification, a brain emulation parameter of a weight matrix that is explicitly identified by a compressed matrix representation of the weight matrix is called a “represented” brain emulation parameter. In this specification, a brain emulation parameter of a weight matrix that is not explicitly identified by a compressed matrix representation of the weight matrix is called an “unrepresented” brain emulation parameter.

For example, the compressed matrix representation122can include data identifying, for each represented brain emulation parameter, the value of the represented brain emulation parameter and the position of the represented brain emulation parameter in the weight matrix. The “position” of a brain emulation parameter in the weight matrix can be represented, e.g., by the row and column index of the brain emulation parameter in the weight matrix, e.g., as a tuple (i,j) where i represents the row index and j represents the column index.

A parameter data store120of the neural network computing system100can store the compressed matrix representation122of the weight matrix of the brain emulation neural network layer108. When the brain emulation neural network102is processing a network input, the brain emulation neural network layer108can obtain the compressed matrix representation122of the weight matrix from the parameter data store120to generate the brain emulation layer output112.

In some implementations, the brain emulation neural network layer108processes the compressed matrix representation122of the weight matrix to re-generate the full representation of the weight matrix, i.e., a representation in which all brain emulation parameters, including the unrepresented brain emulation parameters of the compressed matrix representation122, are represented.

In some other implementations, the neural network computing system100uses a compressed matrix representation that is configured for efficient matrix multiplication. Thus, the brain emulation neural network layer108does not need to “decompress” the compressed matrix representation122, but rather can apply the compressed matrix representation directly to the brain emulation layer input110.

For example, the compressed matrix representation of a matrix can be configured to perform a matrix multiplication between the matrix and another tensor such that the unrepresented elements of the matrix are not explicitly multiplied against any values of the other tensor (as, in implementations in which the unrepresented elements all have value zero, the result of the multiplication is certain to also be zero). That is, a system executing a matrix multiplication using a compressed matrix representation of a matrix does not perform scalar multiplications of the unrepresented elements of the matrix.

For example, the compressed matrix representation112can be a COO matrix representation, where each represented brain emulation parameter is represented by a tuple that include the row of the parameter, the column of the parameter, and the value of the parameter. The COO matrix representation is highly efficient when used to execute matrix multiplications, and is supported by many machine learning computer-language libraries, e.g., TensorFlow.

In some implementations, the computational efficiency of executing a matrix multiplication using the compressed matrix representation122scales linearly or approximately linearly with the sparsity of the weight matrix of the brain emulation neural network layer108. That is, if the weight matrix has a sparsity of 70%, then the neural network computing system can execute the matrix multiplication to generate the brain emulation layer output112using 70% fewer computations.

FIG.2shows an example neural network computing system200. The neural network computing system200is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The neural network computing system200includes a neural network202that has (at least) three subnetworks: (i) a first trained subnetwork204(ii) a brain emulation subnetwork208, and (iii) a second trained subnetwork212. The neural network202is configured to process a network input201to generate a network output214.

The first trained subnetwork204is configured to process the network input201in accordance with a set of model parameters222of the first trained subnetwork204to generate a first subnetwork output206. The brain emulation subnetwork208is configured to process the first subnetwork output206in accordance with a set of model parameters224of the brain emulation subnetwork208to generate a brain emulation subnetwork output210. The second trained subnetwork212is configured to process the brain emulation subnetwork output210in accordance with a set of model parameters226of the second trained subnetwork212to generate the network output214.

The brain emulation subnetwork includes one or more brain emulation neural network layers whose respective architectures are represented by a weight matrix that is represented as a compressed matrix representation. For example, the brain emulation subnetwork208can be configured similarly to the brain emulation neural network102described above with reference toFIG.1.

Although the neural network202depicted inFIG.2includes one trained subnetwork204before the brain emulation subnetwork208and one trained subnetwork212after the brain emulation subnetwork208, in general the neural network202can include any number of trained subnetworks before and/or after the brain emulation subnetwork208. In some implementations, the first trained subnetwork204and/or the second trained subnetwork212can include only one or a few neural network layers (e.g., a single fully-connected layer) that processes the respective subnetwork input to generate the respective subnetwork output.

In implementations where there are zero trained subnetworks before the brain emulation subnetwork208, the brain emulation subnetwork208can receive the network input201directly as input. In implementations where there are zero trained subnetworks after the brain emulation subnetwork208, the brain emulation subnetwork output210can be the network output214.

Although the neural network202depicted inFIG.2includes a single brain emulation subnetwork208, in general the neural network202can include multiple brain emulation subnetwork208. In some implementations, each brain emulation subnetwork208has the same set of model parameters224; in some other implementations, each brain emulation subnetwork208has a different set of model parameters224. In some implementations, each brain emulation subnetwork208has the same network architecture; in some other implementations, each brain emulation subnetwork208has a different network architecture.

In some implementations, the neural network202is a recurrent neural network. In these implementations, the network input201includes a sequence of input elements. The first trained subnetwork204can process, at each of multiple time steps corresponding to respective input elements in the sequence, the input element to generate a respective first subnetwork output206. At each time step, the brain emulation subnetwork208can process the first subnetwork output206to generate a respective brain emulation subnetwork output210. At each time step, the second trained subnetwork212can process the brain emulation subnetwork output210to generate an output element corresponding to the input element.

At each time step, the neural network202can maintain a hidden state220. That is, at each time step, the neural network202updates its hidden state220; then, at the subsequent time step in the sequence of time steps, the neural network202receives as input (i) the input element of the network input201corresponding to the subsequent time step and (ii) the current hidden state220.

In some implementations in which the neural network202is a recurrent neural network (e.g., in the example depicted inFIG.2), the first trained subnetwork204receives both i) the input element of the sequence of the network input201and ii) the hidden state220. For example, the recurrent neural network202can combine the input element and the hidden state220(e.g., through concatenation, addition, multiplication, or an exponential function) to generate a combined input, and then process the combined input using the first trained subnetwork204.

In some implementations in which the neural network202is a recurrent neural network, the brain emulation subnetwork208receives as input the hidden state220and the first subnetwork output206. For example, the neural network202can combine the first subnetwork output206and the hidden state220(e.g., through concatenation, addition, multiplication, or an exponential function) to generate a combined input, and then process the combined input using the brain emulation subnetwork208.

In some implementations in which the neural network202is a recurrent neural network, the second trained subnetwork212receives as input the hidden state220and the brain emulation subnetwork output210. For example, the neural network202can combine the brain emulation subnetwork output210and the hidden state220(e.g., through concatenation, addition, multiplication, or an exponential function) to generate a combined input, and then process the combined input using the second trained subnetwork212.

In some implementations in which the neural network202is a recurrent neural network, the updated hidden state220generated at a time step is the same as the output element generated at the time step. In some other implementations, the hidden state220is an intermediate output of the neural network202. An intermediate output refers to an output generated by a hidden artificial neuron or a hidden neural network layer of the neural network202, i.e., an artificial neuron or neural network layer that is not included in the input layer or the output layer of the neural network202. For example, the hidden state220can be the brain emulation subnetwork output210. In some other implementations, the hidden state220is a combination of the output element and one or more intermediate outputs of the neural network202. For example, the hidden state220can be computed using the output element and the brain emulation subnetwork output210, e.g., by combining the two outputs and applying an activation function.

In some implementations in which the neural network202is a recurrent neural network, after each input element in the network input201has been processed by the recurrent neural network202to generate respective output elements, the recurrent neural network202can generate a network output214corresponding to the network input201. In some such implementations, the network output214is the sequence of generated output elements. In some other implementations, the network output214is a subset of the generated output elements, e.g., the final output element corresponding to the final input element in the sequence of input elements of the network input201. In some other implementations, the recurrent neural network202further processes the sequence of generated output elements to generate the network output214. For example, the network output214can be the mean of the generated output elements.

In some implementations, the brain emulation subnetwork208itself has a recurrent neural network architecture. That is, the brain emulation subnetwork208can process the first subnetwork output206multiple times at respective sub-time steps (referred to as sub-time steps to differentiate from the time steps of the neural network202in implementations where the neural network202is a recurrent neural network).

For example, the architecture of the brain emulation subnetwork208can include a sequence of components (e.g., brain emulation neural network layers or groups of brain emulation neural network layers) such that the architecture includes a connection from each component in the sequence to the next component, and the first and last components of the sequence are identical. In one example, two brain emulation neural network layers that are each directly connected to one another (i.e., where the first layer provides its output the second layer, and the second layer provides its output to the first layer) would form a recurrent loop. A recurrent brain emulation subnetwork208can process the first subnetwork output206over multiple sub-time steps to generate a respective brain emulation subnetwork output210at each sub-time step. In particular, at each sub-time step, the brain emulation subnetwork208can process: (i) the first subnetwork output206(or a component of the first subnetwork output206), and (ii) any outputs generated by the brain emulation subnetwork208at the preceding sub-time step, to generate the brain emulation subnetwork output210for the sub-time step. The neural network202can provide the brain emulation subnetwork output210generated by the brain emulation subnetwork208at the final sub-time step as the input to the second trained subnetwork212. The number of sub-time steps over which the brain emulation subnetwork208processes a network input can be a predetermined hyper-parameter of the neural network computing system200.

In some implementations, in addition to processing the brain emulation subnetwork output210generated by the output layer of the brain emulation subnetwork208, the second trained subnetwork212can additionally process one or more intermediate outputs of the brain emulation subnetwork208.

The neural network computing system200includes a training engine216that is configured to train the neural network202.

In some implementations, the model parameters224for the brain emulation subnetwork208are untrained. Instead, the model parameters224of the brain emulation subnetwork208can be determined before the training of the trained subnetworks204and212based on the weight values of the edges in the synaptic connectivity graph. Optionally, the weight values of the edges in the synaptic connectivity graph can be transformed (e.g., by additive random noise) prior to being used for specifying model parameters224of the brain emulation subnetwork208. This procedure enables the neural network202to take advantage of the information from the synaptic connectivity graph encoded into the brain emulation subnetwork208in performing prediction tasks.

Therefore, rather than training the entire neural network202from end-to-end, the training engine216can train only the model parameters222of the first trained subnetwork204and the model parameters226of the second trained subnetwork212, while leaving the model parameters224of the brain emulation subnetwork208fixed during training.

The training engine216can train the neural network202on a set of training data over multiple training iterations. The training data can include a set of training examples, where each training example specifies: (i) a training network input, and (ii) a target network output that should be generated by the neural network202by processing the training network input.

At each training iteration, the training engine216can sample a batch of training examples from the training data, and process the training inputs specified by the training examples using the neural network202to generate corresponding network outputs214. In particular, for each training input, the neural network202processes the training input using the current model parameter values222of the first trained subnetwork204to generate a first subnetwork output206. The neural network202processes the first subnetwork output206in accordance with the static model parameter values224of the brain emulation subnetwork208to generate a brain emulation subnetwork output210. The neural network202then processes the brain emulation subnetwork output210using the current model parameter values226of the second trained subnetwork212to generate the network output214corresponding to the training input.

The training engine216adjusts the model parameters values222of the first trained subnetwork204and the model parameter values226of the second trained subnetwork212to optimize an objective function that measures a similarity between: (i) the network outputs214generated by the neural network202, and (ii) the target network outputs specified by the training examples. The objective function can be, e.g., a cross-entropy objective function, a squared-error objective function, or any other appropriate objective function.

To optimize the objective function, the training engine216can determine gradients of the objective function with respect to the model parameters222of the first trained subnetwork204and the model parameters226of the second trained subnetwork212, e.g., using backpropagation techniques. The training engine216can then use the gradients to adjust the model parameter values222and226, e.g., using any appropriate gradient descent optimization technique, e.g., an RMSprop or Adam gradient descent optimization technique.

The training engine216can use any of a variety of regularization techniques during training of the neural network202. For example, the training engine216can use a dropout regularization technique, such that certain artificial neurons of the neural network202are “dropped out” (e.g., by having their output set to zero) with a non-zero probability p>0 each time the neural network202processes a network input. Using the dropout regularization technique can improve the performance of the trained neural network202, e.g., by reducing the likelihood of over-fitting. As another example, the training engine216can regularize the training of the neural network202by including a “penalty” term in the objective function that measures the magnitude of the model parameter values222and226of the trained subnetworks204and212. The penalty term can be, e.g., an L1or L2norm of the model parameter values222of the first trained subnetwork204and/or the model parameter values226of the second trained subnetwork212. In some other implementations, the model parameters224for the brain emulation subnetwork208are trained. That is, after initial values for the model parameters224of the brain emulation subnetwork208have been determined based on the weight values of the edges in the synaptic connectivity graph, the training engine216can update the weights of the model parameters, as described above with reference to the parameters222and226of the trained subnetworks, e.g., using backpropagation and stochastic gradient descent.

Because the weight matrices of the brain emulation neural network layers of the brain emulation subnetwork208using the compressed matrix representation, the training engine216can efficiently update the represented brain emulation parameters of the weight matrices while keeping the unrepresented brain emulation parameters of the weight matrices constant, i.e., at zero. That is, if the weight matrices were represented fully and the training engine216executed backpropagation and gradient descent across all the values of the weight matrices, unrepresented brain emulation parameters having value zero would likely be updated to non-zero values. Because the weight matrices represent synaptic connectivity between neurons in the brain of a biological organism, updating a unrepresented, zero-value brain emulation parameter to have a non-zero value corresponds to incorrectly representing synaptic connectivity between the pair of neurons represented by the brain emulation parameter, when no such synaptic connectivity was measured in the brain of the biological organism. Thus, in some implementations in which fidelity to the measured synaptic connectivity is important, representing the weight matrices using the compressed matrix representation allows the training engine216to efficiently update the weight matrices of the brain emulation subnetwork208without inserting representations of new and incorrect synaptic connections.

In some implementations, the training engine216does update which brain emulation parameters of the weight matrices are represented and which brain emulation parameters are unrepresented. That is, during training of the neural network202, the training engine216can update the compressed matrix representations of the weight matrices such that brain emulation parameters are added to the compressed matrix representations (e.g., corresponding to changing a zero-value brain emulation parameter to a non-zero brain emulation parameter) and some brain emulation parameters are removed from the compressed matrix representation (e.g., corresponding to changing a non-zero brain emulation parameter to a zero-value brain emulation parameter).

In particular, for one or more weight matrices of respective brain emulation neural network layers, the training engine216can execute an artificial evolutionary procedure whereby, over multiple training stages, the training engine216iteratively removes the brain emulation parameters representing the weakest synaptic connections in the brain of the biological organism from the compressed matrix representation of the weight matrix. The training engine216can also add new brain emulation parameters to the compressed matrix representation of the weight matrix, where the new brain emulation parameters represent “new” synaptic connections in the brain of the biological organism (i.e., synaptic connections that were not measured in the brain of the biological organism).

This procedure is referred to as “evolutionary” because it simulates, across the multiple training stages, the removal of “weak” brain emulation parameters (e.g., brain emulation parameters with the lowest value or magnitude) and the addition of new brain emulation parameters that may improve the performance of the neural network202. Performing the evolutionary procedure can further reduce the amount of training data and the number of training iterations required to train the neural network202to achieve an acceptable level of performance, e.g., as measured by prediction accuracy.

For example, at each of one or more training stages during the training of the neural network202and for each of one or more weight matrices of the brain emulation subnetwork208, the training engine216can stochastically sample (i.e., select) represented brain emulation parameters of the compressed matrix representation of the weight matrix, and remove the sampled represented brain emulation parameters from the compressed matrix representation.

As a particular example, the training engine216can sample each represented brain emulation parameter with a uniform likelihood. That is, each represented brain emulation parameter can have the same likelihood of being selected, regardless of the value of the parameter of the position of the parameter within the compressed matrix representation. As another particular example, the training engine216can determine the N represented brain emulation parameters that have the lowest respective magnitudes, N>1, and sample the N represented brain emulation parameters uniformly. For instance, N can be a predetermined integer, or N can be a predetermined fraction of the total number of represented brain emulation parameters in the compressed matrix representation.

As another particular example, the training engine216can sample each represented brain emulation parameter with a likelihood that is inversely proportional with the magnitude of its value. That is, represented brain emulation parameters with lower magnitudes can be more likely to be selected than represented brain emulation parameters with higher magnitudes.

In some such implementations, the training engine216can determine the likelihood of sampling each represented brain emulation parameter to be equal to the softmax of the negated magnitude of the represented brain emulation parameter. That is, the training engine216can compute:

where xiis the value of the ithrepresented brain emulation parameter and piis the likelihood with which the ithrepresented brain emulation parameter is sampled by the training engine216.

In some other such implementations, the training engine216can determine the likelihood of sampling each represented brain emulation parameter to be equal to the softmax of the inverse magnitude of the represented brain emulation parameter. That is, the training engine216can compute:

In some other such implementations, the training engine216can determine the N represented brain emulation parameters that have the lowest respective magnitudes, N>1, and sample the N represented brain emulation parameters according to either of the softmax equations described above.

As another particular example, the training engine216can sample each represented brain emulation parameter with a likelihood that is inversely proportional to the rank of the represented brain emulation parameter in a ranking of the represented brain emulation parameters of the weight matrix. That is, represented brain emulation parameters with lower ranks in the ranking of the magnitudes can be more likely to be selected than represented brain parameters with higher ranks in the ranking of the magnitudes. In some such implementations, the training engine216can determine the N represented brain emulation parameters that have the lowest respective ranks in the ranking of the magnitudes, N>1, and sample the N represented brain emulation parameters according to their respective ranks.

As another example, the training engine216can execute a two-step process for stochastically sampling the represented brain emulation parameters of the compressed matrix representation. In the first step of the two-step process, the training engine216can generate a set of candidate represented brain emulation parameters by sampling the represented brain emulation parameters according to a ranking of their magnitudes. In the second step of the two-step process, the training engine216can sample from the set of represented brain emulation parameters according to their magnitudes (e.g., using a softmax function as described above). The training engine216can then remove the candidate represented brain emulation parameters samples in the second step from the compressed matrix representation.

In some implementations, the training engine216removes the same number of represented brain emulation parameters at each training stage. In some other implementations, the training engine216can sample a different number of represented brain emulation parameters at each training stage.

Instead of or in addition to removing represented brain emulation parameters from the compressed matrix representation, the training engine216can add “new” represented brain emulation parameters to the compressed matrix representation at each of one or more training stages. For example, training engine216can randomly sample one or more unrepresented brain emulation parameters of the compressed matrix representation, generate values for the sampled unrepresented brain emulation parameters, and insert the sampled unrepresented brain emulation parameters, having the respective generated values, into the compressed matrix representation as newly-represented brain emulation parameters.

For example, the training engine216can sample a respective value for each new represented brain emulation parameter from a predefined distribution, e.g., a uniform distribution between 0 and 1 or a Normal distribution with mean 0.

As another example, the training engine216can determine the initial value of the new represented brain emulation parameters to be 0. As described above, in some cases, a brain emulation parameter can have a value of zero and still be explicitly represented in the compressed matrix representation. So, the training engine216can explicitly add a representation of the previously-unrepresented brain emulation parameters to the compressed matrix representation, and set their values to be zero. Then, during training of the neural network202, the value of these new non-zero brain emulation parameters can be updated to have non-zero values, e.g., using stochastic gradient descent.

In some implementations, the training engine216samples the same number of unrepresented brain emulation parameters as the number of represented value brain emulation parameters sampled as described above. That is, the compressed matrix representation can include the same number of represented brain emulation parameters before and after the training stage. In some other implementations, the training engine216samples a different number of represented and unrepresented brain emulation parameters during a given training stage, such that the number of represented brain emulation parameters in the compressed matrix representation changes.

In some implementations, the training engine216can sample new represented brain emulation parameters to add to the compressed matrix representation such that the sampled new represented brain emulation parameters are biologically plausible. That is, the training engine216can ensure that each new represented brain emulation parameter represents a pair of neurons that could plausibly share a synaptic connection in the brain of the biological organism. For example, the training engine216can sample represented brain emulation parameters corresponding to pairs of neurons in the same region of the brain of the biological organism.

The neural network202can be configured to perform any appropriate task. A few examples follow.

In one example, the neural network202can be configured to process network inputs201that represent sequences of audio data. For example, each input element in the network input201can be a raw audio sample or an input generated from a raw audio sample (e.g., a spectrogram), and the neural network202can process the sequence of input elements to generate network outputs214representing predicted text samples that correspond to the audio samples. That is, the neural network202can be a “speech-to-text” neural network. As another example, each input element can be a raw audio sample or an input generated from a raw audio sample, and the neural network202can generate a predicted class of the audio samples, e.g., a predicted identification of a speaker corresponding to the audio samples. As a particular example, the predicted class of the audio sample can represent a prediction of whether the input audio example is a verbalization of a predefined work or phrase, e.g., a “wakeup” phrase of a mobile device. In some implementations, one or more weight matrices of the brain emulation subnetwork208can be generated from a subgraph of the synaptic connectivity graph corresponding to an audio region of the brain, i.e., a region of the brain that processes auditory information (e.g., the auditory cortex).

In another example, the neural network202can be configured to process network inputs201that represent sequences of text data. For example, each input element in the network input201can be a text sample (e.g., a character, phoneme, or word) or an embedding of a text sample, and the neural network202can process the sequence of input elements to generate network outputs214representing predicted audio samples that correspond to the text samples. That is, the neural network202can be a “text-to-speech” neural network. As another example, each input element can be an input text sample or an embedding of an input text sample, and the neural network202can generate a network output214representing a sequence of output text samples corresponding to the sequences of input text samples. As a particular example, the output text samples can represent the same text as the input text samples in a different language (i.e., the neural network202can be a machine translation neural network). As another particular example, the output text samples can represent an answer to a question posed by the input text samples (i.e., the neural network202can be a question-answering neural network). As another example, the input text samples can represent two texts (e.g., as separated by a delimiter token), and the neural network202can generate a network output representing a predicted similarity between the two texts. In some implementations, one or more weight matrices of the brain emulation subnetwork208can be generated from a subgraph of the synaptic connectivity graph corresponding to a speech region of the brain, i.e., a region of the brain that is linked to speech production (e.g., Broca's area).

In another example, the neural network202can be configured to process network inputs201representing one or more images, e.g., sequences of video frames. For example, each input element in the network input201can be a video frame or an embedding of a video frame, and the neural network202can process the sequence of input elements to generate a network output214representing a prediction about the video represented by the sequence of video frames. As a particular example, the neural network202can be configured to track a particular object in each of the frames of the video, i.e., to generate a network output214that includes a sequences of output elements, where each output elements represents a predicted location within a respective video frames of the particular object. In some implementations, the brain emulation subnetwork208can be generated from a subgraph of the synaptic connectivity graph corresponding to a visual region of the brain, i.e., a region of the brain that processes visual information (e.g., the visual cortex).

In another example, the neural network202can be configured to process a network input201representing a respective current state of an environment at each of one or more time points, and to generate a network output214representing action selection outputs that can be used to select actions to be performed at respective time points by an agent interacting with the environment. For example, each action selection output can specify a respective score for each action in a set of possible actions that can be performed by the agent, and the agent can select the action to be performed by sampling an action in accordance with the action scores. In one example, the agent can be a mechanical agent interacting with a real-world environment to perform a navigation task (e.g., reaching a goal location in the environment), and the actions performed by the agent cause the agent to navigate through the environment.

In this specification, an embedding is an ordered collection of numeric values that represents an input in a particular embedding space. For example, an embedding can be a vector of floating point or other numeric values that has a fixed dimensionality.

After training, the neural network202can be directly applied to perform prediction tasks. For example, the neural network202can be deployed onto a user device. In some implementations, the neural network202can be deployed directly into resource-constrained environments (e.g., mobile devices). Neural networks202that include brain emulation subnetworks208can generally perform at a high level, e.g., in terms of prediction accuracy, even with very few model parameters compared to other neural networks. For example, neural networks202as described in this specification that have, e.g., 100 or 1000 model parameters can achieve comparable performance to other neural networks that have millions of model parameters. Thus, the neural network202can be implemented efficiently and with low latency on user devices.

In some implementations, after the neural network202has been deployed onto a user device, some of the parameters of the neural network202can be further trained, i.e., “fine-tuned,” using new training example obtained by the user device. For example, some of the parameters can be fine-tuned using training example corresponding to the specific user of the user device, so that the neural network202can achieve a higher accuracy for inputs provided by the specific user. As a particular example, the model parameters222of the first trained subnetwork204and/or the model parameters226of the second trained subnetwork212can be fine-tuned on the user device using new training examples while the model parameters224of the brain emulation subnetwork208are held static, as described above.

FIG.3illustrates an example weight matrix302of a brain emulation neural network layer determined using synaptic connectivity

As described in more detail below with reference toFIG.4B, a system (e.g., the graphing system412depicted inFIG.4B), can generate a synaptic connectivity graph that represents the synaptic connectivity between neurons in the brain of the biological organism. The synaptic connectivity graph can be represented using an adjacency matrix301, all of which or a portion of which can be used as the weight matrix302of the brain emulation neural network layer.

As illustrated inFIG.3, the adjacency matrix301includes n2elements, where n is the number of neurons drawn from the brain of the biological organism. For example, the adjacency matrix301can include hundreds, thousands, tens of thousands, hundreds of thousands, millions, tens of millions, or hundreds of millions of elements.

Each element of the adjacency matrix301represents the synaptic connectivity between a respective pair of neurons in the set of n neurons. That is, each element ci,jidentifies the synaptic connection between neuron i and neuron j. As described in more detail below, in some implementations, each of the elements ci,jare either zero (representing that there is no synaptic connection between the corresponding neurons) or one (representing that there is a synaptic connection between the corresponding neurons), while in some other implementations, each element ci,jis a scalar value representing the strength of the synaptic connection between the corresponding neurons.

As described above with reference toFIG.1, each row of the adjacency matrix301can represent a respective neuron in a first set of neurons of the brain of the biological organism, and each column of the adjacency matrix301can represent a respective neuron in a second set of neurons of the brain of the biological organism. Generally, the first set and the second set can be overlapping or disjoint. As a particular example, the first set and the second set can be the same.

In some implementations (e.g., in implementations in which the synaptic connectivity graph is undirected), the adjacency matrix301is symmetric (i.e., each element ci,jis the same as element while in some other implementations (e.g., in implementations in which the synaptic connectivity graph is directed), the adjacency matrix301is not symmetric (i.e., there may exist elements ci,jand cj,isuch that ci,j≠cj,i).

Although the above description refers to neurons in the brain of the biological organism, generally the elements of the adjacency matrix can correspond to pairs of any appropriate component of the brain of the biological organism. For example, each element can correspond to a pair of voxels in a voxel grid of the brain of the biological organism. As another example, each element can correspond to a pair of sub-neurons of the brain of the biological organism. As another example, each element can correspond to a pair of sets of multiple neurons of the brain of the biological organism.

As described in more detail below with reference toFIG.4B, an architecture mapping system (e.g., the architecture mapping system420depicted inFIG.4B) can generate the weight matrix302from the adjacency matrix301. Generally, the elements of the weight matrix302(i.e., the brain emulation parameters of the brain emulation neural network layer) are a subset of the elements of the adjacency matrix301. For example, as depicted inFIG.3, the weight matrix302includes the elements of the adjacency matrix301representing neuronal connections between the neurons represented by the first three rows and first three columns of the adjacency matrix301. For example, the weight matrix302can represent only neurons of a particular type in the brain of the biological organism. Identifying neurons of a particular type is discussed in more detail below with reference toFIG.7.

For convenience, the weight matrix302is illustrated as including only nine brain emulation parameters; generally, weight matrices of brain emulation neural network layers can have significantly more brain emulation parameters, e.g., hundreds, thousands, or millions of brain emulation parameters. Although the weight matrix302is depicted as square inFIG.3(i.e., the same number of columns and rows), generally the weight matrix302can have any appropriate dimensionality.

The weight matrix can be a sparse matrix, i.e., can include more than a threshold number or proportion of zero-value brain emulation parameters. The weight matrix can thus be represented using a compressed matrix representation, as described above.

In some implementations, the weight matrix302represents the entire synaptic connectivity graph. That is, the weight matrix302can include a respective row and column for each node of the synaptic connectivity graph. Because the weight matrix302will be represented using the compressed matrix representation when applied by the brain emulation neural network layer, representing the entire synaptic connectivity graph is significantly more feasible and efficient than if the weight matrix302were represented fully. Thus, in memory-constrained or computationally-constrained environments, leveraging the compressed matrix representation can allow systems to represent the full brain of the biological organism in implementations in which doing so would otherwise be prohibitive.

FIG.4Aillustrates an example of generating an artificial (i.e., computer implemented) brain emulation neural network409based on a synaptic resolution image405of the brain403of a biological organism401, e.g., a fly. The synaptic resolution image405can be processed to generate a synaptic connectivity graph407, e.g., where each node of the graph407corresponds to a neuron in the brain403, and two nodes in the graph407are connected if the corresponding neurons in the brain403share a synaptic connection. The structure of the graph407can be used to specify the architecture of the brain emulation neural network409. For example, each node of the graph407can mapped to an artificial neuron, a neural network layer, or a group of neural network layers in the brain emulation neural network409. Further, each edge of the graph407can be mapped to a connection between artificial neurons, layers, or groups of layers in the brain emulation neural network409. The brain403of the biological organism401can be adapted by evolutionary pressures to be effective at solving certain tasks, e.g., classifying objects or generating robust object representations, and the brain emulation neural network409can share this capacity to effectively solve tasks.

FIG.4Bshows an example data flow400for generating a synaptic connectivity graph402and a brain emulation neural network404based on the brain406of a biological organism.

As used throughout this document, a brain may refer to any amount of nervous tissue from a nervous system of a biological organism, and nervous tissue may refer to any tissue that includes neurons (i.e., nerve cells). The biological organism can be, e.g., a worm, a fly, a mouse, a cat, or a human.

An imaging system408can be used to generate a synaptic resolution image410of the brain406. An image of the brain406may be referred to as having synaptic resolution if it has a spatial resolution that is sufficiently high to enable the identification of at least some synapses in the brain406. Put another way, an image of the brain406may be referred to as having synaptic resolution if it depicts the brain406at a magnification level that is sufficiently high to enable the identification of at least some synapses in the brain406. The image410can be a volumetric image, i.e., that characterizes a three-dimensional representation of the brain406. The image410can be represented in any appropriate format, e.g., as a three-dimensional array of numerical values.

The imaging system408can be any appropriate system capable of generating synaptic resolution images, e.g., an electron microscopy system. The imaging system408can process “thin sections” from the brain406(i.e., thin slices of the brain attached to slides) to generate output images that each have a field of view corresponding to a proper subset of a thin section. The imaging system408can generate a complete image of each thin section by stitching together the images corresponding to different fields of view of the thin section using any appropriate image stitching technique. The imaging system408can generate the volumetric image410of the brain by registering and stacking the images of each thin section. Registering two images refers to applying transformation operations (e.g., translation or rotation operations) to one or both of the images to align them. Example techniques for generating a synaptic resolution image of a brain are described with reference to: Z. Zheng, et al., “A complete electron microscopy volume of the brain of adultDrosophila melanogaster,” Cell 174, 730-743 (2018).

A graphing system412is configured to process the synaptic resolution image410to generate the synaptic connectivity graph402. The synaptic connectivity graph402specifies a set of nodes and a set of edges, such that each edge connects two nodes. To generate the graph402, the graphing system412identifies each neuron in the image410as a respective node in the graph, and identifies each synaptic connection between a pair of neurons in the image410as an edge between the corresponding pair of nodes in the graph.

The graphing system412can identify the neurons and the synapses depicted in the image410using any of a variety of techniques. For example, the graphing system412can process the image410to identify the positions of the neurons depicted in the image410, and determine whether a synapse connects two neurons based on the proximity of the neurons (as will be described in more detail below). In this example, the graphing system412can process an input including: (i) the image, (ii) features derived from the image, or (iii) both, using a machine learning model that is trained using supervised learning techniques to identify neurons in images. The machine learning model can be, e.g., a convolutional neural network model or a random forest model. The output of the machine learning model can include a neuron probability map that specifies a respective probability that each voxel in the image is included in a neuron. The graphing system412can identify contiguous clusters of voxels in the neuron probability map as being neurons.

Optionally, prior to identifying the neurons from the neuron probability map, the graphing system412can apply one or more filtering operations to the neuron probability map, e.g., with a Gaussian filtering kernel. Filtering the neuron probability map can reduce the amount of “noise” in the neuron probability map, e.g., where only a single voxel in a region is associated with a high likelihood of being a neuron.

The machine learning model used by the graphing system412to generate the neuron probability map can be trained using supervised learning training techniques on a set of training data. The training data can include a set of training examples, where each training example specifies: (i) a training input that can be processed by the machine learning model, and (ii) a target output that should be generated by the machine learning model by processing the training input. For example, the training input can be a synaptic resolution image of a brain, and the target output can be a “label map” that specifies a label for each voxel of the image indicating whether the voxel is included in a neuron. The target outputs of the training examples can be generated by manual annotation, e.g., where a person manually specifies which voxels of a training input are included in neurons.

Example techniques for identifying the positions of neurons depicted in the image410using neural networks (in particular, flood-filling neural networks) are described with reference to: P. H. Li et al.: “Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment,” bioRxiv doi:10.1101/605634 (2019).

The graphing system412can identify the synapses connecting the neurons in the image410based on the proximity of the neurons. For example, the graphing system412can determine that a first neuron is connected by a synapse to a second neuron based on the area of overlap between: (i) a tolerance region in the image around the first neuron, and (ii) a tolerance region in the image around the second neuron. That is, the graphing system412can determine whether the first neuron and the second neuron are connected based on the number of spatial locations (e.g., voxels) that are included in both: (i) the tolerance region around the first neuron, and (ii) the tolerance region around the second neuron. For example, the graphing system412can determine that two neurons are connected if the overlap between the tolerance regions around the respective neurons includes at least a predefined number of spatial locations (e.g., one spatial location). A “tolerance region” around a neuron refers to a contiguous region of the image that includes the neuron. For example, the tolerance region around a neuron can be specified as the set of spatial locations in the image that are either: (i) in the interior of the neuron, or (ii) within a predefined distance of the interior of the neuron.

The graphing system412can further identify a weight value associated with each edge in the graph402. For example, the graphing system412can identify a weight for an edge connecting two nodes in the graph402based on the area of overlap between the tolerance regions around the respective neurons corresponding to the nodes in the image410(e.g., based on a proximity of the respective neurons). The area of overlap can be measured, e.g., as the number of voxels in the image410that are contained in the overlap of the respective tolerance regions around the neurons. The weight for an edge connecting two nodes in the graph402may be understood as characterizing the (approximate) strength of the connection between the corresponding neurons in the brain (e.g., the amount of information flow through the synapse connecting the two neurons).

In addition to identifying synapses in the image410, the graphing system412can further determine the direction of each synapse using any appropriate technique. The “direction” of a synapse between two neurons refers to the direction of information flow between the two neurons, e.g., if a first neuron uses a synapse to transmit signals to a second neuron, then the direction of the synapse would point from the first neuron to the second neuron. Example techniques for determining the directions of synapses connecting pairs of neurons are described with reference to: C. Seguin, A. Razi, and A. Zalesky: “Inferring neural signaling directionality from undirected structure connectomes,” Nature Communications 10, 4289 (2019), doi:10.1038/s41467-019-12201-w.

In implementations where the graphing system412determines the directions of the synapses in the image410, the graphing system412can associate each edge in the graph402with the direction of the corresponding synapse. That is, the graph402can be a directed graph. In some other implementations, the graph402can be an undirected graph, i.e., where the edges in the graph are not associated with a direction.

The graph402can be represented in any of a variety of ways. For example, the graph402can be represented as a two-dimensional array of numerical values with a number of rows and columns equal to the number of nodes in the graph. The component of the array at position (i,j) can have value 1 if the graph includes an edge pointing from node i to node j, and value 0 otherwise. In implementations where the graphing system412determines a weight value for each edge in the graph402, the weight values can be similarly represented as a two-dimensional array of numerical values. More specifically, if the graph includes an edge connecting node i to node j, the component of the array at position (i,j) can have a value given by the corresponding edge weight, and otherwise the component of the array at position (i,j) can have value 0.

An architecture mapping system420can process the synaptic connectivity graph402to determine the architecture of the brain emulation neural network404. For example, the architecture mapping system420can map each node in the graph402to: (i) an artificial neuron, (ii) a neural network layer, or (iii) a group of neural network layers, in the architecture of the brain emulation neural network404. The architecture mapping system420can further map each edge of the graph402to a connection in the brain emulation neural network404, e.g., such that a first artificial neuron that is connected to a second artificial neuron is configured to provide its output to the second artificial neuron. In some implementations, the architecture mapping system420can apply one or more transformation operations to the graph402before mapping the nodes and edges of the graph402to corresponding components in the architecture of the brain emulation neural network404, as will be described in more detail below. An example architecture mapping system is described in more detail below with reference toFIG.5.

The brain emulation neural network404can be provided to a training system414that trains the brain emulation neural network using machine learning techniques, i.e., generates an update to the respective values of one or more parameters of the brain emulation neural network.

In some implementations, the training system414is a supervised training system that is configured to train the brain emulation neural network404using a set of training data. The training data can include multiple training examples, where each training example specifies: (i) a training input, and (ii) a corresponding target output that should be generated by the brain emulation neural network404by processing the training input. In one example, the direct training system414can train the brain emulation neural network404over multiple training iterations using a gradient descent optimization technique, e.g., stochastic gradient descent. In this example, at each training iteration, the direct training system414can sample a “batch” (set) of one or more training examples from the training data, and process the training inputs specified by the training examples to generate corresponding network outputs. The direct training system414can evaluate an objective function that measures a similarity between: (i) the target outputs specified by the training examples, and (ii) the network outputs generated by the brain emulation neural network, e.g., a cross-entropy or squared-error objective function. The direct training system414can determine gradients of the objective function, e.g., using backpropagation techniques, and update the parameter values of the brain emulation neural network404using the gradients, e.g., using any appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam.

In some other implementations, the training system414is an adversarial training system that is configured to train the brain emulation neural network404in an adversarial fashion. For example, the training system414can include a discriminator neural network that is configured to process network outputs generated by the brain emulation neural network404to generate a prediction of whether the network outputs are “real” outputs (i.e., outputs that were not generated by the brain emulation neural network, e.g., outputs that represent data that was captured from the real world) or “synthetic” outputs (i.e., outputs generated by the brain emulation neural network404). The training system can then determine an update to the parameters of the brain emulation neural network in order to increase an error in the prediction of the discriminator neural network; that is, the goal of the brain emulation neural network is to generate synthetic outputs that are realistic enough that the discriminator neural network predicts them to be real outputs. In some implementations, concurrently with training the brain emulation neural network404, the training system414generates updates to the parameters of the discriminator neural network.

In some other implementations, the training system414is a distillation training system that is configured to use the brain emulation neural network404to facilitate training of a “student” neural network having a less complex architecture than the brain emulation neural network404. The complexity of a neural network architecture can be measured, e.g., by the number of parameters required to specify the operations performed by the neural network. The training system414can train the student neural network to match the outputs generated by the brain emulation neural network. After training, the student neural network can inherit the capacity of the brain emulation neural network404to effectively solve certain tasks, while consuming fewer computational resources (e.g., memory and computing power) than the brain emulation neural network404. Typically, the training system414does not update the parameters of the brain emulation neural network404while training the student neural network. That is, in these implementations, the training system414is configured to train the student neural network instead of the brain emulation neural network404.

As a particular example, the training system414can be a distillation training system that trains the student neural network in an adversarial manner. For example, the training system414can include a discriminator neural network that is configured to process network outputs that were generated either by the brain emulation neural network404or the student neural network, and to generate a prediction of whether the network outputs where generated by the brain emulation neural network404or the student neural network. The training system can then determine an update to the parameters of the student neural network in order to increase an error in the prediction of the discriminator neural network; that is, the goal of the student neural network is to generate network outputs that resemble network outputs generated by the brain emulation neural network402so that the discriminator neural network predicts that they were generated by the brain emulation neural network404.

In some implementations, the brain emulation neural network404is a subnetwork of a neural network that includes one or more other neural network layers, e.g., one or more other subnetworks.

For example, the brain emulation neural network404can be a subnetwork of a “reservoir computing” neural network. The reservoir computing neural network can include i) the brain emulation neural network, which includes untrained parameters, and ii) one or more other subnetworks that include trained parameters. For example, the reservoir computing neural network can be configured to process a network input using the brain emulation neural network404to generate an alternative representation of the network input, and process the alternative representation of the network input using a “prediction” subnetwork to generate a network output.

During training of the reservoir computing neural network, the parameter values of the one or more other subnetworks (e.g., the prediction subnetwork) are trained, but the parameter values of the brain emulation neural network404are static, i.e., are not trained. Instead of being trained, the parameter values of the brain emulation neural network404can be determined from the weight values of the edges of the synaptic connectivity graph, as will be described in more detail below. The reservoir computing neural network facilitates application of the brain emulation neural network to machine learning tasks by obviating the need to train the parameter values of the brain emulation neural network404.

After the training system414has completed training the brain emulation neural network404(or a neural network that includes the brain emulation neural network as a subnetwork, or a student neural network trained using the brain emulation neural network), the brain emulation neural network404can be deployed by a deployment system422. That is, the operations of the brain emulation neural network404can be implemented on a device or a system of devices for performing inference, i.e., receiving network inputs and processing the network inputs to generate network outputs. In some implementations, the brain emulation neural network404can be deployed onto a cloud system, i.e., a distributed computing system having multiple computing nodes, e.g., hundreds or thousands of computing nodes, in one or more locations. In some other implementations, the brain emulation neural network404can be deployed onto a user device.

For example, the brain emulation neural network404(or a neural network that includes the brain emulation neural network as a subnetwork, or a student neural network that has been trained using the brain emulation neural network) can be deployed as a recurrent neural network that is configured to process a sequence of network inputs, as described above.

FIG.5shows an example architecture mapping system500. The architecture mapping system500is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The architecture mapping system500is configured to process a synaptic connectivity graph501(e.g., the synaptic connectivity graph402depicted inFIG.4B) to determine a corresponding neural network architecture502of a brain emulation neural network516(e.g., the brain emulation neural network404depicted inFIG.4B). The architecture mapping system500can determine the architecture502using one or more of: a transformation engine504, a feature generation engine506, a node classification engine508, and a nucleus classification engine518, which will each be described in more detail next.

The transformation engine504can be configured to apply one or more transformation operations to the synaptic connectivity graph501that alter the connectivity of the graph501, i.e., by adding or removing edges from the graph. A few examples of transformation operations follow.

In one example, to apply a transformation operation to the graph501, the transformation engine504can randomly sample a set of node pairs from the graph (i.e., where each node pair specifies a first node and a second node). For example, the transformation engine can sample a predefined number of node pairs in accordance with a uniform probability distribution over the set of possible node pairs. For each sampled node pair, the transformation engine504can modify the connectivity between the two nodes in the node pair with a predefined probability (e.g., 0.1%). In one example, the transformation engine504can connect the nodes by an edge (i.e., if they are not already connected by an edge) with the predefined probability. In another example, the transformation engine504can reverse the direction of any edge connecting the two nodes with the predefined probability. In another example, the transformation engine504can invert the connectivity between the two nodes with the predefined probability, i.e., by adding an edge between the nodes if they are not already connected, and by removing the edge between the nodes if they are already connected.

In another example, the transformation engine504can apply a convolutional filter to a representation of the graph501as a two-dimensional array of numerical values. As described above, the graph501can be represented as a two-dimensional array of numerical values where the component of the array at position (i,j) can have value 1 if the graph includes an edge pointing from node i to node j, and value 0 otherwise. The convolutional filter can have any appropriate kernel, e.g., a spherical kernel or a Gaussian kernel. After applying the convolutional filter, the transformation engine504can quantize the values in the array representing the graph, e.g., by rounding each value in the array to 0 or 1, to cause the array to unambiguously specify the connectivity of the graph. Applying a convolutional filter to the representation of the graph501can have the effect of regularizing the graph, e.g., by smoothing the values in the array representing the graph to reduce the likelihood of a component in the array having a different value than many of its neighbors.

In some cases, the graph501can include some inaccuracies in representing the synaptic connectivity in the biological brain. For example, the graph can include nodes that are not connected by an edge despite the corresponding neurons in the brain being connected by a synapse, or “spurious” edges that connect nodes in the graph despite the corresponding neurons in the brain not being connected by a synapse. Inaccuracies in the graph can result, e.g., from imaging artifacts or ambiguities in the synaptic resolution image of the brain that is processed to generate the graph. Regularizing the graph, e.g., by applying a convolutional filter to the representation of the graph, can increase the accuracy with which the graph represents the synaptic connectivity in the brain, e.g., by removing spurious edges.

The architecture mapping system500can use the feature generation engine506and the node classification engine508to determine predicted “types”510of the neurons corresponding to the nodes in the graph501. The type of a neuron can characterize any appropriate aspect of the neuron. In one example, the type of a neuron can characterize the function performed by the neuron in the brain, e.g., a visual function by processing visual data, an olfactory function by processing odor data, or a memory function by retaining information. After identifying the types of the neurons corresponding to the nodes in the graph501, the architecture mapping system500can identify a sub-graph512of the overall graph501based on the neuron types, and determine the neural network architecture502based on the sub-graph512. The feature generation engine506and the node classification engine508are described in more detail next.

The feature generation engine506can be configured to process the graph501(potentially after it has been modified by the transformation engine504) to generate one or more respective node features514corresponding to each node of the graph501. The node features corresponding to a node can characterize the topology (i.e., connectivity) of the graph relative to the node. In one example, the feature generation engine506can generate a node degree feature for each node in the graph501, where the node degree feature for a given node specifies the number of other nodes that are connected to the given node by an edge. In another example, the feature generation engine506can generate a path length feature for each node in the graph501, where the path length feature for a node specifies the length of the longest path in the graph starting from the node. A path in the graph may refer to a sequence of nodes in the graph, such that each node in the path is connected by an edge to the next node in the path. The length of a path in the graph may refer to the number of nodes in the path. In another example, the feature generation engine506can generate a neighborhood size feature for each node in the graph501, where the neighborhood size feature for a given node specifies the number of other nodes that are connected to the node by a path of length at most N. In this example, N can be a positive integer value. In another example, the feature generation engine506can generate an information flow feature for each node in the graph501. The information flow feature for a given node can specify the fraction of the edges connected to the given node that are outgoing edges, i.e., the fraction of edges connected to the given node that point from the given node to a different node.

In some implementations, the feature generation engine506can generate one or more node features that do not directly characterize the topology of the graph relative to the nodes. In one example, the feature generation engine506can generate a spatial position feature for each node in the graph501, where the spatial position feature for a given node specifies the spatial position in the brain of the neuron corresponding to the node, e.g., in a Cartesian coordinate system of the synaptic resolution image of the brain. In another example, the feature generation engine506can generate a feature for each node in the graph501indicating whether the corresponding neuron is excitatory or inhibitory. In another example, the feature generation engine506can generate a feature for each node in the graph501that identifies the neuropil region associated with the neuron corresponding to the node.

In some cases, the feature generation engine506can use weights associated with the edges in the graph in determining the node features514. As described above, a weight value for an edge connecting two nodes can be determined, e.g., based on the area of any overlap between tolerance regions around the neurons corresponding to the nodes. In one example, the feature generation engine506can determine the node degree feature for a given node as a sum of the weights corresponding to the edges that connect the given node to other nodes in the graph. In another example, the feature generation engine506can determine the path length feature for a given node as a sum of the edge weights along the longest path in the graph starting from the node.

The node classification engine508can be configured to process the node features514to identify a predicted neuron type510corresponding to certain nodes of the graph501. In one example, the node classification engine508can process the node features514to identify a proper subset of the nodes in the graph501with the highest values of the path length feature. For example, the node classification engine508can identify the nodes with a path length feature value greater than the 90th percentile (or any other appropriate percentile) of the path length feature values of all the nodes in the graph. The node classification engine508can then associate the identified nodes having the highest values of the path length feature with the predicted neuron type of “primary sensory neuron.” In another example, the node classification engine508can process the node features514to identify a proper subset of the nodes in the graph501with the highest values of the information flow feature, i.e., indicating that many of the edges connected to the node are outgoing edges. The node classification engine508can then associate the identified nodes having the highest values of the information flow feature with the predicted neuron type of “sensory neuron.” In another example, the node classification engine508can process the node features514to identify a proper subset of the nodes in the graph501with the lowest values of the information flow feature, i.e., indicating that many of the edges connected to the node are incoming edges (i.e., edges that point towards the node). The node classification engine508can then associate the identified nodes having the lowest values of the information flow feature with the predicted neuron type of “associative neuron.”

The architecture mapping system500can identify a sub-graph512of the overall graph501based on the predicted neuron types510corresponding to the nodes of the graph501. A “sub-graph” may refer to a graph specified by: (i) a proper subset of the nodes of the graph501, and (ii) a proper subset of the edges of the graph501.FIG.6provides an illustration of an example sub-graph of an overall graph. In one example, the architecture mapping system500can select: (i) each node in the graph501corresponding to particular neuron type, and (ii) each edge in the graph501that connects nodes in the graph corresponding to the particular neuron type, for inclusion in the sub-graph512. The neuron type selected for inclusion in the sub-graph can be, e.g., visual neurons, olfactory neurons, memory neurons, or any other appropriate type of neuron. In some cases, the architecture mapping system500can select multiple neuron types for inclusion in the sub-graph512, e.g., both visual neurons and olfactory neurons.

The type of neuron selected for inclusion in the sub-graph512can be determined based on the task which the brain emulation neural network516will be configured to perform. In one example, the brain emulation neural network516can be configured to perform an image processing task, and neurons that are predicted to perform visual functions (i.e., by processing visual data) can be selected for inclusion in the sub-graph512. In another example, the brain emulation neural network516can be configured to perform an odor processing task, and neurons that are predicted to perform odor processing functions (i.e., by processing odor data) can be selected for inclusion in the sub-graph512. In another example, the brain emulation neural network516can be configured to perform an audio processing task, and neurons that are predicted to perform audio processing (i.e., by processing audio data) can be selected for inclusion in the sub-graph512.

If the edges of the graph501are associated with weight values (as described above), then each edge of the sub-graph512can be associated with the weight value of the corresponding edge in the graph501. The sub-graph512can be represented, e.g., as a two-dimensional array of numerical values, as described with reference to the graph501.

Determining the architecture502of the brain emulation neural network516based on the sub-graph512rather than the overall graph501can result in the architecture502having a reduced complexity, e.g., because the sub-graph512has fewer nodes, fewer edges, or both than the graph501. Reducing the complexity of the architecture502can reduce consumption of computational resources (e.g., memory and computing power) by the brain emulation neural network516, e.g., enabling the brain emulation neural network516to be deployed in resource-constrained environments, e.g., mobile devices. Reducing the complexity of the architecture502can also facilitate training of the brain emulation neural network516, e.g., by reducing the amount of training data required to train the brain emulation neural network516to achieve an threshold level of performance (e.g., prediction accuracy).

In some cases, the architecture mapping system500can further reduce the complexity of the architecture502using a nucleus classification engine518. In particular, the architecture mapping system500can process the sub-graph512using the nucleus classification engine518prior to determining the architecture502. The nucleus classification engine518can be configured to process a representation of the sub-graph512as a two-dimensional array of numerical values (as described above) to identify one or more “clusters” in the array.

A cluster in the array representing the sub-graph512may refer to a contiguous region of the array such that at least a threshold fraction of the components in the region have a value indicating that an edge exists between the pair of nodes corresponding to the component. In one example, the component of the array in position (i,j) can have value 1 if an edge exists from node i to node j, and value 0 otherwise. In this example, the nucleus classification engine518can identify contiguous regions of the array such that at least a threshold fraction of the components in the region have the value 1. The nucleus classification engine518can identify clusters in the array representing the sub-graph512by processing the array using a blob detection algorithm, e.g., by convolving the array with a Gaussian kernel and then applying the Laplacian operator to the array. After applying the Laplacian operator, the nucleus classification engine518can identify each component of the array having a value that satisfies a predefined threshold as being included in a cluster.

Each of the clusters identified in the array representing the sub-graph512can correspond to edges connecting a “nucleus” (i.e., group) of related neurons in brain, e.g., a thalamic nucleus, a vestibular nucleus, a dentate nucleus, or a fastigial nucleus. After the nucleus classification engine518identifies the clusters in the array representing the sub-graph512, the architecture mapping system500can select one or more of the clusters for inclusion in the sub-graph512. The architecture mapping system500can select the clusters for inclusion in the sub-graph512based on respective features associated with each of the clusters. The features associated with a cluster can include, e.g., the number of edges (i.e., components of the array) in the cluster, the average of the node features corresponding to each node that is connected by an edge in the cluster, or both. In one example, the architecture mapping system500can select a predefined number of largest clusters (i.e., that include the greatest number of edges) for inclusion in the sub-graph512.

The architecture mapping system500can reduce the sub-graph512by removing any edge in the sub-graph512that is not included in one of the selected clusters, and then map the reduced sub-graph512to a corresponding neural network architecture, as will be described in more detail below. Reducing the sub-graph512by restricting it to include only edges that are included in selected clusters can further reduce the complexity of the architecture502, thereby reducing computational resource consumption by the brain emulation neural network516and facilitating training of the brain emulation neural network516.

The architecture mapping system500can determine the architecture502of the brain emulation neural network516from the sub-graph512in any of a variety of ways. For example, the architecture mapping system500can map each node in the sub-graph512to a corresponding: (i) artificial neuron, (ii) artificial neural network layer, or (iii) group of artificial neural network layers in the architecture502, as will be described in more detail next.

In one example, the neural network architecture502can include: (i) a respective artificial neuron corresponding to each node in the sub-graph512, and (ii) a respective connection corresponding to each edge in the sub-graph512. In this example, the sub-graph512can be a directed graph, and an edge that points from a first node to a second node in the sub-graph512can specify a connection pointing from a corresponding first artificial neuron to a corresponding second artificial neuron in the architecture502. The connection pointing from the first artificial neuron to the second artificial neuron can indicate that the output of the first artificial neuron should be provided as an input to the second artificial neuron. Each connection in the architecture can be associated with a weight value, e.g., that is specified by the weight value associated with the corresponding edge in the sub-graph. An artificial neuron may refer to a component of the architecture502that is configured to receive one or more inputs (e.g., from one or more other artificial neurons), and to process the inputs to generate an output. The inputs to an artificial neuron and the output generated by the artificial neuron can be represented as scalar numerical values. In one example, a given artificial neuron can generate an output b as:

where σ(·) is a non-linear “activation” function (e.g., a sigmoid function or an arctangent function), {ai}i=1nare the inputs provided to the given artificial neuron, and {wi}i=1nare the weight values associated with the connections between the given artificial neuron and each of the other artificial neurons that provide an input to the given artificial neuron.

In another example, the sub-graph512can be an undirected graph, and the architecture mapping system500can map an edge that connects a first node to a second node in the sub-graph512to two connections between a corresponding first artificial neuron and a corresponding second artificial neuron in the architecture. In particular, the architecture mapping system500can map the edge to: (i) a first connection pointing from the first artificial neuron to the second artificial neuron, and (ii) a second connection pointing from the second artificial neuron to the first artificial neuron.

In another example, the sub-graph512can be an undirected graph, and the architecture mapping system can map an edge that connects a first node to a second node in the sub-graph512to one connection between a corresponding first artificial neuron and a corresponding second artificial neuron in the architecture. The architecture mapping system500can determine the direction of the connection between the first artificial neuron and the second artificial neuron, e.g., by randomly sampling the direction in accordance with a probability distribution over the set of two possible directions.

In some cases, the edges in the sub-graph512is not be associated with weight values, and the weight values corresponding to the connections in the architecture502can be determined randomly. For example, the weight value corresponding to each connection in the architecture502can be randomly sampled from a predetermined probability distribution, e.g., a standard Normal (N(0,1)) probability distribution.

In another example, the neural network architecture502can include: (i) a respective artificial neural network layer corresponding to each node in the sub-graph512, and (ii) a respective connection corresponding to each edge in the sub-graph512. In this example, a connection pointing from a first layer to a second layer can indicate that the output of the first layer should be provided as an input to the second layer. An artificial neural network layer may refer to a collection of artificial neurons, and the inputs to a layer and the output generated by the layer can be represented as ordered collections of numerical values (e.g., tensors of numerical values). In one example, the architecture502can include a respective convolutional neural network layer corresponding to each node in the sub-graph512, and each given convolutional layer can generate an output d as:

where each ci(i=1, . . . , n) is a tensor (e.g., a two- or three- dimensional array) of numerical values provided as an input to the layer, each wi(i=1, . . . , n) is a weight value associated with the connection between the given layer and each of the other layers that provide an input to the given layer (where the weight value for each edge can be specified by the weight value associated with the corresponding edge in the sub-graph), hθ(·) represents the operation of applying one or more convolutional kernels to an input to generate a corresponding output, and σ(·) is a non-linear activation function that is applied element-wise to each component of its input. In this example, each convolutional kernel can be represented as an array of numerical values, e.g., where each component of the array is randomly sampled from a predetermined probability distribution, e.g., a standard Normal probability distribution.

In another example, the architecture mapping system500can determine that the neural network architecture includes: (i) a respective group of artificial neural network layers corresponding to each node in the sub-graph512, and (ii) a respective connection corresponding to each edge in the sub-graph512. The layers in a group of artificial neural network layers corresponding to a node in the sub-graph512can be connected, e.g., as a linear sequence of layers, or in any other appropriate manner.

The neural network architecture502can include one or more artificial neurons that are identified as “input” artificial neurons and one or more artificial neurons that are identified as “output” artificial neurons. An input artificial neuron may refer to an artificial neuron that is configured to receive an input from a source that is external to the brain emulation neural network516. An output artificial neural neuron may refer to an artificial neuron that generates an output which is considered part of the overall output generated by the brain emulation neural network516.

Various operations performed by the described architecture mapping system500are optional or can be implemented in a different order. For example, the architecture mapping system500can refrain from applying transformation operations to the graph501using the transformation engine504, and refrain from extracting a sub-graph512from the graph501using the feature generation engine506, the node classification engine508, and the nucleus classification engine518. In this example, the architecture mapping system500can directly map the graph501to the neural network architecture502, e.g., by mapping each node in the graph to an artificial neuron and mapping each edge in the graph to a connection in the architecture, as described above.

FIG.6illustrates an example graph600and an example sub-graph602. Each node in the graph600is represented by a circle (e.g.,604and606), and each edge in the graph600is represented by a line (e.g.,608and610). In this illustration, the graph600can be considered a simplified representation of a synaptic connectivity graph (an actual synaptic connectivity graph can have far more nodes and edges than are depicted inFIG.6). A sub-graph602can be identified in the graph600, where the sub-graph602includes a proper subset of the nodes and edges of the graph600. In this example, the nodes included in the sub-graph602are hatched (e.g.,606) and the edges included in sub-graph602are dashed (e.g.,610). The nodes included in the sub-graph602can correspond to neurons of a particular type, e.g., neurons having a particular function, e.g., olfactory neurons, visual neurons, or memory neurons. The architecture of the brain emulation neural network can be specified by the structure of the entire graph600, or by the structure of a sub-graph602, as described above.

FIG.7is a flow diagram of an example process700for implementing a brain emulation subnetwork using a compressed matrix representation. For convenience, the process700will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network computing system, e.g., the neural network computing system100depicted inFIG.1, appropriately programmed in accordance with this specification, can perform the process700.

The brain emulation subnetwork is a component of a neural network that is configured to process a network input to generate a network output.

The system process the network input using an input subnetwork of the neural network to generate an embedding of the network input (step702). For example, the input subnetwork can be a trained subnetwork, e.g., the first trained subnetwork204depicted inFIG.2. As another example, the output subnetwork can be another brain emulation subnetwork.

The system processing the embedding of the network input using the brain emulation subnetwork to generate a brain emulation subnetwork output (step704). The brain emulation subnetwork include brain emulation parameters that each correspond to a respective synaptic connection between a respective pair of biological neurons in the brain of a biological organism. Values for the brain emulation parameters can be specified by synaptic connectivity between the biological neurons in the brain of the biological organism.

To generate the brain emulation subnetwork output, the system can obtain a compressed matrix representation of a sparse matrix that includes the brain emulation parameters. The sparse matrix represents the architecture of the brain emulation subnetwork. The system can then apply the compressed matrix representation to the embedding of the network input to generate the brain emulation subnetwork output.

The system processes the brain emulation subnetwork output using an output subnetwork of the neural network to generate the network output (step706). For example, the output subnetwork can be a trained subnetwork, e.g., the second trained subnetwork212depicted inFIG.2. As another example, the output subnetwork can be another brain emulation subnetwork.

FIG.8is a flow diagram of an example process800for generating a brain emulation neural network. For convenience, the process800will be described as being performed by a system of one or more computers located in one or more locations.

The system obtains a synaptic resolution image of at least a portion of a brain of a biological organism (802).

The system processes the image to identify: (i) neurons in the brain, and (ii) synaptic connections between the neurons in the brain (804).

The system generates data defining a graph representing synaptic connectivity between the neurons in the brain (806). The graph includes a set of nodes and a set of edges, where each edge connects a pair of nodes. The system identifies each neuron in the brain as a respective node in the graph, and each synaptic connection between a pair of neurons in the brain as an edge between a corresponding pair of nodes in the graph.

The system determines an artificial neural network architecture corresponding to the graph representing the synaptic connectivity between the neurons in the brain (808).

The system processes a network input using an artificial neural network having the artificial neural network architecture to generate a network output (810).

FIG.9is a flow diagram of an example process900for determining an artificial neural network architecture corresponding to a sub-graph of a synaptic connectivity graph. For convenience, the process900will be described as being performed by a system of one or more computers located in one or more locations. For example, an architecture mapping system, e.g., the architecture mapping system500ofFIG.5, appropriately programmed in accordance with this specification, can perform the process900.

The system obtains data defining a graph representing synaptic connectivity between neurons in a brain of a biological organism (902). The graph includes a set of nodes and edges, where each edge connects a pair of nodes. Each node corresponds to a respective neuron in the brain of the biological organism, and each edge connecting a pair of nodes in the graph corresponds to a synaptic connection between a pair of neurons in the brain of the biological organism.

The system determines, for each node in the graph, a respective set of one or more node features characterizing a structure of the graph relative to the node (904).

The system identifies a sub-graph of the graph (906). In particular, the system selects a proper subset of the nodes in the graph for inclusion in the sub-graph based on the node features of the nodes in the graph.

The system determines an artificial neural network architecture corresponding to the sub-graph of the graph (908).

FIG.10shows an example optimization system1000. The optimization system1000is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The optimization system1000is configured to generate candidate graphs1003using a graph generation engine1002. The graph generation engine1002is configured to process a synaptic connectivity graph1001in accordance with a set of graph generation parameters1012to generate an output graph1004that is added to the set of candidate graphs1003. The optimization system1000iteratively optimizes the parameters1012of the graph generation engine1002using an optimization engine1010to increase the performance measures1008of the output graphs1004generated by the graph generation engine1002, as will be described in more detail below.

The parameters1012of the graph generation engine1002specify transformation operations that are applied to the synaptic connectivity graph1001to generate an output graph1004. The graph generation engine1002may generate the output graph1004by applying transformation operations to a representation of the synaptic connectivity graph1001as a two-dimensional array of numerical values. As described above, a graph may be represented as a two-dimensional array of numerical values with a number of rows and columns equal to the number of nodes in the graph. The component of the array at position (i,j) may have value 1 if the graph includes an edge pointing from node i to node j, and value 0 otherwise.

In one example, as part of generating an output graph1004, the graph generation engine1002may apply a convolutional filtering operation specified by a filtering kernel to the array representing the synaptic connectivity graph1001. In this example, the graph generation parameters1012may specify the components of a matrix defining the filtering kernel.

In another example, as part of generating an output graph1004, the graph generation engine1002may apply a “shifting” operation to the array representing the synaptic connectivity graph1001, e.g., such that each the value in each component of the array is translated “left”, “right”, “up”, or “down”. Components that are shifted outside the bounds of the array may be wrapped around the opposite side of the array. In this example, the graph generation parameters1012may specify the direction and magnitude of the shifting operation.

In another example, as part of generating an output graph, the graph generation engine may apply a cropping operation to the adjacency matrix representing the synaptic connectivity graph, where the cropping operation replaces the adjacency matrix representing the synaptic connectivity graph with an adjacency matrix representing a sub-graph of the synaptic connectivity graph. Generally, a “sub-graph” may refer to a graph specified by: (i) a proper subset of the nodes of the synaptic connectivity graph, and (ii) a proper subset of the edges of the synaptic connectivity graph. The cropping operation may specify a sub-graph of synaptic connectivity graph, e.g., by specifying a proper subset of the rows and a proper subset of the columns of the adjacency matrix representing the synaptic connectivity graph that define a sub-matrix of the adjacency matrix. The sub-graph may include: (i) each edge specified by the sub-matrix, and (ii) each node that is connected by an edge specified by the sub-matrix.

At each of multiple iterations, the graph generation engine1002processes the synaptic connectivity graph1001in accordance with the current values of the graph generation parameters1012to generate an output graph1004which may then be added to the set of candidate graphs1003. The optimization system1000determines a performance measure1008of the output graph1004using an evaluation engine1006, and then provides the performance measure1008of the output graph1004to the optimization engine1010.

In some implementations, each edge of the synaptic connectivity graph may be associated with a weight value that is determined from the synaptic resolution image of the brain, as described above. Each candidate graph may inherit the weight values associated with the edges of the synaptic connectivity graph. For example, each edge in the candidate graph that corresponds to an edge in the synaptic connectivity graph may be associated with the same weight value as the corresponding edge in the synaptic connectivity graph. Edges in the candidate graph that do not correspond to edges in the synaptic connectivity graph may be associated with default or randomly initialized weight values.

The performance measure for a candidate graph characterizes the performance of a neural network that includes a brain emulation neural network layer having an architecture specified by the candidate graph at performing a machine learning task. More specifically, to determine the performance measure of a candidate graph, the evaluation system1006can map the candidate graph to a corresponding brain emulation neural network layer, e.g., using the architecture mapping system420described with reference toFIG.4B, e.g., by mapping each node in the candidate graph to an artificial neuron in the brain emulation neural network layer, each edge in the candidate graph to a connection between a corresponding pair of artificial neurons in the brain emulation neural network layer, and the weight value associated with each edge in the candidate graph to a parameter value associated with the corresponding connection in the brain emulation neural network layer.

The evaluation engine may measure the performance of a neural network that includes a brain emulation neural network layer having an architecture specified by the candidate graph (e.g., the neural network102described with reference toFIG.1), e.g., by training the neural network on a set of training data, and then evaluating the performance of the trained neural network on a set of validation data. Both the training data and the validation data may include training examples, where each training example specifies: (i) a network input, and (ii) a target output, i.e., that should be generated by processing the network input. In determining the performance measure of a neural network, the evaluation engine trains the neural network on the training data, but reserves the validation data for evaluating the performance of the trained neural network (i.e., by not training the neural network on the validation data). The evaluation engine may evaluate the performance of the trained neural network on the validation data, e.g., by using an objective function to measure an error between: (i) the target outputs specified by the validation data, and (ii) the predicted outputs generated by the trained neural network. The objective function may be, e.g., a squared-error objective function, or any other appropriate objective function.

The optimization engine1010is configured to process the performance measures1008of the output graphs1004to determine adjustments to the current values of the graph generation parameters to encourage the generation of output graphs with higher performance measures. Prior to the first iteration, the values of the graph generation parameters1012may be set to default values or randomly initialized. The optimization engine1010may use any appropriate optimization technique, e.g., a “black-box” optimization technique that does not rely on computing gradients of the transformation operations applied by the graph generation engine1002. Examples of black-box optimization techniques which may be implemented by the optimization engine1010are described with reference to: Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D.: “Google vizier: A service for black-box optimization,” In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1487-1495 (2017).

After the final iteration, the optimization system1000may identify a best-performing candidate graph based on the performance measures. For example, the optimization system may identify the best-performing graph as the candidate graph with the highest performance measure. After identifying the best-performing graph, the optimization system may provide the brain emulation neural network layer specified by the best-performing graph for use as part of a neural network, e.g., the neural network102described with reference toFIG.1.

FIG.11is a block diagram of an example computer system1100that can be used to perform operations described previously. The system1100includes a processor1110, a memory1120, a storage device1130, and an input/output device1140. Each of the components1110,1120,1130, and1140can be interconnected, for example, using a system bus1150. The processor1110is capable of processing instructions for execution within the system1100. In one implementation, the processor1110is a single-threaded processor. In another implementation, the processor1110is a multi-threaded processor. The processor1110is capable of processing instructions stored in the memory1120or on the storage device1130.

The memory1120stores information within the system1100. In one implementation, the memory1120is a computer-readable medium. In one implementation, the memory1120is a volatile memory unit. In another implementation, the memory1120is a non-volatile memory unit.

The storage device1130is capable of providing mass storage for the system1100. In one implementation, the storage device1130is a computer-readable medium. In various different implementations, the storage device1130can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (for example, a cloud storage device), or some other large capacity storage device.

The input/output device1140provides input/output operations for the system1100. In one implementation, the input/output device1140can include one or more network interface devices, for example, an Ethernet card, a serial communication device, for example, and RS-232 port, and/or a wireless interface device, for example, and 802.11 card. In another implementation, the input/output device1140can include driver devices configured to receive input data and send output data to other input/output devices, for example, keyboard, printer and display devices1160. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, and set-top box television client devices.

Embodiment 1 is a method comprising:

obtaining a network input; and

processing the network input using a neural network to generate a network output, comprising:processing the network input using an input subnetwork of the neural network to generate an embedding of the network input;processing the embedding of the network input using a brain emulation subnetwork of the neural network, wherein the brain emulation subnetwork has a brain emulation neural network architecture that represents synaptic connectivity between a plurality of biological neurons in a brain of a biological organism, the processing comprising:obtaining a compressed matrix representation of a sparse matrix of brain emulation parameters representing synaptic connectivity between the plurality of biological neurons in the brain of the biological organism; andapplying the compressed matrix representation to the embedding of the network input to generate a brain emulation subnetwork output; andprocessing the brain emulation subnetwork output using an output subnetwork of the neural network to generate the network output.

Embodiment 2 is the method of embodiment 1, wherein the compressed matrix representation identifies only a proper subset of the brain emulation parameters of the sparse matrix.

Embodiment 3 is the method of embodiment 2, wherein the compressed matrix representation identifies only brain emulation parameters in the sparse matrix that have non-zero values, and excludes brain emulation parameters in the sparse matrix having value zero.

Embodiment 4 is the method of any one of embodiments 2 or 3, wherein the compressed matrix representation identifies: (i) all brain emulation parameters in the sparse matrix that have non-zero values, and (ii) a proper subset of brain emulation parameters in the sparse matrix having value zero.

Embodiment 5 is the method of any one of embodiments 1-4, wherein the compressed matrix representation comprises data defining: (i) a respective value, and (ii) a respective position in the sparse matrix, of each brain emulation parameter that is identified in the compressed matrix representation.

Embodiment 6 is the method of any one of embodiments 1-5, wherein the sparse matrix comprises at least 100 million brain emulation parameters.

Embodiment 7 is the method of any one of embodiments 1-6, wherein at least 90% of brain emulation parameters in the sparse matrix have value zero.

Embodiment 8 is the method of any one of embodiments 1-7, wherein the sparse matrix representing synaptic connectivity between the plurality of biological neurons is a two-dimensional matrix of brain emulation parameters arranged into a plurality of rows and a plurality of columns,

wherein each row and each column of the sparse matrix correspond to a respective biological neuron from the plurality of biological neurons, and

wherein each brain emulation parameter of the sparse matrix corresponds to a respective pair of biological neurons in the brain of the biological organism, the pair comprising: (i) the biological neuron corresponding to a row of the brain emulation parameter in the sparse matrix, and (ii) the biological neuron corresponding to a column of the brain emulation parameter in the sparse matrix.

Embodiment 9 is the method of embodiment 8, wherein each brain emulation parameter of the sparse matrix has a respective value that characterizes synaptic connectivity in the brain of the biological organism between the respective pair of biological neurons corresponding to the brain emulation parameter.

Embodiment 10 is the method of embodiment 9, wherein each brain emulation parameter of the sparse matrix that corresponds to a respective pair of biological neurons that are not connected by a synaptic connection in the brain of the biological organism has value zero.

Embodiment 11 is the method of any one of embodiments 9 or 10, wherein each brain emulation parameter of the sparse matrix that corresponds to a respective pair of biological neurons that are connected by a synaptic connection in the brain of the biological organism has a respective non-zero value that is based on a proximity of the pair of biological neurons in the brain of the biological organism.

Embodiment 12 is the method of any one of embodiments 1-11, wherein applying the compressed matrix representation to the embedding of the network input to generate the brain emulation subnetwork output comprises:

determining the brain emulation subnetwork output to be a result of a matrix multiplication of: (i) the sparse matrix represented by the compressed matrix representation, and (ii) the embedding of the network input, without performing any scalar multiplications by brain emulation parameters of the sparse matrix that are not identified in the compressed matrix representation.

Embodiment 13 is the method of any one of embodiments 1-12, further comprising training a plurality of neural network parameters of the neural network to optimize an objective function, the training comprising, at each of a plurality of training iterations:

processing one or more network inputs using the neural network, in accordance with current values of the plurality of neural network parameters, to generate corresponding network outputs;

determining gradients, with respect to the neural network parameters, of an objective function that depends on the network outputs; and

updating the current values of the neural network parameters using the gradients.

Embodiment 14 is the method of embodiment 13, wherein updating the current values of the neural network parameters using the gradients comprises updating current values of the brain emulation parameters of the sparse matrix,

Embodiment 15 is the method of any one of embodiments 13 or 14, wherein the training further comprises, at each of a plurality of training iterations:

removing one or more brain emulation parameters from the compressed matrix representation of the sparse matrix; and

adding one or more new brain emulation parameters to the compressed matrix representation of the sparse matrix.

Embodiment 16 is the method of embodiment 15, wherein removing one or more brain emulation parameters from the compressed matrix representation of the sparse matrix comprises one or more of:

for each brain emulation parameter in the compressed matrix representation, randomly selecting the brain emulation parameter for removal with a likelihood that is inversely proportional with a magnitude of the value of the brain emulation parameter;

for each brain emulation parameter in the compressed matrix representation, randomly selecting the brain emulation parameter for removal with a likelihood that is inversely proportional with a rank of the magnitude of the value of the brain emulation parameter in a ranking of the magnitudes of the values of the brain emulation parameters; or

performing a multi-stage process comprising:a first stage comprising generating a set of candidate brain emulation parameters by randomly sampling the candidate brain emulation parameters according to a ranking of the magnitudes of the values of the brain emulation parameters; anda second stage comprising randomly selecting brain emulations parameters from the set of candidate brain emulation parameters for removal according to the magnitudes of the values of the candidate brain emulation parameters.

Embodiment 17 is the method of any one of embodiments 15 or 16, wherein adding one or more new brain emulation parameters to the compressed matrix representation of the sparse matrix comprises:

randomly selecting one or more brain emulation parameters that are not identified in the compressed matrix representation; and

for each randomly selected brain emulation parameter, adding the randomly selected brain emulation parameter to the compressed matrix representation, and assigning an initial value of zero to the randomly selected brain emulation parameter.

Embodiment 18 is the method of any one of embodiments 1-17, wherein the sparse matrix of brain emulation parameters representing synaptic connectivity between the plurality of biological neurons in the brain of the biological organism is determined from a synaptic resolution image of at least a portion of the brain of the biological organism, the determining comprising:

processing the synaptic resolution image to identify: (i) the plurality of biological neurons, and (ii) a plurality of synaptic connections between pairs of biological neurons; and

determining a respective value for each brain emulation parameter in the sparse matrix, comprising:setting a value of each brain emulation parameter that corresponds to a pair of biological neurons in the brain that are not connected by a synapse to zero; andsetting a value of each brain emulation parameter that corresponds to a pair of biological neurons in the brain that are connected by a synapse based on a proximity of the pair of biological neurons in the brain.

Embodiment 19 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any one of embodiments 1 to 18.

Embodiment 20 is one or more non-transitory computer storage media encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform the method of any one of embodiments 1 to 18.