Title: A SYNAPTIC NEURAL NETWORK AND SYNAPSE LEARNING

Abstract: A Synaptic Neural Network (SynaNN) consists of synapses and neurons. Inspired by the synapse research of neuroscience, we built a synapse model with a nonlinear synapse function of excitatory and inhibitory channel probabilities. Introduced the concept of surprisal space and constructed a commutative diagram, we proved that the inhibitory probability function -log(1-exp(-x)) in surprisal space is the topologically conjugate function of the inhibitory complementary probability 1-x in probability space. Furthermore, we found that the derivative of the synapse over the parameter in the surprisal space is equal to the negative Bose-Einstein distribution. In addition, we constructed a fully connected synapse graph (tensor) as a synapse block of a synaptic neural network. Moreover, we proved the gradient formula of a cross-entropy loss function over parameters, so synapse learning can work with the gradient descent and backpropagation algorithms. In the proof-of-concept experiment, we performed an MNIST training and testing on the MLP model with synapse network as hidden layers.

1. INTRODUCTION:

Synapses play an important role in biological neural networks (Hubel & Kandel (1979) ). They are joint points of neurons' connection with the capability of learning and memory in neural networks. Based on the analysis of excitatory and inhibitory channels of synapses (Hubel & Kandel (1979) ), we proposed a probability model (Feller (2008) for probability introduction) of the synapse together with a non-linear function of excitatory and inhibitory probabilities (Li (2017) (synapse function)). Inspired by the concept of surprisal from (Jones (1979)(self-information), Levine (2009) , Bernstein & Levine (1972) (surprisal analysis), Levy (2008) (surprisal theory in language)) or negative logarithmic space (Miyashita et al. (2016) ), we proposed the concept of surprisal space and represented the synapse function as the addition of the excitatory function and inhibitory function in the surprisal space. By applying a commutative diagram, we figured out the fine structure of inhibitory function and proved that it was the topologically conjugate function of an inhibitory function. Moreover, we discovered (rediscovered) that the derivative of the inhibitory function over parameter was equal to the negative Bose-Einstein distribution (Nave (2018)). Furthermore, we constructed a fully connected synapse graph and figured out its synapse tensor expression. From synapse tensor and a cross-entropy loss function, we found and proved its gradient formula that was the basis for gradient descent learning and using backpropagation algorithm. In surprisal space, the parameter (weight) updating for learning was the addition of the value of the negative Bose-Einstein distribution. Finally, we designed the program to implement a Multiple Layer Perceptrons (MLP) (Minsky et al. (2017) ) for MNIST (LeCun et al. (2010) ) and tested it to achieve the near equal accuracy of standard MLP in the same setting.

1.1 BACKGROUND Hodgkin and Huxley presented a physiological neuron model that described the electronic potential of the membrane between a neuron and a synapse with a differential equation (Hodgkin & Huxley (1952) ). Later, neuron scientists have found that a synapse might have a complicated channel structure with rich chemical and electronic properties (Lodish et al. (1995) (biological synapse), Destexhe et al. (1994) (computing synaptic conductances), Abbott & Nelson (2000) (synaptic plasticity)). Other synapse models based on differential equations had been proposed and been simulated by analogy circuits like Spiking Neural Network (SNN) (Kumar (2017) (differential equations), Lin et al. ( 2018) (Intel's SNN Loihi)). In these approaches, synapses acted as linear amplifiers with adjustable coefficients. An example was the analog circuit implementation of Hopfield neural network (Hopfield & Tank (1986) (analog neural circuits)).

In this paper, we proposed a simple synapse model represented by the joint opening probability of excitatory and inhibitory channels in a synapse. It was described as a non-linear computable synapse function. This neuroscience-inspired model was motivated on our unpublished research to solve optimization problems by neural networks.

To do learning by gradient descent and backpropagation algorithm (Goodfellow et al. (2016) (book on deep learning)), because of the differentiable of the synapse function in the synaptic neural network, we could compute Jacobian matrix explicitly and compute the gradient of the cross-entropy loss function over parameters. Therefore, we provided a detailed proof of the formula of gradients in Appendix A

In the process of analyzing Jacobian matrix, we found that the derivative of the inhibitory function log(1 -e -x ) was equal to the 1/(e x -1) which was the formula of Bose-Einstein distribution (Einstein (1924) (quantum ideal gas)). In statistical physics and thermodynamics, Bose-Einstein distribution had been concluded from the geometric series of the Bose statistics.

A dual space analysis was an efficient scientific method. After successful expressing fully-connected synapse network in a logarithmic matrix, we started to consider log function and log space. The concept of surprisal (where was the first definition of surprisal?), which was the measurement of surprise from Information Theory (Shannon (1948)), gave us hints. Original surprisal was defined on the random variable, however, it was convenient to consider the probability itself as a variable. So we introduced the surprisal space with a mapping function -log(p). The motivation was to transform any points from probability space to surprisal space and in reverse.

In surprisal space, a synapse function was the addition of an excitatory identity function and an inhibitory function. Although we had figured out the inhibitory function being -log(1 -e -x ), we wanted to know its structure and what class it belonged to.

This was a procedure that we rediscovered the way to construct a commutative diagram for synapse inhibitory function Diagram (2.2.3). In 1903, Mathematician Bertrand Russell presented the first commutative diagram in his book (Russell (2009)) before the category theory. You can find a good introduction of applied category theory by (Bradley (2018) ). In this paper, we did not require to know category theory.

The basic idea was to given two spaces and two points in source space which have corresponding points in target space by a continuous and inverse mapping function from source space to target space, plus, a function that maps start point to the endpoint in the same source space. Our question is to find the function that maps the corresponding start point to the corresponding endpoint in the target space (refer to diagram 2.2.3). There are two paths from source start point to target endpoint: one is from top-left, go right and down to bottom-right; another is from top-left, go down and right to bottom-right. The solution is to solve the equation that has the same target endpoint.

We found that the synapse inhibitory function -log(1 -e -x ) was also a topologically conjugate function. Therefore, the synaptic neural network has the same dynamical behavior in both probability space and surprisal space. To convince that the synaptic neural network can work for learning and using the backpropagation algorithm, we proved the gradients of loss function by applying basic calculus. In surprisal space, the negative Bose-Einstein distribution was applied to the updating of parameters in the learning of synaptic neural network. Finally, we implemented a MNIST experiment of MLP to be the proof-of-concept.

1.2. CONTRIBUTIONS:

1) present a neuroscience-inspired synapse model and a synapse function based on the opening probability of channels. 2) defined surprisal space to link information theory to the synaptic neural network. 3) figure out function G(x) = -log(1 -e -x ) as the inhibitory part of a synapse. 4) find the derivative of G(x) to be the formula of negative Bose-Einstein distribution. 5) discover G(x) to be a topologically conjugate function of the complementary probability. 6) represent fully-connected synapse as a synapse tensor. 7) express synapse learning of gradient descent as a negative Bose-Einstein distribution.

2. SYNAPTIC NEURAL NETWORK (SYNANN):

A Synaptic Neural Network (SynaNN) contains non-linear synapse networks that connect to neurons. A synapse consists of an input from the excitatory-channel, an input from the inhibitory-channel, and an output channel which sends a value to other synapses or neurons. Synapses may form a graph to receive inputs from neurons and send outputs to other neurons. In advance, many synapse graphs can connect to neurons to construct a neuron graph. In traditional neural network, its synapse graph is simply the wight matrix or tensor.

2.1. SYNAPSE IN PROBABILITY SPACE:



2.1.1. BIOLOGICAL SYNAPSE:

Changes in neurons and synaptic membranes (i.e. potential gate control channel and chemical gate control channel show selectivity and threshold) explain the interactions between neurons and synapses (Torre & Poggio (1978) ). The process of the chemical tokens (neurotransmitters) affecting the control channel of the chemical gate is accomplished by a random process of mixing tokens of the small bulbs on the membrane. Because of the randomness, a probabilistic model does make sense for the computational model of the biological synapse (Hubel & Kandel (1979) ).

In a synapse, the Na+ channel illustrates the effect of an excitatory-channel. The Na+ channels allow the Na+ ions flow in the membrane and make the conductivity increase, then produce excitatory post-synapse potential. The K+ channels illustrate the effects of inhibitory channels. The K+ channel that lets the K+ ions flow out of the membrane shows the inhibition. This makes the control channel of potential gate closing and generates inhibitory post-potential of the synapse. Other kinds of channels (i.e. Ca channel) have more complicated effects. Biological experiments show that there are only two types of channels in a synapse while a neuron may have more types of channels on the membrane. Experiments illustrate that while a neuron is firing, it generates a series of spiking pulse where the spiking rate (frequency) reflects the strength of stimulation.

2.1.2. NEUROSCIENCE-INSPIRED SYNAPSE MODEL:

From neuroscience, there are many types of chemical channels in the membrane of a synapse. They have the following properties: 1) the opening properties of ion channels reflect the activity of synapses. 2) the excitatory and inhibitory channels are two key types of ion channels. 3) the random properties of channels release the statistical behavior of synapses.

From the basic properties of synapses, we proposed the synapse model below:

1) The open probability x of the excitatory channel (α-channel) is equal to the number of open excitatory channels divided by the total number of excitatory channels of a synapse.

2) The open probability y of the inhibitory channel (β-channel) is equal to the number of open inhibitory channels divided by the total number of inhibitory channels of a synapse. 3) The joint probability of a synapse that affects the activation of the connected output neuron is the product of the probability of excitatory channel and the complementary probability of the inhibitory channel. 4) There are two parameters to control excitatory channel and inhibitory channel respectively.

Given two random variables (X, Y ), their probabilities (x, y), and two parameters (α, β), the joint probability distribution function S(x, y; α, β) for X, Y (the joint probability of a synapse that activates the connected neuron) is defined as

S(x, y; α, β) = αx(1 -βy) (1)

where x ∈ (0, 1) is the open probability of all excitatory channels and α > 0 is the parameter of the excitatory channels; y ∈ (0, 1) is the open probability of all inhibitory channels and β ∈ (0, 1) is the parameter of the inhibitory channels. The symbol semicolon ";" separates the variables and parameters in the definition of function S.

2.2. SYNAPSE IN SURPRISAL SPACE:

Surprisal (self-information) is a measure of the surprise in the unit of bit, nat, or hartley when a random variable is sampled. Surprisal is a fundamental concept of information theory and other basic concepts such as entropy can be represented as the function of surprisal. The concept of surprisal has been successfully used in molecular chemistry and natural language research.

2.2.1. SURPRISAL SPACE:

Given a random variable X with value x, the probability of occurrence of x is p(x). The standard definitions of Surprisal I p (x) is the measure of the surprise in the unit of a bit (base 2), a nat (base e), or a hartley (base 10) when the random variable X is sampled at x. Surprisal is the negative logarithmic probability of x such that I p (x) = -log(p(x)). Ignored random variable X, we can consider p(x) as a variable in Probability Range Space or simply called Probability Space in the context of this paper which is the open interval (0,1) of real numbers.

Surprisal Function is defined as I : (0, 1) → (0, ∞) and I(x) = -log(x) where x ∈ (0, 1) is an open interval in R + . Its inverse function is I -1 (u) = e -u where u ∈ R + . Since surprisal function I(x) is bijective, exists inverse and is continuous, I(x) is a homeomorphism.

Surprisal Space S is the mapping space of the Probability Space P with the negative logarithmic function which is a bijective mapping from the open interval (0, 1) of real numbers to the real open interval (0,

EQUATION

The probability space P and the surprisal space S are topological spaces of real open interval (0,1) and positive real numbers R + that inherit the topology of real line respectively.

2.2.2. SURPRISAL SYNAPSE:

Given variables u, v ∈ S and parameters θ, γ ∈ S which are equal to variables -log(x), -log(y) and parameters -log(α), -log(β) respectively. The Surprisal Synapse LS(u, v; θ, γ) ∈ S is defined as, LS(u, v; θ, γ) = -log(S(x, y; α, β))

Expanding the right side, there is LS(u, v; θ, γ) = (-log(αx)) + (-log(1 -βy)). The first part is an identity mapping plus a parameter. To understand the second part more, we need to figure out its structure and class.

2.2.3. TOPOLOGICAL CONJUGACY:

Theorem 1 (Topologically conjugate function). Given y = F(x) where F(x) = 1 -x; x, y ∈ P, (u, v) = I(x, y) where u, v ∈ S, and the homeomorphism I(x) = -log(x) from P to S, then function

EQUATION

Proof. Building a commutative diagram with the homeomorphism I(x) below,

x ∈ P y ∈ P u ∈ S v ∈ S F I G I

The proof is to figure out the equivalent of two paths from x to v. One path is from top x, go right to y and go down to bottom so v = I(F(x)). Another path is from top x, go down to u and go right to bottom so v = G • I, thus, I(F(x)) = G(I(x)). Let • be the composition of functions, the previous equation is I • F = G • I. Applying I -1 on both right sides and compute G on given functions, we proved Eq.( 4).

Given two topological spaces P and S, continuous function F : P → P and G : S → S as well as homeomorphism Let parametric function be D(u; θ) = u + θ, the surprisal synapse is

I : P → S, if I • F = G • I, then G is called

EQUATION

From Eq.( 5), the universal function of a surprisal synapse is the addition of the excitatory function and the topologically conjugate inhibitory function in surprisal space. By constructed a commutative diagram, we figured out the elegant structure and topological conjugacy of the function -log(1-e -u ), which is a new example of the commutative diagram and the topological conjugate function from synaptic neural network. A bridge has been built to connect the synaptic neural network to the category theory, the topology, and the dynamical system.

2.2.4. BOSE-EINSTEIN DISTRIBUTION:

It is interesting to find the connection between the surprisal synapse and the topologically conjugate function. Furthermore, we are going to figure out the connection between the surprisal synapse and the Bose-Einstein distribution. The Bose-Einstein distribution (BED) is represented as the formula f (E) =

1 Ae E/kT -1 where f(E) is the probability that a particle has the energy E in temperature T. k is Boltzmann constant, A is the coefficient (Nave ( 2018)).

Theorem 2. The BED function is defined as BED(v; γ) = 1 e γ+v -1 where variable v ∈ S, parameter γ ∈ S, and v + γ ≥ ln(2), so that 0 ≤ BED(v; γ) ≤ 1, then there is

∂ ∂γ G(D(v; γ)) = BED(v; γ) or ∂ ∂γ (-log(1 -e -D(v;γ) )) = -1 e D(v;γ) -1 (6)

Proof. Proved by computing the derivative of the function on left side.

Recall that D(v; γ) = v +γ, the derivative of the topologically conjugate function G over parameter γ is equal to the negative Bose-Einstein distribution. The gradient of the surprisal synapse

LS(u, v; θ, γ) is ( ∂ ∂u ; ∂ ∂θ , ∂ ∂v ; ∂ ∂γ )LS(u, v; θ, γ) = (1; 1, -BED(v; γ); -BED(v, γ)) (7)

This is a connection between surprisal synapse and statistical physics. In physics, BED(v; γ) can be thought of as the probability that boson particles remain in v energy level with an initial value γ.

2.3. SYNAPSE GRAPH AND TENSOR:

Generally, a biological neuron consists of a soma, an axon, and dendrites. Synapses are distributed on dendritic trees and the axon connects to other neurons in the longer distance. A synapse graph is the set of synapses on dendritic trees of a neuron. A synapse can connect its output to an input of a neuron or to an input of another synapse. A synapse has two inputs: one is excitatory input and another is inhibitory input. Typically neurons receive signals via the synapses on dendrites and send out spiking plus to an axon (Hubel & Kandel (1979) ).

Assume that the total number of input of the synapse graph equals the total number of outputs, the fully-connected synapse graph is defined as

EQUATION

where

x = (x 1 , • • • , x n ), x i ∈ (0, 1) and y = (y 1 , • • • , y n ) are row vectors of probability distribution; β β β i = (β i1 , • • • , β in ), 0 < β ij < 1 are row vectors of parameters; β β β = matrix{β ij }

is the matrix of all parameters. α α α = 1 is assigned to Eq.1 to simplify the computing.

An output y i of the fully-connected synapse graph is constructed by linking the output of a synapse to the excitatory input of another synapse in a chain while the inhibitory input of each synapse is the output of neuron x i in series. In the case of the diagonal value β ii is zero, there is no self-correlated factor in the ith item.

This fully-connected synapse graph represents that only neuron itself acts as excitation all its connected synapses act as inhibition. This follows the observation of neuroscience that most synapses act as inhibition.

Theorem 3 (Synapse tensor formula). The following synapse tensor formula Eq.9 is equivalent to fully-connected synapse graph defined in the Eq.8

EQUATION

or

I(y) = I(x) + 1 |x| * I(1 |β| -diag(x) * β β β T )

where x, y, and β β β are distribution vectors and parameter matrix. β β β T is the transpose of the matrix β β β. 1 |x| is the row vector of all real number ones and 1 |β| is the matrix of all real number ones that have the same size and dimension of x and β β β respectively. Moreover, the * is the matrix multiplication, diag(x) is the diagonal matrix of the row vector x, and the log is the logarithm of the tensor (matrix).

Proof. Applying the log on both sides of the definition Eq.( 8) and completes the matrix multiplications in the fully-connected synapse graph, we proved the formula Eq.( 9). Furthermore, by the definition of I(x), we have the expression of surprisal synapse.

2.4. SYNAPSE LEARNING:

To prove that synapse learning of synaptic neural network is compatible with the standard backpropagation algorithm, we are going to apply cross-entropy as the loss function and use gradient descent to minimize that loss function.

2.4.1. GRADIENT OF LOSS FUNCTION:

The basic idea of deep learning is to apply gradient descent optimization algorithm to update the parameters of the deep neural network and achieve a global minimization of the loss function (Goodfellow et al. (2016) ).

Theorem 4 (Gradient equation). Let the loss function L(ô ô ô, o o o) of the fully-connected synapse graph Eq.8 be equal to the sum of cross-entropy L(ô ô ô, o o o) =i ôi log(o i ), then its parameter gradient is

∂L(ô ô ô, o o o) ∂β ij = (o i τ -ôi ) -y i x j 1 -β ij x j (10) or ∂L(ô ô ô, o o o)/∂β ij = (o i τ -ôi )∂log(S(y i , x j ; 1, β ij ))/∂β ij

where ô ô ô is the target vector and o o o is the output vector and the fully-connected synapse graph outputs through a softmax activation function that is o j = sof tmax(y j ).

Proof. The proof is given in the Appendix A.

2.4.2. GRADIENT AND BOSE-EINSTEIN DISTRIBUTION:

Considering the surprisal space, let (u k , v k , γ ki ) = -log(x k , y k , β ki ), the fully-connected synapse graph is denoted as v k = u k + i (-log(1 -e -(γ ki +ui) )) . Compute the gradient over parameters

EQUATION

because only when k = p and i = q, two δ are 1, so ∂vp ∂γpq = -e -(γpq +uq )

1-e -(γpq +uq ) . Replacing the indexes and reformulating, we have

∂v i ∂γ ij = -1 e γij +uj -1 (12)

The right side of Eq.( 12) is the negative Bose-Einstein Distribution in the surprisal space.

To compute the loss function in surprisal space, we convert the target vector ô ô ô and output 

EQUATION

We can apply error back-propagation to implement gradient descent for synapse learning.

EQUATION

)

where η is the learning rate.

The equation Eq.( 14) illustrates that the learning of synaptic neural network follows the Bose-Einstein statistics in the surprisal space. This paper "Memory as an equilibrium Bose gas" by (Fröhlich (1968) , Pascual-Leone ( 1970)) showed that memory maybe possible to be represented as the equilibrium of Bose gas. SynaMLP has an input layer for down-sampling and an output layer for classification. The hidden layer is a block of fully-connected synapse tensor. The inputs of the block are neurons from the input layer and the outputs of the block are neurons to the output layer. The block is the implementation of synapse tensor in Eq.( 9). The activation functions are connected synapse tensor to the output layer. Moreover, the input of the block is a probability distribution. The block can be thought of the replacement of the weight layer in a standard neural network. In the comparison experiment, SynaNN MLP and traditional MLP generated the similar test accuracy of around 98%. We applied a softmax activation function in front of the input of synapse to avoid the error of NAN (computing value out of the domain). In fact, synaptic neural network handles a probability distribution (vector from neurons).

3. EXPERIMENTS:



3.2. MNIST EXPERIMENT:



4. CONCLUSION:

In this paper, we presented and analyzed a Synaptic Neural Network (SynaNN). We found the fine structure of synapse and the construction of synapse network as well as the BE distribution in the gradient descent learning. In surprisal space, the input of a neuron is the addition of the identity function and the sum of topologically conjugate functions of inhibitory synapses which is the sum of bits of information. The formula of surprisal synapse function is defined as

EQUATION

The non-linear synaptic neural network may be implemented by physical or chemical components. Instead of using a simple linear synapse function, more synapse functions maybe found in the researches and applications of neural network. 

where ô ô ô is the target vector and o o o is the output vector and the fully-connected synapse graph outputs through a softmax activation function. where δ jk is the Kronecker delta and τ = j ôj is a constant. After applying log on both sides of Eq.8, we can compute the partial derivative over parameters as, ∂log(y k )

EQUATION

then applying log derivative on the left side and multiple y k on both sides, we have (o k τ -ôk ) -x q y k 1 -β kq x q δ kp = (o p τ -ôp ) -x q y p 1 -β pq x q (22) replace index p, q by i, j in Eq.22, we proved the gradient formula Eq.10 which is Eq.4 as well.

EQUATION

:



Figures and Tables:

the topologically conjugated function of the function F in the standard definition. From Theorem 1, specially G(u) = -log(1 -e -u ) is the topologically conjugate function of the complementary probability function 1 -x. Features: i) The iterated function F and its topologically conjugate function G have the same dynamics. ii) They have the same mapped fixed point where F : x = 1/2 and G : u = -log(1/2). iii) I(x) = -log(x) is a infinite differentiable and continuous function in real open interval (0,1).

SYNAMLP : SYNAPTIC MULTIPLE LAYER PERCEPTRONS We are going to illustrate a Synaptic Neural Network implementation SynaMLP with the connection of Multiple Layer Perceptrons (MLP) (Minsky et al. (2017)).

Figure 1: SynaMLP: (green, blue, red) dots are (input, hidden, output) layers.

Given o j = sof tmax(y j ), then ∂log(o j ) ∂y k = δ jk -o k , ∂L(ô ô ô, o o o) ∂y k = o k τ -ôk(18)

vector o o o to surprisal space as (ô ô ô, o o o), so the new loss function is L( t t t, t t t) = k tk * t k . The log function has been removed in L( t t t, t t t) because log is implied in the surprisal space. Without using an activation function, there is t k = v k . By Eq.(12),

Table content: <table><tr><td>∂L( t t t, t t t) ∂γ ij</td><td>=</td><td>k</td><td>∂L( t t t, t t t) ∂v k</td></tr></table>

To proof-of-concept, we implemented the SynaMLP with MNIST. Hand-written digital MNIST data sets are used for training and testing in machine learning. It is split into three parts: 60,000 data points of training data (mnist.train), 10,000 points of test data (mnist.test), and 5,000 points of validation data (mnist.validation) (LeCun et al. (2010)). SynaMLP MNIST TestingThe MNIST SynaMLP training and testing is implemented by Python, Keras and Tensorflow(Abadi et al. (2016)) from the revision of the example of mnist_mlp.py in Keras distribution. The synapse tensor is designed to be a class to replace Dense in Keras. The layer sequence is as below,

Table content: <table><tr><td/><td colspan="2">Synapse MLP</td><td>Dense MLP</td></tr><tr><td>Iteration</td><td>10001</td><td/><td>10001</td></tr><tr><td>Test loss</td><td colspan="3">0.0942904587357543 0.0906127168575471</td></tr><tr><td colspan="4">Test accuracy 0.9802000087499618 0.9830000066757202</td></tr><tr><td colspan="2">Layer Type</td><td colspan="2">Output Shape Param #</td></tr><tr><td>Dense[1]</td><td/><td colspan="2">(None, 300)</td><td>235500</td></tr><tr><td colspan="4">Batch-normalization[1] (None, 300)</td><td>1200</td></tr><tr><td colspan="2">Activation[1]</td><td colspan="2">(None, 300)</td><td>0</td></tr><tr><td colspan="2">Synapse[1]</td><td colspan="2">(None, 300)</td><td>90000</td></tr><tr><td colspan="4">Batch-normalization[2] (None, 300)</td><td>1200</td></tr><tr><td>Dense[2]</td><td/><td colspan="2">(None, 10)</td><td>3010</td></tr><tr><td>Optimizer</td><td/><td>Adam</td><td>default</td></tr></table>

SynaMLP Layers

Table content: <table/>

Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv:1603.01025, 2016. Carl R. Nave. The bose-einstein distribution, 2018. URL http://hyperphysics.phy-astr. gsu.edu/hbase/quantum/disbe.html. Juan Pascual-Leone. A mathematical model for the transition rule in piaget's developmental stages. Acta psychologica, 32:301-345, 1970. Bertrand Russell. Principles of mathematics. Routledge, 2009. Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 27(3):379-423, 1948. V Torre and T Poggio. A synaptic mechanism possibly underlying directional selectivity to motion. Proc. R. Soc. Lond. B, 202(1148):409-416, 1978. APPENDIX A GRADIENT EQUATION THEOREM Let the loss function L(ô ô ô, o o o) of the fully-connected synapse graph Eq.8 be equal to the sum of cross-entropy L(ô ô ô, o o o) =i ôi log(o i ), then its item of parameter gradient is ∂L(ô ô ô, o o o) ∂β ij = (o i τ -ôi ) -y i x j 1 -β ij x j

Table content: <table><tr><td/><td/><td>(16)</td></tr><tr><td>or</td><td>∂L(ô ô ô, o o o) ∂β ij</td><td>= (o</td></tr></table>

Peer review (iclr2019):
 The authors propose a hybrid neural network, composed of a synapse graph that can be embedded into and a standard neural network, such that the entire architecture can be trained in a way that is compatible with the gradient descent and backpropagation of. As a proof of concept, the hybrid architecture is trained to classify MNIST. I am not convinced by the way this work is motivated. What problem are the authors actually addressing? Just because biological neurons use synapses does not mean we should try hard to put a certain instance of them into deep neural networks. Clearly this is not an attempt to add to neuroscience, as beyond the inspiration of neurons having synapses, there is little attempt to biologically plausible. As an attempt to add to machine learning research, the neuroscience motivation is unconvincing. Provided that the math works out (and I admit that I did not attempt to follow the detailed derivations), this looks like an interesting intellectual exercise, but it also seems a bit like a discovery of a hammer that is in need for nails to be applied to. And it’s not even clear to me how practical the hammer would actually be, even if we had a convincing problem setting at hand. How scalable is it beyond toy-settings? The final sentence makes a tantalizing claim, but at this stage the work has to resort to promising potential, rather than being able to demonstrate that it is practically useful. Moreover, this work is not presented right for the venue and audience, and would need substantial rewriting and restructuring to make the central claims and contributions sufficiently clear.