Patent Publication Number: US-2023147781-A1

Title: Sound Signal Processing Using A Neuromorphic Analog Signal Processor

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation-in-part of and claims priority to U.S. application Ser. No. 17/196,960, filed Mar. 9, 2021, entitled “Analog Hardware Realization of Trained Neural Networks for Voice Clarity,” which is a continuation-in-part of and claims priority to U.S. application Ser. No. 17/189,109, filed Mar. 1, 2021, entitled “Analog Hardware Realization of Neural Networks,” which is a continuation of PCT Application PCT/RU2020/000306, filed Jun. 25, 2020, entitled “Analog Hardware Realization of Neural Networks,” each of which is incorporated by reference herein in its entirety. U.S. application Ser. No. 17/189,109 is also a continuation-in-part of PCT Application PCT/EP2020/067800, filed Jun. 25, 2020, entitled “Analog Hardware Realization of Neural Networks,” which is incorporated by reference herein in its entirety. 
    
    
     TECHNICAL FIELD 
     The disclosed implementations relate generally to neural networks, and more specifically to systems and methods for hardware realization of trained neural networks for sound signal processing, classification, and enhancement. 
     BACKGROUND 
     Conventional hardware has failed to keep pace with innovation in neural networks and the growing popularity of machine learning based applications. The complexity of neural networks continues to outpace CPU and GPU computational power as digital microprocessor advances are plateauing. Neuromorphic processors based on spike neural networks, such as Loihi and True North, are limited in their applications. For GPU-like architectures, power and speed of such architectures are limited by data transmission speed. Data transmission can consume up to 80% of chip power and can significantly impact the speed of calculations. Edge applications demand low power consumption, but there are currently no known performant hardware implementations that consume less than 50 milliwatts of power. 
     Memristor-based architectures that use cross-bar technology remain impractical for manufacturing recurrent and feed-forward neural networks. For example, memristor-based cross-bars have several disadvantages, including high latency and leakage of currents during operation, which make them impractical. Also, there are reliability issues in manufacturing memristor-based cross-bars, especially when neural networks have both negative and positive weights. For large neural networks with many neurons, at high dimensions, memristor-based cross-bars cannot be used for simultaneous propagation of different signals, which in turn complicates summation of signals, when neurons are represented by operational amplifiers. Furthermore, memristor-based analog integrated circuits have a number of limitations, such as a small number of resistive states, first cycle problems when forming memristors, complexity with channel formation when training the memristors, unpredictable dependency on the dimensions of the memristors, slow operation of memristors, and drift of state of resistance. 
     Additionally, the training process required for neural networks presents unique challenges for hardware realization of neural networks. A trained neural network is used for specific inferencing tasks, such as classification. Once a neural network is trained, a hardware equivalent is manufactured. When the neural network is retrained, the hardware manufacturing process is repeated, driving up costs. Although some reconfigurable hardware solutions exist, such hardware cannot be easily mass produced, and costs a lot more (e.g., five times more) than hardware that is not reconfigurable. Further, edge environments, such as smart-home applications, do not require re-programmability as such. For example, 85% of all applications of neural networks do not require any retraining during operation, so on-chip learning is not that useful. Furthermore, edge applications include noisy environments, which can cause reprogrammable hardware to become unreliable. 
     Voice transmissions comprise the majority of communications between humans and human-machine interfaces, and substantially surpass video and hand-typed communications. Clarity of voice transmission needs to be maintained while voice signals are compressed or digitized for transmission. Traditionally, multiple noise suppression and noise filtering methods and apparatuses process the unclear voice signals and remove at least some of the unwanted noise. Some conventional techniques use microphones that capture noise and generate sounds that effectively cancel out the unwanted noises detected around a listener. Such techniques are more prevalent in headphones, and specifically in noise-cancelling headphones. There are also techniques that suppress certain noises based on spectra qualities of specific noise sources, or using more elaborate algorithms, such as Markov processes, Fast Fourier Transform methods, and various noise-detecting adaptive algorithms. 
     More recently, neural networks have been used to analyze signals containing a mix of voice and noise, and to effectively extract mostly voice-containing signals, based on the specific features attributable to voice. Such neural networks need to be trained and are implemented substantially as programs running on powerful computers. These computers consume substantial electric and computing power. Conventional solutions are often limited by training features, fail to provide real-time processing, and are limited to processing specific recorded voice signals. Currently, voice communications are predominantly performed via cellular or land-line phones. Conventional equipment lacks computing power and/or electrical power for effectively processing voice signals and suppressing unwanted noises. Even with sophisticated noise-cancelling technologies, the types of noises that can be effectively suppressed are substantially limited. It is common to have unwanted disturbances, such as dog barks, door slams, emergency sirens, car honks, and similar unpredictable interferences, which are still background noises for the purpose of transmitting clear voice signals. 
     Non-voice noises or signals can originate in the vicinity of a speaker, near the microphone, or on another device used to transform sound into electrical signals. Such noises are generally referred to as the background noises at-origin. For such noise signals, any background conversations, or voices of persons, farther from the microphone, could be considered noise. Other non-voice noises can originate during processing and transmission of the signals, such as compression, analog-to-digital conversion, spectrum limitation, breakdown in packets limited by length, spectrum, or information size. Such noises occur during transmission, as well as during corresponding reversing steps. When several voice/noise signals are mixed together, such as in conference calls or multi-person communications, noises associated with each signal are mixed, further complicating the task and challenge of voice clarification. In addition, when the voice signals are further processed to result in the actual sound generated near the ear of the recipient (e.g., for human-to-human communications), via either speakers, headphones, or other apparatuses or methods, further noises or unwanted signals may be introduced by the ambient environment near the recipient. Although voice commands have become popular, wearable devices lack advanced sound processing capabilities. Conventional devices need Internet connection and have power limitations. There is also a security concern using Internet-connected voice processing devices. 
     SUMMARY 
     Accordingly, there is a need for methods, circuits and/or interfaces that address at least some of the deficiencies identified above. Analog circuits that model trained neural networks and manufactured according to the techniques described herein, can provide improved performance per watt advantages, can be useful in implementing hardware solutions in edge environments, and can tackle a variety of applications, such as drone navigation and autonomous cars. The cost advantages provided by the proposed manufacturing methods and/or analog network architectures are even more pronounced with larger neural networks. Also, analog hardware implementations of neural networks provide improved parallelism and neuromorphism. Moreover, neuromorphic analog components are not sensitive to noise and temperature changes, when compared to digital counterparts. A neurovoice processor is described herein, according to some implementations. The processor consumes low power (e.g., 200 micro-watt or less) and can work locally (without the Internet), thereby providing privacy. 
     Chips manufactured according to the techniques described herein provide orders of magnitude improvement over conventional systems in size, power, and performance, and are ideal for edge environments, including for retraining purposes. Such analog neuromorphic chips can be used to implement edge computing applications or in Internet-of-Things (IoT) environments. Due to the analog hardware, initial processing (e.g., formation of descriptors for image recognition), that can consume over 80-90% of power, can be moved onto the chip, thereby decreasing energy consumption and network load, which can open new markets for applications. 
     Various edge applications can benefit from the use of such analog hardware. For example, for video processing, the techniques described herein can be used to include direct connection to CMOS sensors without a digital interface. Various other video processing applications include road sign recognition for automobiles, camera-based true depth and/or simultaneous localization and mapping for robots, room access control without a server connection, and always-on solutions for security and healthcare. Such chips can be used for data processing from radars and lidars, and for low-level data fusion. Such techniques can be used to implement battery management features for large battery packs, sound/voice processing without connection to data centers, voice recognition on mobile devices, wake up speech instructions for IoT sensors, translators that translate one language to another, large sensor arrays of IoT with low signal intensity, and/or configurable process controls with hundreds of sensors. 
     Neuromorphic analog chips can be mass produced after standard software-based neural network simulations/training, according to some implementations. A client&#39;s neural network can be easily ported, regardless of the structure of the neural network, with customized chip design and production. Moreover, a library of ready to make on-chip solutions (network emulators) are provided, according to some implementations. Such solutions require only training, one lithographic mask change, following which chips can be mass produced. For example, during chip production, only part of the lithography masks needs to be changed. 
     The techniques described herein can be used to design and/or manufacture an analog neuromorphic integrated circuit that is mathematically equivalent to a trained neural network (either feed-forward or recurrent neural networks). According to some implementations, the process begins with a trained neural network that is first converted into a transformed network comprised of standard elements. Operation of the transformed network are simulated using software with known models representing the standard elements. The software simulation is used to determine the individual resistance values for each of the resistors in the transformed network. Lithography masks are laid out based on the arrangement of the standard elements in the transformed network. Each of the standard elements are laid out in the masks using an existing library of circuits corresponding to the standard elements to simplify and speed up the process. In some implementations, the resistors are laid out in one or more masks separate from the masks including the other elements (e.g., operational amplifiers) in the transformed network. In this manner, if the neural network is retrained, only the masks containing the resistors, or other types of fixed-resistance elements, representing the new weights in the retrained neural network need to be regenerated, which simplifies and speeds up the process. The lithography masks are then sent to a fab for manufacturing the analog neuromorphic integrated circuit. 
     In one aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method incudes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. 
     In some implementations, generating the schematic model includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. 
     In some implementations, the method further includes obtaining new weights for the trained neural network, computing a new weight matrix for the equivalent analog network based on the new weights, and generating a new resistance matrix for the new weight matrix. 
     In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: for each layer of the one or more layers of neurons: (i) identifying one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and (ii) generating a respective multi-layer network of analog neurons based on arranging the one or more function blocks. Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multi-layer network is connected to one or more analog neurons of a second layer of the multi-layer network. 
     In some implementations, the one or more function blocks include one or more basic function blocks selected from the group consisting of: (i) a weighted summation block with a block output V out =ReLU(Σw i ·V i   in +bias). ReLU is Rectified Linear Unit (ReLU) activation function or a similar activation function, V i  represents an i-th input, w i  represents a weight corresponding to the i-th input, and bias represents a bias value, and E is a summation operator; (ii) a signal multiplier block with a block output V out =coeff·V i ·V j ·V i  represents an i-th input and V i  represents a j-th input, and coeff is a predetermined coefficient; (iii) a sigmoid activation block with a block output 
     
       
         
           
             
               V 
               
                 o 
                 ⁢ 
                 u 
                 ⁢ 
                 t 
               
             
             = 
             
               
                 A 
                 
                   1 
                   + 
                   
                     e 
                     
                       - 
                       
                         B 
                         . 
                         V 
                       
                     
                   
                 
               
               . 
             
           
         
       
     
     V represents an input, and A and B are predetermined coefficient values of the sigmoid activation block; (iv) a hyperbolic tangent activation block with a block output V out =A*tanh (B*V in ). V in  represents an input, and A and B are predetermined coefficient values; and (v) a signal delay block with a block output U(t)=V(t−dt). t represents a current time-period, V(t−dt) represents an output of the signal delay block for a preceding time period t−dt, and dt is a delay value. 
     In some implementations, identifying the one or more function blocks includes selecting the one or more function blocks based on a type of the respective layer. 
     In some implementations, the neural network topology includes one or more layers of neurons, each layer of neurons computing respective outputs based on a respective mathematical function, and transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function; and (ii) for each sub-layer of the first layer of the neural network topology: (a) selecting one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and (b) generating a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network. 
     In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting the one or more weights such that combining the one or more intermediate functions results in the mathematical function. 
     In some implementations, the method further includes: (i) generating equivalent digital network of digital components for one or more output layers of the neural network topology; and (ii) connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components. 
     In some implementations, the analog components include a plurality of operational amplifiers and a plurality of resistors, each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. 
     In some implementations, selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors. 
     In some implementations, the neural network topology includes one or more GRU or LSTM neurons, and transforming the neural network topology includes generating one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons. 
     In some implementations, the one or more signal delay blocks are activated at a frequency that matches a predetermined input signal frequency for the neural network topology. 
     In some implementations, the neural network topology includes one or more layers of neurons that perform unlimited activation functions, and transforming the neural network topology includes applying one or more transformations selected from the group consisting of: (i) replacing the unlimited activation functions with limited activation; and (ii) adjusting connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized. 
     In some implementations, the method further includes generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix. 
     In some implementations, the method further includes: (i) obtaining new weights for the trained neural network; (ii) computing a new weight matrix for the equivalent analog network based on the new weights; (iii) generating a new resistance matrix for the new weight matrix; and (iv) generating a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix. 
     In some implementations, the trained neural network is trained using software simulations to generate the weights. 
     In another aspect, a method for hardware realization of neural networks is provided, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes calculating one or more connection constraints based on analog integrated circuit (IC) design constraints. The method also includes transforming the neural network topology to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints. The method also includes computing a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network. 
     In some implementations, transforming the neural network topology to the equivalent sparsely connected network of analog components includes deriving a possible input connection degree N i  and output connection degree N o , according to the one or more connection constraints. 
     In some implementations, the neural network topology includes at least one densely connected layer with K inputs and L outputs and a weight matrix U. In such cases, transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, and ┌log N     i    K┐+┌log N     o    L┐−1 layers, such that input connection degree does not exceed N i , and output connection degree does not exceed N o . 
     In some implementations, the neural network topology includes at least one densely connected layer with K inputs and L outputs and a weight matrix U. In such cases, transforming the at least one densely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, and M≥max(┌log N     i   L┐, ┌log N     o   K┐) layers. Each layer m is represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed and output connection degree does not exceed N o . The equation U=Π m=1 . . . M  U m  is satisfied with a predetermined precision. 
     In some implementations, the neural network topology includes a single sparsely connected layer with K inputs and L outputs, a maximum input connection degree of P i , a maximum output connection degree of P o , and a weight matrix of U, where absent connections are represented with zeros. In such cases, transforming the single sparsely connected layer includes constructing the equivalent sparsely connected network with K inputs, L outputs, M≥max(┌log N     i   P i ┐, ┌log N     o   P o ┐) layers, each layer m represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed N i , and output connection degree does not exceed N o . The equation U=Π m=1 . . . M  U m  is satisfied with a predetermined precision. 
     In some implementations, the neural network topology includes a convolutional layer with K inputs and L outputs. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the convolutional layer into a single sparsely connected layer with K inputs, L outputs, a maximum input connection degree of P i , and a maximum output connection degree of P o . P i ≤N i  and P o ≤N o . 
     In some implementations, generating a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix. 
     In some implementations, the neural network topology includes a recurrent neural layer. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes transforming the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections. 
     In some implementations, the neural network topology includes a recurrent neural layer. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with K inputs and L output and a weight matrix U, where absent connections are represented with zeros. 
     In some implementations, the neural network topology includes K inputs, a weight vector U∈R K , and a single layer perceptron with a calculation neuron with an activation function F. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating a number of layers m for the equivalent sparsely connected network using the equation m=┌log N  K┐; and (iii) constructing the equivalent sparsely connected network with the K inputs, m layers and the connection degree N. The equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes calculating a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation 
     
       
         
           
             
               S 
               = 
               
                 K 
                 ⁡ 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
             
             . 
           
         
       
     
     In some implementations, the neural network topology includes K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating number of layers m for the equivalent sparsely connected network using the equation m=┌log N  K┐; (iii) decomposing the single layer perceptron into L single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network of the L single layer perceptron networks: (a) constructing a respective equivalent pyramid-like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (b) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network of the L single layer perceptron networks: (i) setting a weight vector U=V i , i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network; and (ii) calculating a weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 K 
                 ⁡ 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     In some implementations, the neural network topology includes K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i  and corresponding weight matrices V i  that includes a row of weights for each calculation neuron of the L, calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing the multi-layer perceptron into Q=Σ i=1,S (L i ) single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons. Decomposing the multi-layer perceptron includes duplicating one or more input of the K inputs that are shared by the Q calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating a number of layers m for a respective equivalent pyramid-like sub-network using the equation m=┌log N  K i j ┐. K i,j  is number of inputs for the respective calculation neuron in the multi-layer perceptron; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i,j  inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of them layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i,j  inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes: for each single layer perceptron network of the Q single layer perceptron networks: (i) setting a weight vector U=V i   j , the i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the multi-layer perceptron; and (ii) calculating a weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K i,j  equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 
                   K 
                   
                     i 
                     , 
                     j 
                   
                 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     In some implementations, the neural network topology includes a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i  and corresponding weight matrices V i  that includes a row of weights for each calculation neuron of the L i  calculation neurons. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing the CNN into Q=Σ i=1,S (L i ) single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons. Decomposing the CNN includes duplicating one or more input of the K inputs that are shared by the Q calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating number of layers m for a respective equivalent pyramid-like sub-network using the equation m=┌log N  K i,j ┐. j is the corresponding layer of the respective calculation neuron in the CNN, and K i,j  is number of inputs for the respective calculation neuron in the CNN; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i,j  inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i,j  inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network of the Q single layer perceptron networks: (i) setting a weight vector U=V i   j , the i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the CNN; and (ii) calculating weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K i,j  equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 
                   K 
                   
                     i 
                     , 
                     j 
                   
                 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     In some implementations, the neural network topology includes K inputs, a layer L p  with K neurons, a layer L n  with L neurons, and a weight matrix W∈R L×K , where R is the set of real numbers, each neuron of the layer L p  is connected to each neuron of the layer L n , each neuron of the layer L n  performs an activation function F, such that output of the layer L n  is computed using the equation Y o =F(W·x) for an input x. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes performing a trapezium transformation that includes: (i) deriving a possible input connection degree N I &gt;1 and a possible output connection degree N O &gt;1, according to the one or more connection constraints; (ii) in accordance with a determination that K·L&lt;L·N I +K·N O , constructing a three-layered analog network that includes a layer LA p  with K analog neurons performing identity activation function, a layer LA h  with 
     
       
         
           
             M 
             = 
             
               ⌈ 
               
                 max 
                 ⁡ 
                 ( 
                 
                   
                     
                       K 
                       · 
                       
                         N 
                         I 
                       
                     
                     
                       N 
                       O 
                     
                   
                   , 
                   
                     
                       L 
                       · 
                       
                         N 
                         O 
                       
                     
                     
                       N 
                       I 
                     
                   
                 
                 ) 
               
               ⌉ 
             
           
         
       
     
     analog neurons performing identity activation function, and a layer LA o  with L analog neurons performing the activation function F, such that each analog neuron in the layer LA p  has No outputs, each analog neuron in the layer LA h  has not more than N I  inputs and N O  outputs, and each analog neuron in the layer LA o  has N I  inputs. Also, in such cases, computing the weight matrix for the equivalent sparsely connected network includes generating a sparse weight matrices W o  and W h  by solving a matrix equation W o ·W h =W that includes K·L equations in K·N O +L·N 1  variables, so that the total output of the layer LA o  is calculated using the equation Y o =F(W o ·W h ·x). The sparse weight matrix W o ∈R K×M  represents connections between the layers LA p  and LA h , and the sparse weight matrix W h ∈R M×L  represents connections between the layers LA h  and LA o . 
     In some implementations, performing the trapezium transformation further includes: in accordance with a determination that K·L≥L·N I +K·N O : (i) splitting the layer L p  to obtain a sub-layer L p1  with K′ neurons and a sub-layer L p2  with (K-K′) neurons such that K′·L≥L·N I +K′·N O ; (ii) for the sub-layer L p1  with K′ neurons, performing the constructing, and generating steps; and (iii) for the sub-layer L p2  with K-K′ neurons, recursively performing the splitting, constructing, and generating steps. 
     In some implementations, the neural network topology includes a multilayer perceptron network. In such cases, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. 
     In some implementations, the neural network topology includes a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation. In such cases, the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation. 
     In some implementations, the neural network topology includes a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations. In such cases, the method further includes performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations. 
     In some implementations, the neural network topology includes a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers and (ii) one or more fully-connected layers. In such cases, the method further includes: (i) transforming the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and (ii) for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. 
     In some implementations, the neural network topology includes K inputs, L output neurons, and a weight matrix U∈R L×K , where R is the set of real numbers, each output neuron performs an activation function F. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes performing an approximation transformation that includes: (i) deriving a possible input connection degree N I &gt;1 and a possible output connection degree N O &gt;1, according to the one or more connection constraints; (ii) selecting a parameter p from the set {0, 1, . . . , ┌log N     I   K┐−1}; (iii) in accordance with a determination that p&gt;0, constructing a pyramid neural network that forms first p layers of the equivalent sparsely connected network, such that the pyramid neural network has N p =┌K/N I   p ┐ neurons in its output layer. Each neuron in the pyramid neural network performs identity function; and (iv) constructing a trapezium neural network with N p  inputs and L outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function. In such cases, computing the weight matrix for the equivalent sparsely connected network includes: (i) generating weights for the pyramid neural network including (a) setting weights of every neuron i of the first layer of the pyramid neural network according to following rule: (a) w ik     i     (1) =C. C is a non-zero constant and k i =(i−1)N I +1; and (b) 
     
       
         
           
             
               
                 w 
                 
                   i 
                   ⁢ 
                   j 
                 
                 
                   ( 
                   1 
                   ) 
                 
               
               = 
               
                 
                   1 
                   L 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       l 
                       = 
                       1 
                     
                     L 
                   
                   
                     
                       
                         U 
                         lj 
                       
                       
                         U 
                         
                           lk 
                           i 
                         
                       
                     
                     ⁢ 
                     C 
                   
                 
               
             
             , 
           
         
       
     
     for all weights j of the neuron except k i ; and (b) setting all other weights of the pyramid neural network to 1; and (ii) generating weights for the trapezium neural network including (a) setting weights of each neuron i of the first layer of the trapezium neural network according to the equation 
     
       
         
           
             
               
                 w 
                 
                   i 
                   ⁢ 
                   
                     k 
                     i 
                   
                 
                 
                   ( 
                   
                     p 
                     + 
                     1 
                   
                   ) 
                 
               
               = 
               
                 
                   U 
                   
                     i 
                     ⁢ 
                     
                       k 
                       i 
                     
                   
                 
                 C 
               
             
             ; 
           
         
       
     
     and (b) setting other weights of the trapezium neural network to 1. 
     In some implementations, the neural network topology includes a multilayer perceptron with the K inputs, S layers, and L i=1,S  calculation neurons in i-th layer, and a weight matrix U i=1,S ∈R L     i     ×L     i-1    for the i-th layer, where L 0 =K. In such cases, transforming the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer j of the S layers of the multilayer perceptron: (i) constructing a respective pyramid-trapezium network PTNNX j  by performing the approximation transformation to a respective single layer perceptron consisting of L j-1  inputs, L j  output neurons, and a weight matrix U j ; and (ii) constructing the equivalent sparsely connected network by stacking each pyramid trapezium network. 
     In another aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. 
     In some implementations, generating the resistance matrix for the weight matrix includes: (i) obtaining a predetermined range of possible resistance values {R min , R max } and selecting an initial base resistance value R base  within the predetermined range; (ii) selecting a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights 
     
       
         
           
             
               w 
               
                 i 
                 , 
                 j 
               
             
             = 
             
               
                 R 
                 
                   b 
                   ⁢ 
                   a 
                   ⁢ 
                   s 
                   ⁢ 
                   e 
                 
               
               ( 
               
                 
                   1 
                   
                     R 
                     i 
                   
                 
                 - 
                 
                   1 
                   
                     R 
                     j 
                   
                 
               
               ) 
             
           
         
       
     
     within the range [−R base , R base ] for all combinations of {R i , R j } within the limited length set of resistance values; (iii) selecting a resistance value R + =R − , from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias w max  of each neuron or for each layer of the equivalent analog network, such that R + =R −  is the closest resistor set value to R base *w max ; and (iv) for each element of the weight matrix, selecting a respective first resistance value R 1  and a respective second resistance value R 2  that minimizes an error according to equation 
     
       
         
           
             err 
             = 
             
               
                 
                   ( 
                   
                     
                       
                         R 
                         + 
                       
                       
                         R 
                         1 
                       
                     
                     + 
                     
                       
                         R 
                         - 
                       
                       
                         R 
                         2 
                       
                     
                   
                   ) 
                 
                 · 
                 
                   r 
                   
                     e 
                     ⁢ 
                     r 
                     ⁢ 
                     r 
                   
                 
               
               + 
               
                 
                   ❘ 
                   &#34;\[LeftBracketingBar]&#34; 
                 
                 
                   w 
                   - 
                   
                     
                       R 
                       + 
                     
                     
                       R 
                       1 
                     
                   
                   + 
                   
                     
                       R 
                       - 
                     
                     
                       R 
                       2 
                     
                   
                 
                 
                   ❘ 
                   &#34;\[RightBracketingBar]&#34; 
                 
               
             
           
         
       
     
     for all possible values of R 1  and R 2  within the predetermined range of possible resistance values. w is the respective element of the weight matrix, and r err  is a predetermined relative tolerance value for resistances. 
     In some implementations, the predetermined range of possible resistance values includes resistances according to nominal series E24 in the range 100 KΩ to 1 MΩ. 
     In some implementations, R +  and R −  are chosen independently for each layer of the equivalent analog network. 
     In some implementations, R +  and R −  are chosen independently for each analog neuron of the equivalent analog network. 
     In some implementations, a first one or more weights of the weight matrix and a first one or more inputs represent one or more connections to a first operational amplifier of the equivalent analog network. In such cases, the method further includes, prior to generating the resistance matrix: (i) modifying the first one or more weights by a first value; and (ii) configuring the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function. 
     In some implementations, the method further includes: (i) obtaining a predetermined range of weights; and (ii) updating the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input. 
     In some implementations, the trained neural network is trained so that each layer of the neural network topology has quantized weights. 
     In some implementations, the method further includes retraining the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network. 
     In some implementations, the method further includes retraining the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. 
     In another aspect, a method is provided for hardware realization of neural networks, according to some implementations. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. The method also includes generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. The method also includes pruning the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components. 
     In some implementations, pruning the equivalent analog network includes substituting, with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. 
     In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. 
     In some implementations, pruning the equivalent analog network includes removing one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero. 
     In some implementations, pruning the equivalent analog network further includes removing one or more analog neurons of the equivalent analog network without any input connections. 
     In some implementations, pruning the equivalent analog network includes: (i) ranking analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets; (ii) selecting one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing the one or more analog neurons from the equivalent analog network. 
     In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring propagation of analog signals by using the model to generate calculations for the one or more data sets. 
     In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring output signals of the model by using the model to generate calculations for the one or more data sets. 
     In some implementations, detecting use of the analog neurons includes: (i) building a model of the equivalent analog network using a modelling software; and (ii) measuring power consumed by the analog neurons by using the model to generate calculations for the one or more data sets. 
     In some implementations, the method further includes subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix. 
     In some implementations, the method further includes, for each analog neuron of the equivalent analog network: (i) computing a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing the respective analog neuron with a linear junction in the equivalent analog network. 
     In some implementations, the method further includes reducing number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections from one or more analog neurons of the equivalent analog network. 
     In some implementations, the method further includes pruning the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components. 
     In some implementations, the pruning is performed iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network. 
     In some implementations, the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing network knowledge extraction. 
     In another aspect, an integrated circuit is provided, according to some implementations. The integrated circuit includes an analog network of analog components fabricated by a method that includes: (i) obtaining a neural network topology and weights of a trained neural network; (ii) transforming the neural network topology to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron; (iii) computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection; (iv) generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix; (v) generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix; and (vi) fabricating the circuit based on the one or more lithographic masks using a lithographic process. 
     In some implementations, the integrated circuit further includes one or more digital to analog converters configured to generate analog input for the equivalent analog network of analog components based on one or more digital. 
     In some implementations, the integrated circuit further includes an analog signal sampling module configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit. 
     In some implementations, the integrated circuit further includes a voltage converter module to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers. 
     In some implementations, the integrated circuit further includes a tact signal processing module configured to process one or more frames obtained from a CCD camera. 
     In some implementations, the trained neural network is a long short-term memory (LSTM) network. In such cases, the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing. 
     In some implementations, the integrated circuit further includes one or more analog to digital converters configured to generate digital signal based on output of the equivalent analog network of analog components. 
     In some implementations, the integrated circuit further includes one or more signal processing modules configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications. 
     In some implementations, the trained neural network is trained, using training datasets containing signals of arrays of gas sensors on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected. In such cases, the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections. 
     In some implementations, the trained neural network is trained, using training datasets containing thermal aging time series data for different MOSFETs, for predicting remaining useful life (RUL) of a MOSFET device. In such cases, the neural network topology includes 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons, and (v) between 123,000 and 124,000 connections. 
     In some implementations, the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS). In such cases, the neural network topology includes an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons, and (v) between 51,000 and 52,000 connections. 
     In some implementations, the trained neural network is trained, using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries, for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS). In such cases, the neural network topology includes an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons, and (v) between 2,200 and 2,400 connections. 
     In some implementations, the trained neural network is trained, using training datasets containing speech commands, for identifying voice commands. In such cases, the neural network topology is a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron. In such cases, the equivalent analog network includes: (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections. 
     In some implementations, the trained neural network is trained, using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor, for determining pulse rate during physical exercises based on PPG sensor data and 3-axis accelerometer data. In such cases, the neural network topology includes two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively. In such cases, the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons, and (vi) between 12,000 and 12,500 connections. 
     In some implementations, the trained neural network is trained to classify different objects based on pulsed Doppler radar signal. In such cases, the neural network topology includes multi-scale LSTM neural network. 
     In some implementations, the trained neural network is trained to perform human activity type recognition, based on inertial sensor data. In such cases, the neural network topology includes three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes. In such cases, the equivalent analog network includes: (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons, and (vi) between 20,000 and 21,000 connections. 
     In some implementations, the trained neural network is further trained to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation. 
     In another aspect, a method is provided for generating libraries for hardware realization of neural networks. The method includes obtaining a plurality of neural network topologies, each neural network topology corresponding to a respective neural network. The method also includes transforming each neural network topology to a respective equivalent analog network of analog components. The method also includes generating a plurality of lithographic masks for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components. 
     In some implementations, the method further includes obtaining a new neural network topology and weights of a trained neural network. The method also includes selecting one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies. The method also includes computing a weight matrix for a new equivalent analog network based on the weights. The method also includes generating a resistance matrix for the weight matrix. The method also includes generating a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks. 
     In some implementations, the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithographic masks is further based on comparing each subnetwork topology with each network topology of the plurality of network topologies. 
     In some implementations, one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare with any network topology of the plurality of network topologies. In such cases, the method further includes: (i) transforming each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and (ii) generating one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components. 
     In some implementations, transforming a respective network topology to a respective equivalent analog network includes: (i) decomposing the respective network topology to a plurality of subnetwork topologies; (ii) transforming each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing each equivalent analog subnetwork to obtain the respective equivalent analog network. 
     In some implementations, decomposing the respective network topology includes identifying one or more layers of the respective network topology as the plurality of subnetwork topologies. 
     In some implementations, each circuit is obtained by: (i) generating schematics for a respective equivalent analog network of analog components; and (ii) generating a respective circuit layout design based on the schematics. 
     In some implementations, the method further includes combining one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits. 
     In another aspect, a method is provided for optimizing energy efficiency of analog neuromorphic circuits, according to some implementations. The method includes obtaining an integrated circuit implementing an analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network, each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. The method also include generating inferences using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network. The method also includes, while generating inferences using the integrated circuit: (i) determining if a level of signal output of the plurality of operational amplifiers is equilibrated; and (ii) in accordance with a determination that the level of signal output is equilibrated: (a) determining an active set of analog neurons of the analog network influencing signal formation for propagation of signals; and (turning off power for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time. 
     In some implementations, determining the active set of analog neurons is based on calculating delays of signal propagation through the analog network. 
     In some implementations, determining the active set of analog neurons is based on detecting the propagation of signals through the analog network. 
     In some implementations, the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power for one or more layers prior to the active layer of the analog network. 
     In some implementations, the predetermined period of time is calculated based on simulating propagation of signals through the analog network, accounting for signal delays. 
     In some implementations, the trained neural network is a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors. In such cases, the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power, for the one or more analog components, for the predetermined period of time. 
     In some implementations, the method further includes turning on power for the one or more analog neurons of the analog network after the predetermined period of time. 
     In some implementations, determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level. 
     In some implementations, the method further includes repeating the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences. 
     In some implementations, the method further includes: (i) in accordance with a determination that the level of signal output is equilibrated, for each inference cycle: (a) during a first time interval, determining a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (b) turning off power for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and (ii) during a second time interval subsequent to the first time interval, turning off power for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period. 
     In some implementations, the one or more analog neurons consist of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers. 
     According to some implementations, a method and apparatus are provided to clear voice signal from undesired noises, in order to clarify transmission for the benefit of a recipient (e.g., a human or machine interface) receiving such clarified signal. The techniques described herein can be applied at origin, at the interim transmission point(s), or near the recipient. To process the voice clarification, a neural network is trained to separate voice signal from other unwanted signals, in an input signal. Such neural network is transformed into an equivalent analog neural network using techniques described herein. Such transformed equivalent analog neural network is implemented in the form of an analog integrated circuit. 
     In another aspect, a method is provided for analog hardware realization of trained convolutional neural networks for voice clarity. The method includes obtaining a neural network topology and weights of a trained neural network. The method also includes transforming the neural network topology into an equivalent analog network of analog components. The method also includes computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network. For example, for dense layers, one weight matrix element represents a single connection. For convolutional layers, on the other hand, one weight matrix element represents multiple connections. To further illustrate, suppose a layer multiplies N input signals by a single weight value w. In this case, the input layer size is N, output layer size is N, and there are N connections each with weight w. In this way, one weight value represents multiple connections. The method also includes generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. 
     In some implementations, the neural network topology includes a Fourier transformation layer and an inverse Fourier transformation layer. Fourier transformation layers are useful for any voice-based application because voice signal is based on frequencies, so frequencies can be extracted with FFT (Fast Fourier Transformation). In some implementations, FFTs are implemented using a dense layer. 
     In some implementations, the neural network topology includes one or more of: a convolutional layer, a max-pooling layer, and a densely connected layer. 
     In some implementations, the neural network topology includes a convolutional layer. In such cases, transforming the neural network topology includes: for each output of the convolutional layer: defining dependency relations between the respective output and a related subset of inputs. The related subset of inputs is defined by filters, kernel, padding, and strides parameters of the convolutional layer; and defining a respective subset of weights according to the dependency relations of the respective output; and constructing a layer of analog neurons such that (i) each analog neuron corresponds to a respective output of the convolutional layer, (ii) each analog neuron is connected to a related subset of inputs of a previous layer of analog neurons of the equivalent analog network, and (iii) incoming connections for each analog neuron are weighted according to a respective subset of weights of a corresponding output of the convolutional layer. 
     To illustrate these steps, consider a convolutional layer, where each output depends on a subset of inputs and each subset of inputs affects a subset of outputs. In other words, this relationship is a many-to-many relation, unlike an all-to-all relationship in a dense layer. For a single batch, suppose there is a matrix of inputs (for 1D convolution, this matrix is a two-dimensional matrix; for 2D convolution, this is a three-dimensional matrix) and a matrix of outputs with the same number of dimensions as the matrix of inputs. The first and second dimensions are sometimes called spatial dimensions, the last dimension is sometimes called channels. Suppose a kernel size (e.g., one dimensional kernel for 1D convolution, two-dimensional kernel for 2D convolution) is provided. A weight matrix is defined and sized using the equation: (kernel size times channels times filters). Here, kernel size and filters are parameters of the weight matrix and channels is a measure of inputs shape. For each filter F, a subset of the weight matrix [:, :, F] is applied over input data, and slid along spatial dimensions with a step defined by strides parameter. For 1D convolution, kernel is of size 3, spatial dimension X is 7, and stride=2. The weight matrix is applied over spatial coordinates {1,2,3}, {3,4,5}, {5,6,7} and over the channel dimension. Each time, the weight matrix is applied, a single output element is calculated. This element connections have weights as weights matrix subset and are connected to subset of input data, as described earlier. For more filters, some implementations leave the input as is and duplicate kernel and output as many times as there are filters, with different kernel values. It is noted that the description here is only provided for illustration purposes, and various other implementations are possible. 
     In some implementations, the neural network topology includes a max-pooling layer. In such instances, transforming the neural network topology includes generating a multi-layer network of analog neurons, for the max-pooling layer, that have maximum input counts. In some implementations, generating the multi-layer network of analog neurons includes generating a two-input schematic comprising two SNMs arranged in two layers, where an SNM of the last layer has a maximum of two inputs. In some implementations, generating the multi-layer network of analog neurons includes generating a three-input schematic comprising three SNMs arranged in three layers, where an SNM of the last layer has a maximum of three inputs. In some implementations, generating the multi-layer network of analog neurons includes generating a four-input schematic comprising four SNMs arranged in three layers, where an SNM of the last layer has a maximum of four inputs. In some implementations, the method further includes transforming the max-pooling layer into a calculation tree in which each node of the calculation tree is selected from the group consisting of: a two-input schematic comprising two SNMs arranged in two layers, where an SNM of the last layer has a maximum of two inputs; a three-input schematic comprising three SNMs arranged in three layers, where an SNM of the last layer has a maximum of three inputs; and a four-input schematic comprising four SNMs arranged in three layers, where an SNM of the last layer has a maximum of four inputs. In some implementations, the method further includes minimizing a number of layers of the calculation tree. In some implementations, the method further includes prioritizing use of the four-input SNMs over use of three-input SNMs and two-input SNMs. 
     In some implementations, the method further includes (i) defining an analog neuron of a last layer of the multi-layer analog network to perform an activation function other than ReLU, and (ii) defining all other neurons of the multi-layer analog network to perform ReLU without changing final output of the multi-layer network. 
     In some implementations, each layer of the trained neural network computes respective outputs based on a respective mathematical function. In such cases, transforming the neural network topology to the equivalent analog network of analog components includes: for each layer of the trained neural network: identifying one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function; and generating a respective multi-layer network of analog neurons based on arranging the one or more function blocks, wherein each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the respective multi-layer network is connected to one or more analog neurons of a second layer of the respective multi-layer network. In some implementations, the one or more function blocks include a weighted summation block with a block output V out =ReLU(Σw i ·V i   in +bias), where ReLU is a Rectified Linear Unit (ReLU) activation function or a similar activation function, V i  represents an i-th input, w i  represents a weight corresponding to the i-th input, bias represents a bias value, and Σ is a summation operator. In some implementations, the one or more function blocks include a weighted summation block with a block output V out =ReLU_X(Σw i ·V i   in +bias), where ReLU_X is a Rectified Linear Unit (ReLU) activation function, or a similar activation function, that limits output signal by the positive value X, V i  represents an i-th input, w i  represents a weight corresponding to the i-th input, bias represents a bias value, and Σ is a summation operator. 
     In some implementations, the neural network topology includes a convolutional layer having K inputs and L outputs. In such cases, transforming the neural network topology to the equivalent analog network includes: deriving a possible input connection degree N i  and output connection degree N o , according to one or more connection constraints based on analog integrated circuit (IC) design constraints; and transforming the convolutional layer includes decomposing the convolutional layer into a single sparsely connected layer with K inputs, L outputs, a maximum input connection degree of P i , and a maximum output connection degree of P o , where P i ≤N i  and P o ≤N o . 
     In some implementations, the analog components include a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. Generating the schematic model includes generating a resistance matrix from the weight matrix. Each element of the resistance matrix (i) represents a respective resistance value and (ii) corresponds to a respective weight of the weight matrix. Selecting component values of the analog components includes performing a gradient descent method to identify possible resistance values for the plurality of resistors. 
     In some implementations, the method further includes: generating an equivalent digital network of digital components for one or more output layers of the neural network topology; and connecting output of one or more layers of the equivalent analog network to the equivalent digital network of digital components. 
     In another aspect, a system is provided for hardware realization of neural networks. The system includes one or more processors, memory that stores one or more programs configured for execution by the one or more processors. The one or more programs include instructions for: obtaining a neural network topology and weights of a trained neural network; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; and generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. 
     In another aspect, a voice-transmission device is provided, and includes an integrated circuit for voice clarification. The integrated circuit includes an analog network of analog components fabricated by a method comprising the steps of: obtaining a neural network topology and weights of a trained neural network; transforming the neural network topology into an equivalent analog network of analog components; computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents one or more connections between analog components of the equivalent analog network; generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components; and fabricating the circuit, according to the schematic model, using a lithographic process. 
     In some implementations of the voice-transmission device, generating the schematic model further includes: generating a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix; and generating one or more lithographic masks for fabricating the circuit implementing the equivalent analog network of analog components based on the resistance matrix. 
     In some implementations of the voice-transmission device, the voice-transmission device is integrated into a cell phone. 
     In some implementations of the voice-transmission device, input from a microphone of the cell phone is input to the integrated circuit. 
     In some implementations of the voice-transmission device, output from the integrated circuit is input to a speaker of the cell phone. 
     In some implementations of the voice-transmission device, the integrated circuit is coupled to one or more other noise cancelling devices. 
     In some implementations of the voice-transmission device, the integrated circuit is coupled to one or more noise reduction software programs executing on the voice transmission device. 
     In some implementations, a computer system has one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein. 
     In some implementations, a non-transitory computer readable storage medium stores one or more programs configured for execution by a computer system having one or more processors, memory, and a display. The one or more programs include instructions for performing any of the methods described herein. 
     Thus, methods, systems, and devices are disclosed that are used for hardware realization of trained neural networks. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       For a better understanding of the aforementioned systems, methods, and graphical user interfaces, as well as additional systems, methods, and graphical user interfaces that provide data visualization analytics and data preparation, reference should be made to the Description of Implementations below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures. 
         FIG.  1 A  is a block diagram of a system for hardware realization of trained neural networks using analog components, according to some implementations.  FIG.  1 B  is a block diagram of an alternative representation of the system of  FIG.  1 A  for hardware realization of trained neural networks using analog components, according to some implementations.  FIG.  1 C  is a block diagram of another representation of the system of  FIG.  1 A  for hardware realization of trained neural networks using analog components, according to some implementations. 
         FIG.  2 A  is a system diagram of a computing device in accordance with some implementations.  FIG.  2 B  shows optional modules of the computing device, according to some implementations. 
         FIG.  3 A  shows an example process for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations.  FIG.  3 B  shows an example manual prototyping process used for generating a target chip model, according to some implementations. 
         FIGS.  4 A,  4 B, and  4 C  show examples of neural networks that are transformed to mathematically equivalent analog networks, according to some implementations. 
         FIG.  5    shows an example of a math model for a neuron, according to some implementations. 
         FIGS.  6 A- 6 C  illustrate an example process for analog hardware realization of a neural network for computing an XOR of input values, according to some implementations. 
         FIG.  7    shows an example perceptron, according to some implementations. 
         FIG.  8    shows an example Pyramid-Neural Network, according to some implementations. 
         FIG.  9    shows an example Pyramid Single Neural Network, according to some implementations. 
         FIG.  10    shows an example of a transformed neural network, according to some implementations. 
         FIGS.  11 A- 11 C  show an application of a T-transformation algorithm for a single layer neural network, according to some implementations. 
         FIG.  12    shows an example Recurrent Neural Network (RNN), according to some implementations. 
         FIG.  13 A  is a block diagram of a LSTM neuron, according to some implementations. 
         FIG.  13 B  shows delay blocks, according to some implementations. 
         FIG.  13 C  is a neuron schema for a LSTM neuron, according to some implementations. 
         FIG.  14 A  is a block diagram of a GRU neuron, according to some implementations. 
         FIG.  14 B  is a neuron schema for a GRU neuron, according to some implementations. 
         FIGS.  15 A and  15 B  are neuron schema of variants of a single Conv1D filter, according to some implementations. 
         FIG.  16    shows an example architecture of a transformed neural network, according to some implementations. 
         FIGS.  17 A- 17 C  provide example charts illustrating dependency between output error and classification error or weight error, according to some implementations. 
         FIG.  18    provides an example scheme of a neuron model used for resistors quantization, according to some implementations. 
         FIG.  19 A  shows a schematic diagram of an operational amplifier made on CMOS, according to some implementations.  FIG.  19 B  shows a table of description for the example circuit shown in  FIG.  19 A , according to some implementations. 
         FIGS.  20 A- 20 E  show a schematic diagram of a LSTM block, according to some implementations.  FIG.  20 F  shows a table of description for the example circuit shown in  FIG.  20 A- 20 D , according to some implementations. 
         FIGS.  21 A- 21 I  show a schematic diagram of a multiplier block, according to some implementations.  FIG.  21 J  shows a table of description for the schematic shown in  FIGS.  21 A- 21 I , according to some implementations. 
         FIG.  22 A  shows a schematic diagram of a sigmoid neuron, according to some implementations.  FIG.  22 B  shows a table of description for the schematic diagram shown in  FIG.  22 A , according to some implementations. 
         FIG.  23 A  shows a schematic diagram of a hyperbolic tangent function block, according to some implementations.  FIG.  23 B  shows a table of description for the schematic diagram shown in  FIG.  23 A , according to some implementations. 
         FIGS.  24 A- 24 C  show a schematic diagram of a single neuron CMOS operational amplifier, according to some implementations.  FIG.  24 D  shows a table of description for the schematic diagram shown in  FIG.  24 A- 24 C , according to some implementations. 
         FIGS.  25 A- 25 D  show a schematic diagram of a variant of a single neuron CMOS operational amplifiers according to some implementations.  FIG.  25 E  shows a table of description for the schematic diagram shown in  FIG.  25 A- 25 D , according to some implementations. 
         FIGS.  26 A- 26 K  show example weight distribution histograms, according to some implementations. 
         FIGS.  27 A- 27 J  show a flowchart of a method for hardware realization of neural networks, according to some implementations. 
         FIGS.  28 A- 28 S  show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. 
         FIGS.  29 A- 29 F  show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. 
         FIGS.  30 A- 30 M  show a flowchart of a method for hardware realization of neural networks according to hardware design constraints, according to some implementations. 
         FIGS.  31 A- 31 Q  show a flowchart of a method for fabricating an integrated circuit that includes an analog network of analog components, according to some implementations. 
         FIGS.  32 A- 32 E  show a flowchart of a method for generating libraries for hardware realization of neural networks, according to some implementations. 
         FIGS.  33 A- 33 K  show a flowchart of a method for optimizing energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations. 
         FIG.  34    shows a table describing the MobileNet v1 architecture, according to some implementations. 
         FIG.  35    shows an example 1-D convolutional neural network used for voice clarification, according to some implementations. 
         FIG.  36    shows an example T-transformation of fully connected layers of the neural network shown in  FIG.  35   , according to some implementations. 
         FIG.  37    shows an example T-transformation of 1-D convolution of the neural network shown in  FIG.  35   , according to some implementations. 
         FIG.  38    shows an example T-transformation of a MaxPooling operator of the neural network shown in  FIG.  35   , according to some implementations. 
         FIG.  39    shows an example system for sound signal processing using neuormorphic signal processors, according to some implementations. 
     
    
    
     Reference will now be made to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without requiring these specific details. 
     DESCRIPTION OF IMPLEMENTATIONS 
       FIG.  1 A  is a block diagram of a system  100  for hardware realization of trained neural networks using analog components, according to some implementations. The system includes transforming ( 126 ) trained neural networks  102  to analog neural networks  104 . In some implementations, analog integrated circuit constraints  184  constrain ( 146 ) the transformation ( 126 ) to generate the analog neural networks  104 . Subsequently, the system derives (calculates or generates) weights  106  for the analog neural networks  104  by a process that is sometimes called weight quantization ( 128 ). In some implementations, the analog neural network includes a plurality of analog neuron, each analog neuron represented by an analog component, such as an operational amplifier, and each analog neuron connected to another analog neuron via a connection. In some implementations, the connections are represented using resistors that reduce the current flow between two analog neurons. In some implementations, the system transforms ( 148 ) the weights  106  to resistance values  112  for the connections. The system subsequently generates ( 130 ) one or more schematic models  108  for implementing the analog neural networks  104  based on the weights  106 . In some implementations, the system optimizes resistance values  112  (or the weights  106 ) to form optimized analog neural networks  114  which is further used to generate ( 150 ) the schematic models  108 . In some implementations, the system generates ( 132 ) lithographic masks  110  for the connections and/or generates ( 136 ) lithographic masks  120  for the analog neurons. In some implementations, the system fabricates ( 134  and/or  138 ) analog integrated circuits  118  that implement the analog neural networks  104 . In some implementations, the system generates ( 152 ) libraries of lithographic masks  116  based on the lithographic masks for connections  110  and/or lithographic masks  120  for the analog neurons. In some implementations, the system uses ( 154 ) the libraries of lithographic masks  116  to fabricate the analog integrated circuits  118 . In some implementations, when the trained neural networks  142  are retrained ( 142 ), the system regenerates (or recalculates) ( 144 ) the resistance values  112  (and/or the weights  106 ), the schematic model  108 , and/or the lithographic masks for connections  110 . In some implementations, the system reuses the lithographic masks  120  for the analog neurons  120 . In other words, in some implementations, only the weights  106  (or the resistance values  112  corresponding to the changed weights), and/or the lithographic masks for the connections  110  are regenerated. Since only the connections, weights, the schematic model, and/or the corresponding lithographic masks for the connections are regenerated, as indicated by the dashed line  156 , the process for (or the path to) fabricating analog integrated circuits for the retrained neural networks is substantially simplified, and the time to market for re-spinning hardware for neural networks is reduced, when compared to conventional techniques for hardware realization of neural networks. 
       FIG.  1 B  is a block diagram of an alternative representation of the system  100  for hardware realization of trained neural networks using analog components, according to some implementations. The system includes training ( 156 ) neural networks in software, determining weights of connections, generating ( 158 ) electronic circuit equivalent to the neural network, calculating ( 160 ) resistor values corresponding to weights of each connection, and subsequently generating ( 162 ) lithography mask with resistor values. 
       FIG.  1 C  is a block diagram of another representation of the system  100  for hardware realization of trained neural networks using analog components, according to some implementations. The system is distributed as a software development kit (SDK)  180 , according to some implementations. A user develops and trains ( 164 ) a neural network and inputs the trained neural net  166  to the SDK  180 . The SDK estimates ( 168 ) complexity of the trained neural net  166 . If the complexity of the trained neural net can be reduced (e.g., some connections and/or neurons can be removed, some layers can be reduced, or the density of the neurons can be changed), the SDK  180  prunes ( 178 ) the trained neural net and retrains ( 182 ) the neural net to obtain an updated trained neural net  166 . Once the complexity of the trained neural net is reduced, the SDK  180  transforms ( 170 ) the trained neural net  166  into a sparse network of analog components (e.g., a pyramid- or a trapezia-shaped network). The SDK  180  also generates a circuit model  172  of the analog network. In some implementations, the SDK estimates ( 176 ) a deviation in an output generated by the circuit model  172  relative to the trained neural network for a same input, using software simulations. If the estimated error exceeds a threshold error (e.g., a value set by the user), the SDK  180  prompts the user to reconfigure, redevelop, and/or retrain the neural network. In some implementations, although not shown, the SDK automatically reconfigures the trained neural net  166  so as to reduce the estimated error. This process is iterated multiple times until the error is reduced below the threshold error. In  FIG.  1 C , the dashed line from the block  176  (“Estimation of error raised in circuitry”) to the block  164  (“Development and training of neural network”) indicates a feedback loop. For example, if the pruned network did not show desired accuracy, some implementations prune the network differently, until accuracy exceeds a predetermined threshold (e.g., 98% accuracy) for a given application. In some implementations, this process includes recalculating the weights, since pruning includes retraining of the whole network. 
     In some implementations, components of the system  100  described above are implemented in one or more computing devices or server systems as computing modules.  FIG.  2 A  is a system diagram of a computing device  200  in accordance with some implementations. As used herein, the term “computing device” includes both personal devices  102  and servers. A computing device  200  typically includes one or more processing units/cores (CPUs)  202  for executing modules, programs, and/or instructions stored in the memory  214  and thereby performing processing operations; one or more network or other communications interfaces  204 ; memory  214 ; and one or more communication buses  212  for interconnecting these components. The communication buses  212  may include circuitry that interconnects and controls communications between system components. A computing device  200  may include a user interface  206  comprising a display device  208  and one or more input devices or mechanisms  210 . In some implementations, the input device/mechanism  210  includes a keyboard; in some implementations, the input device/mechanism includes a “soft” keyboard, which is displayed as needed on the display device  208 , enabling a user to “press keys” that appear on the display  208 . In some implementations, the display  208  and input device/mechanism  210  comprise a touch screen display (also called a touch sensitive display). In some implementations, the memory  214  includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices. In some implementations, the memory  214  includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some implementations, the memory  214  includes one or more storage devices remotely located from the CPU(s)  202 . The memory  214 , or alternatively the non-volatile memory device(s) within the memory  214 , comprises a computer readable storage medium. In some implementations, the memory  214 , or the computer readable storage medium of the memory  214 , stores the following programs, modules, and data structures, or a subset thereof:
         an operating system  216 , which includes procedures for handling various basic system services and for performing hardware dependent tasks;   a communications module  218 , which is used for connecting the computing device  200  to other computers and devices via the one or more communication network interfaces  204  (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;   trained neural networks  220  that includes weights  222  and neural network topologies  224 . Examples of input neural networks are described below in reference to  FIGS.  4 A- 4 C ,  FIG.  12   ,  FIGS.  13 A, and  14 A , according to some implementations;   a neural network transformation module  226  that includes transformed analog neural networks  228 , mathematical formulations  230 , the basic function blocks  232 , analog models  234  (sometimes called neuron models), and/or analog integrated circuit (IC) design constraints  236 . Example operations of the neural network transformation module  226  are described below in reference to at least  FIGS.  5 ,  6 A- 6 C,  7 ,  8 ,  9 ,  10 , and  11 A- 11 C , and the flowcharts shown in  FIGS.  27 A- 27 J , and  FIGS.  28 A- 28 S ; and/or   a weight matrix computation (sometimes called a weight quantization) module  238  that includes weights  272  of transformed networks, and optionally includes resistance calculation module  240 , resistance values  242 . Example operations of the weight matrix computation module  238  and/or weight quantization are described in reference to at least  FIGS.  17 A- 17 C ,  FIG.  18   , and  FIGS.  29 A- 29 F , according to some implementations.       

     Some implementations include one or more optional modules  244  as shown in  FIG.  2 B . Some implementations include an analog neural network optimization module  246 . Examples of analog neural network optimization are described below in reference to  FIGS.  30 A- 30 M , according to some implementations. 
     Some implementations include a lithographic mask generation module  248  that further includes lithographic masks  250  for resistances (corresponding to connections), and/or lithographic masks for analog components (e.g., operational amplifiers, multipliers, delay blocks, etc.) other than the resistances (or connections). In some implementations, lithographic masks are generated based on chip design layout following chip design using Cadence, Synopsys, or Mentor Graphics software packages. Some implementations use a design kit from a silicon wafer manufacturing plant (sometimes called a fab). Lithographic masks are intended to be used in that particular fab that provides the design kit (e.g., TSMC 65 nm design kit). The lithographic mask files that are generated are used to fabricate the chip at the fab. In some implementations, the Cadence, Mentor Graphics, or Synopsys software packages-based chip design is generated semi-automatically from the SPICE or Fast SPICE (Mentor Graphics) software packages. In some implementations, a user with chip design skill drives the conversion from the SPICE or Fast SPICE circuit into Cadence, Mentor Graphics or Synopsis chip design. Some implementations combine Cadence design blocks for single neuron unit, establishing proper interconnects between the blocks. 
     Some implementations include a library generation module  254  that further includes libraries of lithographic masks  256 . Examples of library generation are described below in reference to  FIGS.  32 A- 32 E , according to some implementations. 
     Some implementations include Integrated Circuit (IC) fabrication module  258  that further includes Analog-to-Digital Conversion (ADC), Digital-to-Analog Conversion (DAC), or similar other interfaces  260 , and/or fabricated ICs or models  262 . Example integrated circuits and/or related modules are described below in reference to  FIGS.  31 A- 31 Q , according to some implementations. 
     Some implementations include an energy efficiency optimization module  264  that further includes an inferencing module  266 , a signal monitoring module  268 , and/or a power optimization module  270 . Examples of energy efficiency optimizations are described below in reference to  FIGS.  33 A- 33 K , according to some implementations. 
     Each of the above identified executable modules, applications, or sets of procedures may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise rearranged in various implementations. In some implementations, the memory  214  stores a subset of the modules and data structures identified above. Furthermore, in some implementations, the memory  214  stores additional modules or data structures not described above. 
     Although  FIG.  2 A  shows a computing device  200 ,  FIG.  2 A  is intended more as a functional description of the various features that may be present rather than as a structural schematic of the implementations described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. 
     Example Process for Generating Schematic Models of Analog Networks 
       FIG.  3 A  shows an example process  300  for generating schematic models of analog networks corresponding to trained neural networks, according to some implementations. As shown in  FIG.  3 A , a trained neural network  302  (e.g., MobileNet) is converted ( 322 ) to a target or equivalent analog network  304  (using a process that is sometimes called T-transformation). The target neural network (sometimes called a T-network)  304  is exported ( 324 ) to SPICE (as a SPICE model  306 ) using a single neuron model (SNM), which is exported ( 326 ) from SPICE to CADENCE and full on-chip designs using a CADENCE model  308 . The CADENCE model  308  is cross-validated ( 328 ) against the initial neural network for one or more validation inputs. 
     In the description above and below, a math neuron is a mathematical function which receives one or more weighted inputs and produces a scalar output. In some implementations, a math neuron can have memory (e.g., long short-term memory (LSTM), recurrent neuron). A trivial neuron is a math neuron that performs a function, representing an ‘ideal’ mathematical neuron, V out =f(Σ(V i   in ·ω i +bias), where f(x) is an activation function. A SNM is a schematic model with analog components (e.g., operational amplifiers, resistors R 1 , . . . , R n , and other components) representing a specific type of math neuron (for example, trivial neuron) in schematic form. SNM output voltage is represented by a corresponding formula that depends on K input voltages and SNM component values V out =g(V i   in , . . . , V K   in , R 1  . . . R n ). According to some implementations, with properly selected component values, SNM formula is equivalent to math neuron formula, with a desired weights set. In some implementations, the weights set is fully determined by resistors used in a SNM. A target (analog) neural network  304  (sometimes called a T-network) is a set of math neurons which have defined SNM representation, and weighted connections between them, forming a neural network. A T-network follows several restrictions, such as an inbound limit (a maximum limit of inbound connections for any neuron within the T-network), an outbound limit (a maximum limit of outbound connections for any neuron within the T-network), and a signal range (e.g., all signals should be inside pre-defined signal range). T-transformation ( 322 ) is a process of converting some desired neural network, such as MobileNet, to a corresponding T-network. A SPICE model  306  is a SPICE Neural Network model of a T-network  304 , where each math neuron is substituted with corresponding one or more SNMs. A Cadence NN model  310  is a Cadence model of the T-network  304 , where each math neuron is substituted with a corresponding one or more SNMs. Also, as described herein, two networks L and M have mathematical equivalence, if for all neuron outputs of these networks |V i   L −V i   M |&lt;eps, where eps is relatively small (e.g., between 0.1-1% of operating voltage range). Also, two networks L and M have functional equivalence, if for a given validation input data set {I 1 , . . . , I n }, the classification results are mostly the same, i.e., P(L(I k )=M(I k ))=1−eps, where eps is relatively small. 
       FIG.  3 B  shows an example manual prototyping process used for generating a target chip model  320  based on a SNM model on Cadence  314 , according to some implementations. Note that although the following description uses Cadence, alternate tools from Mentor Graphic design or Synopsys (e.g., Synopsys design kit) may be used in place of Cadence tools, according to some implementations. The process includes selecting SNM limitations, including inbound and outbound limits and signal limitation, selecting analog components (e.g., resistors, including specific resistor array technology) for connections between neurons, and developing a Cadence SNM model  314 . A prototype SNM model  316  (e.g., a PCB prototype) is developed ( 330 ) based on the SNM model on Cadence  314 . The prototype SNM model  316  is compared with a SPICE model for equivalence. In some implementations, a neural network is selected for an on-chip prototype, when the neural network satisfies equivalence requirements. Because the neural network is small in size, the T-transformation can be hand-verified for equivalence. Subsequently, an on-chip SNM model  318  is generated ( 332 ) based on the SNM model prototype  316 . The on-chip SNM model is optimized as possible, according to some implementations. In some implementations, an on-chip density for the SNM model is calculated prior to generating ( 334 ) a target chip model  320  based on the on-chip SNM model  318 , after finalizing the SNM. During the prototyping process, a practitioner may iterate selecting neural network task or application and specific neural network (e.g., a neural network having in the order of 0.1 to 1.1 million neurons), performing T-transformation, building a Cadence neural network model, designing interfaces and/or the target chip model. 
     Example Input Neural Networks 
       FIGS.  4 A,  4 B, and  4 C  show examples of trained neural networks (e.g., the neural networks  220 ) that are input to the system  100  and transformed to mathematically equivalent analog networks, according to some implementations.  FIG.  4 A  shows an example neural network (sometimes called an artificial neural network) that are composed of artificial neurons that receive input, combine the input using an activation function, and produce one or more outputs. The input includes data, such as images, sensor data, and documents. Typically, each neural network performs a specific task, such as object recognition. The networks include connections between the neurons, each connection providing the output of a neuron as an input to another neuron. After training, each connection is assigned a corresponding weight. As shown in  FIG.  4 A , the neurons are typically organized into multiple layers, with each layer of neurons connected only to the immediately preceding and following layer of neurons. An input layer of neurons  402  receives external input (e.g., the input X 1 , X 2 , . . . , X n ). The input layer  402  is followed by one or more hidden layers of neurons (e.g., the layers  404  and  406 ), that is followed by an output layer  408  that produces outputs  410 . Various types of connection patterns connect neurons of consecutive layers, such as a fully-connected pattern that connects every neuron in one layer to all the neurons of the next layer, or a pooling pattern that connect output of a group of neurons in one layer to a single neuron in the next layer. In contrast to the neural network shown in  FIG.  4 A  that are sometimes called feedforward networks, the neural network shown in  FIG.  4 B  includes one or more connections from neurons in one layer to either other neurons in the same layer or neurons in a preceding layer. The example shown in  FIG.  4 B  is an example of a recurrent neural network, and includes two input neurons  412  (that accepts an input X1) and  414  (that accepts an input X2) in an input layer followed by two hidden layers. The first hidden layer includes neurons  416  and  418  that is fully connected with neurons in the input layer, and the neurons  420 ,  422 , and  424  in the second hidden layer. The output of the neuron  420  in the second hidden layer is connected to the neuron  416  in the first hidden layer, providing a feedback loop. The hidden layer including the neurons  420 ,  422 , and  424  are input to a neuron  426  in the output layer that produces an output y. 
       FIG.  4 C  shows an example of a convolutional neural network (CNN), according to some implementations. In contrast to the neural networks shown in  FIGS.  4 A and  4 B , the example shown in  FIG.  4 C  includes different types of neural network layers, that includes a first stage of layers for feature learning, and a second stage of layers for classification tasks, such as object recognition. The feature learning stage includes a convolution and Rectified Linear Unit (ReLU) layer  430 , followed by a pooling layer  432 , that is followed by another convolution and ReLU layer  434 , which is in turn followed by another pooling layer  436 . The first layer  430  extracts features from an input  428  (e.g., an input image or portions thereof), and performs a convolution operation on its input, and one or more non-linear operations (e.g., ReLU, tanh, or sigmoid). A pooling layer, such as the layer  432 , reduces the number of parameters when the inputs are large. The output of the pooling layer  436  is flattened by the layer  438  and input to a fully connected neural network with one or more layers (e.g., the layers  440  and  442 ). The output of the fully-connected neural network is input to a softmax layer  444  to classify the output of the layer  442  of the fully-connected network to produce one of many different output  446  (e.g., object class or type of the input image  428 ). 
     Some implementations store the layout or the organization of the input neural networks including number of neurons in each layer, total number of neurons, operations or activation functions of each neuron, and/or connections between the neurons, in the memory  214 , as the neural network topology  224 . 
       FIG.  5    shows an example of a math model  500  for a neuron, according to some implementations. The math model includes incoming signals  502  input multiplied by synaptic weights  504  and summed by a unit summation  506 . The result of the unit summation  506  is input to a nonlinear conversion unit  508  to produce an output signal  510 , according to some implementations. 
       FIGS.  6 A- 6 C  illustrate an example process for analog hardware realization of a neural network for computing an XOR (classification of XOR results) of input values, according to some implementations.  FIG.  6 A  shows a table  600  of possible input values X 1  and X 2  along x- and y-axis, respectively. The expected result values are indicated by hollow circle (represents a value of 1) and a filled or dark circle (represents a value of 0)— this is a typical XOR problem with 2 input signals and 2 classes. Only if either, not both, of the values X 1  and X 2  are 1, the expected result is 1, and 0, otherwise. Training set consists of 4 possible input signal combinations (binary values for the X 1  and X 2  inputs).  FIG.  6 B  shows a ReLU-based neural network  602  to solve the XOR classification of  FIG.  6 A , according to some implementations. The neurons do not use any bias values, and use ReLU activation. Inputs  604  and  606  (that correspond to X 1  and X 2 , respectively) are input to a first ReLU neuron  608 - 2 . The inputs  604  and  606  are also input to a second ReLU neuron  608 - 4 . The results of the two ReLU neurons  608 - 2  and  608 - 4  are input to a third neuron  608 - 6  that performs linear summation of the input values, to produce an output value  510  (the Out value). The neural network  602  has the weights −1 and 1 (for the input values X 1  and X 2 , respectively) for the ReLU neuron  608 - 2 , the weights 1 and −1 (for the input values X 1  and X 2 , respectively) for the ReLU neuron  608 - 4 , and the weights 1 and 1 (for the output of the ReLU neurons  608 - 2  and  608 - 4 , respectively). In some implementations, the weights of trained neural networks are stored in memory  214 , as the weights  222 . 
       FIG.  6 C  shows an example equivalent analog network for the network  602 , according to some implementations. The analog equivalent inputs  614  and  616  of the X 1  and X 2  inputs  604  and  606  are input to analog neurons N1  618  and N2  620  of a first layer. The neurons N1 and N2 are densely connected with neurons N3 and N4 of a second layer. The neurons of a second layer (i.e. neuron N3  622  and neuron N4  624 ) are connected with an output neuron N5  626  that produces the output Out (equivalent to the output  610  of the network  602 ). The neurons N1, N2, N3, N4 and N5 have ReLU (maximum value=1) activation function. 
     Some implementations use Keras learning that converges in approximately 1000 iterations, and results in weights for the connections. In some implementations, the weights are stored in memory  214 , as part of the weights  222 . In the following example, data format is ‘Neuron [1 st  link weight, 2 nd  link weight, bias]’.
         N1 [−0.9824321, 0.976517, −0.00204677];   N2 [1.0066702, −1.0101418, −0.00045485];   N3 [1.0357606, 1.0072469, −0.00483723];   N4 [−0.07376373, −0.7682612, 0.0]; and   N5 [1.0029935, −1.1994369, −0.00147767].       

     Next, to compute resistor values for connections between the neurons, some implementations compute resistor range. Some implementations set resistor nominal values (R+, R−) of 1 MΩ, possible resistor range of 100 KΩ to 1 MΩ and nominal series E24. Some implementations compute w1, w2, wbias resistor values for each connection as follows. For each weight value wi (e.g., the weights  222 ), some implementations evaluate all possible (Ri−, Ri+) resistor pairs options within the chosen nominal series and choose a resistor pair which produces minimal error value 
     
       
         
           
             
               e 
               ⁢ 
               r 
               ⁢ 
               r 
             
             = 
             
               Abs 
               ⁢ 
               
                 
                   ( 
                   
                     
                       w 
                       i 
                     
                     - 
                     
                       1 
                       
                         R 
                         i 
                         + 
                       
                     
                     + 
                     
                       1 
                       
                         R 
                         i 
                         - 
                       
                     
                   
                   ) 
                 
                 . 
               
             
           
         
       
     
     The following table provides example values for the weights w1, w2, and bias, for each connection, according to some implementations. 
     
       
         
           
               
               
               
               
               
             
               
                   
                   
               
               
                   
                   
                   
                   
                 Implemented 
               
               
                   
                 Model value 
                 R− (MΩ) 
                 R+ (MΩ) 
                 value 
               
               
                   
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
               
            
               
                 N1_w1 
                 −0.9824321 
                 0.36 
                 0.56 
                 −0.992063 
               
               
                 N1_w2 
                 0.976517 
                 0.56 
                 0.36 
                 0.992063 
               
               
                 N1_bias 
                 −0.00204677 
                 0.1 
                 0.1 
                 0.0 
               
               
                 N2_w1 
                 1.0066702 
                 0.43 
                 0.3 
                 1.007752 
               
               
                 N2_w2 
                 −1.0101418 
                 0.18 
                 0.22 
                 −1.010101 
               
               
                 N2_bias 
                 −0.00045485 
                 0.1 
                 0.1 
                 0.0 
               
               
                 N3_w1 
                 1.0357606 
                 0.91 
                 0.47 
                 1.028758 
               
               
                 N3_w2 
                 1.0072469 
                 0.43 
                 0.3 
                 1.007752 
               
               
                 N3_bias 
                 −0.00483723 
                 0.1 
                 0.1 
                 0.0 
               
               
                 N4_w1 
                 −0.07376373 
                 0.91 
                 1.0 
                 −0.098901 
               
               
                 N4_w2 
                 −0.7682612 
                 0.3 
                 0.39 
                 −0.769231 
               
               
                 N4_bias 
                 0.0 
                 0.1 
                 0.1 
                 0.0 
               
               
                 N5_w1 
                 1.0029935 
                 0.43 
                 0.3 
                 1.007752 
               
               
                 N5_w2 
                 −1.1994369 
                 0.3 
                 0.47 
                 −1.205674 
               
               
                 N5_bias 
                 −0.00147767 
                 0.1 
                 0.1 
                 0.0 
               
               
                   
               
            
           
         
       
     
     Example Advantages of Transformed Neural Networks 
     Before describing examples of transformation, it is worth noting some of the advantages of the transformed neural networks over conventional architectures. As described herein, the input trained neural networks are transformed to pyramid- or trapezium-shaped analog networks. Some of the advantages of pyramid or trapezium over cross bars include lower latency, simultaneous analog signal propagation, possibility for manufacture using standard integrated circuit (IC) design elements, including resistors and operational amplifiers, high parallelism of computation, high accuracy (e.g., accuracy increases with the number of layers, relative to conventional methods), tolerance towards error(s) in each weight and/or at each connection (e.g., pyramids balance the errors), low RC (low Resistance Capacitance delay related to propagation of signal through network), and/or ability to manipulate biases and functions of each neuron in each layer of the transformed network. Also, pyramids are excellent computation block by itself, since it is a multi-level perceptron, which can model any neural network with one output. Networks with several outputs are implemented using different pyramids or trapezia geometry, according to some implementations. A pyramid can be thought of as a multi-layer perceptron with one output and several layers (e.g., N layers), where each neuron has n inputs and 1 output. Similarly, a trapezium is a multilayer perceptron, where each neuron has n inputs and m outputs. Each trapezium is a pyramid-like network, where each neuron has n inputs and m outputs, where n and m are limited by IC analog chip design limitations, according to some implementations. 
     Some implementations perform lossless transformation of any trained neural network into subsystems of pyramids or trapezia. Thus, pyramids and trapezia can be used as universal building blocks for transforming any neural networks. An advantage of pyramid- or trapezia-based neural networks is the possibility to realize any neural network using standard IC analog elements (e.g., operational amplifiers, resistors, signal delay lines in case of recurrent neurons) using standard lithography techniques. It is also possible to restrict the weights of transformed networks to some interval. In other words, lossless transformation is performed with weights limited to some predefined range, according to some implementations. Another advantage of using pyramids or trapezia is the high degree of parallelism in signal processing or the simultaneous propagation of analog signals that increases the speed of calculations, providing lower latency. Moreover, many modern neural networks are sparsely connected networks and are much better (e.g., more compact, have low RC values, absence of leakage currents) when transformed into pyramids than into cross-bars, Pyramids and trapezia networks are relatively more compact than cross-bar based memristor networks. 
     Furthermore, analog neuromorphic trapezia-like chips possess a number of properties, not typical for analog devices. For example, signal to noise ratio is not increasing with the number of cascades in analog chip, the external noise is suppressed, and influence of temperature is greatly reduced. Such properties make trapezia-like analog neuromorphic chips analogous to digital circuits. For example, individual neurons, based on operational amplifier, level the signal and are operated with the frequencies of 20,000-100,000 Hz, and are not influenced by noise or signals with frequency higher than the operational range, according to some implementations. Trapezia-like analog neuromorphic chip also perform filtration of output signal due to peculiarities in how operational amplifiers function. Such trapezia-like analog neuromorphic chip suppresses the synphase noise. Due to low-ohmic outputs of operational amplifiers, the noise is also significantly reduced. Due to the leveling of signal at each operational amplifier output and synchronous work of amplifiers, the drift of parameters, caused by temperature does not influence the signals at final outputs. Trapezia-like analogous neuromorphic circuit is tolerant towards the errors and noise in input signals and is tolerant towards deviation of resistor values, corresponding to weight values in neural network. Trapezia-like analog neuromorphic networks are also tolerant towards any kind of systemic error, like error in resistor value settings, if such error is same for all resistors, due to the very nature of analog neuromorphic trapezia-like circuits, based on operational amplifiers. 
     Example Lossless Transformation (T-Transformation) of Trained Neural Networks 
     In some implementations, the example transformations described herein are performed by the neural network transformation module  226  that transform trained neural networks  220 , based on the mathematical formulations  230 , the basic function blocks  232 , the analog component models  234 , and/or the analog design constraints  236 , to obtain the transformed neural networks  228 . 
       FIG.  7    shows an example perceptron  700 , according to some implementations. The perceptron includes K=8 inputs and 8 neurons  702 - 2 , . . . ,  702 - 16  in an input layer that receives the 8 inputs. There is an output layer with 4 neurons  704 - 2 , . . . ,  704 - 8 , in an output layer, that correspond to L=4 outputs. The neurons in the input layer are fully connected to the neurons in the output layer, making 8 times 4=32 connections. Suppose the weights of the connections are represented by a weight matrix WP (element WP i,j  corresponds to the weight of the connection between the i-th neuron in the input layer and the j-th neuron in the output layer). Suppose further each neuron performs an activation function F. 
       FIG.  8    shows an example Pyramid-Neural Network (P-NN)  800 , a type of Target-Neural Network (T-NN, or TNN), that is equivalent to the perceptron shown in  FIG.  7   , according to some implementations. To perform this transformation of the perceptron ( FIG.  7   ) to the PN-NN architecture ( FIG.  8   ), suppose, for the T-NN, that number of inputs is restricted to Ni=4 and number of outputs is restricted to No=2. The T-NN includes an input layer LTI of neurons  802 - 2 , . . . ,  802 - 34 , that is a concatenation of two copies of the input layer of neurons  802 - 2 , . . . ,  802 - 16 , for a total of 2 times 8=16 input neurons. The set of neurons  804 , including neurons  802 - 20 , . . . ,  802 - 34 , is a copy of the neurons  802 - 2 , . . . ,  802 - 18 , and the input is replicated. For example, the input to the neuron  802 - 2  is also input to the neuron  802 - 20 , the input  20  the neuron  802 - 4  is also input to the neuron  802 - 22 , and so on.  FIG.  8    also includes a hidden layer LTH1 of neurons  806 - 02 , . . . ,  806 - 16  (2 times 16 divided by 4=8 neurons) that are linear neurons. Each group of Ni neurons from the input layer LTI are fully connected to two neurons from the LTH1 layer.  FIG.  8    also includes an output layer LTO with 2 times 8 divided by 4=4 neurons  808 - 02 , . . . ,  808 - 08 , each neuron performing the activation function F. Each neuron in the layer LTO is connected to distinct neurons from different groups in the layer LTH1. The network shown in  FIG.  8    includes 40 connections. Some implementations perform weight matrix calculation for the P-NN in  FIG.  8   , as follows. Weights for the hidden layer LTH1 (WTH1) are calculated from the weight matrix WP, and weights corresponding to the output layer LTO (WTO) form a sparse matrix with elements equal to 1. 
       FIG.  9    shows a Pyramid Single Neural Network (PSNN)  900  corresponding to an output neuron of  FIG.  8   , according to some implementations. The PSNN includes a layer (LPSI) of input neurons  902 - 02 , . . . ,  902 - 16  (corresponding to the 8 input neurons in the network  700  of  FIG.  7   ). A hidden layer LPSH1 includes 8 divided by 4=2 linear neurons  904 - 02  and  904 - 04 , and each group of Ni neurons from LTI is connected to one neuron of the LPSH1 layer. An output layer LPSO consists of 1 neuron  906  with an activation function F, that is connected to both the neurons  904 - 02  and  904 - 04  of the hidden layer. For calculating weight matrix for the PSNN  900 , some implementations compute a vector WPSH1 that is equal to the first row of WP, for the LPSH1 layer. For the LPSO layer, some implementations compute a weight vector WPSO with 2 elements, each element equal to 1. The process is repeated for the first, second, third, and fourth output neurons. A P-NN, such as the network shown in  FIG.  8   , is a union of the PSNNs (for the 4 output neurons). Input layer for every PSNN is a separate copy of P&#39;s input layer. For this example, the P-NN  800  includes an input layer with 8 times 4=32 inputs, a hidden layer with 2 times 4=8 neurons, and an output layer with 4 neurons. 
     Example Transformations with Target Neurons with N Inputs and 1 Output 
     In some implementations, the example transformations described herein are performed by the neural network transformation module  226  that transform trained neural networks  220 , based on the mathematical formulations  230 , the basic function blocks  232 , the analog component models  234 , and/or analog design constraints  236 , to obtain the transformed neural networks  228 . 
     Single Layer Perceptron with One Output 
     Suppose a single layer perceptron SLP(K, l) includes K inputs and one output neuron with activation function F. Suppose further U∈R K  is a vector of weights for SLP(K, l). The following algorithm Neuron2TNN1 constructs a T-neural network from T-neurons with N inputs and 1 output (referred to as TN(N, l)). 
     Algorithm Neuron2TNN1 
     1. Construct an input layer for T-NN by including all inputs from SLP(K,l). 
     2. If K&gt;N then:
         a. Divide K input neurons into       

     
       
         
           
             
               m 
               1 
             
             = 
             
               ⌈ 
               
                 K 
                 N 
               
               ⌉ 
             
           
         
       
         
         
           
              groups such that every group consists of no more than N inputs. 
             b. Construct the first hidden layer LTH 1  of the T-NN from m 1  neurons, each neuron performing an identity activation function. 
             c. Connect input neurons from every group to corresponding neuron from the next layer. So every neuron from the LTH 1  has no more than N input connections. 
             d. Set the weights for the new connections according the following equation: 
           
         
       
    
     
       
         
           
             
               
                 w 
                 ij 
               
               = 
               
                 u 
                 j 
               
             
             , 
             
               j 
               = 
               
                 
                   
                     ( 
                     
                       i 
                       - 
                       1 
                     
                     ) 
                   
                   * 
                   N 
                 
                 + 
                 1 
               
             
             , 
             … 
                 
             , 
             
               i 
               * 
               N 
             
           
         
       
       
         
           
             
               i 
               = 
               1 
             
             , 
             … 
                 
             , 
             
               ⌈ 
               
                 K 
                 N 
               
               ⌉ 
             
           
         
       
     
     3. Else (i.e., if K&lt;=N) then):
         a. Construct the output layer with 1 neuron calculating activation function F   b. Connect input neurons to the single output neuron. It has K≤N connections.   c. Set the weights of the new connections by means of the following equation:       

         w   j   1   =u   j   ,j= 1, . . . , K            d. Terminate the algorithm       
     4. Set l=1 
     5. If m l &gt;N:
         a. Divide m l  neurons into       

     
       
         
           
             
               m 
               
                 l 
                 + 
                 1 
               
             
             = 
             
               ⌈ 
               
                 
                   m 
                   l 
                 
                 N 
               
               ⌉ 
             
           
         
       
         
         
           
              groups, every group consists of no more than N neurons. 
             b. Construct the hidden layer LTH l+1  of the T-NN from m l+1  neurons, every neuron has identity activation function. 
             c. Connect input neurons from every group to the corresponded neuron from the next layer. 
             d. Set the weights of the new connections according the following equation: 
           
         
       
    
     
       
         
           
             
               w 
               ij 
               
                 l 
                 + 
                 1 
               
             
             = 
             1 
           
         
       
       
         
           
             
               i 
               = 
               1 
             
             , 
             … 
                 
             , 
             
               ⌈ 
               
                 
                   m 
                   l 
                 
                 N 
               
               ⌉ 
             
           
         
       
         
         
           
             e. Set l=l+1 
           
         
       
    
     6. Else (if m&gt;=N):
         a. Construct the output layer with 1 neuron calculating activation function F   b. Connect all LTH l &#39;s neurons to the single output neuron.   c. Set the weights of the new connections according the following equation:       

         w   j   l+1 =1  a.
         d. Terminate the algorithm       

     7. Repeat steps 5 and 6. 
     Here ┌x┐—minimum integer number being no less than x. Number of layers in T-NN constructed by means of the algorithm Neuron2TNN1 is h=┌log N K┐. The total number of weights in T-NN is: 
     
       
         
           
             S 
             = 
             
               K 
               ⁢ 
               
                 
                   
                     N 
                     m 
                   
                   - 
                   1 
                 
                 
                   
                     N 
                     
                       m 
                       - 
                       1 
                     
                   
                   ( 
                   
                     N 
                     - 
                     1 
                   
                   ) 
                 
               
             
           
         
       
     
       FIG.  10    shows an example of the constructed T-NN, according to some implementations. All layers except the first one perform identity transformation of their inputs. Weight matrices of the constructed T-NN have the following forms, according to some implementations. 
     Layer 1 (e.g., layer  1002 ): 
     
       
         
           
             
               w 
               1 
             
             = 
             
               
                 
                   
                     u 
                     1 
                   
                 
                 
                   
                     u 
                     2 
                   
                 
                 
                   … 
                 
                 
                   
                     u 
                     N 
                   
                 
                 
                   0 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   0 
                 
               
               
                 
                   0 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   
                     u 
                     
                       N 
                       + 
                       1 
                     
                   
                 
                 
                   
                     u 
                     
                       N 
                       + 
                       2 
                     
                   
                 
                 
                   … 
                 
                 
                   
                     u 
                     
                       2 
                       ⁢ 
                       N 
                     
                   
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
               
               
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
               
               
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                     
                 
                 
                   
                     u 
                     
                       ( 
                       
                         h 
                         - 
                       
                     
                   
                 
                 
                     
                 
                 
                     
                 
               
               
                 
                   0 
                 
                 
                   0 
                 
                 
                     
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   
                       
                     
                       
                         
                           1 
                           ) 
                         
                         ⁢ 
                         N 
                       
                       + 
                       1 
                     
                   
                 
                 
                   … 
                 
                 
                   
                     u 
                     K 
                   
                 
               
             
           
         
       
     
     Layers i=2, 3, . . . , h (e.g., layers  1004 ,  1006 , 1008 , and  1010 ): 
     
       
         
           
             
               w 
               i 
             
             = 
             
               
                 
                   1 
                 
                 
                   1 
                 
                 
                   … 
                 
                 
                   1 
                 
                 
                   0 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   0 
                 
               
               
                 
                   0 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   1 
                 
                 
                   1 
                 
                 
                   … 
                 
                 
                   1 
                 
                 
                   0 
                 
                 
                   … 
                 
                 
                   0 
                 
               
               
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
               
               
                 
                   0 
                 
                 
                   0 
                 
                 
                     
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   … 
                 
                 
                   0 
                 
                 
                   1 
                 
                 
                   … 
                 
                 
                   1 
                 
               
             
           
         
       
     
     Output value of the T-NN is calculated according the following formula: 
         y=F ( W   m   W   m-1    . . . W   2   W   1   x ) 
     Output for the first layer is calculated as an output vector according to the following formula: 
     
       
         
           
             
               
                 W 
                 1 
               
               ⁢ 
               x 
             
             = 
             
               
                 ( 
                 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       N 
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                   , 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         
                           N 
                           + 
                           1 
                         
                       
                       
                         2 
                         ⁢ 
                         N 
                       
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                   , 
                   … 
                       
                   , 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         
                           
                             
                               ( 
                               
                                 
                                   m 
                                   1 
                                 
                                 - 
                                 1 
                               
                               ) 
                             
                             * 
                             N 
                           
                           + 
                           1 
                         
                       
                       K 
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                   , 
                 
                 ) 
               
               T 
             
           
         
       
     
     Multiplying the obtained vector by the weight matrix of the second layer: 
     
       
         
           
             
               W 
               2 
             
             ⁢ 
             
               W 
               1 
             
             ⁢ 
             x 
           
         
       
       
         
           
             = 
             
               
                 ( 
                 
                   
                     
                       ∑ 
                       
                         l 
                         = 
                         1 
                       
                       N 
                     
                     
                       1 
                       ⁢ 
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             
                               
                                 
                                   ( 
                                   
                                     l 
                                     - 
                                     1 
                                   
                                   ) 
                                 
                                 * 
                                 N 
                               
                               + 
                               1 
                             
                           
                           lN 
                         
                         
                           
                             u 
                             j 
                           
                           ⁢ 
                           
                             x 
                             j 
                           
                         
                       
                     
                   
                   , 
                   
                     
                       ∑ 
                       
                         l 
                         = 
                         
                           N 
                           + 
                           1 
                         
                       
                       
                         2 
                         ⁢ 
                         N 
                       
                     
                     
                       1 
                       ⁢ 
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             
                               
                                 
                                   ( 
                                   
                                     l 
                                     - 
                                     1 
                                   
                                   ) 
                                 
                                 * 
                                 N 
                               
                               + 
                               1 
                             
                           
                           lN 
                         
                         
                           
                             u 
                             j 
                           
                           ⁢ 
                           
                             x 
                             j 
                           
                         
                       
                     
                   
                   , 
                   … 
                       
                   , 
                   
                     
                       ∑ 
                       
                         l 
                         = 
                         
                           
                             
                               ( 
                               
                                 
                                   m 
                                   2 
                                 
                                 - 
                                 1 
                               
                               ) 
                             
                             ⁢ 
                             N 
                           
                           + 
                           1 
                         
                       
                       
                         m 
                         1 
                       
                     
                     
                       1 
                       ⁢ 
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             
                               
                                 
                                   ( 
                                   
                                     l 
                                     - 
                                     1 
                                   
                                   ) 
                                 
                                 * 
                                 N 
                               
                               + 
                               1 
                             
                           
                           lN 
                         
                         
                           
                             u 
                             j 
                           
                           ⁢ 
                           
                             x 
                             j 
                           
                         
                       
                     
                   
                 
                 ) 
               
               T 
             
           
         
       
       
         
           
             = 
             
               
                 ( 
                 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       
                         N 
                         2 
                       
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                   , 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         
                           
                             N 
                             2 
                           
                           + 
                           1 
                         
                       
                       
                         2 
                         ⁢ 
                         
                           N 
                           2 
                         
                       
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                   , 
                   … 
                       
                   , 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         
                           
                             
                               ( 
                               
                                 
                                   m 
                                   2 
                                 
                                 - 
                                 1 
                               
                               ) 
                             
                             * 
                             
                               N 
                               2 
                             
                           
                           + 
                           1 
                         
                       
                       K 
                     
                     
                       
                         u 
                         j 
                       
                       ⁢ 
                       
                         x 
                         j 
                       
                     
                   
                 
                 ) 
               
               T 
             
           
         
       
     
     Every subsequent layer outputs a vector with components equal to linear combination of some sub-vector of x. 
     Finally, the T-NN&#39;s output is equal to: 
     
       
         
           
             y 
             = 
             
               
                 F 
                 ⁡ 
                 ( 
                 
                   
                     W 
                     m 
                   
                   ⁢ 
                   
                     W 
                     
                       m 
                       - 
                       1 
                     
                   
                   ⁢ 
                   … 
                   ⁢ 
                       
                   
                     W 
                     2 
                   
                   ⁢ 
                   
                     W 
                     1 
                   
                   ⁢ 
                   x 
                 
                 ) 
               
               = 
               
                 F 
                 ⁡ 
                 ( 
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     K 
                   
                   
                     
                       u 
                       j 
                     
                     ⁢ 
                     
                       x 
                       j 
                     
                   
                 
                 ) 
               
             
           
         
       
     
     This is the same value as the one calculated in SLP(K,l) for the same input vector x. So output values of SLP(K,l) and constructed T-NN are equal. 
     Single Layer Perceptron with Several Outputs 
     Suppose there is a single layer perceptron SLP(K, L) with K inputs and L output neurons, each neuron performing an activation function F. Suppose further U∈R L×K  is a weight matrix for SLP(K, L). The following algorithm Layer2TNN1 constructs a T-neural network from neurons TN(N, l). 
     Algorithm Layer2TNN1 
     1. For every output neuron i=1, . . . ,L
         a. Apply the algorithm Neuron2TNN1 to SLP i (K, 1) consisting on K inputs, l output neuron and weight vector U ij , j=1, 2, . . . ,K. A TNN i  is constructed as a result.       

     2. Construct PTNN by composing all TNN i  into one neural net:
         a. Concatenate input vectors of all TNN i , so the input of PTNN has L groups of K inputs, with each group being a copy of the SLP(K, L)&#39;s input layer.       

     Output of the PTNN is equal to the SLP(K, L)&#39;s output for the same input vector because output of every pair SLP i (K, l) and TNN i  are equal. 
     Multilayer Perceptron 
     Suppose a multilayer perceptron (MLP) includes K inputs, S layers and L i  calculation neurons in i-th layer, represented as MLP(K, S, L I , . . . L S ). Suppose U i ∈R L     i     ×L     i-1    is a weight matrix for the i-th layer. 
     The following is an example algorithm to construct a T-neural network from neurons TN(N, l), according to some implementations. 
     Algorithm MLP2TNN1 
     1. For every layer i=1, . . . ,S
         a. Apply the algorithm Layer2TNN1 to SLP i (L i-1 , L i ) consisting of L i-1  inputs, L i  output neurons, and a weight matrix U i , constructing PTNN i  as a result.       

     2. Construct MTNN by stacking all PTNN i  into one neural net; output of a TNN i-1  is set as input for TNN i . 
     Output of the MTNN is equal to the MLP(K, S, L l , . . . L S )&#39;s output for the same input vector because output of every pair SLP i (L i-1 , L i ) and PTNN i  are equal. 
     Example T-Transformations with Target Neurons with N I  Inputs and N O  Outputs 
     In some implementations, the example transformations described herein are performed by the neural network transformation module  226  that transform trained neural networks  220 , based on the mathematical formulations  230 , the basic function blocks  232 , the analog component models  234 , and/or the analog design constraints  236 , to obtain the transformed neural networks  228 . 
     Example Transformation of Single Layer Perceptron with Several Outputs 
     Suppose a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each neuron performing an activation function F. Suppose further U∈R L×K  is a weight matrix for SLP(K,L). The following algorithm constructs a T-neural network from neurons TN(N I , N O ), according to some implementations. 
     Algorithm Layer2TNNX 
     
         
         
           
             1. Construct a PTNN from SLP(K,L) by using the algorithm Layer2TNN1 (see description above). PTNN has an input layer consisting of L groups of K inputs. 
             2. Compose 
           
         
       
    
     
       
         
           
             ⌈ 
             
               L 
               
                 N 
                 o 
               
             
             ⌉ 
           
         
       
         
         
           
              subsets from L groups. Each subset contains no more than N O  groups of input vector copies. 
             3. Replace groups in every subset with one copy of input vector. 
             4. Construct PTNNX by rebuild connections in every subset by making N O  output connections from every input neuron. 
           
         
       
    
     According to some implementations, output of the PTNNX is calculated by means of the same formulas as for PTNN (described above), so the outputs are equal. 
       FIGS.  11 A- 11 C  show an application  1100  of the above algorithm for a single layer neural network (NN) with 2 output neurons and TN(N I , 2), according to some implementations.  FIG.  11 A  shows an example source or input NN, according to some implementations. K inputs are input to two neurons 1 and 2 belonging to a layer  1104 .  FIG.  11 B  shows a PTNN constructed after the first step of the algorithm, according to some implementations. The PTNN consists of two parts implementing subnets corresponding to the output neuron 1 and neuron 2 of the NN shown in  FIG.  11 A . In  FIG.  11 B , the input  1102  is replicated and input to two sets of input neurons  1106 - 2  and  1106 - 4 . Each set of input neurons is connected to a subsequent layer of neurons with two sets of neurons  1108 - 2  and  1108 - 4 , each set of neurons including mi neurons. The input layer is followed by identity transform blocks  1110 - 2  and  1110 - 4 , each block containing one or more layers with identity weight matrix. The output of the identity transform block  1110 - 2  is connected to the output neuron  1112  (corresponding to the output neuron 1 in  FIG.  11 A ), and the output of the identity transform block  1110 - 4  is connected to the output neuron  1114  (corresponding to the output neuron 1 in  FIG.  11 A ).  FIG.  11 C  shows application of the final steps of the algorithm, including replacing two copies of the input vector ( 1106 - 2  and  1106 - 4 ) with one vector  1116  (step 3), and rebuilding connections in the first layer  1118  by making two output links from every input neuron: one link connects to subnet related to output  1  and another link connects to subnet for the output  2 . 
     Example Transformation of Multilayer Perceptron 
     Suppose a multilayer perceptron (MLP) includes K inputs, S layers and Li calculation neurons in ith layer, represented as MLP(K, S, L l , . . . L S ). Suppose U i ∈R L     i     ×L     i-1    is a weight matrix for i-th layer. The following example algorithm constructs a T-neural network from neurons TN(N I , N O ), according to some implementations. 
     Algorithm MLP2TNNX 
     1. For every layer i=1, . . . ,S:
         a. Apply the algorithm Layer2TNNX to SLP i (L i-1 , L i ) consisting on L i-1  inputs, L i  output neuron and weight matrix U i . PTNNX i  is constructed as a result.       

     2. Construct MTNNX by stacking all PTNNX i  into one neural net:
         a. Output of a TNNX i-1  is set as input for TNNX.       

     According to some implementations. output of the MTNNX is equal to the MLP(K, S, L l , . . . L S )&#39;s output for the same input vector, because output of every pair SLP i (L i-1 , L i ) and PTNNX i  are equal. 
     Example Transformation of Recurrent Neural Network 
     A Recurrent Neural Network (RNN) contains backward connection allowing saving information.  FIG.  12    shows an example RNN  1200 , according to some implementations. The example shows a block  1204  performing an activation function A, that accepts an input X t    1206  and performs an activation function A, and outputs a value h t    1202 . The backward arrow from the block  1204  to itself indicates a backward connection, according to some implementations. An equivalent network is shown on the right up to the point in time when the activation block receives the input X t    1206 . At time 0, the network accepts input X t    1208  and performs the activation function A  1204 , and outputs a value h o    1210 ; at time 1, the network accepts input X 1    1212  and the output of the network at time 0, and performs the activation function A  1204 , and outputs a value h 1    1214 ; at time 2, the network accepts input X 2    1216  and the output of the network at time 1, and performs the activation function A  1204 , and outputs a value h 1    1218 . This process continues until time t, at which time the network accepts the input X t    1206  and the output of the network at time t−1, and performs the activation function A  1204 , and outputs the value h t    1202 , according to some implementations. 
     Data processing in an RNN is performed by means of the following formula: 
         h   t   =f ( W   (hh)   h   t-1   +W   (hx)   x   t ) 
     In the equation above, x t  is a current input vector, and h t-1  is the RNN&#39;s output for the previous input vector x t-1 . This expression consists of the several operations: calculation of linear combination for two fully connected layers W (hh) h t_1  and W (hx) x t , element-wise addition, and non-linear function calculation (f). The first and third operations can be implemented by trapezium-based network (one fully connected layer is implemented by pyramid-based network, a special case of trapezium networks). The second operation is a common operation that can be implemented in networks of any structure. 
     In some implementations, the RNN&#39;s layer without recurrent connections is transformed by means of Layer2TNNX algorithm described above. After transformation is completed, recurrent links are added between related neurons. Some implementations use delay blocks described below in reference to  FIG.  13 B . 
     Example Transformation of LSTM Network 
     A Long Short-Term Memory (LSTM) neural network is a special case of a RNN. A LSTM network&#39;s operations are represented by the following equations: 
         f   t =σ( W   f [ h   t-1   ,x   t ]+ b   f );
 
         i   t =σ( W   i [ h   t-1   ,x   t ]+ b   i );
 
         D   t =tanh( W   D [ h   t-1   ,x   t ]+ b   D ); 
         C   t =( f   t   ×C   t-1   +i   t   ×D   t ); 
         o   t =σ( W   o [ h   t-1   ,x   t ]+ b   o ); and
 
         h   t   =o   t ×tanh( C   t ).
 
     In the equations above, W f , W i , W D , and W O  are trainable weight matrices, b f , b i , b D , and b O  are trainable biases, x t  is a current input vector, h t-1  is an internal state of the LSTM calculated for the previous input vector x t-1 , and o t  is output for the current input vector. In the equations, the subscript t denotes a time instance t, and the subscript t−1 denotes a time instance t−1. 
       FIG.  13 A  is a block diagram of a LSTM neuron  1300 , according to some implementations. A sigmoid (σ) block  1318  processes the inputs h t-1    1330  and x t    1332 , and produces the output f t    1336 . A second sigmoid (σ) block  1320  processes the inputs h t-1    1330  and x t    1332 , and produces the output i t    1338 . A hyperbolic tangent (tanh) block  1322  processes the inputs h t-1    1330  and x t    1332 , and produces the output D t    1340 . A third sigmoid (σ) block  1328  processes the inputs h t-1    1330  and x t    1332 , and produces the output O t    1342 . A multiplier block  1304  processes f t    1336  and the output of a summing block  1306  (from a prior time instance) C t-1    1302  to produce an output that is in turn summed by the summing block  1306  along with the output of a second multiplier block  1314  that multiplies the outputs i t    1338  and D t    1340  to produce the output C t    1310 . The output C t    1310  is input to another tanh block  1312  that produces an output that is multiplied a third multiplier block  1316  with the output O t    1342  to produce the output h t    1334 . 
     There are several types of operations utilized in these expressions: (i) calculation of linear combination for several fully connected layers, (ii) elementwise addition, (iii) Hadamard product, and (iv) non-linear function calculation (e.g., sigmoid (σ) and hyperbolic tangent (tanh)). Some implementations implement the (i) and (iv) operations by a trapezium-based network (one fully connected layer is implemented by a pyramid-based network, a special case of trapezium networks). Some implementations use networks of various structures for the (ii) and (iii) operations which are common operations. 
     The layer in an LSTM layer without recurrent connections is transformed by using the Layer2TNNX algorithm described above, according to some implementations. After transformation is completed, recurrent links are added between related neurons, according to some implementations. 
       FIG.  13 B  shows delay blocks, according to some implementations. As described above, some of the expressions in the equations for the LSTM operations depend on saving, restoring, and/or recalling an output from a previous time instance. For example, the multiplier block  1304  processes the output of the summing block  1306  (from a prior time instance) C t-1    1302 .  FIG.  13 B  shows two examples of delay blocks, according to some implementations. The example  1350  includes a delay block  1354  on the left accepts input x t    1352  at time t, and outputs the input after a delay of dt indicated by the output x t-dt    1356 . The example  1360  on the right shows cascaded (or multiple) delay blocks  1364  and  1366  outputs the input x t    1362  after 2 units of time delays, indicated by the output x t-2dt    1368 , according to some implementations. 
       FIG.  13 C  is a neuron schema for a LSTM neuron, according to some implementations. The schema includes weighted summator nodes (sometimes called adder blocks)  1372 ,  1374 ,  1376 ,  1378 , and  1396 , multiplier blocks  1384 ,  1392 , and  1394 , and delay blocks  1380  and  1382 . The input x t    1332  is connected to the adder blocks  1372 ,  1374 ,  1376 , and  1378 . The output h t-1    1330  for a prior input x t-1  is also input to the adder blocks  1372 ,  1374 ,  1376 , and  1378 . The adder block  1372  produces an output that is input to a sigmoid block  1394 - 2  that produces the output f t    1336 . Similarly, the adder block  1374  produces an output that is input to the sigmoid block  1386  that produces the output i t    1338 . Similarly, the adder block  1376  produces an output that is input to a hyperbolic tangent block  1388  that produces the output D t    1340 . Similarly, the adder block  1378  produces an output that is input to the sigmoid block  1390  that produces the output O t    1342 . The multiplier block  1392  uses the outputs i t    1338 , f t    1336 , and output of the adder block  1396  from a prior time instance C t-1    1302  to produce a first output. The multiplier block  1394  uses the outputs i t    1338  and D t    1340  to produce a second output. The adder block  1396  sums the first output and second output to produce the output C t    1310 . The output C t    1310  is input to a hyperbolic tangent block  1398  that produces an output that is input, along with the output of the sigmoid block  1390 , O t    1342 , to the multiplier block  1384  to produce the output h t    1334 . The delay block  1382  is used to recall (e.g., save and restore) the output of the adder block  1396  from a prior time instance. Similarly, the delay block  1380  is used to recall or save and restore the output of the multiplier block  1384  for a prior input x t-1  (e.g., from a prior time instance). Examples of delay blocks are described above in reference to  FIG.  13 B , according to some implementations. 
     Example Transformation of GRU Networks 
     A Gated Recurrent Unit) (GRU) neural network is a special case of RNN. A RNN&#39;s operations are represented by the following expressions: 
         z   t =σ( W   z   x   t   +U   z   h   t-1 );
 
         r   t =σ( W   r   x   t   +U   r   h   t-1 );
 
         j   t =tanh( Wx   t   +r   t   ×Uh   t-1 ); 
         h   t   =z   t   ×h   t-1 +(1− z   t )× j   t ).
 
     In the equations above, x t  is a current input vector, and h t-1  is an output calculated for the previous input vector x t-1 . 
       FIG.  14 A  is a block diagram of a GRU neuron, according to some implementations. A sigmoid (σ) block  1418  processes the inputs h t-1    1402  and x t    1422 , and produces the output r t    1426 . A second sigmoid (σ) block  1420  processes the inputs h t-1    1402  and x t    1422 , and produces the output z t    1428 . A multiplier block  1412  multiplies the output r t    1426  and the input h t-1    1402  to produce and output that is input (along with the input x t    1422 ) to a hyperbolic tangent (tanh) block  1424  to produce the output j t    1430 . A second multiplier block  1414  multiplies the output j t    1430  and the output z t    1428  to produce a first output. The block  1410  computes 1−the output z t    1428  to produce an output that is input to a third multiplier block  1404  that multiplies the output and the input h t-1    1402  to produce a product that is input to an adder block  1406  along with the first output (from the multiplier block  1414 ) to produce the output h t    1408 . The input h t-1    1402  is the output of the GRU neuron from a prior time interval output t−1. 
       FIG.  14 B  is a neuron schema for a GRU neuron  1440 , according to some implementations. The schema includes weighted summator nodes (sometimes called adder blocks)  1404 ,  1406 ,  1410 ,  1406 , and  1434 , multiplier blocks  1404 ,  1412 , and  1414 , and delay block  1432 . The input x t    1422  is connected to the adder blocks  1404 ,  1410 , and  1406 . The output h t-1    1402  for a prior input x t-1  is also input to the adder blocks  1404  and  1406 , and the multiplier blocks  1404  and  1412 . The adder block  1404  produces an output that is input to a sigmoid block  1418  that produces the output Z t    1428 . Similarly, the adder block  1406  produces an output that is input to the sigmoid block  1420  that produces the output r t    1426  that is input to the multiplier block  1412 . The output of the multiplier block  1412  is input to the adder block  1410  whose output is input to a hyperbolic tangent block  1424  that produces an output  1430 . The output  1430  as well as the output of the sigmoid block  1418  are input to the multiplier block  1414 . The output of the sigmoid block  1418  is input to the multiplier block  1404  that multiplies that output with the input from the delay block  1432  to produce a first output. The multiplier block produces a second output. The adder block  1434  sums the first output and the second output to produce the output h t    1408 . The delay block  1432  is used to recall (e.g., save and restore) the output of the adder block  1434  from a prior time instance. Examples of delay blocks are described above in reference to  FIG.  13 B , according to some implementations. 
     Operation types used in GRU are the same as the operation types for LSTM networks (described above), so GRU is transformed to trapezium-based networks following the principles described above for LSTM (e.g., using the Layer2TNNX algorithm), according to some implementations. 
     Example Transformation of Convolutional Neural Network 
     In general, Convolutional Neural Networks (CNN) include several basic operations, such as convolution (a set of linear combinations of image&#39;s (or internal map&#39;s) fragments with a kernel), activation function, and pooling (e.g., max, mean, etc.). Every calculation neuron in a CNN follows the general processing scheme of a neuron in an MLP: linear combination of some inputs with subsequent calculation of activation function. So a CNN is transformed using the MLP2TNNX algorithm described above for multilayer perceptrons, according to some implementations. 
     Conv1D is a convolution performed over time coordinate.  FIGS.  15 A and  15 B  are neuron schema of variants of a single Conv1D filter, according to some implementations. In  FIG.  15 A , a weighted summator node  1502  (sometimes called adder block, marked ‘+’) has 5 inputs, so it corresponds to 1Dconvolution with a kernel of 5. The inputs are x t    1504  from time t, x t-1    1514  from time t−1 (obtained by inputting the input to a delay block  1506 ), x t-2    1516  from time t−2 (obtained by inputting the output of the delay block  1506  to another delay block  1508 ), x t-3    1518  from time t−3 (obtained by inputting the output of the delay block  1508  to another delay block  1510 ), and x t-4    1520  from time t−4 (obtained by inputting the output of the delay block  1510  to another delay block  1512 . For large kernels, it is sometimes beneficial to utilize different frequency delay blocks, so that some of the blocks produce bigger delays. Some implementations substitute several small delay blocks for one large delay block, as shown in  FIG.  15 B . In addition to the delay blocks in  FIG.  15 A , the example uses a delay_3 block  1524  that produces x t-3    1518  from time t−3, and another delay block  1526  that produces the x t-5    1522  from time t−5. The delay_3  1524  block is an example of multiple delay blocks, according to some implementations. This operation does not decrease total number of blocks, but it may decrease total number of consequent operations performed over the input signal and reduce accumulation of errors, according to some implementations. 
     In some implementations, convolutional layers are represented by trapezia-like neurons and fully connected layer is represented by cross-bar of resistors. Some implementations use cross-bars, and calculate resistance matrix for the cross-bars. 
     Example Approximation Algorithm for Single Layer Perceptron with Multiple Outputs 
     In some implementations, the example transformations described herein are performed by the neural network transformation module  226  that transform trained neural networks  220 , and/or the analog neural network optimization module  246 , based on the mathematical formulations  230 , the basic function blocks  232 , the analog component models  234 , and/or the analog design constraints  236 , to obtain the transformed neural networks  228 . 
     Suppose a single layer perceptron SLP(K, L) includes K inputs and L output neurons, each output neuron performing an activation function F. Suppose further that U∈R L×K  is a weight matrix for SLP(K, L). The following is an example for constructing a T-neural network from neurons TN(N I , N O ) using an approximation algorithm Layer2TNNX_Approx, according to some implementations. The algorithm applies Layer2TNN1 algorithm (described above) at the first stage in order to decrease a number of neurons and connections, and subsequently applies Layer2TNNX to process the input of the decreased size. The outputs of the resulted neural net are calculated using shared weights of the layers constructed by the Layer2TNN1 algorithm. The number of these layers is determined by the value p, a parameter of the algorithm. If p is equal to 0 then Layer2TNNX algorithm is applied only and the transformation is equivalent. If p&gt;0, then p layers have shared weights and the transformation is approximate. 
     Algorithm Layer2TNNX_Approx 
     
         
         
           
             1. Set the parameter p with a value from the set {0,1, . . . , ┌log N     I   K┐−1}. 
             2. If p&gt;0 apply the algorithm Layer2TNN1 with neuron TN(N I , l) to the net SLP(K, L) and construct first p layers of the resulted subnet (PNN). The net PNN has 
           
         
       
    
     
       
         
           
             
               N 
               p 
             
             = 
             
               ⌈ 
               
                 K 
                 
                   N 
                   I 
                   p 
                 
               
               ⌉ 
             
           
         
       
         
         
           
              neurons in the output layer. 
             3. Apply the algorithm Layer2TNNX with a neuron TN(N I , N O ) and construct a neural subnet TNN with N p  inputs and L outputs. 
             4. Set the weights of the PNN net. The weights of every neuron i of the first layer of the PNN are set according to the rule w k     i     (1) =C. Here, C is any constant not equal to zero, 
           
         
       
    
     
       
         
           
             
               
                 k 
                 i 
               
               = 
               
                 
                   
                     ( 
                     
                       i 
                       - 
                       1 
                     
                     ) 
                   
                   ⁢ 
                   
                     N 
                     I 
                   
                 
                 + 
                 1 
               
             
             , 
             
               
                 and 
                 ⁢ 
                 
                     
                      
                 
                 ⁢ 
                 
                   w 
                   ij 
                   
                     ( 
                     1 
                     ) 
                   
                 
               
               = 
               
                 
                   1 
                   L 
                 
                 ⁢ 
                 
                   
                     ∑ 
                     
                       l 
                       = 
                       1 
                     
                     L 
                   
                   
                     
                       
                         U 
                         lj 
                       
                       
                         U 
                         
                           lk 
                           i 
                         
                       
                     
                     ⁢ 
                     
                       C 
                       , 
                     
                   
                 
               
             
           
         
       
         
         
           
             for all weights j of this neuron except k i . All other weights of the PNN net are set to 1. w ik     i     (1)  represents a weight for the first layer (as denoted by the superscript (1)) for the connection between the neuron i and the neuron k i  in the first layer. 
             5. Set the weights of the TNN subnet. The weights of every neuron i of the first layer of the TNN (considering the whole net this is (p+1)th layer) are set according to the equation 
           
         
       
    
     
       
         
           
             
               w 
               
                 ik 
                 i 
               
               
                 ( 
                 
                   p 
                   + 
                   1 
                 
                 ) 
               
             
             = 
             
               
                 
                   U 
                   
                     ik 
                     i 
                   
                 
                 C 
               
               . 
             
           
         
       
         
         
           
              All other weights of the TNN are set to 1. 
             6. Set activation functions for all neurons of the last layer of the TNN subnet as F. Activation functions of all other neurons are identity. 
           
         
       
    
       FIG.  16    shows an example architecture  1600  of the resulting neural net, according to some implementations. The example includes a PNN  1602  connected to a TNN  1606 . The PNN  1602  includes a layer for K inputs and produce N p  outputs, that is connected as input  1612  to the TNN  1606 . The TNN  1606  generates L outputs  1610 , according to some implementations. 
     Approximation Algorithm for Multilayer Perceptron with Several Outputs 
     Suppose a multilayer perceptron (MLP) includes K inputs, S layers and Li calculation neurons in i-th layer, represented as MLP(K, S, L l , . . . L S ). Suppose further U i ∈R L     i     ×L     i-1    is a weight matrix for the i-th layer. The following example algorithm constructs a T-neural network from neurons TN(N I , N O ), according to some implementations. 
     Algorithm MLP2TNNX_Approx 
     
         
         
           
             1. For every layer i=1, . . . ,S:
           a. Apply the algorithm Layer2TNNX_Approx (described above) to SLP i (L i-1 , L i ) consisting of L i-1  inputs, L i  output neuron, and weight matrix U i . If i=1, then L 0 =K. Suppose this step constructs PTNNX i  as a result.   
         
             2. Construct a MTNNX (a multilayer perceptron) by stacking all PTNNX i  into one neural net, where output of a TNNX i-1  is set as input for TNNX i . 
           
         
       
    
     Example Methods of Compression of Transformed Neural Networks 
     In some implementations, the example transformations described herein are performed by the neural network transformation module  226  that transform trained neural networks  220 , and/or the analog neural network optimization module  246 , based on the mathematical formulations  230 , the basic function blocks  232 , the analog component models  234 , and/or the analog design constraints  236 , to obtain the transformed neural networks  228 . 
     This section describes example methods of compression of transformed neural networks, according to some implementations. Some implementations compress analog pyramid-like neural networks in order to minimize the number of operational amplifiers and resistors, necessary to realize the analog network on chip. In some implementations, the method of compression of analog neural networks is pruning, similar to pruning in software neural networks. There is nevertheless some peculiarities in compression of pyramid-like analog networks, which are realizable as IC analog chip in hardware. Since the number of elements, such as operational amplifiers and resistors, define the weights in analog based neural networks, it is crucial to minimize the number of operational amplifiers and resistors to be placed on chip. This will also help minimize the power consumption of the chip. Modern neural networks, such as convolutional neural networks, can be compressed 5-200 times without significant loss of the accuracy of the networks. Often, whole blocks in modern neural networks can be pruned without significant loss of accuracy. The transformation of dense neural networks into sparsely connected pyramid or trapezia or cross-bar like neural networks presents opportunities to prune the sparsely connected pyramid or trapezia-like analog networks, which are then represented by operational amplifiers and resistors in analog IC chips. In some implementations, such techniques are applied in addition to conventional neural network compression techniques. In some implementations, the compression techniques are applied based on the specific architecture of the input neural network and/or the transformed neural networks (e.g., pyramids versus trapezia versus cross-bars). 
     For example, since the networks are realized by means of analog elements, such as operational amplifiers, some implementations determine the current which flows through the operational amplifier when the standard training dataset is presented, and thereby determine if a knot (an operational amplifier) is needed for the whole chip or not. Some implementations analyze the SPICE model of the chip and determine the knots and connections, where no current is flowing and no power is consumed. Some implementations determine the current flow through the analog IC network and thus determine the knots and connections, which are then pruned. Besides, some implementations also remove the connections if the weight of connection is too high, and/or substitute resistor to direct connector if the weight of connection is too low. Some implementations prune the knot if all connections leading to this knot have weights that are lower than a predetermined threshold (e.g., close to 0), deleting the connections where an operational amplifier always provides zero at output, and/or changing an operational amplifier to a linear junction if the amplifier gives linear function without amplification. 
     Some implementations apply compression techniques specific to pyramid, trapezia, or cross-bar types of neural networks. Some implementations generate pyramids or trapezia with larger amount of inputs (than without the compression), thus minimizing the number of layers in pyramid or trapezia. Some implementations generate a more compact trapezia network by maximizing the number of outputs of each neuron. 
     Example Generation of Optimal Resistor Set 
     In some implementations, the example computations described herein are performed by the weight matrix computation or weight quantization module  238  (e.g., using the resistance calculation module  240 ) that compute the weights  272  for connections of the transformed neural networks, and/or corresponding resistance values  242  for the weights  272 . 
     This section describes an example of generating an optimal resistor set for a trained neural network, according to some implementations. An example method is provided for converting connection weights to resistor nominals for implementing the neural network (sometimes called a NN model) on a microchip with possibly less resistor nominals and possibly higher allowed resistor variance. 
     Suppose a test set ‘Test’ includes around 10,000 values of input vector (x and y coordinates) with both coordinates varying in the range [0; 1], with a step of 0.01. Suppose network NN output for given input X is given by Out=NN(X). Suppose further that input value class is found as follows: Class_nn(X)=NN(X)&gt;0.61 ? 1: 0. 
     The following compares a mathematical network model M with a schematic network model S. The schematic network model includes possible resistor variance of rv and processes the ‘Test’ set, each time producing a different vector of output values S(Test)=Out_s. Output error is defined by the following equation: 
     
       
         
           
             
               Err 
               out 
             
             = 
             
               Mean 
               ⁢ 
               
                   
                    
               
               ( 
               
                 
                   ∑ 
                   
                     i 
                     = 
                     1 
                   
                   N 
                 
                 
                   
                     
                       ❘ 
                       &#34;\[LeftBracketingBar]&#34; 
                     
                     
                       
                         S 
                         ⁡ 
                         ( 
                         
                           X 
                           i 
                         
                         ) 
                       
                       - 
                       
                         M 
                         ⁡ 
                         ( 
                         
                           X 
                           i 
                         
                         ) 
                       
                     
                     
                       ❘ 
                       &#34;\[RightBracketingBar]&#34; 
                     
                   
                   N 
                 
               
               ) 
             
           
         
       
     
     Classification error is defined by the following equation: 
     
       
         
           
             
               Err 
               class 
             
             = 
             
               Mean 
               ⁢ 
                   
               
                 ( 
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     N 
                   
                   
                     
                       
                         Class_s 
                         ⁢ 
                         
                           ( 
                           
                             X 
                             i 
                           
                           ) 
                         
                       
                       ≠ 
                       
                         Class_m 
                         ⁢ 
                         
                           ( 
                           
                             X 
                             i 
                           
                           ) 
                         
                       
                     
                     N 
                   
                 
                 ) 
               
             
           
         
       
     
     Some implementations set the desired classification error as no more than 1%. 
     Example Error Analysis 
       FIG.  17 A  shows an example chart  1700  illustrating dependency between output error and classification error on the M network, according to some implementations. In  FIG.  17 A , the x-axis corresponds to classification margin  1704 , and the y-axis corresponds to total error  1702  (see description above). The graph shows total error (difference between output of model M and real data) for different classification margins of output signal. For this example, according to the chart, the optimal classification margin  1706  is 0.610. 
     Suppose another network O produces output values with a constant shift versus relevant M output values, there would be classification error between O and M. To keep the classification error below 1%, this shift should be in the range of [−0.045, 0.040]. Thus, possible output error for S is 45 mV. 
     Possible weight error is determined by analyzing dependency between weight/bias relative error over the whole network and output error. The charts  1710  and  1720  shown in  FIGS.  17 B and  17 C , respectively, are obtained by averaging 20 randomly modified networks over the ‘Test’ set, according to some implementations. In these charts, x-axis represents the absolute weight error  1712  and y-axis represents the absolute output error  1714 . As can be seen from the charts, output error limit of 45 mV (y=0.045) allows for 0.01 relative or 0.01 absolute error value (value of x) for each weight. Maximum weight modulus (maximum of absolute value of weights among all weights) for the neural network is 1.94. 
     Example Process for Choosing Resistor Set 
     A resistor set together with a {R+, R−} pair chosen from this set has a value function over the required weight range [−wlim; wlim] with some degree of resistor error r_err. In some implementations, value function of a resistor set is calculated as follows:
         Possible weight options array is calculated together with weight average error dependent on resistor error;   The weight options in the array is limited to the required weight range [−wlim; wlim];   Values that are worse than neighboring values in terms of weight error are removed;   An array of distances between neighboring values is calculated; and   The value function is a composition of square mean or maximum of the distances array.       

     Some implementations iteratively search for an optimal resistor set by consecutively adjusting each resistor value in the resistor set on a learning rate value. In some implementations, the learning rate changes over time. In some implementations, an initial resistor set is chosen as uniform (e.g., [1; 1; . . . ; 1]), with minimum and maximum resistor values chosen to be within two orders of magnitude range (e.g., [1; 100] or [0.1; 10]). Some implementation choose R+=R−. In some implementations, the iterative process converges to a local minimum. In one case, the process resulted in the following set: [0.17, 1.036, 0.238, 0.21, 0.362, 1.473, 0.858, 0.69, 5.138, 1.215, 2.083, 0.275]. This is a locally optimal resistor set of 12 resistors for the weight range [−2; 2] with rmin=0.1 (minimum resistance), rmax=10 (maximum resistance), and r_err=0.001 (an estimated error in the resistance). Some implementations do not use the whole available range [rmin; rmax] for finding a good local optimum. Only part of the available range (e.g., in this case [0.17; 5.13]) is used. The resistor set values are relative, not absolute. Is this case, relative value range of 30 is enough for the resistor set. 
     In one instance, the following resistor set of length 20 is obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02]. In this example, the value 1.763 is also the R−=R+ value. This set is subsequently used to produce weights for NN, producing corresponding model S. The model S&#39;s mean square output error was 11 mV given the relative resistor error is close to zero, so the set of 20 resistors is more than required. Maximum error over a set of input data was calculated to be 33 mV. In one instance, S, DAC, and ADC converters with 256 levels were analyzed as a separate model, and the result showed 14 mV mean square output error and 49 mV max output error. An output error of 45 mV on NN corresponds to a relative recognition error of 1%. The 45 mV output error value also corresponds to 0.01 relative or 0.01 absolute weight error, which is acceptable. Maximum weight modulus in NN is 1.94. In this way, the optimal (or near optimal) resistor set is determined using the iterative process, based on desired weight range [−wlim; wlim], resistors error (relative), and possible resistors range. 
     Typically, a very broad resistor set is not very beneficial (e.g., between 1-⅕ orders of magnitude is enough) unless different precision is required within different layers or weight spectrum parts. For example, suppose weights are in the range of [0, 1], but most of the weights are in the range of [0, 0.001], then better precision is needed within that range. In the example described above, given the relative resistor error is close to zero, the set of 20 resistors is more than sufficient for quantizing the NN network, with given precision. In one instance, on a set of resistors [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] (note values are relative), an average S output error of 11 mV was obtained. 
     Example Process for Quantization of Resistor Values 
     In some implementations, the example computations described herein are performed by the weight matrix computation or weight quantization module  238  (e.g., using the resistance calculation module  240 ) that compute the weights  272  for connections of the transformed neural networks, and/or corresponding resistance values  242  for the weights  272 . 
     This section describes an example process for quantizing resistor values corresponding to weights of a trained neural network, according to some implementations. The example process substantially simplifies the process of manufacturing chips using analog hardware components for realizing neural networks. As described above, some implementations use resistors to represent neural network weights and/or biases for operational amplifiers that represent analog neurons. The example process described here specifically reduces the complexity in lithographically fabricating sets of resistors for the chip. With the procedure of quantizing the resistor values, only select values of resistances are needed for chip manufacture. In this way, the example process simplifies the overall process of chip manufacture and enables automatic resistor lithographic mask manufacturing on demand. 
       FIG.  18    provides an example scheme of a neuron model  1800  used for resistors quantization, according to some implementations. In some implementations, the circuit is based on an operational amplifier  1824  (e.g., AD824 series precision amplifier) that receives input signals from negative weight fixing resistors (R1−  1804 , R2−  1806 , Rb− bias  1816 , Rn−  1818 , and R−  1812 ), and positive weight fixing resistors (R1+  1808 , R2+  1810 , Rb+ bias  1820 , Rn+  1822 ), and R+  1814 ). The positive weight voltages are fed into direct input of the operational amplifier  1824  and negative weights voltages are fed into inverse input of the operational amplifier  1824 . The operational amplifier  1824  is used to allow weighted summation operation of weighted outputs from each resistor, where negative weights are subtracted from positive weights. The operational amplifier  1824  also amplifies signal to the extent necessary for the circuit operation. In some implementations, the operational amplifier  1824  also accomplishes RELU transformation of output signal at it&#39;s output cascade. 
     The following equations determine the weights, based on resistor values: 
     Voltage at the output of neuron is determined by the following equation: 
     
       
         
           
             
               U 
               out 
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   1 
                 
                 N 
               
               
                 
                   ( 
                   
                     
                       
                         R 
                         + 
                       
                       
                         R 
                         i 
                         + 
                       
                     
                     - 
                     
                       
                         R 
                         - 
                       
                       
                         R 
                         i 
                         - 
                       
                     
                   
                   ) 
                 
                 ⁢ 
                 
                   U 
                   i 
                 
               
             
           
         
       
     
     The weights of each connection are determined by following equation: 
     
       
         
           
             
               w 
               i 
             
             = 
             
               
                 
                   R 
                   + 
                 
                 
                   R 
                   i 
                   + 
                 
               
               - 
               
                 
                   R 
                     
                   - 
                 
                 
                   R 
                   i 
                   - 
                 
               
             
           
         
       
     
     The following example optimization procedure quantizes the values of each resistance and minimize the error of neural network output, according to some implementations:
         1. Obtain a set of connection weights and biases {wl, . . . , wn, b}.   2. Obtain possible minimum and maximum resistor values {Rmin, Rmax}. These parameters are determined based on the technology used for manufacturing. Some implementations use TaN or Tellurium high resistivity materials. In some implementations, the minimum value of resistor is determined by minimum square that can be formed lithographically. The maximum value is determined by length, allowable for resistors (e.g., resistors made from TaN or Tellurium) to fit to the desired area, which is in turn determined by the area of an operational amplifier square on lithographic mask. In some implementations, the area of arrays of resistors is smaller than the area of one operational amplifier, since the arrays of resistors are stacked (e.g., one in BEOL, another in FEOL).   3. Assume that each resistor has r_err relative tolerance value   4. The goal is to select a set of resistor values {R1, . . . , Rn} of given length N within the defined [Rmin; Rmax], based on {w1, . . . , wn, b} values. An example search algorithm is provided below to find sub-optimal {R1, . . . , Rn} set based on particular optimality criteria.   5. Another algorithm chooses {Rn, Rp, Rni, Rpi} for a network given that {R1 . . . Rn} is determined.       

     Example {R1, . . . , Rn} Search Algorithm 
     Some implementations use an iterative approach for resistor set search. Some implementations select an initial (random or uniform) set {R1, . . . , Rn} within the defined range. Some implementations select one of the elements of the resistor set as a R−=R+ value. Some implementations alter each resistor within the set by a current learning rate value until such alterations produce ‘better’ set (according to a value function). This process is repeated for all resistors within the set and with several different learning rate values, until no further improvement is possible. 
     Some implementations define the value function of a resistor set as follows:
         Possible weight options are calculated according to the formula (described above):       

     
       
         
           
             
               w 
               i 
             
             = 
             
               
                 
                   R 
                   + 
                 
                 
                   R 
                   i 
                   + 
                 
               
               - 
               
                 
                   R 
                     
                   - 
                 
                 
                   R 
                   i 
                   - 
                 
               
             
           
         
       
         
         
           
             Expected error value for each weight option is estimated based on potential resistor relative error r_err determined by IC manufacturing technology. 
             Weight options list is limited or restricted to [−wlim; wlim] range Some values, which have expected error beyond a high threshold (e.g., 10 times r_err), are removed 
             Value function is calculated as a square mean of distance between two neighboring weight options. So, value function is minimal when weight options are distributed uniformly within [−wlim; wlim] range 
           
         
       
    
     Suppose the required weight range [−wlim; wlim] for a model is set to [−5; 5], and the other parameters include N=20, r_err=0.1%, rmin=100 KΩ, rmax=5 Ma Here, rmin and rmax are minimum and maximum values for resistances, respectively. 
     In one instance, the following resistor set of length 20 was obtained for abovementioned parameters: [0.300, 0.461, 0.519, 0.566, 0.648, 0.655, 0.689, 0.996, 1.006, 1.048, 1.186, 1.222, 1.261, 1.435, 1.488, 1.524, 1.584, 1.763, 1.896, 2.02] MΩ. R−=R+=1.763 Ma 
     Example {Rn, Rp, Rni, Rpi} Search Algorithm 
     Some implementations determine Rn and Rp using an iterative algorithm such as the algorithm described above. Some implementations set Rp=Rn (the tasks to determine Rn and Rp are symmetrical—the two quantities typically converge to a similar value). Then for each weight w i , some implementations select a pair of resistances {Rni, Rpi} that minimizes the estimated weight error value: 
     
       
         
           
             
               w 
               err 
             
             = 
             
               
                 
                   ( 
                   
                     
                       
                         R 
                         + 
                       
                       
                         R 
                         i 
                         + 
                       
                     
                     - 
                     
                       
                         R 
                           
                         - 
                       
                       
                         R 
                         i 
                         - 
                       
                     
                   
                   ) 
                 
                 · 
                 
                   r 
                   err 
                 
               
               + 
               
                 
                   ❘ 
                   &#34;\[LeftBracketingBar]&#34; 
                 
                 
                   
                     w 
                     i 
                   
                   - 
                   
                     
                       R 
                       + 
                     
                     
                       R 
                       i 
                       + 
                     
                   
                   - 
                   
                     
                       R 
                         
                       - 
                     
                     
                       R 
                       i 
                       - 
                     
                   
                 
                 
                   ❘ 
                   &#34;\[RightBracketingBar]&#34; 
                 
               
             
           
         
       
     
     Some implementations subsequently use the {Rni; Rpi; Rn; Rp} values set to implement neural network schematics. In one instance, the schematics produced mean square output error (sometimes called S mean square output error, described above) of 11 mV and max error of 33 mV over a set of 10,000 uniformly distributed input data samples, according to some implementations. In one instance, S model was analyzed along with digital-to-analog converters (DAC), analog-to-digital converters (ADC), with 256 levels as a separate model. The model produced 14 mV mean square output error and 49 mV max output error on the same data set, according to some implementations. DAC and ADC have levels because they convert analog value to bit value and vice-versa. 8 bits of digital value is equal to 256 levels. Precision cannot be better than 1/256 for 8-bit ADC. 
     Some implementations calculate the resistance values for analog IC chips, when the weights of connections are known, based on Kirchhoff&#39;s circuit laws and basic principles of operational amplifiers (described below in reference to  FIG.  19 A ), using Mathcad or any other similar software. In some implementations, operational amplifiers are used both for amplification of signal and for transformation according to the activation functions (e.g., ReLU, sigmoid, Tangent hyperbolic, or linear mathematical equations), 
     Some implementations manufacture resistors in a lithography layer where resistors are formed as cylindrical holes in the SiO2 matrix and the resistance value is set by the diameter of hole. Some implementations use amorphous TaN, TiN of CrN or Tellurium as the highly resistive material to make high density resistor arrays. Some ratios of Ta to N Ti to N and Cr to N provide high resistance for making ultra-dense high resistivity elements arrays. For example, for TaN, Ta5N6, Ta3N5, the higher the N ratio to Ta, the higher is the resistivity. Some implementations use Ti2N, TiN, CrN, or Cr5N, and determine the ratios accordingly. TaN deposition is a standard procedure used in chip manufacturing and is available at all major Foundries. 
     Example Operational Amplifier 
       FIG.  19 A  shows a schematic diagram of an operational amplifier made on CMOS (CMOS OpAmp)  1900 , according to some implementations. In  FIG.  19 A , In+ (positive input or pos)  1404 , and In− (negative input or neg)  1406 , and Vdd− (positive supply voltage relative to GND)  1402  are contact inputs. Contact Vss− (negative supply voltage or GND) is indicated by the label  1408 . The circuit output is Out  1410  (contact output). Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel) to W (the width of the gate channel), examples of which are shown in the Table shown in  FIG.  19 B  (described below). The current mirror is made on NMOS transistors M11  1944 , M12  1946 , and resistor R1  1921  (with an example resistance value of 12 kΩ), and provides the offset current of the differential pair (M1  1926  and M3  1930 ). The differential amplifier stage (differential pair) is made on the NMOS transistors M1  1926  and M3  1930 . Transistors M1, M3 are amplifying, and PMOS transistors M2  1928  and M4  1932  play the role of active current load. From the M3 transistor, the signal is input to the gate of the output PMOS transistor M7  1936 . From the transistor M1, the signal is input to the PMOS transistor M5 (inverter)  1934  and the active load on the NMOS transistor M6  1934 . The current flowing through the transistor M5  1934  is the setting for the NMOS transistor M8  1938 . Transistors M7  1936  is included in the scheme with a common source for a positive half-wave signal. The M8 transistors  1938  are enabled by a common source circuit for a negative half-wave signal. To increase the overall load capacity of the operational amplifier, the M7  1936  and M8  1938  outputs include an inverter on the M9  1940  and M10  1942  transistors. Capacitors C1  1912  and C2  1914  are blocking. 
       FIG.  19 B  shows a table  1948  of description for the example circuit shown in  FIG.  19 A , according to some implementations. The values for the parameters are provided as examples, and various other configurations are possible. The transistors M1, M3, M6, M8, M10, M11, and M12 are N-Channel MOSFET transistors with explicit substrate connection. The other transistors M2, M4, M5, M7, and M9 are P-Channel MOSFET transistors with explicit substrate connection. The Table shows example shutter ratio of length (L, column 1) and width (W, column 2) are provided for each of the transistors (column 3). 
     In some implementations, operational amplifiers such as the example described above are used as the basic element of integrated circuits for hardware realization of neural networks. In some implementations, the operational amplifiers are of the size of 40 square microns and fabricated according to 45 nm node standard. 
     In some implementations, activation functions, such as ReLU, Hyperbolic Tangent, and Sigmoid functions are represented by operational amplifiers with modified output cascade. For example, RELU, Sigmoid, or Tangent function is realized as an output cascade of an operational amplifier (sometimes called OpAmp) using corresponding well-known analog schematics, according to some implementations. 
     In the examples described above and below, in some implementations, the operational amplifiers are substituted by inverters, current mirrors, two-quadrant or four quadrant multipliers, and/or other analog functional blocks, that allow weighted summation operation. 
     Example Scheme of a LSTM Block 
       FIGS.  20 A- 20 E  show a schematic diagram of a LSTM neuron  20000 , according to some implementations. The inputs of the neuron are Vin1  20002  and Vin2  20004  that are values in the range [−0.1,0.1]. The LSTM neuron also input the value of the result of calculating the neuron at time H(t−1) (previous value; see description above for LST neuron)  20006  and the state vector of the neuron at time C(t−1) (previous value)  20008 . Outputs of the neuron LSTM (shown in  FIG.  20 B ) include the result of calculating the neuron at the present time H(t)  20118  and the state vector of the neuron at the present time C(t)  20120 . The scheme includes:
         a “neuron O” assembled on the operational amplifiers U1  20094  and U2  20100 , shown in  FIG.  20 A . Resistors R_Wo1  20018 , R_Wo2  20016 , R_Wo3  20012 , R_Wo4  20010 , R_Uop1  20014 , R_Uom1  20020 , Rr  20068  and Rf2  20066  set the weights of connections of the single “neuron O”. The “neuron O” uses a sigmoid (module X1  20078 ,  FIG.  20 B ) as a nonlinear function;   a “neuron C” assembled on the operational amplifiers U3  20098  (shown in  FIG.  20 C ) and U4  20100  (shown in  FIG.  20 A ). Resistors R_Wc1  20030 , R_Wc2  20028 , R_Wc3  20024 , R_Wc4  20022 , R_Ucp1  20026 , R_Ucm1  20032 , Rr  20122 , and Rf2  20120 , set the weights of connections of the “neuron C”. The “neuron C” uses a hyperbolic tangent (module X2  22080 ,  FIG.  2 B ) as a nonlinear function;   a “neuron I” assembled on the operational amplifiers U5  20102  and U6  20104 , shown in  FIG.  20 C . Resistors R_Wi1  20042 , R_Wi2  20040 , R_Wi3  20036 , and R_Wi4  20034 , R_Uip1  20038 , R_Uim1  20044 , Rr  20124 , and Rf2  20126  set the weights of connections of the “neuron I”. The “neuron I” uses a sigmoid (module X3  20082 ) as a nonlinear function; and   a “neuron f” assembled on the operational amplifiers U7  20106  and U8  20108 , as shown in  FIG.  20 D . Resistors R_Wf1  20054 , R_Wf2  20052 , R_Wf3  20048 , R_Wf4  20046 , R_Ufp1  20050 , R_Ufm1  20056 , Rr  20128  and Rf2  20130  set the weights of connections of the “neuron f”. The “neuron f” uses a sigmoid (module X4  20084 ) as a nonlinear function.       

     The outputs of modules X2  20080  ( FIG.  20 B ) and X3  20082  ( FIG.  20 C ) are input to the X5 multiplier module  20086  ( FIG.  20 B ). The outputs of modules X4  20084  ( FIG.  20 D ) and buffer to U9  20010  are input to the multiplier module X6  20088 . The outputs of the modules X5  20086  and X6  20088  are input to the adder (U10  20112 ). A divider  10  is assembled on the resistors R1  20070 , R2  20072 , and R3  20074 . A nonlinear function of hyperbolic tangent (module X7  20090 ,  FIG.  20 B ) is obtained with the release of the divisor signal. The output C(t)  20120  (a current state vector of the LSTM neuron) is obtained with the buffer-inverter on the U11  20114  output signal. The outputs of modules X1  20078  and X7  20090  is input to a multiplier (module X8  20092 ) whose output is input to a buffer divider by 10 on the U12  20116 . The result of calculating the LSTM neuron at the present time H(t)  20118  is obtained from the output signal of U12  20116 . 
       FIG.  20 E  shows example values for the different configurable parameters (e.g., voltages) for the circuit shown in  FIGS.  20 A- 20 D , according to some implementations. Vdd  20058  is set to +1.5V, Vss  20064  is set to −1.5V, Vdd1  20060  is set to +1.8V, Vss1  20062  is set to −1.0V, and GND 20118 is set to GND, according to some implementations. 
       FIG.  20 F  shows a table  20132  of description for the example circuit shown in  FIG.  20 A- 20 D , according to some implementations. The values for the parameters are provided as examples, and various other configurations are possible. The transistors U1-U12 are CMOS OpAmps (described above in reference to  FIGS.  19 A and  19 B ). X1, X3, and X4 are modules that perform the Sigmoid function. X2 and X7 are modules that perform the Hyperbolic Tangent function. X5 and X8 are modules that perform the multiplication function. Example resistor ratings include: Rw=10 kΩ, and Rr=1.25 kΩ. The other resistor values are expressed relative to Rw. For example, Rf2=12 times Rw, R_Wo4=5 times Rw, R_Wo3=8 times Rw, R_Uop1=2.6 times Rw, R_Wo2=12 times Rw, R W1=w times Rw, and R_Uom1=2.3 times Rw, R_wc4=4 times Rw, R_Wc3=5.45 times Rw, R_Ucp1=3 times Rw, R_Wc2=12 times Rw, R_Wc1=2.72 times Rw, R_Ucm1=3.7 times Rw, R_Wi4=4.8 times Rw, W_Wi3=6 times Rw, W_Uip1=2 times Rw, R_Wi2=12 times Rw, R_Wi1=3 times Rw, R_Uim1=2.3 times Rw, R_Wf4=2.2 times Rw, R_Wf3=5 times Rw, R_Wfp=4 times Rw, R_Wf2=2 times Rw, R_Wf1=5.7 times Rw, and Rfm1=4.2 times Rw. 
     Example Scheme of a Multiplier Block 
       FIGS.  21 A- 21 I  show a schematic diagram of a multiplier block  21000 , according to some implementations. The neuron  21000  is based on the principle of a four-quadrant multiplier, assembled using operational amplifiers U1  21040  and U2  21042  (shown in  FIG.  21 B ), U3  21044  (shown in  FIG.  21 H ), and U4  21046  and U5  21048  (shown in  FIG.  21 I ), and CMOS transistors M1  21052  through M68  21182 . The inputs of the multiplier include V_one  21020   21006  and V two  21008  (shown in  FIG.  21 B ), and contact Vdd (positive supply voltage, e.g., +1.5 V relative to GND)  21004  and contact Vss (negative supply voltage, e.g., −1.5 V relative to GND)  21002 . In this scheme, additional supply voltages are used: contact Input Vdd1 (positive supply voltage, e.g., +1.8 V relative to GND), contact Vss1 (negative supply voltage, e.g., −1.0 V relative to GND). The result of the circuit calculations are output at mult_out (output pin)  21170  (shown in  FIG.  21 I ). 
     Referring to  FIG.  21 B , input signal (V_one) from V_one  21006  is connected to the inverter with a single gain made on U1  21040 , the output of which forms a signal negA  21006 , which is equal in amplitude, but the opposite sign with the signal V_one. Similarly, the signal (V_two) from the input V_two  21008  is connected to the inverter with a single gain made on U2  21042 , the output of which forms a signal negB  21012  which is equal in amplitude, but the opposite sign with the signal V_two. Pairwise combinations of signals from possible combinations (V_one, V_two, negA, negB) are output to the corresponding mixers on CMOS transistors. 
     Referring back to  FIG.  21 A , V two  21008  and negA  21010  are input to a multiplexer assembled on NMOS transistors M19  21086 , M20  21088 , M21  21090 , M22  21092 , and PMOS transistors M23  21094  and M24  21096 . The output of this multiplexer is input to the NMOS transistor M6  21060  ( FIG.  21 D ). 
     Similar transformations that occur with the signals include:
         negB  21012  and V_one  21020  are input to a multiplexer assembled on NMOS transistors M11  21070 , M12  2072 , M13  2074 , M14  21076 , and PMOS transistors M15  2078  and M16  21080 . The output of this multiplexer is input to the M5  21058  NMOS transistor (shown in  FIG.  21 D );   V_one  21020  and negB  21012  are input to a multiplexer assembled on PMOS transistors M18  21084 , M48  21144 , M49  21146 , and M50  21148 , and NMOS transistors M17  21082 , M47  21142 . The output of this multiplexer is input to the M9 PMOS transistor  21066  (shown in  FIG.  21 D );   negA  21010  and V_two  21008  are input to a multiplexer assembled on PMOS transistors M52  21152 , M54  21156 , M55  21158 , and M56  21160 , and NMOS transistors M51  21150 , and M53  21154 . The output of this multiplexer is input to the M2 NMOS transistor  21054  (shown in  FIG.  21 C );   negB  21012  and V_one  21020  are input to a multiplexer assembled on NMOS transistors M11  21070 , M12  21072 , M13  21074 , and M14  21076 , and PMOS transistors M15  21078 , and M16  21080 . The output of this multiplexer is input to the M10 NMOS transistor  21068  (shown in  FIG.  21 D );   negB  21012  and negA  21010  are input to a multiplexer assembled on NMOS transistors M35  21118 , M36  21120 , M37  21122 , and M38  21124 , and PMOS transistors M39 2 1126 , and M40  21128 . The output of this multiplexer is input to the M27 PMOS transistor  21102  (shown in  FIG.  21 H );   V_two  21008  and V_one  21020  are input to a multiplexer assembled on NMOS transistors M41  21130 , M42  21132 , M43  21134 , and M44  21136 , and PMOS transistors M45  21138 , and M46  21140 . The output of this multiplexer is input to the M30 NMOS transistor  21108  (shown in  FIG.  21 H );   V_one  21020  and V_two  21008  are input to a multiplexer assembled on PMOS transistors M58  21162 , M60  21166 , M61  21168 , and M62  21170 , and NMOS transistors M57  21160 , and M59  21164 . The output of this multiplexer is input to the M34 PMOS transistor  21116  (shown in  FIG.  21 H ); and   negA  21010  and negB  21012  are input to a multiplexer assembled on PMOS transistors M64  21174 , M66  21178 , M67  21180 , and M68  21182 , and NMOS transistors M63 2 1172 , and M65  21176 . The output of this multiplexer is input to the PMOS transistor M33  21114  (shown in  FIG.  21 H ).       

     The current mirror (transistors M1  21052 , M2  21053 , M3  21054 , and M4 2 1056 ) powers the portion of the four quadrant multiplier circuit shown on the left, made with transistors M5  21058 , M6  21060 , M7  21062 , M8  21064 , M9  21066 , and M10  21068 . Current mirrors (on transistors M25  21098 , M26  21100 , M27  21102 , and M28  21104 ) power supply of the right portion of the four-quadrant multiplier, made with transistors M29  21106 , M30  21108 , M31  21110 , M32  21112 , M33  21114 , and M34  21116 . The multiplication result is taken from the resistor Ro  21022  enabled in parallel to the transistor M3  21054  and the resistor Ro  21188  enabled in parallel to the transistor M28  21104 , supplied to the adder on U3  21044 . The output of U3  21044  is supplied to an adder with a gain of 7,1, assembled on U5  21048 , the second input of which is compensated by the reference voltage set by resistors R1  21024  and R2  21026  and the buffer U4  21046 , as shown in  FIG.  21 I . The multiplication result is output via the Mult_Out output  21170  from the output of U5  21048 . 
       FIG.  21 J  shows a table  21198  of description for the schematic shown in  FIGS.  21 A- 21 I , according to some implementations. U1-U5 are CMOS OpAmps. The N-Channel MOSFET transistors with explicit substrate connection include transistors M1, M2, M25, and M26 (with shutter ratio of length (L)=2.4 u, and shutter ratio of width (W)=1.26 u), transistors M5, M6, M29, and M30 (with L=0.36 u, and W=7.2 u), transistors M7, M8, M31, and M32 (with L=0.36 u, and W=199.98 u), transistors M11-M14, M19-M22, M35-M38, and M41-M44 (with L=0.36 u and W=0.4 u), and transistors M17, M47, M51, M53, M57, M59, M43, and M64 (with L=0.36 u and W=0.72 u). The P-Channel MOSFET transistors with explicit substrate connection include transistors M3, M4, M27, and M28 (with shutter ratio of length (L)=2.4 u, and shutter ratio of width (W)=1.26 u), transistors M9, M10, M33, and M34 (with L=0.36 u, and W=7.2 u), transistors M18, M48, M49, M50, M52, M54, M55, M56, M58, M60, M61, M62, M64, M66, M67, and M68 (with L=0.36 u, and W=0.8 u), and transistors M15, M16, M23, M24, M39, M40, M45, and M46 (with L=0.36 u and W=0.72 u). Example resistor ratings include Ro=1 kΩ, Rin=1 kΩ, Rf=1 kΩ, Rc4=2 kΩ, and Rc5=2 kΩ, according to some implementations. 
     Example Scheme of a Sigmoid Block 
       FIG.  22 A  shows a schematic diagram of a sigmoid block  2200 , according to some implementations. The sigmoid function (e.g., modules X1  20078 , X3  20082 , and X4  20084 , described above in reference to  FIGS.  20 A- 20 F ) is implemented using operational amplifiers U1  2250 , U2  2252 , U3  2254 , U4  2256 , U5  2258 , U6  2260 , U7,  2262 , and U8  2264 , and NMOS transistors M1  2266 , M2  2268 , and M3  2270 . Contact sigm_in  2206  is module input, contact Input Vdd1  2222  is positive supply voltage +1.8 V relative to GND  2208 , and contact Vss1  2204  is negative supply voltage −1.0 V relative to GND. In this scheme, U4  2256  has a reference voltage source of −0.2332 V, and the voltage is set by the divider R10  2230  and R11  2232 . The U5  2258  has a reference voltage source of 0.4 V, and the voltage is set by the divider R12  2234  and R13  2236 . The U6  2260  has a reference voltage source of 0.32687 V, the voltage is set by the divider R14  2238  and R15  2240 . The U7  2262  has a reference voltage source of −0.5 V, the voltage is set by the divider R16  2242  and R17  2244 . The U8  2264  has a reference voltage source of −0.33 V, the voltage is set by the divider R18  2246  and R19  2248 . 
     The sigmoid function is formed by adding the corresponding reference voltages on a differential module assembled on the transistors M1  2266  and M2  2268 . A current mirror for a differential stage is assembled with active regulation operational amplifier U3  2254 , and the NMOS transistor M3  2270 . The signal from the differential stage is removed with the NMOS transistor M2 and resistor R5  2220  is input to the adder U2  2252 . The output signal sigm_out  2210  is removed from the U2 adder  2252  output. 
       FIG.  22 B  shows a table  2278  of description for the schematic diagram shown in  FIG.  22 A , according to some implementations. U1-U8 are CMOS OpAmps. M1, M2, and M3 are N-Channel MOSFET transistors with a shutter ratio of length (L)=0.18 u, and shutter ration of width (W)=0.9 u, according to some implementations. 
     Example Scheme of a Hyperbolic Tangent Block 
       FIG.  23 A  shows a schematic diagram of a hyperbolic tangent function block  2300 , according to some implementations. The hyperbolic tangent function (e.g., the modules X2  20080 , and X7  20090  described above in reference to  FIGS.  20 A- 20 F ) is implemented using operational amplifiers (U1  2312 , U2  2314 , U3  2316 , U4  2318 , U5  2320 , U6  2322 , U7  2328 , and U8  2330 ) and NMOS transistors (M1  2332 , M2  2334 , and M3  2336 ). In this scheme, contact tanh_in  2306  is module input, contact Input Vdd1  2304  is positive supply voltage +1.8 V relative to GND  2308 , and contact Vss1  2302  is negative supply voltage −1.0 V relative to GND. Further, in this scheme, U4  2318  has a reference voltage source of −0.1 V, the voltage set by the divider R10  2356  and R11  2358 . The U5  2320  has a reference voltage source of 1.2 V, the voltage set by the divider R12  2360  and R13  2362 . The U6  2322  has a reference voltage source of 0.32687 V, the voltage set by the divider R14  2364  and R15  2366 . The U7  2328  has a reference voltage source of −0.5 V, the voltage set by the divider R16  2368  and R17  2370 . The U8  2330  has a reference voltage source of −0.33 V, the voltage set by the divider R18  2372  and R19  2374 . The hyperbolic tangent function is formed by adding the corresponding reference voltages on a differential module made on transistors M1  2332  and M2  2334 . A current mirror for a differential stage is obtained with active regulation operational amplifier U3  2316 , and NMOS transistor M3  2336 . With NMOS transistor M2  2334  and resistor R5  2346 , the signal is removed from the differential stage and input to the adder U2  2314 . The output signal tanh_out  2310  is removed from the U2 adder  2314  output. 
       FIG.  23 B  shows a table  2382  of description for the schematic diagram shown in  FIG.  23 A , according to some implementations. U1-U8 are CMOS OpAmps, and M1, M2, and M3 are N-Channel MOSFET transistors, with a shutter ratio of length (L)=0.18 u, and the shutter ratio of width (W)=0.9 u. 
     Example Scheme of a Single Neuron OP1 CMOS OpAmp 
       FIGS.  24 A- 24 C  show a schematic diagram of a single neuron OP1 CMOS OpAmp  2400 , according to some implementations. The example is a variant of a single neuron on an operational amplifier, made on CMOS according to an OP1 scheme described herein. In this scheme, contacts V1  2410  and V2  2408  are inputs of a single neuron, contact bias  2406  is voltage +0.4 V relative to GND, contact Input Vdd  2402  is positive supply voltage +5.0 V relative to GND, contact Vss  2404  is GND, and contact Out  2474  is output of a single neuron. Parameters of CMOS transistors are determined by the ratio of geometric dimensions: L (the length of the gate channel), and W (the width of the gate channel). This Op Amp has two current mirrors. The current mirror on NMOS transistors M3  2420 , M6  2426 , and M13  2440  provides the offset current of the differential pair on NMOS transistors M2  2418  and M5  2424 . The current mirror in the PMOS transistors M7  2428 , M8  2430 , and M15  2444  provides the offset current of the differential pair on the PMOS transistors M9  2432  and M10  2434 . In the first differential amplifier stage, NMOS transistors M2  2418  and M5  2424  are amplifying, and PMOS transistors M1  2416  and M4  2422  play the role of active current load. From the M5  2424  transistor, the signal is output to the PMOS gate of the transistor M13  2440 . From the M2  2418  transistor, the signal is output to the right input of the second differential amplifier stage on PMOS transistors M9  2432  and M10  2434 . NMOS transistors M11  2436  and M12  2438  play the role of active current load for the M9  2432  and M10  2434  transistors. The M17  2448  transistor is switched on according to the scheme with a common source for a positive half-wave of the signal. The M18  2450  transistor is switched on according to the scheme with a common source for the negative half-wave of the signal. To increase the overall load capacity of the Op Amp, an inverter on the M17  2448  and M18  2450  transistors is enabled at the output of the M13  2440  and M14  2442  transistors. 
       FIG.  24 D  shows a table  2476  of description for the schematic diagram shown in  FIG.  24 A- 24 C , according to some implementations. The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratio: w1=(Rp/R1+)— (Rn/R1−); w2=(Rp/R2+)— (Rn/R2−); w bias=(Rp/Rbias+)−(Rn/Rbias−). Normalizing resistors (Rnorm− and Rnorm+) are necessary to obtain exact equality: (Rn/R1−)+(Rn/R2−)+(Rn/Rbias−)+(Rn/Rnorm−)=(Rp/R1+)+(Rp/R2+)+(Rp/Rbias+)+(Rp/Rnorm+). N-Channel MOSFET transistors with explicit substrate connection include transistors M2 and M5 with L=0.36 u and W=3.6 u, transistors M3, M6, M11, M12, M14, and M16 with L=0.36 u and W=1.8 u, and transistor M18 with L=0.36 u and W=18 u. P-Channel MOSFET transistors with explicit substrate connection include transistors M1, M4, M7, M8, M13, and M15 with L=0.36 u and W=3.96 u, transistors M9 and M10 with L=0.36 u and W=11.88 u, and transistor M17 with L=0.36 u and W=39.6 u. 
     Example Scheme of a Single Neuron OP3 CMOS OpAmp 
       FIGS.  25 A- 25 D  show a schematic diagram of a variant of a single neuron  25000  on operational amplifiers, made on CMOS according to an OP3 scheme, according to some implementations. The single neuron consists of three simple operational amplifiers (OpAmps), according to some implementations. The unit Neuron adder is performed on two Opamps with bipolar power supply and the RELU activation function is performed on an OpAmp with unipolar power supply and with a gain of =10. Transistors M1  25028 -M16  25058  are used for summation of negative connections of the neuron. Transistors M17  25060 -M32  25090  are used for adding the positive connections of the neuron. The RELU activation function is performed on the transistors M33  25092 -M46  25118 . In the scheme, contacts V1  25008  and V2  25010  are inputs of the single neuron, contact bias  25002  is voltage +0.4 V relative to GND, contact Input Vdd  25004  is positive supply voltage +2.5 V relative to GND, contact Vss  25006  is negative supply voltage −2.5 V, and contact Out  25134  is output of the single neuron. Parameters of CMOS transistors used in a single neuron are determined by the ratio of geometric dimensions: L (the length of the gate channel) and W (the width of the gate channel). Consider the operation of the simplest OpAmp included in a single neuron. Each op amp has two current mirrors. The current mirror on NMOS transistors M3  25032  (M19  25064 , M35  25096 ), M6  25038  (M22  25070 , M38  25102 ) and M16  25058  (M32  25090 , M48  25122 ) provides the offset current of the differential pair on NMOS transistors M2  25030  (M18  25062 , M34  25094 ) and M5  25036  (M21  25068 , M35  25096 ). The current mirror in PMOS transistors M7  25040  (M23  25072 , M39  25104 ), M8  25042  (M24  25074 , M40  25106 ) and M15  25056  (M31  2588 ) provides the offset current of the differential pair on PMOS transistors M9  25044  (M25  25076 , M41  25108 ) and M10  25046  (M26  25078 , M42  25110 ). In the first differential amplifier stage, NMOS transistors M2  25030  (M18  25062 , M34  25094 ) and M5  25036  (M21  25068 , M37  25100 ) are amplifying, and PMOS transistors M1  25028  (M17  25060 , M33  25092 ) and M4  25034  (M20  25066 , M36  25098 ) play the role of active current load. From the transistor M5  25036  (M21  25068 , M37  25100 ), the signal is input to the PMOS gate of the transistor M13  25052  (M29  25084 , M45  25116 ). From the transistor M2  25030  (M18  25062 , M34  25094 ), the signal is input to the right input of the second differential amplifier stage on PMOS transistors M9  25044  (M25  25076 , M41  25108 ) and M10  25046  (M26  25078 , M42  25110 ). NMOS transistors M11  25048  (M27  25080 , M43  25112 ) and M12  25048  (M28  25080 , M44  25114 ) play the role of active current load for transistors M9  25044  (M25  25076 , M41  25108 ) and M10  25046  (M26  25078 , M42  25110 ). Transistor M13  25052  (M29  25082 , M45  25116 ) is included in the scheme with a common source for a positive half-wave signal. The transistor M14  25054  (M30  25084 , M46  25118 ) is switched on according to the scheme with a common source for the negative half-wave of the signal. 
     The weights of the connections of a single neuron (with two inputs and one output) are set by the resistor ratio: w1=(R feedback/R1+)−(R feedback/R1−); w2=(R feedback/R2+)−(R feedback/R2−); wbias=(R feedback/Rbias+)−(R feedback/Rbias−); w1=(R p*K amp/R1+)−(R n*K amp/R1−); w2=(R p*K amp/R2+)−(R n*K amp/R2−); wbias=(R p*K amp/Rbias+)−(R n*K amp/Rbias−), where K amp=R1ReLU/R2ReLU. R feedback=100 k—used only for calculating w1, w2, wbias. According to some implementations, example values include: R feedback=100 k, Rn=Rp=Rcom=10 k, K amp ReLU=1+90 k/10 k=10, w1=(10 k*10/22.1 k)−(10 k*10/21.5 k)=−0.126276, w2=(10 k*10/75 k)−(10 k*10/71.5 k)=−0.065268, wbias=(10 k*10/71.5 k)−(10 k*10/78.7 k)=0.127953. 
     The input of the negative link adder of the neuron (M1-M17) is received from the positive link adder of the neuron (M17-M32) through the Rcom resistor. 
       FIG.  25 E  shows a table  25136  of description for the schematic diagram shown in  FIG.  25 A- 25 D , according to some implementations. N-Channel MOSFET transistors with explicit substrate connection include transistors M2, M5, M18, M21, M34, and M37, with L=0.36 u and W=3.6 u, transistors M3, M6, M11, M12, M14, M16, M19, M22, M27, M28, M32, M38, M35, M38, M43, M44, M46, and M48, with L=0.36 u and W=1.8 u. P-Channel MOSFET transistors with explicit substrate connection include transistors M1, M4, M7, M8, M13, M15, M17, M20, M23, M24, M29, M31, M33, M36, M39, M40, M45, and M47 with L=0.36 u and W=3.96 u, and transistor M9, M10, M25, M26, M41, and M42, with L=0.36 u and W=11.88 u. 
     Example Methods for Analog Hardware Realization of Trained Neural Networks 
       FIGS.  27 A- 27 J  show a flowchart of a method  2700  for hardware realization ( 2702 ) of neural networks, according to some implementations. The method is performed ( 2704 ) at the computing device  200  (e.g., using the neural network transformation module  226 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . The method includes obtaining ( 2706 ) a neural network topology (e.g., the topology  224 ) and weights (e.g., the weights  222 ) of a trained neural network (e.g., the networks  220 ). In some implementations, the trained neural network is trained ( 2708 ) using software simulations to generate the weights. 
     The method also includes transforming ( 2710 ) the neural network topology to an equivalent analog network of analog components. Referring next to  FIG.  27 C , in some implementations, the neural network topology includes ( 2724 ) one or more layers of neurons. Each layer of neurons computing respective outputs based on a respective mathematical function. In such cases, transforming the neural network topology to the equivalent analog network of analog components includes, performing ( 2726 ) a sequence of steps for each layer of the one or more layers of neurons. The sequence of steps include identifying ( 2728 ) one or more function blocks, based on the respective mathematical function, for the respective layer. Each function block has a respective schematic implementation with block outputs that conform to outputs of a respective mathematical function. In some implementations, identifying the one or more function blocks includes selecting ( 2730 ) the one or more function blocks based on a type of the respective layer. For example, a layer can consist of neurons, and the layer&#39;s output is a linear superposition of its inputs. Selecting the one or more function blocks is based on this identification of a layer type, if a layer&#39;s output is a linear superposition, or similar pattern identification. Some implementations determine if number of output&gt;1, then use either a trapezium or a pyramid transformation. 
     Referring next to  FIG.  27 D , in some implementations, the one or more function blocks include one or more basic function blocks (e.g., the basic function blocks  232 ) selected ( 2734 ) from the group consisting of: (i) a weighted summation block ( 2736 ) with a block output V out =ReLU(Σw i ·V i   in +bias). ReLU is Rectified Linear Unit (ReLU) activation function or a similar activation function (e.g., ReLU with a threshold), V i  represents an i-th input, w i  represents a weight corresponding to the i-th input, and bias represents a bias value, and Σ is a summation operator; (ii) a signal multiplier block ( 2738 ) with a block output V out =coeff·V i ·V j ·V i  represents an i-th input and V j  represents a j-th input, and coeff is a predetermined coefficient; (iii) a sigmoid activation block ( 2740 ) with a block output 
     
       
         
           
             
               V 
               out 
             
             = 
             
               
                 A 
                 
                   1 
                   + 
                   
                     e 
                     
                       - 
                       
                         B 
                         . 
                         V 
                       
                     
                   
                 
               
               . 
               V 
             
           
         
       
     
     represents an input, and A and B are predetermined coefficient values (e.g., A=−0.1; B=11.3) of the sigmoid activation block; (iv) a hyperbolic tangent activation block ( 2742 ) with a block output V out =A*tanh (B*V in ). V in  represents an input, and A and B are predetermined coefficient values (e.g., A=0.1, B=−10.1); and a signal delay block ( 2744 ) with a block output U(t)=V(t−dt). t represents a current time-period, V(t−1) represents an output of the signal delay block for a preceding time period t−1, and dt is a delay value. 
     Referring now back to  FIG.  27 C , the sequence of steps also includes generating ( 2732 ) a respective multilayer network of analog neurons based on arranging the one or more function blocks. Each analog neuron implements a respective function of the one or more function blocks, and each analog neuron of a first layer of the multilayer network is connected to one or more analog neurons of a second layer of the multilayer network. 
     Referring now back to  FIG.  27 A , for some networks, such as GRU and LSTM, transforming ( 2710 ) the neural network topology to an equivalent analog network of analog components requires more complex processing, according to some implementations. Referring next to  FIG.  27 E , suppose the neural network topology includes ( 2746 ) one or more layers of neurons. Suppose further that each layer of neurons computes respective outputs based on a respective mathematical function. In such cases, transforming the neural network topology to the equivalent analog network of analog components includes: (i) decomposing ( 2748 ) a first layer of the neural network topology to a plurality of sub-layers, including decomposing a mathematical function corresponding to the first layer to obtain one or more intermediate mathematical functions. Each sub-layer implements an intermediate mathematical function. In some implementations, the mathematical function corresponding to the first layer includes one or more weights, and decomposing the mathematical function includes adjusting ( 2750 ) the one or more weights such that combining the one or more intermediate functions results in the mathematical function.; and (ii) performing ( 2752 ) a sequence of steps for each sub-layer of the first layer of the neural network topology. The sequence of steps includes selecting ( 2754 ) one or more sub-function blocks, based on a respective intermediate mathematical function, for the respective sub-layer; and generating ( 2756 ) a respective multilayer analog sub-network of analog neurons based on arranging the one or more sub-function blocks. Each analog neuron implements a respective function of the one or more sub-function blocks, and each analog neuron of a first layer of the multilayer analog sub-network is connected to one or more analog neurons of a second layer of the multilayer analog sub-network. 
     Referring next to  FIG.  27 H , suppose the neural network topology includes ( 2768 ) one or more GRU or LSTM neurons. In that case, transforming the neural network topology includes generating ( 2770 ) one or more signal delay blocks for each recurrent connection of the one or more GRU or LSTM neurons. In some implementations, an external cycle timer activates the one or more signal delay blocks with a constant time period (e.g., 1, 5, or 10 time steps). Some implementations use multiple delay blocks over one signal for producing additive time shift. In some implementations, the activation frequency of the one or more signal delay blocks is/are synchronized to network input signal frequency. In some implementations, the one or more signal delay blocks are activated ( 2772 ) at a frequency that matches a predetermined input signal frequency for the neural network topology. In some implementations, this predetermined input signal frequency may be dependent on the application, such as Human Activity Recognition (HAR) or PPG. For example, the predetermined input signal frequency is 30-60 Hz for video processing, around 100 Hz for HAR and PPG, 16 KHz for sound processing, and around 1-3 Hz for battery management. Some implementations activate different signal delay blocks activate at different frequencies. 
     Referring next to  FIG.  27 I , suppose the neural network topology includes ( 2774 ) one or more layers of neurons that perform unlimited activation functions. In some implementations, in such cases, transforming the neural network topology includes applying ( 2776 ) one or more transformations selected from the group consisting of: replacing ( 2778 ) the unlimited activation functions with limited activation (e.g., replacing ReLU with a threshold ReLU); and adjusting ( 2780 ) connections or weights of the equivalent analog network such that, for predetermined one or more inputs, difference in output between the trained neural network and the equivalent analog network is minimized. 
     Referring now back to  FIG.  27 A , the method also includes computing ( 2712 ) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent analog network. 
     The method also includes generating ( 2714 ) a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component values for the analog components. Referring next to  FIG.  27 B , in some implementations, generating the schematic model includes generating ( 2716 ) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. In some implementations, the method includes regenerating just the resistance matrix for the resistors for a retrained network. In some implementations, the method further includes obtaining ( 2718 ) new weights for the trained neural network, computing ( 2720 ) a new weight matrix for the equivalent analog network based on the new weights, and generating ( 2722 ) a new resistance matrix for the new weight matrix. 
     Referring next to  FIG.  27 J , in some implementations, the method further includes generating ( 2782 ) one or more lithographic masks (e.g., generating the masks  250  and/or  252  using the mask generation module  248 ) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix. In some implementations, the method includes regenerating just the masks for resistors (e.g., the masks  250 ) for retrained networks. In some implementations, the method further includes: (i) obtaining ( 2784 ) new weights for the trained neural network; (ii) computing ( 2786 ) a new weight matrix for the equivalent analog network based on the new weights; (iii) generating ( 2788 ) a new resistance matrix for the new weight matrix; and (iv) generating ( 2790 ) a new lithographic mask for fabricating the circuit implementing the equivalent analog network of analog components based on the new resistance matrix. 
     Referring now back to  FIG.  27 G , the analog components include ( 2762 ) a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. Some implementations include other analog components, such as four-quadrant multipliers, sigmoid and hyperbolic tangent function circuits, delay lines, summers, and/or dividers. In some implementations, selecting ( 2764 ) component values of the analog components includes performing ( 2766 ) a gradient descent method and/or other weight quantization methods to identify possible resistance values for the plurality of resistors. 
     Referring now back to  FIG.  27 F , in some implementations, the method further includes implementing certain activation functions (e.g., Softmax) in output layer in digital. In some implementations, the method further includes generating ( 2758 ) equivalent digital network of digital components for one or more output layers of the neural network topology, and connecting ( 2760 ) output of one or more layers of the equivalent analog network to the equivalent digital network of digital components. 
     Example Methods for Constrained Analog Hardware Realization of Neural Networks 
       FIGS.  28 A- 28 S  show a flowchart of a method  28000  for hardware realization ( 28002 ) of neural networks according to hardware design constraints, according to some implementations The method is performed ( 28004 ) at the computing device  200  (e.g., using the neural network transformation module  226 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . The method includes obtaining ( 28006 ) a neural network topology (e.g., the topology  224 ) and weights (e.g., the weights  222 ) of a trained neural network (e.g., the networks  220 ). 
     The method also includes calculating ( 28008 ) one or more connection constraints based on analog integrated circuit (IC) design constraints (e.g., the constraints  236 ). For example, IC design constraints can set the current limit (e.g.,  1 A), and neuron schematics and operational amplifier (OpAmp) design can set the OpAmp output current in the range [0-10 mA], so this limits output neuron connections to 100. This means that the neuron has 100 outputs which allow the current to flow to the next layer through 100 connections, but current at the output of the operational amplifier is limited to 10 mA, so some implementations use a maximum of 100 outputs (0.1 mA times 100=10 mA). Without this constraint, some implementations use current repeaters to increase number of outputs to more than 100, for example. 
     The method also includes transforming ( 28010 ) the neural network topology (e.g., using the neural network transformation module  226 ) to an equivalent sparsely connected network of analog components satisfying the one or more connection constraints. 
     In some implementations, transforming the neural network topology includes deriving ( 28012 ) a possible input connection degree N i  and output connection degree N o , according to the one or more connection constraints. 
     Referring next to  FIG.  28 B , in some implementations, the neural network topology includes ( 28018 ) at least one densely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U, and transforming ( 28020 ) the at least one densely connected layer includes constructing ( 28022 ) the equivalent sparsely connected network with K inputs, L outputs, and ┌log N     i    K┐+┌log N     o    L┐−1 layers, such that input connection degree does not exceed N i , and output connection degree does not exceed N o . 
     Referring next to  FIG.  28 C , in some implementations, the neural network topology includes ( 28024 ) at least one densely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U, and transforming ( 28026 ) the at least one densely connected layer includes: constructing ( 28028 ) the equivalent sparsely connected network with K inputs, L outputs, and M≥max(┌log N     i   L┐, ┌log N     o   K┐) layers. Each layer m is represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed N i , and output connection degree does not exceed N o . The equation U=Π m=1 . . . M  U m  is satisfied with a predetermined precision. The predetermined precision is a reasonable precision value that statistically guarantees that altered networks output differs from referent network output by no more than allowed error value, and this error value is task-dependent (typically between 0.1% and 1%). 
     Referring next to  FIG.  28 D , in some implementations, the neural network topology includes ( 28030 ) a single sparsely connected layer with K inputs and L outputs, a maximum input connection degree of P i , a maximum output connection degree of P o , and a weight matrix of U, where absent connections are represented with zeros. In such cases, transforming ( 28032 ) the single sparsely connected layer includes constructing ( 28034 ) the equivalent sparsely connected network with K inputs, L outputs, M≥max(┌log N     i   P i ┐, ┌log N     o   P o ┐) layers. Each layer m is represented by a corresponding weight matrix U m , where absent connections are represented with zeros, such that input connection degree does not exceed N i , and output connection degree does not exceed N o , and the equation U=Π n=1 . . . M  U m  is satisfied with a predetermined precision. 
     Referring next to  FIG.  28 E , in some implementations, the neural network topology includes ( 28036 ) a convolutional layer (e.g., a Depthwise convolutional layer, or a Separable convolutional layer) with K inputs (neurons in previous layer) and L outputs (neurons in current layer). In such cases, transforming ( 28038 ) the neural network topology to the equivalent sparsely connected network of analog components includes decomposing ( 28040 ) the convolutional layer into a single sparsely connected layer with K inputs, L outputs, a maximum input connection degree of P i , and a maximum output connection degree of P o , where P i ≤N i  and P o ≤N o . 
     Referring back to  FIG.  28 A , the method also includes computing ( 28014 ) a weight matrix for the equivalent sparsely connected network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection between analog components of the equivalent sparsely connected network. 
     Referring now to  FIG.  28 F , in some implementations, the neural network topology includes ( 28042 ) a recurrent neural layer, and transforming ( 28044 ) the neural network topology to the equivalent sparsely connected network of analog components includes transforming ( 28046 ) the recurrent neural layer into one or more densely or sparsely connected layers with signal delay connections. 
     Referring next to  FIG.  28 G , in some implementations, the neural network topology includes a recurrent neural layer (e.g., a long short-term memory (LSTM) layer or a gated recurrent unit (GRU) layer), and transforming the neural network topology to the equivalent sparsely connected network of analog components includes decomposing the recurrent neural layer into several layers, where at least one of the layers is equivalent to a densely or sparsely connected layer with K inputs (neurons in previous layer) and L outputs (neurons in current layer) and a weight matrix U, where absent connections are represented with zeros. 
     Referring next to  FIG.  28 H , in some implementations, the method includes performing a transformation of a single layer perceptron with one calculation neurons. In some implementations, the neural network topology includes ( 28054 ) K inputs, a weight vector U∈R K , and a single layer perceptron with a calculation neuron with an activation function F. In such cases, transforming ( 28056 ) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving ( 28058 ) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating ( 28060 ) a number of layers m for the equivalent sparsely connected network using the equation m=┌log N  K┐; and (iii) constructing ( 28062 ) the equivalent sparsely connected network with the K inputs, m layers and the connection degree N. The equivalent sparsely connected network includes respective one or more analog neurons in each layer of the m layers. Each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function F of the calculation neuron of the single layer perceptron. Furthermore, in such cases, computing ( 28064 ) the weight matrix for the equivalent sparsely connected network includes calculating ( 28066 ) a weight vector W for connections of the equivalent sparsely connected network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 K 
                 ⁡ 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     Referring next to  FIG.  28 I , in some implementations, the method includes performing a transformation of a single layer perceptron with L calculation neurons. In some implementations, the neural network topology includes ( 28068 ) K inputs, a single layer perceptron with L calculation neurons, and a weight matrix V that includes a row of weights for each calculation neuron of the L calculation neurons. In such cases, transforming ( 28070 ) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving ( 28072 ) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) calculating ( 28074 ) number of layers m for the equivalent sparsely connected network using the equation m=┌log N  K┐; (iii) decomposing ( 28076 ) the single layer perceptron into L single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the L calculation neurons; (iv) for each single layer perceptron network ( 28078 ) of the L single layer perceptron networks, constructing ( 28080 ) a respective equivalent pyramid-like sub-network for the respective single layer perceptron network with the K inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron; and (v) constructing ( 28082 ) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating an input of each equivalent pyramid-like sub-network for the L single layer perceptron networks to form an input vector with L*K inputs. Furthermore, in such cases, computing ( 28084 ) the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network ( 28086 ) of the L single layer perceptron networks, (i) setting ( 28088 ) a weight vector U=V i , i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, and (ii) calculating ( 28090 ) a weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 K 
                 ⁡ 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     Referring next to  FIG.  28 J , in some implementations, the method includes performing a transformation algorithm for multi-layer perceptron. In some implementations, the neural network topology includes ( 28092 ) K inputs, a multi-layer perceptron with S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i  and corresponding weight matrices V that includes a row of weights for each calculation neuron of the L i  calculation neurons. In such cases, transforming ( 28094 ) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving ( 28096 ) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing ( 28098 ) the multi-layer perceptron into Q=Σ i=1,S (L i ) single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons. Decomposing the multi-layer perceptron includes duplicating one or more input of the K inputs that are shared by the Q calculation neurons; (iii) for each single layer perceptron network ( 28100 ) of the Q single layer perceptron networks, (a) calculating ( 28102 ) a number of layers m for a respective equivalent pyramid-like sub-network using the equation m=┌log N  K i,j ┐. K i,j  is number of inputs for the respective calculation neuron in the multi-layer perceptron, and (b) constructing ( 28104 ) the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i,j  inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of them layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing ( 28106 ) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i,j  inputs. In such cases, computing ( 28108 ) the weight matrix for the equivalent sparsely connected network includes: for each single layer perceptron network ( 28110 ) of the Q single layer perceptron networks, (i) setting ( 28112 ) a weight vector U=V i   j , the i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the multi-layer perceptron; and (ii) calculating ( 28114 ) a weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K i,j  equations with S variables, and S is computed using the equation 
     
       
         
           
             S 
             = 
             
               
                 
                   K 
                   
                     i 
                     , 
                     j 
                   
                 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     Referring next to  FIG.  28 K , in some implementations, the neural network topology includes ( 28116 ) a Convolutional Neural Network (CNN) with K inputs, S layers, each layer i of the S layers includes a corresponding set of calculation neurons L i  and corresponding weight matrices V i  that includes a row of weights for each calculation neuron of the L i  calculation neurons. In such cases, transforming ( 28118 ) the neural network topology to the equivalent sparsely connected network of analog components includes: (i) deriving ( 28120 ) a connection degree N for the equivalent sparsely connected network according to the one or more connection constraints; (ii) decomposing ( 28122 ) the CNN into Q=Σ i=1,S (L i ) single layer perceptron networks. Each single layer perceptron network includes a respective calculation neuron of the Q calculation neurons. Decomposing the CNN includes duplicating one or more input of the K inputs that are shared by the Q calculation neurons; (iii) for each single layer perceptron network of the Q single layer perceptron networks: (a) calculating number of layers m for a respective equivalent pyramid-like sub-network using the equation m=┌log N  K i,j ┐. j is the corresponding layer of the respective calculation neuron in the CNN, and K i,j  is number of inputs for the respective calculation neuron in the CNN; and (b) constructing the respective equivalent pyramid-like sub-network for the respective single layer perceptron network with K i,j  inputs, the m layers and the connection degree N. The equivalent pyramid-like sub-network includes one or more respective analog neurons in each layer of the m layers, each analog neuron of first m−1 layers implements identity transform, and an analog neuron of last layer implements the activation function of the respective calculation neuron corresponding to the respective single layer perceptron network; and (iv) constructing ( 28130 ) the equivalent sparsely connected network by concatenating each equivalent pyramid-like sub-network including concatenating input of each equivalent pyramid-like sub-network for the Q single layer perceptron networks to form an input vector with Q*K i,j  inputs. In such cases, computing ( 28132 ) the weight matrix for the equivalent sparsely connected network includes, for each single layer perceptron network ( 28134 ) of the Q single layer perceptron networks: (i) setting a weight vector U=V i   j , the i th  row of the weight matrix V corresponding to the respective calculation neuron corresponding to the respective single layer perceptron network, where j is the corresponding layer of the respective calculation neuron in the CNN; and (ii) calculating weight vector W i  for connections of the respective equivalent pyramid-like sub-network by solving a system of equations based on the weight vector U. The system of equations includes K i,j  equations with S variables, and S is computed using me equation 
     
       
         
           
             S 
             = 
             
               
                 
                   K 
                   
                     i 
                     , 
                     j 
                   
                 
                 ( 
                 
                   
                     
                       N 
                       m 
                     
                     - 
                     1 
                   
                   
                     
                       N 
                       
                         m 
                         - 
                         1 
                       
                     
                     ( 
                     
                       N 
                       - 
                       1 
                     
                     ) 
                   
                 
                 ) 
               
               . 
             
           
         
       
     
     Referring next to  FIG.  28 L , in some implementations, the method includes transforming two layers to trapezium-based network. In some implementations, the neural network topology includes ( 28140 ) K inputs, a layer L p  with K neurons, a layer L n  with L neurons, and a weight matrix W∈R L×K , where R is the set of real numbers, each neuron of the layer L p  is connected to each neuron of the layer L n , and each neuron of the layer L n  performs an activation function F, such that output of the layer L n  is computed using the equation Y o =F(W·x) for an input x. In such cases, transforming ( 28142 ) the neural network topology to the equivalent sparsely connected network of analog components includes performing a trapezium transformation that includes: (i) deriving ( 28144 ) a possible input connection degree N I &gt;1 and a possible output connection degree N O &gt;1, according to the one or more connection constraints; and (ii) in accordance with a determination that K·L&lt;L·N I +K·N O , constructing ( 28146 ) a three-layered analog network that includes a layer LA p  with K analog neurons performing identity activation function, a layer LA h  with 
     
       
         
           
             M 
             = 
             
               ⌈ 
               
                 max 
                 ⁡ 
                 ( 
                 
                   
                     
                       K 
                       · 
                       
                         N 
                         I 
                       
                     
                     
                       N 
                       O 
                     
                   
                   , 
                   
                     
                       L 
                       · 
                       
                         N 
                         O 
                       
                     
                     
                       N 
                       I 
                     
                   
                 
                 ) 
               
               ⌉ 
             
           
         
       
     
     analog neurons performing identity activation function, and a layer LA o  with L analog neurons performing the activation function F, such that each analog neuron in the layer LA p  has N O  outputs, each analog neuron in the layer LA h  has not more than N I  inputs and N O  outputs, and each analog neuron in the layer LA o  has N I  inputs. In some such cases, computing ( 28148 ) the weight matrix for the equivalent sparsely connected network includes generating ( 2850 ) a sparse weight matrices W o  and W h  by solving a matrix equation W o ·W h =W that includes K·L equations in K·N O +L·N I  variables, so that the total output of the layer LA o  is calculated using the equation Y o =F(W o ·W h ·x). The sparse weight matrix W o ∈R K×M  represents connections between the layers LA p  and LA h , and the sparse weight matrix W h ∈R M×L  represents connections between the layers LA h  and LA o . 
     Referring next to  FIG.  28 M , in some implementations, performing the trapezium transformation further includes: in accordance with a determination that K·L≥L·N I +K·N O : (i) splitting ( 28154 ) the layer L p  to obtain a sub-layer L p1  with K′ neurons and a sub-layer L p2  with (K-K′) neurons such that K′·L≥L·N I +K′·N O ; (ii) for the sub-layer L p1  with K′ neurons, performing ( 28156 ) the constructing, and generating steps; and (iii) for the sub-layer L p2  with K-K′ neurons, recursively performing ( 28158 ) the splitting, constructing, and generating steps. 
     Referring next to  FIG.  28 N , the method includes transforming multilayer perceptron to trapezium-based network. In some implementations, the neural network topology includes ( 28160 ) a multilayer perceptron network, the method further includes, for each pair of consecutive layers of the multilayer perceptron network, iteratively performing ( 28162 ) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. 
     Referring next to  FIG.  28 O , the method includes transforming recurrent neural network to trapezium-based network. In some implementations, the neural network topology includes ( 28164 ) a recurrent neural network (RNN) that includes (i) a calculation of linear combination for two fully connected layers, (ii) element-wise addition, and (iii) a non-linear function calculation. In such cases, the method further includes performing ( 28166 ) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the two fully connected layers, and (ii) the non-linear function calculation. Element-wise addition is a common operation that can be implemented in networks of any structure, examples of which are provided above. Non-linear function calculation is a neuron-wise operation that is independent of the No and Ni restrictions, and are usually calculated with ‘sigmoid’ or ‘tank’ block on each neuron separately. 
     Referring next to  FIG.  28 P , the neural network topology includes ( 28168 ) a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network that includes (i) a calculation of linear combination for a plurality of fully connected layers, (ii) element-wise addition, (iii) a Hadamard product, and (iv) a plurality of non-linear function calculations (sigmoid and hyperbolic tangent operations). In such cases, the method further includes performing ( 28170 ) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network, for (i) the plurality of fully connected layers, and (ii) the plurality of non-linear function calculations. Element-wise addition and Hadamard products are common operations that can be implemented in networks of any structure described above. 
     Referring next to  FIG.  28 Q , the neural network topology includes ( 28172 ) a convolutional neural network (CNN) that includes (i) a plurality of partially connected layers (e.g., sequence of convolutional and pooling layers; each pooling layer is assumed to be a convolutional later with stride larger than 1) and (ii) one or more fully-connected layers (the sequence ends in the fully-connected layers). In such cases, the method further includes (i) transforming ( 28174 ) the plurality of partially connected layers to equivalent fully-connected layers by inserting missing connections with zero weights; and for each pair of consecutive layers of the equivalent fully-connected layers and the one or more fully-connected layers, iteratively performing ( 28176 ) the trapezium transformation and computing the weight matrix for the equivalent sparsely connected network. 
     Referring next to  FIG.  28 R , the neural network topology includes ( 28178 ) K inputs, L output neurons, and a weight matrix U∈R L×K  where R is the set of real numbers, each output neuron performs an activation function F. In such cases, transforming ( 28180 ) the neural network topology to the equivalent sparsely connected network of analog components includes performing an approximation transformation that includes: (i) deriving ( 28182 ) a possible input connection degree N I &gt;1 and a possible output connection degree N O &gt;1, according to the one or more connection constraints; (ii) selecting ( 28184 ) a parameter p from the set {0, 1, . . . , ┌log N     I   K┐−1}; (iii) in accordance with a determination that p&gt;0, constructing ( 28186 ) a pyramid neural network that forms first p layers of the equivalent sparsely connected network, such that the pyramid neural network has N p =┌K/N I   p ┐ neurons in its output layer. Each neuron in the pyramid neural network performs identity function; and (iv) constructing ( 28188 ) a trapezium neural network with N p  inputs and L outputs. Each neuron in the last layer of the trapezium neural network performs the activation function F and all other neurons perform identity function. Also, in such cases, computing ( 28190 ) the weight matrix for the equivalent sparsely connected network includes: (i) generating ( 28192 ) weights for the pyramid neural network including (i) setting weights of every neuron i of the first layer of the pyramid neural network according to following rule: (a) w ik     i     (1) =C. C is a non-zero constant and 
     
       
         
           
             
               
                 
                   k 
                   i 
                 
                 = 
                 
                   
                     
                       ( 
                       
                         i 
                         - 
                         1 
                       
                       ) 
                     
                     ⁢ 
                     
                       N 
                       I 
                     
                   
                   + 
                   1 
                 
               
               ; 
               
                 
                   and 
                   ⁢ 
                       
                   
                     ( 
                     b 
                     ) 
                   
                   ⁢ 
                   
                     w 
                     ij 
                     
                       ( 
                       1 
                       ) 
                     
                   
                 
                 = 
                 
                   
                     1 
                     L 
                   
                   ⁢ 
                   
                     
                       ∑ 
                       
                         l 
                         = 
                         1 
                       
                       L 
                     
                     
                       
                         
                           U 
                           lj 
                         
                         
                           U 
                           
                             lk 
                             i 
                           
                         
                       
                       ⁢ 
                       C 
                     
                   
                 
               
             
             , 
           
         
       
     
     for all weights j of the neuron except k i ; and (ii) setting all other weights of the pyramid neural network to 1; and (ii) generating ( 28194 ) weights for the trapezium neural network including (i) setting weights of each neuron i of the first layer of the trapezium neural network (considering the whole net, this is (p+1)th layer) according to the equation 
     
       
         
           
             
               
                 w 
                 
                   ik 
                   i 
                 
                 
                   ( 
                   
                     p 
                     + 
                     1 
                   
                   ) 
                 
               
               = 
               
                 
                   U 
                   
                     lk 
                     i 
                   
                 
                 C 
               
             
             , 
           
         
       
     
     and (ii) setting other weights of the trapezium neural network to 1. 
     Referring next to  FIG.  28 S , in some implementations, the neural network topology includes ( 28196 ) a multilayer perceptron with the K inputs, S layers, and L i=1,S  calculation neurons in i-th layer, and a weight matrix U i=1,S ∈R L     i     ×L     i-1    for the i-th layer, where L 0 =K. In such cases, transforming ( 28198 ) the neural network topology to the equivalent sparsely connected network of analog components includes: for each layer j ( 28200 ) of the S layers of the multilayer perceptron, constructing ( 28202 ) a respective pyramid-trapezium network PTNNX j  by performing the approximation transformation to a respective single layer perceptron consisting of L j-1  inputs, L j  output neurons, and a weight matrix U j ; and (ii) constructing ( 28204 ) the equivalent sparsely connected network by stacking each pyramid trapezium network (e.g., output of a pyramid trapezium network PTNNXj−1 is set as an input for PTNNXj). 
     Referring back to  FIG.  28 A , In some implementations, the method further includes generating ( 28016 ) a schematic model for implementing the equivalent sparsely connected network utilizing the weight matrix. 
     Example Methods of Calculating Resistance Values for Analog Hardware Realization of Trained Neural Networks 
       FIGS.  29 A- 29 F  show a flowchart of a method  2900  for hardware realization ( 2902 ) of neural networks according to hardware design constraints, according to some implementations. The method is performed ( 2904 ) at the computing device  200  (e.g., using the weight quantization module  238 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . 
     The method includes obtaining ( 2906 ) a neural network topology (e.g., the topology  224 ) and weights (e.g., the weights  222 ) of a trained neural network (e.g., the networks  220 ). In some implementations, weight quantization is performed during training. In some implementations, the trained neural network is trained ( 2908 ) so that each layer of the neural network topology has quantized weights (e.g., a particular value from a list of discrete values; e.g., each layer has only 3 weight values of +1, 0, −1). 
     The method also includes transforming ( 2910 ) the neural network topology (e.g., using the neural network transformation module  226 ) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. 
     The method also includes computing ( 2912 ) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. 
     The method also includes generating ( 2914 ) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix and represents a resistance value. 
     Referring next to  FIG.  29 B , in some implementations, generating the resistance matrix for the weight matrix includes a simplified gradient-descent based iterative method to find a resistor set. In some implementations, generating the resistance matrix for the weight matrix includes: (i) obtaining ( 2916 ) a predetermined range of possible resistance values {R min , R max } and selecting an initial base resistance value R base  within the predetermined range. For example, the range and the base resistance are selected according to values of elements of the weight matrix; the values are determined by the manufacturing process; ranges—resistors that can be actually manufactured; large resistors are not preferred; quantization of what can be actually manufactured. In some implementations, the predetermined range of possible resistance values includes ( 2918 ) resistances according to nominal series E24 in the range 100 KΩ to 1 MΩ; (ii) selecting ( 2920 ) a limited length set of resistance values, within the predetermined range, that provide most uniform distribution of possible weights 
     
       
         
           
             
               w 
               
                 i 
                 , 
                 j 
               
             
             = 
             
               
                 R 
                 
                   base 
                     
                 
               
               ( 
               
                 
                   1 
                   
                     R 
                     i 
                   
                 
                 - 
                 
                   1 
                   
                     R 
                     j 
                   
                 
               
               ) 
             
           
         
       
     
     within the range [−R base , R base ] for all combinations of {R i ,R j } within the limited length set of resistance values. In some implementations, weight values are outside this range, but the square average distance between weights within this range is minimum; (iii) selecting ( 2922 ) a resistance value R + =R − , from the limited length set of resistance values, either for each analog neuron or for each layer of the equivalent analog network, based on maximum weight of incoming connections and bias w max  of each neuron or for each layer of the equivalent analog network, such that R + =R −  is the closest resistor set value to R base *W max . In some implementations, R +  and R −  are chosen ( 2924 ) independently for each layer of the equivalent analog network. In some implementations, R +  and R −  are chosen ( 2926 ) independently for each analog neuron of the equivalent analog network; and (iv) for each element of the weight matrix, selecting ( 2928 ) a respective first resistance value R 1  and a respective second resistance value R 2  that minimizes an error according to equation 
     
       
         
           
             err 
             = 
             
               
                 
                   ( 
                   
                     
                       
                         R 
                         + 
                       
                       
                         R 
                         1 
                       
                     
                     + 
                     
                       
                         R 
                         - 
                       
                       
                         R 
                         2 
                       
                     
                   
                   ) 
                 
                 · 
                 
                   r 
                   err 
                 
               
               + 
               
                 
                   ❘ 
                   &#34;\[LeftBracketingBar]&#34; 
                 
                 
                   w 
                   - 
                   
                     
                       R 
                       + 
                     
                     
                       R 
                       1 
                     
                   
                   + 
                   
                     
                       R 
                       - 
                     
                     
                       R 
                       2 
                     
                   
                 
                 
                   ❘ 
                   &#34;\[RightBracketingBar]&#34; 
                 
               
             
           
         
       
     
     for all possible values of R 1  and R 2  within the predetermined range of possible resistance values. w is the respective element of the weight matrix, and r err  is a predetermined relative tolerance value for the possible resistance values. 
     Referring next to  FIG.  29 C , some implementations perform weight reduction. In some implementations, a first one or more weights of the weight matrix and a first one or more inputs represent ( 2930 ) one or more connections to a first operational amplifier of the equivalent analog network. The method further includes: prior to generating ( 2932 ) the resistance matrix, (i) modifying ( 2934 ) the first one or more weights by a first value (e.g., dividing the first one or more weights by the first value to reduce weight range, or multiplying the first one or more weights by the first value to increase weight range); and (ii) configuring ( 2936 ) the first operational amplifier to multiply, by the first value, a linear combination of the first one or more weights and the first one or more inputs, before performing an activation function. Some implementations perform the weight reduction so as to change multiplication factor of one or more operational amplifiers. In some implementations, the resistor values set produce weights of some range, and in some parts of this range the error will be higher than in others. Suppose there are only 2 nominals (e.g., 1Ω and 4Ω), these resistors can produce weights [−3; −0.75; 0; 0.75; 3]. Suppose the first layer of a neural network has weights of {0, 9} and the second layer has weights of {0, 1}, some implementations divide the first layer&#39;s weights by 3 and multiply the second layer&#39;s weights by 3 to reduce overall error. Some implementations consider restricting weight values during training, by adjusting loss function (e.g., using 11 or 12 regularizer), so that resulting network does not have weights too large for the resistor set. 
     Referring next to  FIG.  29 D , the method further includes restricting weights to intervals. For example, the method further includes obtaining ( 2938 ) a predetermined range of weights, and updating ( 2940 ) the weight matrix according to the predetermined range of weights such that the equivalent analog network produces similar output as the trained neural network for same input. 
     Referring next to  FIG.  29 E , the method further includes reducing weight sensitivity of network. For example, the method further includes retraining ( 2942 ) the trained neural network to reduce sensitivity to errors in the weights or the resistance values that cause the equivalent analog network to produce different output compared to the trained neural network. In other words, some implementations include additional training for an already trained neural network in order to give it less sensitivity to small randomly distributed weight errors. Quantization and resistor manufacturing produce small weight errors. Some implementations transform networks so that the resultant network is less sensitive to each particular weight value. In some implementations, this is performed by adding a small relative random value to each signal in at least some of the layers during training (e.g., similar to a dropout layer). 
     Referring next to  FIG.  29 F , some implementations include reducing weight distribution range. Some implementations include retraining ( 2944 ) the trained neural network so as to minimize weight in any layer that are more than mean absolute weight for that layer by larger than a predetermined threshold. Some implementations perform this step via retraining. Example penalty function include a sum over all layers (e.g., A*max(abs(w))/mean(abs(w)), where max and mean are calculated over a layer. Another example include order of magnitude higher and above. In some implementations, this function impacts weight quantization and network weight sensitivity. For e.g., small relative changes of weights due to quantization might cause high output error. Example techniques include introducing some penalty functions during training that penalize network when it has such weight outcasts. 
     Example Methods of Optimizations for Analog Hardware Realization of Trained Neural Networks 
       FIGS.  30 A- 30 M  show a flowchart of a method  3000  for hardware realization ( 3002 ) of neural networks according to hardware design constraints, according to some implementations. The method is performed ( 3004 ) at the computing device  200  (e.g., using the analog neural network optimization module  246 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . 
     The method includes obtaining ( 3006 ) a neural network topology (e.g., the topology  224 ) and weights (e.g., the weights  222 ) of a trained neural network (e.g., the networks  220 ). 
     The method also includes transforming ( 3008 ) the neural network topology (e.g., using the neural network transformation module  226 ) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron of the equivalent analog network, and each resistor represents a connection between two analog neurons. 
     Referring next to  FIG.  30 L , in some implementations, the method further includes pruning the trained neural network. In some implementations, the method further includes pruning ( 3052 ) the trained neural network to update the neural network topology and the weights of the trained neural network, prior to transforming the neural network topology, using pruning techniques for neural networks, so that the equivalent analog network includes less than a predetermined number of analog components. In some implementations, the pruning is performed ( 3054 ) iteratively taking into account accuracy or a level of match in output between the trained neural network and the equivalent analog network. 
     Referring next to  FIG.  30 M , in some implementations, the method further includes, prior to transforming the neural network topology to the equivalent analog network, performing ( 3056 ) network knowledge extraction. Knowledge extraction is unlike stochastic/learning like pruning, but more deterministic than pruning. In some implementations, knowledge extraction is performed independent of the pruning step. In some implementations, prior to transforming the neural network topology to the equivalent analog network, connection weights are adjusted according to predetermined optimality criteria (such as preferring zero weights, or weights in a particular range, over other weights) through methods of knowledge extraction, by derivation of causal relationships between inputs and outputs of hidden neurons. Conceptually, in a single neuron or a set of neurons, on particular data set, there might be causal relationships between inputs and outputs which allows readjustment of weights in such a manner, that (1) new set of weights produces the same network output, and (2) new set of weights is easier to implement with resistors (e.g., more uniformly distributed values, more zero values or no connection). For example, if some neuron output is always 1 on some dataset, some implementations remove this neuron&#39;s output connections (and the neuron as a whole), and instead adjust bias weight of the neurons following the neuron. In this way, knowledge extraction step is different to pruning, because pruning requires re-learning after removing a neuron, and learning is stochastic, while knowledge extraction is deterministic. 
     Referring back to  FIG.  30 A , the method also includes computing ( 3010 ) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. 
     Referring next to  FIG.  30 J , in some implementations, the method further includes removing or transforming neurons based on bias values. In some implementations, the method further includes, for each analog neuron of the equivalent analog network: (i) computing ( 3044 ) a respective bias value for the respective analog neuron based on the weights of the trained neural network, while computing the weight matrix; (ii) in accordance with a determination that the respective bias value is above a predetermined maximum bias threshold, removing ( 3046 ) the respective analog neuron from the equivalent analog network; and (iii) in accordance with a determination that the respective bias value is below a predetermined minimum bias threshold, replacing ( 3048 ) the respective analog neuron with a linear junction in the equivalent analog network. 
     Referring next to  FIG.  30 K , in some implementations, the method further includes minimizing number of neurons or compacting the network. In some implementations, the method further includes reducing ( 3050 ) number of neurons of the equivalent analog network, prior to generating the weight matrix, by increasing number of connections (inputs and outputs) from one or more analog neurons of the equivalent analog network. 
     Referring back to  FIG.  30 A , the method also includes generating ( 3012 ) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. 
     The method also includes pruning ( 3014 ) the equivalent analog network to reduce number of the plurality of operational amplifiers or the plurality of resistors, based on the resistance matrix, to obtain an optimized analog network of analog components. 
     Referring next to  FIG.  30 B , in some implementations, the method includes substituting insignificant resistances with conductors. In some implementations, pruning the equivalent analog network includes substituting ( 3016 ), with conductors, resistors corresponding to one or more elements of the resistance matrix that have resistance values below a predetermined minimum threshold resistance value. 
     Referring next to  FIG.  30 C , in some implementations, the method further includes removing connections with very high resistances. In some implementations, pruning the equivalent analog network includes removing ( 3018 ) one or more connections of the equivalent analog network corresponding to one or more elements of the resistance matrix that are above a predetermined maximum threshold resistance value. 
     Referring next to  FIG.  30 D , in some implementations, pruning the equivalent analog network includes removing ( 3020 ) one or more connections of the equivalent analog network corresponding to one or more elements of the weight matrix that are approximately zero. In some implementations, pruning the equivalent analog network further includes removing ( 3022 ) one or more analog neurons of the equivalent analog network without any input connections. 
     Referring next to  FIG.  30 E , in some implementations, the method includes removing unimportant neurons. In some implementations, pruning the equivalent analog network includes (i) ranking ( 3024 ) analog neurons of the equivalent analog network based on detecting use of the analog neurons when making calculations for one or more data sets. For example, training data set used to train the trained neural network; typical data sets; data sets developed for pruning procedure. Some implementations perform ranking of neurons for pruning based on frequency of use of given neuron or block of neurons when subjected to training data set. For example, (a) if there is no signal at given neuron never, when using test data set—meaning this neuron or block of neurons was never in use and are pruned; (b) if the frequency of use of the neuron is very low, then the neuron is pruned without significant loss of accuracy; and (c) the neuron is always in use, then the neuron cannot be pruned); (ii) selecting ( 3026 ) one or more analog neurons of the equivalent analog network based on the ranking; and (iii) removing ( 3028 ) the one or more analog neurons from the equivalent analog network. 
     Referring next to  FIG.  30 F , in some implementations, detecting use of the analog neurons includes: (i) building ( 3030 ) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring ( 3032 ) propagation of analog signals (currents) by using the model (remove the blocks where the signal is not propagating when using special training sets) to generate calculations for the one or more data sets. 
     Referring next to  FIG.  30 G , in some implementations, detecting use of the analog neurons includes: (i) building ( 3034 ) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring ( 3036 ) output signals (currents or voltages) of the model (e.g., signals at outputs of some blocks or amplifiers in SPICe model or in real circuit, and deleting the areas where output signal for training set is always zero volts) by using the model to generate calculations for the one or more data sets. 
     Referring next to  FIG.  30 H , in some implementations, detecting use of the analog neurons includes: (i) building ( 3038 ) a model of the equivalent analog network using a modelling software (e.g., SPICe or similar software); and (ii) measuring ( 3040 ) power consumed by the analog neurons (e.g., power consumed by certain neurons or blocks of neurons, represented by operational amplifiers either in a SPICE model or in real circuit and deleting the neurons or blocks of neurons which did not consume any power) by using the model to generate calculations for the one or more data sets. 
     Referring next to  FIG.  30 I , in some implementations, the method further includes, subsequent to pruning the equivalent analog network, and prior to generating one or more lithographic masks for fabricating a circuit implementing the equivalent analog network, recomputing ( 3042 ) the weight matrix for the equivalent analog network and updating the resistance matrix based on the recomputed weight matrix. 
     Example Analog Neuromorphic Integrated Circuits and Fabrication Methods 
     Example Methods for Fabricating Analog Integrated Circuits for Neural Networks 
       FIGS.  31 A- 31 Q  show a flowchart of a method  3100  for fabricating an integrated circuit  3102  that includes an analog network of analog components, according to some implementations. The method is performed at the computing device  200  (e.g., using the IC fabrication module  258 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . The method includes obtaining ( 3104 ) a neural network topology and weights of a trained neural network. 
     The method also includes transforming ( 3106 ) the neural network topology (e.g., using the neural network transformation module  226 ) to an equivalent analog network of analog components including a plurality of operational amplifiers and a plurality of resistors (for recurrent neural networks, also use signal delay lines, multipliers, Tanh analog block, Sigmoid Analog Block). Each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. 
     The method also includes computing ( 3108 ) a weight matrix for the equivalent analog network based on the weights of the trained neural network. Each element of the weight matrix represents a respective connection. 
     The method also includes generating ( 3110 ) a resistance matrix for the weight matrix. Each element of the resistance matrix corresponds to a respective weight of the weight matrix. 
     The method also includes generating ( 3112 ) one or more lithographic masks (e.g., generating the masks  250  and/or  252  using the mask generation module  248 ) for fabricating a circuit implementing the equivalent analog network of analog components based on the resistance matrix, and fabricating ( 3114 ) the circuit (e.g., the ICs  262 ) based on the one or more lithographic masks using a lithographic process. 
     Referring next to  FIG.  31 B , in some implementations, the integrated circuit further includes one or more digital to analog converters ( 3116 ) (e.g., the DAC converters  260 ) configured to generate analog input for the equivalent analog network of analog components based on one or more digital signals (e.g., signals from one or more CCD/CMOS image sensors). 
     Referring next to  FIG.  31 C , in some implementations, the integrated circuit further includes an analog signal sampling module ( 3118 ) configured to process 1-dimensional or 2-dimensional analog inputs with a sampling frequency based on number of inferences of the integrated circuit (number of inferences for the IC is determined by product Spec—we know sampling rate from Neural Network operation and exact task the chip is intended to solve). 
     Referring next to  FIG.  31 D , in some implementations, the integrated circuit further includes a voltage converter module ( 3120 ) to scale down or scale up analog signals to match operational range of the plurality of operational amplifiers. 
     Referring next to  FIG.  31 E , in some implementations, the integrated circuit further includes a tact signal processing module ( 3122 ) configured to process one or more frames obtained from a CCD camera. 
     Referring next to  FIG.  31 F , in some implementations, the trained neural network is a long short-term memory (LSTM) network, AND the integrated circuit further includes one or more clock modules to synchronize signal tacts and to allow time series processing. 
     Referring next to  FIG.  31 G , in some implementations, the integrated circuit further includes one or more analog to digital converters ( 3126 ) (e.g., the ADC converters  260 ) configured to generate digital signal based on output of the equivalent analog network of analog components. 
     Referring next to  FIG.  31 H , in some implementations, the integrated circuit includes one or more signal processing modules ( 3128 ) configured to process 1-dimensional or 2-dimensional analog signals obtained from edge applications. 
     Referring next to  FIG.  31 I , the trained neural network is trained ( 3130 ), using training datasets containing signals of arrays of gas sensors (e.g., 2 to 25 sensors) on different gas mixture, for selective sensing of different gases in a gas mixture containing predetermined amounts of gases to be detected (in other words, the operation of trained chip is used to determine each of known to neural network gases in the gas mixture individually, despite the presence of other gases in the mixture). In some implementations, the neural network topology is a 1-Dimensional Deep Convolutional Neural network (1D-DCNN) designed for detecting 3 binary gas components based on measurements by 16 gas sensors, and includes ( 3132 ) 16 sensor-wise 1-D convolutional blocks, 3 shared or common 1-D convolutional blocks and 3 dense layers. In some implementations, the equivalent analog network includes ( 3134 ): (i) a maximum of 100 input and output connections per analog neuron, (ii) delay blocks to produce delay by any number of time steps, (iii) a signal limit of 5, (iv) 15 layers, (v) approximately 100,000 analog neurons, and (vi) approximately 4,900,000 connections. 
     Referring next to  FIG.  31 J , the trained neural network is trained ( 3136 ), using training datasets containing thermal aging time series data for different MOSFETs (e.g., NASA MOSFET dataset that contains thermal aging time series for 42 different MOSFETs; data is sampled every 400 ms and typically several hours of data for each device), for predicting remaining useful life (RUL) of a MOSFET device. In some implementations, the neural network topology includes ( 3138 ) 4 LSTM layers with 64 neurons in each layer, followed by two dense layers with 64 neurons and 1 neuron, respectively. In some implementations, the equivalent analog network includes ( 3140 ): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 18 layers, (iv) between 3,000 and 3,200 analog neurons (e.g., 3137 analog neurons), and (v) between 123,000 and 124,000 connections (e.g., 123,200 connections). 
     Referring next to  FIG.  31 K , the trained neural network is trained ( 3142 ), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery), for monitoring state of health (SOH) and state of charge (SOC) of Lithium Ion batteries to use in battery management systems (BMS). In some implementations, the neural network topology includes ( 3144 ) an input layer, 2 LSTM layers with 64 neurons in each layer, followed by an output dense layer with 2 neurons for generating SOC and SOH values. The equivalent analog network includes ( 3146 ): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 9 layers, (iv) between 1,200 and 1,300 analog neurons (e.g., 1271 analog neurons), and (v) between 51,000 and 52,000 connections (e.g., 51,776 connections). 
     Referring next to  FIG.  31 L , the trained neural network is trained ( 3148 ), using training datasets containing time series data including discharge and temperature data during continuous usage of different commercially available Li-Ion batteries (e.g., NASA battery usage dataset; the dataset presents data of continuous usage of 6 commercially available Li-Ion batteries; network operation is based on analysis of discharge curve of battery), for monitoring state of health (SOH) of Lithium Ion batteries to use in battery management systems (BMS). In some implementations, the neural network topology includes ( 3150 ) an input layer with 18 neurons, a simple recurrent layer with 100 neurons, and a dense layer with 1 neuron. In some implementations, the equivalent analog network includes ( 3152 ): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 4 layers, (iv) between 200 and 300 analog neurons (e.g., 201 analog neurons), and (v) between 2,200 and 2,400 connections (e.g., 2,300 connections). 
     Referring next to  FIG.  31 M , the trained neural network is trained ( 3154 ), using training datasets containing speech commands (e.g., Google Speech Commands Dataset), for identifying voice commands (e.g., 10 short spoken keywords, including “yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”). In some implementations, the neural network topology is ( 3156 ) a Depthwise Separable Convolutional Neural Network (DS-CNN) layer with 1 neuron. In some implementations, the equivalent analog network includes ( 3158 ): (i) a maximum of 100 input and output connections per analog neuron, (ii) a signal limit of 5, (iii) 13 layers, (iv) approximately 72,000 analog neurons, and (v) approximately 2.6 million connections. 
     Referring next to  FIG.  31 N , the trained neural network is trained ( 3160 ), using training datasets containing photoplethysmography (PPG) data, accelerometer data, temperature data, and electrodermal response signal data for different individuals performing various physical activities for a predetermined period of times and reference heart rate data obtained from ECG sensor (e.g., PPG data from the PPG-Dalia dataset (CHECK LICENSE). Data is collected for 15 individuals performing various physical activities during 1-4 hours each. Wrist-based sensor data contains PPG, 3-axis accelerometer, temperature and electrodermal response signals sampled from 4 to 64 Hz, and a reference heartrate data obtained from ECG sensor with sampling around 2 Hz. Original data was split into sequences of 1000 timesteps (around 15 seconds), with a shift of 500 timesteps, thus getting 16541 samples total. Dataset was split into 13233 training samples and 3308 test samples), for determining pulse rate during physical exercises (e.g., jogging, fitness exercises, climbing stairs) based on PPG sensor data and 3-axis accelerometer data. The neural network topology includes ( 3162 ) two Conv1D layers each with 16 filters and a kernel of 20, performing time series convolution, two LSTM layers each with 16 neurons, and two dense layers with 16 neurons and 1 neuron, respectively. In some implementations, the equivalent analog network includes ( 3164 ): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) a signal limit of 5, (iv) 16 layers, (v) between 700 and 800 analog neurons (e.g., 713 analog neurons), and (vi) between 12,000 and 12,500 connections (e.g., 12,072 connections). 
     Referring next to  FIG.  31 O , the trained neural network is trained ( 3166 ) to classify different objects (e.g., humans, cars, cyclists, scooters) based on pulsed Doppler radar signal (remove clutter and provide noise to Doppler radar signal), and the neural network topology includes ( 3168 ) multi-scale LSTM neural network. 
     Referring next to  FIG.  31 P , the trained neural network is trained ( 3170 ) to perform human activity type recognition (e.g., walking, running, sitting, climbing stairs, exercising, activity tracking), based on inertial sensor data (e.g., 3-axes accelerometers, magnetometers, or gyroscope data, from fitness tracking devices, smart watches or mobile phones; 3-axis accelerometer data as input, sampled at up to 96 Hz frequency. Network was trained on 3 different publicly available datasets, presenting such activities as “open then close the dishwasher”, “drink while standing”, “close left hand door”, “jogging”, “walking”, “ascending stairs” etc.). In some implementations, the neural network topology includes ( 3172 ) three channel-wise convolutional networks each with a convolutional layer of 12 filters and a kernel dimension of 64, and each followed by a max pooling layer, and two common dense layers of 1024 neurons and N neurons, respectively, where N is a number of classes. In some implementations, the equivalent analog network includes ( 3174 ): (i) delay blocks to produce any number of time steps, (ii) a maximum of 100 input and output connections per analog neuron, (iii) an output layer of 10 analog neurons, (iv) signal limit of 5, (v) 10 layers, (vi) between 1,200 and 1,300 analog neurons (e.g., 1296 analog neurons), and (vi) between 20,000 and 21,000 connections (e.g., 20,022 connections). 
     Referring next to  FIG.  31 Q , the trained neural network is further trained ( 3176 ) to detect abnormal patterns of human activity based on accelerometer data that is merged with heart rate data using a convolution operation (so as to detect pre-stroke or pre-heart attack states or signal in case of sudden abnormal patterns, caused by injuries or malfunction due to medical reasons, like epilepsy, etc). 
     Some implementations include components that are not integrated into the chip (i.e., these are external elements, connected to the chip) selected from the group consisting of: voice recognition, video signal processing, image sensing, temperature sensing, pressure sensing, radar processing, LIDAR processing, battery management, MOSFET circuits current and voltage, accelerometers, gyroscopes, magnetic sensors, heart rate sensors, gas sensors, volume sensors, liquid level sensors, GPS satellite signal, human body conductance sensor, gas flow sensor, concentration sensor, pH meter, and IR vision sensors. 
     Examples of analog neuromorphic integrated circuits manufactured according to the processes described above are provided in the following section, according to some implementations. 
     Example Analog Neuromorphic IC for Keyword Spotting 
     In some implementations, a neuromorphic IC is manufactured according to the processes described above. The neuromorphic IC can be used for keyword spotting. 
     The input network is a neural network with 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio mel-spectrogram of size 49 times 10. In some implementations, the network includes 5 convolutional layers, 4 depthwise convolutional layers, an average pooling layer, and a final dense layer. 
     In some implementations, the networks are pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”) from Google Speech Commands Dataset, with a recognition accuracy of 94.4%. 
     In some implementations, the Integration Circuit is manufactured based on Depthwise Separable Convolutional Neural Network (DS-CNN) for the voice command identification. In some implementations, the original DS-CNN network is T-transformed with following parameters: maximum input and output connections per neuron=100, signal limit of 5. In some implementations, the resulting T-network had following properties: 13 layers, approximately 72,000 neurons, and approximately 2.6 million connections. 
     Example DS-CNN Keyword Spotting Network 
     In one instance, a keyword spotting network is transformed to a T-network, according to some implementations. The network is a neural network of 2-D Convolutional and 2-D Depthwise Convolutional layers, with input audio spectrogram of size 49×10. Network consists of 5 convolutional layers, 4 depthwise convolutional layers, average pooling layer and final dense layer. Network is pre-trained to recognize 10 short spoken keywords (yes”, “no”, “up”, “down”, “left”, “right”, “on”, “off”, “stop”, “go”) from Google Speech Commands Dataset https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html. There are 2 additional classes which correspond to ‘silence’ and ‘unknown’. Network output is a softmax of length 12. 
     The trained neural network (input to the transformation) had a recognition accuracy of 94.4%, according to some implementations. In the neural network topology, each convolutional layer is followed with BatchNorm layer and ReLU layer, and ReLU activations are unbounded, and included around 2.5 million multiply-add operations. 
     After transformation, the transformed analog network was tested with a test set of 1000 samples (100 of each spoken command). All test samples are also used as test samples in the original dataset. Original DS-CNN network gave close to 5.7% recognition error for this test set. Network was converted to a T-network of trivial neurons. BatchNormalization layers in ‘test’ mode produce simple linear signal transformation, so can be interpreted as weight multiplier+some additional bias. Convolutional, AveragePooling and Dense layers are T-transformed quite straightforwardly. Softmax activation function was not implemented in T-network and was applied to T-network output separately. 
     Resulting T-network had 12 layers including an Input layer, approximately 72,000 neurons and approximately 2.5 million connections. 
       FIGS.  26 A- 26 K  show example histograms  2600  for absolute weights for the layers 1 through 11, respectively, according to some implementations. The weight distribution histogram (for absolute weights) was calculated for each layer. The dashed lines in the charts correspond to a mean absolute weight value for the respective layer. After conversion (i.e., T transformation), the average output absolute error (calculated over test set) of converted network vs original is calculated to be 4.1e-9. 
     Various examples for setting network limitations for the transformed network are described herein, according to some implementations. For signal limit, as ReLU activations used in the network are unbounded, and some implementations use a signal limit on each layer. 
     This could potentially affect mathematical equivalence. For this, some implementations use a signal limit of 5 on all layers which corresponds to power voltage of 5 in relation to input signal range. 
     For quantizing the weights, some implementations use a nominal set of 30 resistors [0.001, 0.003, 0.01, 0.03, 0.1, 0.324, 0.353, 0.436, 0.508, 0.542, 0.544, 0.596, 0.73, 0.767, 0.914, 0.985, 0.989, 1.043, 1.101, 1.149, 1.157, 1.253, 1.329, 1.432, 1.501, 1.597, 1.896, 2.233, 2.582, 2.844]. 
     Some implementations select R− and R+ values (see description above) separately for each layer. For each layer, some implementations select a value which delivers most weight accuracy. In some implementations, subsequently all the weights (including bias) in the T-network are quantized (e.g., set to the closest value which can be achieved with the input or chosen resistors). 
     Some implementations convert the output layer as follows. Output layer is a dense layer that does not have ReLU activation. The layer has softmax activation which is not implemented in T-conversion and is left for digital part, according to some implementations. Some implementations perform no additional conversion. 
     Example Transformation of Modular Net Structure for Generating Libraries 
     A modular structure of converted neural networks is described herein, according to some implementations. Each module of a modular type neural network is obtained after transformation of (a whole or a part of) one or more trained neural network. In some implementations, the one or more trained neural networks is subdivided into parts, and then subsequently transformed into an equivalent analog network. Modular structure is typical for some of the currently used neural networks, and modular division of neural networks corresponds to a trend in neural network development. Each module can have an arbitrary number of inputs or connections of input neurons to output neurons of a connected module, and an arbitrary number of outputs connected to input layers of a subsequent module. In some implementations, a library of preliminary (or a seed list of) transformed modules is developed, including lithographic masks for manufacture of each module. A final chip design is obtained as a combination of (or by connecting) preliminary developed modules. Some implementations perform commutation between the modules. In some implementations, the neurons and connections within the module are translated into chip design using ready-made module design templates. This significantly simplifies the manufacture of the chip, accomplished by just connecting corresponding modules. 
     Some implementations generate libraries of ready-made T-converted neural networks and/or T-converted modules. For example, a layer of CNN network is a modular building block, LSTM chain is another building block, etc. Larger neural networks NNs also have modular structure (e.g., LSTM module and CNN module). In some implementations, libraries of neural networks are more than by-products of the example processes, and can be sold independently. For example, a third-party can manufacture a neural network starting with the analog circuits, schematics, or designs in the library (e.g., using CADENCE circuits, files and/or lithography masks). Some implementations generate T-converted neural networks (e.g., networks transformable to CADENCE or similar software) for typical neural networks, and the converted neural networks (or the associated information) are sold to a third-party. In some instances, a third-party chooses not to disclose structure and/or purpose of the initial neural network, but uses the conversion software (e.g., SDK described above) to converts the initial network into trapezia-like networks and passes the transformed networks to a manufacturer to the fabricate the transformed network, with a matrix of weights obtained using one of the processes described above, according to some implementations. As another example, where the library of ready-made networks are generated according to the processes described herein, corresponding lithographic masks are generated and a customer can train one of the available network architectures for his task, perform lossless transformation (sometimes called T transformation) and provide the weights to a manufacturer for fabricating a chip for the trained neural networks. 
     In some implementations, the modular structure concept is also used in the manufacture of multi-chip systems or the multi-level 3D chips, where each layer of the 3D chip represents one module. The connections of outputs of modules to the inputs of connected modules in case of 3D chips will be made by standard interconnects that provide ohmic contacts of different layers in multi-layer 3D chip systems. In some implementations, the analog outputs of certain modules is connected to analog inputs of connected modules through interlayer interconnects. In some implementations, the modular structure is used to make multi-chip processor systems as well. A distinctive feature of such multi-chip assemblies is the analog signal data lines between different chips. The analog commutation schemes, typical for compressing several analog signals into one data line and corresponding de-commutation of analog signals at receiver chip, is accomplished using standard schemes of analog signal commutation and de-commutation, developed in analog circuitry. 
     One main advantage of a chip manufactured according to the techniques described above, is that analog signal propagation can be broadened to multi-layer chips or multi-chip assemblies, where all signal interconnects and data lines transfer analog signals, without a need for analog-to-digital or digital-to-analog conversion. In this way, the analog signal transfer and processing can be extended to 3D multi-layer chips or multi-chip assemblies. 
     Example Methods for Generating Libraries for Hardware Realization of Neural Networks 
       FIGS.  32 A- 32 E  show a flowchart of a method  3200  for generating ( 3202 ) libraries for hardware realization of neural networks, according to some implementations. The method is performed ( 3204 ) at the computing device  200  (e.g., using the library generation module  254 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . 
     The method includes obtaining ( 3206 ) a plurality of neural network topologies (e.g., the topologies  224 ), each neural network topology corresponding to a respective neural network (e.g., a neural network  220 ). 
     The method also includes transforming ( 3208 ) each neural network topology (e.g., using the neural network transformation module  226 ) to a respective equivalent analog network of analog components. 
     Referring next to  FIG.  32 D , in some implementations, transforming ( 3230 ) a respective network topology to a respective equivalent analog network includes: (i) decomposing ( 3232 ) the respective network topology to a plurality of subnetwork topologies. In some implementations, decomposing the respective network topology includes identifying ( 3234 ) one or more layers (e.g., LSTM layer, fully connected layer) of the respective network topology as the plurality of subnetwork topologies; (ii) transforming ( 3236 ) each subnetwork topology to a respective equivalent analog subnetwork of analog components; and (iii) composing ( 3238 ) each equivalent analog subnetwork to obtain the respective equivalent analog network. 
     Referring back to  FIG.  32 A , the method also includes generating ( 3210 ) a plurality of lithographic masks (e.g., the masks  256 ) for fabricating a plurality of circuits, each circuit implementing a respective equivalent analog network of analog components. 
     Referring next to  FIG.  32 E , in some implementations, each circuit is obtained by: (i) generating ( 3240 ) schematics for a respective equivalent analog network of analog components; and (ii) generating ( 3242 ) a respective circuit layout design based on the schematics (using special software, e.g., CADENCE). In some implementations, the method further includes combining ( 3244 ) one or more circuit layout designs prior to generating the plurality of lithographic masks for fabricating the plurality of circuits. 
     Referring next to  FIG.  32 B , in some implementations, the method further includes: (i) obtaining ( 3212 ) a new neural network topology and weights of a trained neural network; (ii) selecting ( 3214 ) one or more lithographic masks from the plurality of lithographic masks based on comparing the new neural network topology to the plurality of neural network topologies. In some implementations, the new neural network topology includes a plurality of subnetwork topologies, and selecting the one or more lithographic masks is further based on comparing ( 3216 ) each subnetwork topology with each network topology of the plurality of network topologies; (iii) computing ( 3218 ) a weight matrix for a new equivalent analog network based on the weights; (iv) generating ( 3220 ) a resistance matrix for the weight matrix; and (v) generating ( 3222 ) a new lithographic mask for fabricating a circuit implementing the new equivalent analog network based on the resistance matrix and the one or more lithographic masks. 
     Referring next to  FIG.  32 C , one or more subnetwork topologies of the plurality of subnetwork topologies fails to compare ( 3224 ) with any network topology of the plurality of network topologies, and the method further includes: (i) transforming ( 3226 ) each subnetwork topology of the one or more subnetwork topologies to a respective equivalent analog subnetwork of analog components; and generating ( 3228 ) one or more lithographic masks for fabricating one or more circuits, each circuit of the one or more circuits implementing a respective equivalent analog subnetwork of analog components. 
     Example Methods for Optimizing Enemy Efficiency of Neuromorphic Analog Integrated Circuits 
       FIGS.  33 A- 33 J  show a flowchart of a method  3300  for optimizing ( 3302 ) energy efficiency of analog neuromorphic circuits (that model trained neural networks), according to some implementations. The method is performed ( 3204 ) at the computing device  200  (e.g., using the energy efficiency optimization module  264 ) having one or more processors  202 , and memory  214  storing one or more programs configured for execution by the one or more processors  202 . 
     The method includes obtaining ( 3306 ) an integrated circuit (e.g., the ICs  262 ) implementing an analog network (e.g., the transformed analog neural network  228 ) of analog components including a plurality of operational amplifiers and a plurality of resistors. The analog network represents a trained neural network (e.g., the neural networks  220 ), each operational amplifier represents a respective analog neuron, and each resistor represents a respective connection between a respective first analog neuron and a respective second analog neuron. 
     The method also includes generating ( 3308 ) inferences (e.g., using the inferencing module  266 ) using the integrated circuit for a plurality of test inputs, including simultaneously transferring signals from one layer to a subsequent layer of the analog network. In some implementations, the analog network has layered structure, with the signals simultaneously coming from previous layer to the next one. During inference process, the signals propagate through the circuit layer by layer; simulation at device level; time delays every minute. 
     The method also includes, while generating inferences using the integrated circuit, determining ( 3310 ) if a level of signal output of the plurality of operational amplifiers is equilibrated (e.g., using the signal monitoring module  268 ). Operational amplifiers go through a transient period (e.g., a period that lasts less than 1 millisecond from transient to plateau signal) after receiving inputs, after which the level of signal is equilibrated and does not change. In accordance with a determination that the level of signal output is equilibrated, the method also includes: (i) determining ( 3312 ) an active set of analog neurons of the analog network influencing signal formation for propagation of signals. The active set of neurons need not be part of a layer/layers. In other words, the determination step works regardless of whether the analog network includes layers of neurons; and (ii) turning off power ( 3314 ) (e.g., using the power optimization module  270 ) for one or more analog neurons of the analog network, distinct from the active set of analog neurons, for a predetermined period of time. For example, some implementations switch off power (e.g., using the power optimization module  270 ) of operational amplifiers which are in layers behind an active layer (to where signal propagated at the moment), and which do not influence the signal formation on the active layer. This can be calculated based on RC delays of signal propagation through the IC. So all the layers behind the operational one (or the active layer) are switched off to save power. So the propagation of signals through the chip is like surfing—the wave of signal formation propagate through chip, and all layers which are not influencing signal formation are switched off. In some implementations, for layer-by-layer networks, signal propagates layer to layer, and the method further includes decreasing power consumption before a layer corresponding to the active set of neurons because there is no need for amplification before the layer. 
     Referring next to  FIG.  33 B , in some implementations, in some implementations, determining the active set of analog neurons is based on calculating ( 3316 ) delays of signal propagation through the analog network. Referring next to  FIG.  33 C , in some implementations, determining the active set of analog neurons is based on detecting ( 3318 ) the propagation of signals through the analog network. 
     Referring next to  FIG.  33 D , in some implementations, the trained neural network is a feed-forward neural network, and the active set of analog neurons belong to an active layer of the analog network, and turning off power includes turning off power ( 3320 ) for one or more layers prior to the active layer of the analog network. 
     Referring next to  FIG.  33 E , in some implementations, the predetermined period of time is calculated ( 3322 ) based on simulating propagation of signals through the analog network, accounting for signal delays (using special software, e.g., CADENCE). 
     Referring next to  FIG.  33 F , in some implementations, the trained neural network is ( 3324 ) a recurrent neural network (RNN), and the analog network further includes one or more analog components other than the plurality of operational amplifiers, and the plurality of resistors. In such cases, the method further includes, in accordance with a determination that the level of signal output is equilibrated, turning off power ( 3326 ) (e.g., using the power optimization module  270 ), for the one or more analog components, for the predetermined period of time. 
     Referring next to  FIG.  33 G , in some implementations, the method further includes turning on power ( 3328 )) (e.g., using the power optimization module  270 ) for the one or more analog neurons of the analog network after the predetermined period of time. 
     Referring next to  FIG.  33 H , in some implementations, determining if the level of signal output of the plurality of operational amplifiers is equilibrated is based on detecting ( 3330 ) if one or more operational amplifiers of the analog network is outputting more than a predetermined threshold signal level (e.g., power, current, or voltage). 
     Referring next to  FIG.  33 I , in some implementations, the method further includes repeating ( 3332 )) (e.g., by the power optimization module  270 ) the turning off for the predetermined period of time and turning on the active set of analog neurons for the predetermined period of time, while generating the inferences. 
     Referring next to  FIG.  33 J , in some implementations, the method further includes, in accordance with a determination that the level of signal output is equilibrated, for each inference cycle ( 3334 ): (i) during a first time interval, determining ( 3336 ) a first layer of analog neurons of the analog network influencing signal formation for propagation of signals; and (ii) turning off power ( 3338 )) (e.g., using the power optimization module  270 ) for a first one or more analog neurons of the analog network, prior to the first layer, for the predetermined period of time; and during a second time interval subsequent to the first time interval, turning off power ( 3340 )) (e.g., using the power optimization module  270 ) for a second one or more analog neurons including the first layer of analog neurons and the first one or more analog neurons of the analog network, for the predetermined period. 
     Referring next to  FIG.  33 K , in some implementations, the one or more analog neurons consist ( 3342 ) of analog neurons of a first one or more layers of the analog network, and the active set of analog neurons consist of analog neurons of a second layer of the analog network, and the second layer of the analog network is distinct from layers of the first one or more layers. 
     Some implementations include means for delaying and/or controlling signal propagation from layer to layer of the resulting hardware-implemented neural network. 
     Example Transformation of MobileNet v.1 
     An example transformation of MobileNet v.1 into an equivalent analog network is described herein, according to some implementations. In some implementations, single analog neurons are generated, then converted into SPICE schematics with a transformation of weights from MobileNet into resistor values. MobileNet v1 architecture is depicted in the Table shown in  FIG.  34   . In the Table, the first column  3402  corresponds to type of layer and stride, the second column  3404  corresponds to filter shape for the corresponding layer, and the third column  3406  corresponds to input size for the corresponding layer. In MobileNet v.1, each convolutional layer is followed by a batch normalization layer and a ReLU 6 activation function (y=max(0, min(6, x)). The network consists of 27 convolutional layers, 1 dense layer, and has around 600 million multiply-accumulate operations for a 224×224×3 input image. Output values are the result of softmax activation function which means the values are distributed in the range [0, 1] and the sum is 1. Some implementations accept as input MobileNet 32×32 and alpha=1 for the transformation. In some implementations, the network is pre-trained for CIFAR-10 task (50,000 32×32×3 images divided into 10 non-intersecting classes). Batch normalization layers operate in ‘test’ mode to produce simple linear signal transformation, so the layers are interpreted as weight multiplier+some additional bias. Convolutional, AveragePooling and Dense layers are transformed using the techniques described above, according to some implementations. In some implementations, Softmax activation function is not implemented in transformed network but applied to output of the transformed network (or the equivalent analog network) separately. 
     In some implementations, the resulting transformed network included 30 layers including an input layer, approximately 104,000 analog neurons, and approximately 11 million connections. After transformation, the average output absolute error (calculated over 100 random samples) of transformed network versus MobileNet v.1 was 4.9e-8. 
     As every convolutional and other layers of MobileNet have an activation function ReLU6, the output signal on each layer of the transformed network is also limited by the value 6. As part of the transformation, the weights are brought into accordance with a resistor nominal set. Under each nominal set, different weight values are possible. Some implementations use resistor nominal sets e24, e48 and e96, within the range of [0.1-1] Mega Ohm. Given that the weight ranges for each layer vary, and for most layers weight values do not exceed 1-2, in order to achieve more weight accuracy, some implementations decrease R− and R+ values. In some implementations, the R− and R+ values are chosen separately for each layer from the set [0.05, 0.1, 0.2, 0.5, 1] Mega Ohm. In some implementations, for each layer, a value which delivers most weight accuracy is chosen. Then all the weights (including bias) in the transformed network are ‘quantized’, i.e., set to the closest value which can be achieved with used resistors. In some implementations, this reduced transformed network accuracy versus original MobileNet according to the Table shown below. The Table shows mean square error of transformed network, when using different resistor sets, according to some implementations. 
     
       
         
           
               
               
               
             
               
                   
                   
               
               
                   
                 Resistor set 
                 Mean Square Error 
               
               
                   
                   
               
             
            
               
                   
               
            
           
           
               
               
               
            
               
                   
                 E24 0.1-1 MΩ 
                 0.01 
               
               
                   
                 E24 0.1-5 MΩ 
                 0.004 
               
               
                   
                 E48 0.1-1 MΩ 
                 0.007 
               
               
                   
                 E96 0.1-1 MΩ 
                 0.003 
               
               
                   
                   
               
            
           
         
       
     
     Example Analog Hardware Realization of Trained Neural Networks for Voice Clarity 
     Some implementations provide a method for fabricating a neuromorphic Integrated Circuit for voice clarification, using techniques described above. Various types of trained neural networks can be used for this purpose. For example, a neural network can be trained to identify only one voice, suppressing and removing everything else. In particular, the neural network can identify the voice that is the closest to the microphone. As another example, a neural network can be trained to identify several voices, suppressing and removing everything else. Voices can be identified and preserved regardless of their distance from the microphone(s). Alternatively, voices can be prioritized by their distances from the microphone(s) and given different weights in the output signal, based on their respective distances form the microphone. As another alternative, voices can be identified and preserved regardless of their relative strength (e.g., volume). As yet another alternative, voices can be prioritized by their relative strength and be given different weights in the output signal, based on their respective relative strength. A neural network can process the signal that is originating from the microphone(s). Such a signal may include analog and/or digital signals. A neural network can process an analog and/or a digital signal that is transmitted over a transmission media and received by the neural network. Such a signal can be transmitted across wireless or digital/internet networks for the purposes of phone communication. Such a signal can also be input after pre- and post-processing of the original voice(s), either before the signal is ready to be transmitted, or after the signal has been transmitted and delivered to the recipient. As another example, a neural network can process a signal that is a mix of several voice signals, with associated noises. In particular, such a mix can be delivered to the recipient from several different sources. Such a signal can be pre- and post-processed by different methods for different components. As yet another example, a neural network can process the signal that is a mix of several external voice signals, with associated noises, combined with the own voice(s) on the recipient side. In particular, such a mix can be delivered to the recipient from several different sources, including the recipient&#39;s own voice overlapped with recipient&#39;s own noises. Such a signal can be pre- and/or post-processed by different methods for different components. The clarification of voice(s) can be performed for the combined signal. As another example, a neural network can process a signal that includes voice(s) from the recipient side. In particular, such a signal can be processed before it is transmitted to the other party. Such a signal can be processed by the neural network before it is pre- and/or post-processed by different methods prior to transmission. 
     Example Methods for Extracting Voice from Inbound or Outbound Analog Noisy Signal 
     Described herein are example techniques for the extraction of voice from a noisy signal, both inbound and outbound, where noise can be either stationary or non-stationary, using a neuromorphic analog Integrated Circuit. Such a circuit implements a noise suppression neural network at the hardware level. The circuit design of the analog neuromorphic Integrated Circuit is realized by converting (using techniques described above) a noise suppression (or voice extraction) neural network. 
     As described in the Background and Summary sections, the task of extracting the voice from noisy signal is of great importance for communication in smartphones, smartwatches, notebooks, or other voice transmitting devices. There are conventional realizations of noise cancellation or active noise suppression using dual microphone scheme, where the signal from one microphone is used to cancel noise at a main microphone. But these solutions do not cancel all noises, especially non-stationary ones, since not all noise is canceled in such combination of two microphones. There are also filters, which can filter out stationary noise from inbound or outbound analog signal. There are also software realizations of neural networks, which extract voice from a noisy signal by converting some part of the signal using Fourier transformation, thereby reducing components that are not similar to voice. These products are realized as software applications, which can be installed on smartphones or notebook computers, and can effectively suppress noise coming from a microphone. However, such applications require high computational power and consequently lead to higher power consumption. Also, such applications require powerful processors, which cannot be installed in earbuds or other miniature devices. 
     Described herein are techniques for voice extraction using a specially designed Integrated Circuit, realized from a trained neural network. The Integrated Circuit is realized as a hardware solution and is represented by a set of operational amplifiers and resistors, connected in such a way that the resulting neuromorphic hardware chip operates similarly to the initial neural network (e.g., the neural network realized in software), with the absolute error not exceeding a maximum threshold percentage (e.g., 1% absolute) from the error corresponding to the software neural network. The schematics of the Integration Circuit are obtained using techniques described above, thus ensuring full equivalency of analog neuromorphic hardware realization of the neural network and its initial software neural network model. The analog Integrated Circuit may be used for voice extraction from noisy analog inbound or outbound signals, with low latency and low power consumption. 
     In some implementations, the hardware realization of a voice extraction neural network can be used to process both inbound and outbound noisy signals. In some implementations, the Integrated Circuit has direct analog input and is placed adjacent to a microphone or a speaker of a smartphone, smartwatch, earbuds, notebook computer, or similar device. The Integrated Circuit provides telecommunication voice transfer, extracting voice from noisy analog signals. Such a solution suppresses both stationary and non-stationary noise from inbound or outbound analog signals (e.g., signals from a microphone or signals directed to a speaker or earbuds) and is characterized by excellent noise suppression, unlike conventional methods. 
     The resulting hardware realization of a voice extraction algorithm is characterized by low power operation, small latency, and small die area, which makes analog hardware realization an advantageous solution for noise reduction in smartphones, earbuds, notebook computers, tablets, or other voice transmitting devices, in comparison with software neural network voice extraction algorithms. The small die area makes it possible to include the Integrated Circuit application in true wireless (TWS) earbuds or other miniature devices. Such analog Integrated Circuits may also be used for two-way voice extraction (noise reduction) in Notebook PCs or Smartphones, where a neuromorphic analog integration circuit is installed both at the analog output of the microphone and at the analog input of the speaker or earbuds. 
     Example Neuromorphic Analog Integrated Circuit for Voice Clarity 
     Some implementations obtain a convolutional neural network with 1D convolutions (e.g., as described in “Single Channel Speech Enhancement Using A Convolutional Neural Network,” by T. Kounovsky and J. Malek, 2017), an example of which is shown in  FIG.  35   . The architecture  3500  shown in  FIG.  35    performs Fourier transformation of an incoming analog signal, to obtain input features  3502  that form a network input  3504 . Subsequently, the architecture uses convolution  3506 , maxpooling  3508 , convolution  3510 , and fully connected layers (layers  3512  and  3514 ), to obtain the output  3516 . An inverse Fourier transformation is applied on the output  3516  to obtain an analog output signal. Some implementations convert this example network  3500  into a network of analog components using techniques described above and herein. Some implementations apply the techniques described above for fabricating a neuromorphic analog integrated circuit based on the network of analog components. Simulations have shown the resulting integrated circuit occupied a 30 square millimeter die area, consumed approximately 150 micro-Watts of power, and had a signal latency of 3 milliseconds. The resulting integrated circuit can be used for inbound and outbound voice extraction for smartphones, earbuds, notebook computers, smartwatches, or other telecommunication devices. In experiments, when the quality is measured according to PESQ criteria, an improvement of voice signal up to 30% was accomplished. The small power consumption allows the use of such integrated circuits in battery-powered devices. The small die area allows installing the device into TWS earbuds or other miniature devices. 
     The network architecture shown in  FIG.  35    includes convolution, max pooling, and fully-connected layers. Transformation of convolutional layers is described above, according to some implementations. The following sections describe example transformation techniques for various components of the network shown in  FIG.  35   , according to some implementations. 
     Example Transformation of Fully Connected (Dense) Layers 
       FIG.  36    shows an example transformation  3600  of a fully-connected or dense layer, according to some implementations. A dense layer of a neural network is represented using the formula  3620 , Output=f(Input·W+b), where the Input  3602  (corresponding to Input 1  3604 , Input 2  3606 , . . . , Input i  3608 ) and Output  3612  (corresponding to SNM 1  3614 , SNM 2  3616 , . . . , SNM j  3618 ) are vectors. W is a weight matrix  3610 , and each input is connected to every output. Each weight W[i][j] corresponds to an edge or a connection between the Input i and the Output j. Bias b is a bias vector and f(x) is an element-wise activation function, typically ReLU. This mathematical formula is transformed into a 2-layer fully connected mesh of SNMs using techniques described above, where the first layer represents the Input and the second layer represents the Output, according to some implementations. 
     Example Transformation of Conv1D Layers 
       FIG.  37    shows example transformations  3700  of a Conv1D layer of a Convolutional Neural Network, according to some implementations. The Conv1D layer of a neural network is a convolution applied over a 1-dimensional (e.g., the X dimension  3708 ) sequence of data (shown as input  3702 , which may include multiple input channels  3712 ) to obtain output  3704  (with output channels  3714 ). Each element of an output tensor is a product of a particular subset of the input data  3702  and a slice of a weight tensor called the kernel  3706  (sometimes called a filter kernel or a filter). For T-transforming the Conv1D layer, some implementations evaluate the output tensor shape and construct a layer of SNMs (e.g., the layer  3710 ) with the same shape (the number of SNMs equals the product of the tensor dimensions). Each of these SNMs has incoming connections weighted according to a relevant slice of the weight tensor and is connected to a relevant subset of input SNMs of the previous layer. The Output-to-Input SNMs relations may depend on Conv1D layer parameters, such as kernel size, stride, padding, and filters. The shape of the output layer SNMs is the same as output tensor shape in mathematical representation, where each SNM has a number of incoming weighted connections from the previous layer. 
     Example Transformation of MaxPooling Layers 
       FIG.  38    shows example transformations  3800  of a MaxPooling layer of a Convolutional Neural Network, according to some implementations. A MaxPooling layer performs a number of calculations of the form Output j =f(max{Input i }), where “max” refers to computing a maximum of a subset of input values. The subset of input values is defined by pool size and strides parameters (e.g., as specified by a machine learning framework where the original Convolutional Neural Network is defined). Some implementations define dependency relations between each output value and a related subset of input values, and for each output value, build a maximum subtree (sometimes called a Max subtree), thereby converting the MaxPooling layer into a multilayer SNM structure according to following algorithm:
         a) Define a schematic with 2 layers and 2 SNMs (e.g., SNM  3808  and SNM  3810 ) performing a max operation (e.g., the Max2 operation  3802 ) over 2 Input elements (e.g., Input 1  3804  and Input 2  3806 ).   b) Define a schematic with 3 layers and 3 SNMs (e.g., SNM  3820 , SNM  3822 , and SNM  3824 ) performing a max operation (e.g., the Max3 operation  3812 ) over 3 Input elements (e.g., Input 1  3814 , Input 2  3816 , and Input 3  3818 ).   c) Define a schematic with 3 layers and 4 SNMs (e.g., SNM  3836 , SNM  3838 , SNM  3840 , and SNM  3842 ) performing a max operation (e.g., the Max4 operation  3826 ) over 4 Input elements (e.g., Input 1  3828 , Input 2  3830 , Input 3  3832 , and Input 4  3834 ).   d) Because max({x_i}) is symmetric with respect to the arguments (e.g., max(x,y,z)=max(max(x,y), z)), perform transformation of the max({Input_i}) calculation into a calculation tree, where each tree node is a Max2, Max3, or Max4 schematic. This tree is built in a manner that minimizes total tree layers and prioritizes the use of the Max4 schematic, according to some implementations. For instance, max(1,2,3,4,5,6,7,8,9) is transformed into max(max(1,2,3,4), max(5,6,7,8), 9) producing a structure of 6 layers with 11 SNMs.   e) An activation function other than ReLU can be applied over the output neuron. ReLUs are applied over each SNM without changing the final output value.       

     Example Sound Signal Processing Using Neuromorphic Analog Signal Processors 
     Some implementations include a supervisor switch (sometimes referred to as a digital switch, an artificial intelligence (AI) sound supervisor, or an AI sound supervisor switch) for sound processing, multiplexing, and/or controlling, one or more analog neural cores that provide voice, music, and/or acoustic event detection, voice extraction from a noisy stream, voice command recognition, and/or speaker identification. In some implementations, a neurovoice processor includes one or more neuromorphic analog signal processors. 
     Described above are example techniques for voice extraction using a specially designed Integrated Circuit, realized from a trained neural network. The Integrated Circuit is realized as a hardware solution and is represented by a set of operational amplifiers and resistors, connected in such a way that the resulting neuromorphic hardware chip operates similarly to the initial neural network (e.g., the neural network realized in software), with the absolute error not exceeding a maximum threshold percentage (e.g., 1% absolute) from the error corresponding to the software neural network. The schematics of the Integrated Circuit are obtained using example techniques described above, thus ensuring full equivalency of analog neuromorphic hardware realization of the neural network and its initial software neural network model. The analog Integrated Circuit may be used for voice extraction from noisy analog inbound or outbound signals, with low latency and low power consumption. 
     In some implementations, the hardware realization of a voice extraction neural network is used to process both inbound and outbound noisy signals. In some implementations, the Integrated Circuit has direct analog input and is placed adjacent to a microphone or a speaker of a smartphone, smartwatch, earbuds, notebook computer, or similar device. The Integrated Circuit provides telecommunication voice transfer, extracting voice from noisy analog signals. Such a solution suppresses both stationary and non-stationary noise from inbound or outbound analog signals (e.g., signals from a microphone or signals directed to a speaker or earbuds) and is characterized by excellent noise suppression, unlike conventional methods. Figures described above provide example networks and transformation techniques for generating and/or fabricating circuits and devices using analog components, based on trained neural networks, according to some implementations. In particular,  FIGS.  35 - 38    provide specific examples for analog processors for performing voice clarification. It is noted that the neuromorphic analog signal processors described herein may each be similarly produced, starting with a respective trained neural network, transforming the respective trained neural network to an equivalent analog network of analog components. Such trained neural networks may include recurrent neural networks and convolutional neural networks, layers of such networks, combinations of such networks, including any other supporting components, as described above in reference to  FIGS.  35 - 38   . In particular, example Depthwise Separable Convolutional Neural Networks (DS-CNN) and associated transformations are described above in reference to a keyword spotting network, and example Recurrent Neural Networks (RNNs) and associated transformations are described above in reference to  FIGS.  4 B,  12 , and  28 O , according to some implementations. Each neuromorphic analog signal processor or core referred to herein may include any such network or combination of such networks. An example analog hardware realization of a trained DS-CNN network architecture for keyword spotting is described herein for illustration. Keyword spotting networks based on DS-CNN are known. An example is described in an article titled “Hello Edge: Keyword Spotting on Microcontrollers” by Zhang et al., available on arxiv.org. This network contains convolutional and max-pooling layers. The pre-trained network is converted to an analog network of analog components using techniques described above and weights provided by third-party provdiers, without any re-training. A subset of the dataset is used for testing in order to perform validation. Such networks may also contain pre-processing and post-processing steps, such as Fast Fourier Transform (FFT) and inverse FFT. FFT and inverse FFT are converted to analog form by decomposing the operations into scalar linear operations. In this way, an equivalent analog network for the pre-trained neural network architecture is obtained. The cores used for sound signal processing are based on such equivalent analog networks. 
     Some implementations include a music voice acoustic event detector, which is an AI-based low-power hardware programmed neuromorphic module, which analyzes a sound stream and provides a digital logical signal as output, to indicate if voice, music, or an acoustic event are present in the sound stream. Some implementations include a voice extractor, which is another AI-based neuromorphic low power module configured to enhance voice in an audio stream, and/or suppress noises. Some implementations include a keyword spotting module, which is an AI-based neuromorphic low power module that recognizes several spoken words in speech. Some implementations include a wake word detection module, which is an AI-based neuromorphic low power module that recognizes one or more wake words. Some implementations include a speaker identification module, which is an analog neuromorphic low power module that authenticates a speaker. 
     A sound supervisor (sometimes referred to as a supervisor switch or an AI sound supervisor) orchestrates voice processing blocks of a system to provide different features as output. There are several examples where the systems, methods, and techniques described herein may be advantageous. Consider a hearing aid where a user hears sounds enhanced for their hearing configured by certain digital signal processing (DSP) presets. Voices are cleared but background sound is left in suppressed form. When there is no voice, a user needs just environment sounds. Suppose the user needs to listen to music. With conventional systems, the user has to manually change a hearing aid DSP preset to hear music unchanged, and the user has to manually change the system settings to a voice preset after the music. Another problem with true wireless stereo (TWS) that is controlled by voice is that voice appears in the sound stream. The music sound should be suppressed and voice enhanced in order to recognize voice commands. In other cases, noise should be suppressed. Conventional systems rely on micro-controllers or DSPs, but such devices consume a high amount of power and are not suitable for wearable devices. 
       FIG.  39    shows an example system  3900  (sometimes referred to as neurovoice or a neurovoice system) for sound signal processing using one or more neuromorphic analog signal processors, according to some implementations. The example system includes an AI sound switch  3910  (sometimes referred to as an AI sound supervisor or a digital switch) that receives inputs  3912  (e.g.,  1  to  4  mono streams or  2  stereo streams) from one or more sound sources (e.g., speaker systems, human voices, or background sounds and noises). In some implementations, the AI sound supervisor  3910  transmits the input  3912  as unprocessed sound input or a processed sound stream  3914  to a music voice acoustic event detector (MVED)  3902  (sometimes referred to as an music voice acoustic event detection module) which processes the input  3914  and outputs voice, speaker, music, and/or event signals  3916  to the AI sound supervisor  3910 . In some implementations, the AI sound supervisor  3910  transmits the input  3912  and/or the MVED output  3916  to a voice extractor (VE)  3904  (sometimes referred to as a voice extraction module) as input  3918 , which processes the input  3918  to output processed voice extraction output  3920  to the AI sound supervisor  3910 . In some implementations, the AI sound supervisor  3910  transmits the input  3912 , the MVED output  3916 , and/or the VE output  3920  to a wake word detector (WWD)  3906  (sometimes referred to as a wake word detection module) as input  3922 , which processes the input  3922  to output detected wake words (or signals indicating detection of wake words)  3924  to the AI sound supervisor  3910 . In some implementations, the AI sound supervisor  3910  transmits the input  3912 , the MVED output  3916 , the VE output  3920 , and/or the WWD output  3924 , to a keyword spotter (KWS)  3908  (sometimes referred to as a keyword spotting module), as input  3926  which processes the input  3926  to output spotted keywords (or signal indicating spotting of keywords)  3928  to the AI sound supervisor  3910 . In some implementations, the AI sound switch  3910  outputs signals  3930  (e.g., voice, music, acoustic events, wake words spotted, and/or keywords spotted), based on the input signal  3912 , the MVED output signal  3916 , the VE output signal  3920 , the WWD output signal  3924 , and/or the KWS output signal  3924 . In some implementations, the AI sound switch  3910  outputs 1 to 4 mono streams (or 2 stereo streams)  3932 . Various other combinations and/or routing of signals (not shown in  FIG.  39   ) are possible. The MVED  3902 , the VE  3904 , the WWD  3906 , and/or the KWS  3908 , are analog neuromorphic cores that implement trained neural networks using analog components, such as operational amplifiers and resistors. 
     In some implementations, the AI sound supervisor  3910  controls the operation and timing of the system  3900 . Some implementations pass audio to the neuromorphic analog signal processor cores according to different operating modes of the system. For example, the system  3900  operates in several operation modes defining how input signals pass through, how and in what order the signals are passed to the neuromorphic analog signal processor cores, and/or how the signals are conditioned on output. The AI sound supervisor  3910  controls one or more aspects of these operation modes. Some implementations pass input audio with suppressed volume or unchanged input audio, according to the operating mode of the system. In some implementations, a logging module (e.g., a module implemented in the AI sound supervisor  3910 ) continuously logs audio parameters (e.g., audio volume at different stages, before and after signal conditioning), configuration (e.g., operating mode configuration for the system), and states of activated neural cores (e.g., configuration for a specific core, such as a threshold for false/true decision, a sensitity threshold for silence, a sensitivity for voice processing, and/or sensitivity for word detection). In some implementations, the AI sound supervisor switch  3910  modifies and routes audio steams between neuromorphic analog signal processor cores. In some implementations, the AI sound supervisor switch  3910  downsamples audio to 16 kilohertz (kHz) before processing, stores an input audio stream, stores the voice extractor processed audio stream  3920 , and/or selects input or voice extractor processed audio for wake word detection or keyword spotting. In some implementations, the AI sound supervisor  3910  combines input audio and the voice extractor-processed audio stream according to its configuration and signals. In some implementations, the AI sound supervisor  3910  upsamples the audio stream for output. 
     In some implementations, the AI sound supervisor  3910  is a digital logic unit connected to several neural cores to provide flexible conditional sound processing depending on whether voice, music, or alarm sound, are present. In some implementations, the AI sound supervisor  3910  is based on an AI music/voice/acoustic event detection module (e.g., the MVED  3902 ), which recognizes voice, music, and/or acoustic events, in the input audio stream. Depending on the task, a voice is recognized and enhanced in the audio stream and/or other sounds are suppressed. In other configurations, music is detected and a device preset is changed. For example, a hearing aid has a preset for voice and music, with different digital signal processor (DSP) configurations for comfortable hearing. With conventional devices, such presets have to be configured by a user. In contrast, devices based on techniques described herein can use the output signal  3930  to select presets for voice or music automatically. In some implementations, the AI sound supervisor  3910  is a digital logic unit, connected to neuromorphic analog signal processor cores, which perform depending on flags generated by an AI music/voice/acoustic event detector. If there is only ambient noise at the input, as determined by the MVED  3902 , some implementations translate ambient sounds or suppress them using a voice extractor module (e.g., the VE  3904 ). Some implementations use speaker identification using the music voice acoustic event detector module (e.g., the MVED  3902 ), due to which personalization modes can be added to listening devices. In some implementations, the AI sound supervisor  3910  also upsamples or downsamples the sound stream based on the task. Neuromorphic analog signal processors described above typically operate with 16 kHz signals. 
     In some implementations, the AI music/voice/acoustic event detection module  3902  analyses an audio stream (e.g., with a latency of approximately 4 milliseconds or less) and sets an output flag for the AI sound supervisor  3910  to recognize. 
     Voice commands have become popular but are still generally unavailable in wearable devices without an Internet connection because of power limitations. There is also a security concern using Internet-connected voice processing devices. The neurovoice processor described herein has a very low power consumption (e.g., 200 micro-watt or less) and works locally, thus providing privacy, according to some implementations. 
     In some implementations, the AI sound supervisor  3910  transforms a sound stream into a modified stream, where voice, noise and music are enhanced or suppressed depending on the operation mode, which may be user configurable or automatically configured. 
     Some implementations include a music/voice/acoustic event detection module (e.g., the MVED  3902 ), which provides output to the sound supervisor  3910  (which is a digital control unit). In some implementations, a music/voice/acoustic event detection module  3902  analyzes a sound stream and provides a flag to indicate music, voice, or an acoustic event, with latency of 4 milliseconds or less, which allows fast real time control of sound streams. 
     Some implementations dispatch sounds using the AI sound supervisor  3910 , in real-time, due to low latency of the MVED module  3902 . For example, the MVED  3902  may take between 16 to 200 milliseconds to detect voice, music, or an acoustic event. In some implementations, supervised analog neuromorphic blocks for sound autonomous processing consume ultra-low power (e.g., 200 microwatts or less) when compared to digital processors. 
     In some implementations, a hearing aid device is provided. The device includes separate DSP presets for voice, music, and no-voice conditions. Voice presets can be used to increase voice frequencies in an audio spectrogram. For certain hearing disorders, specific frequencies need to be increased. Music presets typically do not include an increase in frequencies. Conventional devices cannot distinguish between these conditions, so a user must switch between them manually. In some implementations, a hearing aid sound supervisor is responsible for voice extraction in the sound stream based on the MVED module  3902 . In some implementations, if there is a voice, the noise is suppressed by means of the voice extraction module  3904  and a hearing aid preset is activated. If there is no voice in the input stream, ambient sound is transmitted as the output signal. If there is music, a hearing aid preset is activated. Some implementations operate using wake word detection (e.g., the WWD  3906 ) and keyword spotting modules (e.g., the KWS  3908 ). Some implementations use neuromorphic analog signal processor-based voice and music activity detection in a hearing aid device, which can automatically change DSP presets. Voice extraction is activated optionally to enhance voice quality if noise is detected, according to some implementations. In some implementations, neuro-voice continuously monitors and logs sound parameters, its configuration and state, to enable future device personalization for the hearing specifics of the current user. Some implementations provide speaker identification to ensure that only the device owner can control the device. 
     There are true wireless stereo (TWS) models which are controlled by voice commands. Such models can be processed using the apparatus described herein. Moreover, some implementations provide a switch between music and voice. If a voice is recognized, voice extraction is switched on if there is a voice at the input of the TWS earphones. In some implementations, the AI-based music voice acoustic event detection module detects voice in the input stream, provides the signal to the AI sound supervisor  3910 , which suppresses the music and amplifies voice to make commands clear for a keyword spotting module (e.g., the KWS  3908 ) or a wake-word detection module (e.g., the WWD  3906 ) or it switches on the voice extraction module to suppress noise in the input voice stream. The MVED module provides speaker identification to ensure that only the device owner can control the device. 
     Some implementations provide passive noise cancellation and active noise cancellation (ANC) or environmental noise cancellation (ENC) features that isolate the user from all environmental sounds. A user can miss important voice information in this mode. A sound supervisor core can detect voice using an ANC/ENC microphone, to trigger a watchdog signal and instruct a subsystem to switch off ANC/ENC and mute music playback. Some implementations perform voice extraction for voice clarification (examples of which are described above). 
     According to some implementations, a hardware apparatus  3900  is provided. The hardware apparatus  3900  includes a digital switch  3910  coupled to a plurality of analog neuromorphic cores (e.g., cores corresponding to the MVED  3902 , the VE  3904 , the WWD  3906 , and/or the KWS  3908 ). The digital switch  3910  is configured to obtain one or more sound streams  3912  from one or more sound sources. The digital switch  3910  is also configured to transmit data based on the one or more sound streams to the plurality of analog neuromorphic cores. The digital switch  3910  is also configured to receive output from the plurality of analog neuromorphic cores. The digital switch  3910  is also configured to output one or more modified sound streams based on the output received from the plurality of analog neuromorphic cores. Each analog neuromorphic core comprises a respective analog network of analog components and is configured to (i) receive respective input data from the digital switch  3910 , (ii) perform a respective voice-related function, and (iii) transmit a respective output to the digital switch  3910 , for the one or more sound streams. 
     In some implementations, the digital switch  3910  is further configured to switch on or off at least one of the plurality of analog neuromorphic cores. 
     In some implementations, the analog components include a plurality of operational amplifiers and a plurality of resistors. Each operational amplifier represents an analog neuron, and each resistor represents a connection between two analog neurons. 
     In some implementations, the plurality of analog neuromorphic cores includes (i) a first core (e.g., a core corresponding to the MVED  3902 ) configured to detect music, voice and acoustic events, in the one or more sound streams, and (ii) a second core (e.g., a core corresponding to the VE  3904 ) configured to extract and/or enhance voice in the one or more sound streams. 
     In some implementations, the digital switch  3910  is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal (e.g., the output  3916 ) detecting a voice, subsequently transmit data based on the signal to the second core, and, in response, receive an enhanced voice signal (e.g., the output  3920 ) from the second core. In some implementations, the digital switch  3910  is further configured to inter-operate with a neuromorphic analog core configured to perform voice extraction, a neuromorphic analog core configured to perform voice activation detection, a neuromorphic analog core configured to perform wake-word detection, a neuromorphic analog core configured to perform keyword spotting, in any order, and/or any combinations thereof. 
     In some implementations, the plurality of analog neuromorphic cores further includes (i) a third core (e.g., a core corresponding to the KWS  3908 ) configured to spot keywords in the one or more sound streams, and (ii) a fourth core (e.g., a core corresponding to the WWD  3906 ) configured to detect wake words in the one or more sound streams. 
     In some implementations, the digital switch  3910  is further configured to: transform the one or more sound streams to normalize volume (e.g., automatic gain control), to obtain a sound stream suitable for processing in an analog neuromorphic core; and transmit the sound stream to the analog neuromorphic core. For example, input audio may have a low volume that is not suitable for voice extraction. To improve voice extraction quality, input audio volume is increased and passed to the voice extraction module. 
     In some implementations, the digital switch  3910  is further configured to log information related to audio parameters, configuration, and one or more states of activated neural cores amongst the plurality of analog neuromorphic cores. 
     In some implementations, the digital switch  3910  is further configured to: down-sample the one or more streams to 16 KHz, to obtain down-sampled data; transmit the down-sampled data to the plurality of analog neuromorphic cores; and up-sample audio stream output from the plurality of analog neuromorphic cores, for output (e.g., the output  3932 ). 
     In some implementations, the plurality of analog neuromorphic cores includes (i) a first core configured to detect music, voice, and acoustic events, in the one or more sound streams, and (ii) a second core configured to extract and/or enhance voice in the one or more sound streams. The digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; in response to receiving, from the first core, a signal detecting a voice, subsequently transmit data based on the signal to the second core, and, in response, receive an enhanced voice signal from the second core with noises suppressed; and in response to receiving, from the first core, a signal detecting no voice, subsequently output ambient noise in the one or more sound streams. 
     In some implementations, the plurality of analog neuromorphic cores includes a first core configured to detect a voice, music, or no voice. The system includes different operating modes, for voice, music, and no voice conditions. The digital switch  3910  is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal detecting a voice, music, or no voice, select a different operating mode based on the signal. 
     In some implementations, the plurality of analog neuromorphic cores includes a first core configured to detect the voice of a user. The digital switch  3910  is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal detecting the voice of the user, only then activate another core of the plurality of analog neuromorphic cores. 
     In some implementations, the plurality of analog neuromorphic cores includes (i) a first core configured to detect voice, (ii) a second core configured to enhance voice signals, and (iii) a third core configured to either spot keywords or detect wake words. In some implementations, the plurality of analog neuromorphic cores includes separate cores for spotting keywords and wake word detection, and the core for keyword spotting reacts to an output of the core for wake word detection. The digital switch  3910  is further configured to: initially transmit the data based on the one or more sound streams to the first core; in response to receiving, from the first core, a signal detecting a voice, subsequently transmit data based on the signal to a second core, and, in response, receive an enhanced voice signal from the second core with music suppressed and voice amplified; and in response to receiving, from the second core, the enhanced voice signal, transmit the enhanced voice signal to the third core for either spotting keywords or detecting wake words. 
     In some implementations, the plurality of analog neuromorphic cores includes one or more cores including: (i) a first core that implements a trained neural network (e.g., depth wise separable convolutional neural network (DS-CNN)) trained to spot keywords, (ii) a second core that implements a trained neural network (e.g., a recurrent neural network (RNN)) trained to detect wake words, (iii) a third core that implements a trained neural network (e.g., a recurrent neural network (RNN)) trained for voice activity detection, and/or (iv) a fourth core that implements a trained neural network (e.g., recurrent neural network) to extract voice from noisy sound streams. 
     In another aspect, a method is provided for sound signal processing. The method includes, at the digital switch  3910  coupled to a plurality of analog neuromorphic cores (e.g., cores corresponding to the MVED  3902 , the VE  3904 , the WWD  3906 , and/or the KWS  3908 ): obtaining one or more sound streams from one or more sound sources; transmitting data based on the one or more sound streams to the plurality of analog neuromorphic cores; receiving output from the plurality of analog neuromorphic cores; and outputting one or more modified sound streams based on the output received from the plurality of analog neuromorphic cores. Each analog neuromorphic core includes a respective analog network of analog components. Each analog neuromorphic core receives input data from the digital switch; performs a respective voice-related function; and transmits a respective output to the digital switch, for the one or more sound streams. 
     In some implementations, the method further includes, at the digital switch  3910 , switching on or off at least one of the plurality of analog neuromorphic cores. 
     In some implementations, the analog components include a plurality of operational amplifiers and a plurality of resistors, where each operational amplifier represents an analog neuron, and each resistor represents a connection between two analog neurons. 
     In some implementations, the method includes, at a core of the plurality of analog neuromorphic cores, detecting music, voice, and acoustic events, in the one or more sound streams. 
     In some implementations, the method further includes, at a core of the plurality of analog neuromorphic cores, extracting and/or enhancing voice in the one or more sound streams. 
     In some implementations, the order of operation/control (e.g., voice activation detection followed by voice enhancement) changes adaptively. For example, initially voice activation detection is performed followed by voice enhancement. Subsequently, voice enhancement is performed first then voice activation detection is performed. Following that, the system returns to original order of processing, and so on, and the digital switch changes control based on the application and/or the environment in which the hardware apparatus is operating and/or user preferences. 
     In some implementations, the method further includes, at the digital switch  3910 : initially transmitting the data based on the one or more sound streams to a first core for detecting music, voice and acoustic events, in the one or more sound streams; and in response to receiving, from the first core, a signal detecting a voice, subsequently transmitting data based on the signal to a second core, and, in response, receive an enhanced voice signal from the second core. 
     In some implementations, the method further includes, at the digital switch  3910 : initially transmitting the data based on the one or more sound streams to a first core for enhancing voice signals, in the one or more sound streams; and in response to receiving, from the first core, an enhanced voice signal, subsequently transmitting data based on the signal to a second core, and, in response, receive a signal from the second core for detecting music, voice and acoustic events. 
     In some implementations, the method further includes, at a core of the plurality of analog neuromorphic cores: detecting wake words in the one or more sound streams. 
     In some implementations, the method further includes, at a core of the plurality of analog neuromorphic cores, spotting keywords in the one or more sound streams. Typically, detecting wake words is followed by keyword spotting. 
     In some implementations, the digital switch  3910  transmits the sound stream directly to a wake word detection and/or keyword spotting neural network without voice activation detection and/or voice extraction. 
     The present application discloses subject-matter in correspondence with the following numbered clauses: 
     (A1) A hardware apparatus comprising: a digital switch coupled to a plurality of analog neuromorphic cores, the digital switch configured to: obtain one or more sound streams from one or more sound sources; transmit data based on the one or more sound streams to the plurality of analog neuromorphic cores; receive output from the plurality of analog neuromorphic cores; and output one or more modified sound streams based on the output received from the plurality of analog neuromorphic cores; and the plurality of analog neuromorphic cores, each analog neuromorphic core comprising a respective analog network of analog components and configured to (i) receive a respective input data from the digital switch, (ii) perform a respective voice-related function, and (iii) transmit a respective output to the digital switch, for the one or more sound streams. 
     (A2) The hardware apparatus as recited in clause (A1), wherein the digital switch is further configured to switch on or off at least one of the plurality of analog neuromorphic cores. 
     (A3) The hardware apparatus as recited in any of clauses (A1)-(A2), wherein the analog components include a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron, and each resistor represents a connection between two analog neurons. 
     (A4) The hardware apparatus as recited in any of clauses (A1)-(A3), wherein the plurality of analog neuromorphic cores includes (i) a first core configured to detect music, voice and acoustic events, in the one or more sound streams, and (ii) a second core configured to extract and/or enhance voice in the one or more sound streams. 
     (A5) The hardware apparatus as recited in clause (A4), wherein the digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal detecting a voice, subsequently transmit data based on the signal to the second core, and, in response, receive an enhanced voice signal from the second core. 
     (A6) The hardware apparatus as recited in clause (A4), wherein the plurality of analog neuromorphic cores further includes (i) a third core configured to spot keywords in the one or more sound streams, and (ii) a fourth core configured to detect wake words in the one or more sound streams. 
     (A7) The hardware apparatus as recited in any of clauses (A1)-(A6), wherein the digital switch is further configured to: transform the one or more sound streams to normalize volume, to obtain a sound stream suitable for processing in an analog neuromorphic core; and transmit the sound stream of data to the analog neuromorphic core. 
     (A8) The hardware apparatus as recited in any of clauses (A1)-(A7), wherein the digital switch is further configured to: log information related to audio parameters, configuration, and one or more states of activated neural cores amongst the plurality of analog neuromorphic cores. 
     (A9) The hardware apparatus as recited in any of clauses (A1)-(A8), wherein the digital switch is further configured to: down-sample the one or more streams to 16 KHz, to obtain down-sampled data; transmit the down-sampled data to the plurality of analog neuromorphic cores; and up-sample audio stream output from the plurality of analog neuromorphic cores, for output. 
     (A10) The hardware apparatus as recited in any of clauses (A1)-(A9), wherein the plurality of analog neuromorphic cores includes (i) a first core configured to detect music, voice and acoustic events, in the one or more sound streams, and (ii) a second core configured to extract and/or enhance voice in the one or more sound streams, wherein the digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; in response to receiving, from the first core, a signal detecting a voice, subsequently transmit data based on the signal to the second core, and, in response, receive an enhanced voice signal from the second core with noises suppressed; and in response to receiving, from the first core, a signal detecting no voice, subsequently output ambient noise in the one or more sound streams. 
     (A11) The hardware apparatus as recited in any of clauses (A1)-(A10), wherein the plurality of analog neuromorphic cores includes a first core configured to detect a voice, music, or no voice, wherein the digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal detecting a voice, music, or no voice, select a different operating mode based on the signal. 
     (A12) The hardware apparatus as recited in any of clauses (A1)-(A11), wherein the plurality of analog neuromorphic cores includes a first core configured to detect a voice of a user, wherein the digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; and in response to receiving, from the first core, a signal detecting the voice of the user, only then activate another core of the plurality of analog neuromorphic cores. 
     (A13) The hardware apparatus as recited in any of clauses (A1)-(A12), wherein the plurality of analog neuromorphic cores includes (i) a first core configured to detect a voice, (ii) a second core configured to enhance voice signals, and (iii) a third core configured to either spot keywords or detect wake words, wherein the digital switch is further configured to: initially transmit the data based on the one or more sound streams to the first core; in response to receiving, from the first core, a signal detecting a voice, subsequently transmit data based on the signal to a second core, and, in response, receive an enhanced voice signal from the second core with music suppressed and voice amplified; and in response to receiving, from the second core, the enhanced voice signal, transmit the enhanced voice signal to the third core for either spotting keywords or detecting wake words. 
     (A14) The hardware apparatus as recited in any of clauses (A1)-(A13), wherein the plurality of analog neuromorphic cores includes one or more cores selected from the group consisting of: (i) a first core that implements a trained neural network trained to spot keywords, (ii) a second core that implements a trained neural network trained to detect wake words, (iii) a third core that implements a trained neural network trained for voice activity detection, and (iv) a fourth core that implements a trained neural network to extract voice from noisy sound streams. 
     (A15) The hardware apparatus as recited in clause (A14), wherein (i) the first core implements a trained depth wise separable convolutional neural network (DS-CNN) trained to spot keywords, (ii) the second core implements a trained recurrent neural network (RNN) trained to detect wake words, (iii) the third core implements a trained recurrent neural network (RNN) trained for voice activity detection, and (iv) the fourth core implements a trained recurrent neural network trained to extract voice from noisy sound stream. 
     (B1) A method comprising: at a digital switch coupled to a plurality of analog neuromorphic cores: obtaining one or more sound streams from one or more sound sources; transmitting data based on the one or more sound streams to the plurality of analog neuromorphic cores; receiving output from the plurality of analog neuromorphic cores; and outputting one or more modified sound streams based on the output received from the plurality of analog neuromorphic cores; and at each of the plurality of analog neuromorphic cores, each analog neuromorphic core comprising a respective analog network of analog components: receiving a respective input data from the digital switch; performing a respective voice-related function; and transmitting a respective output to the digital switch, for the one or more sound streams. 
     (B2) The method as recited in clause (B1), further comprising: at the digital switch: switching on or off at least one of the plurality of analog neuromorphic cores. 
     (B3) The method as recited in any of clauses (B1)-(B2), wherein the analog components includes a plurality of operational amplifiers and a plurality of resistors, wherein each operational amplifier represents an analog neuron, and each resistor represents a connection between two analog neurons. 
     (B4) The method as recited in any of clauses (B1)-(B3), further comprising: at a core of the plurality of analog neuromorphic cores: detecting music, voice and acoustic events, in the one or more sound streams. 
     (B5) The method as recited in any of clauses (B1)-(B4), further comprising: at a core of the plurality of analog neuromorphic cores: extracting and/or enhancing voice in the one or more sound streams. 
     (B6) The method as recited in any of clauses (B1)-(B5), further comprising: at the digital switch: initially transmitting the data based on the one or more sound streams to a first core for detecting music, voice and acoustic events, in the one or more sound streams; and in response to receiving, from the first core, a signal detecting a voice, subsequently transmitting data based on the signal to a second core, and, in response, receive an enhanced voice signal from the second core. 
     (B7) The method as recited in any of clauses (B1)-(B6), further comprising: at the digital switch: initially transmitting the data based on the one or more sound streams to a first core for enhancing voice signals, in the one or more sound streams; and in response to receiving, from the first core, an enhanced voice signal, subsequently transmitting data based on the signal to a second core, and, in response, receive a signal from the second core for detecting music, voice and acoustic events. 
     (B8) The method as recited in any of clauses (B1)-(B7), further comprising: at a core of the plurality of analog neuromorphic cores: detecting wake words in the one or more sound streams. 
     (B9) The method as recited in any of clauses (B1)-(B8), further comprising: at a core of the plurality of analog neuromorphic cores: spotting keywords in the one or more sound streams. 
     The terminology used in the description of the invention herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. 
     The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various implementations with various modifications as are suited to the particular use contemplated.