A Training Scheme for the Uncertain Neuromorphic Computing Chips

Uncertainty is a very important feature of the intelligence and helps the brain become a flexible , creative and powerful intelligent system . The crossbar-based neuromorphic computing chips , in which the computing is mainly performed by analog circuits , have the uncertainty and can be used to imitate the brain . However , most of the current deep neural networks have not taken the uncertainty of the neuromorphic computing chip into consideration . Therefore , their performances on the neuromorphic computing chips are not as good as on the original platforms ( CPUs/GPUs ) . In this work , we proposed the uncertainty adaptation training scheme ( UATS ) that tells the uncertainty to the neural network in the training process . The experimental results show that the neural networks can achieve comparable inference performances on the uncertain neuromorphic computing chip compared to the results on the original platforms , and much better than the performances without this training scheme . 1 INTRODUCTION . Uncertainty reasoning is the essence of human thinking activities and a key aspect of the intelligence . There are two kind of uncertainties in intelligent systems . One is the fuzziness , the other is the stochasticity . The fuzziness helps the brain deal with the real world efficiently by ignoring the enormous redundant information . When we try to distinguish a cat or a dog , we do not need to know the expressions and the number of the legs . Although such information can be easily captured by our visual system with a glance , it will be ignored for efficiency . The stochasticity endows the brain the creativity and enables us not always failed in an unfamiliar field . Our decisions may change when we do not sure . These characteristics are not available in most existing artificial intelligence ( AI ) systems , such as a classifier based on a deep neural network ( DNN ) . The 32-bit or 64-bit floating numbers are used to describe the weights and activations . While some researchers found that the 8- bit integer is enough for many applications Banner et al . ( 2018 ) ; Wu et al . ( 2018 ) . Moreover , after the training procedure , the results will be the same no matter how many times it performs , although the margin is very small and the answer is wrong . There are some methods to address these issues , such as the network quantization and the Bayesian network . In addition , the neuromorphic computing chip has provide a hardware approach to supplement the missing uncertainty in DNN . In recent years , the emerging nanotechnology device and crossbar structure based neuromorphic computing chips have developed a lot Fuller et al . ( 2019 ) ; Boybat et al . ( 2018 ) ; Yao et al . ( 2017 ) . The Ohms law and Kirchhoffs law make the crossbar structure very efficient when doing the vectormatrix multiplication ( VMM ) , and the emerging nanoscale nonvolatile memory ( NVM ) device at each cross point provides additional storage capability ( Figure 1 ) . The crossbar holds the devices conductances as memory in peacetime , and performs the computing function when applied voltages . The so-called computing in memory ( CIM ) architecture can relieve the memory bottleneck , which is the most serious problem in the von Neumann architecture , and make the neuromorphic computing chips more energy and area efficiency . Therefore , the neuromorphic computing has become a promising approach to realize the AI applications , which is full of VMMs and great memory requirement . Besides the energy and area efficiency , the uncertainty is also an important and intrinsic feature of the neuromorphic computing chips and is not well utilized . The uncertainty in the neuromorphic computing chips comes from two aspects . The fuzziness is mainly caused by the analog to digital converters ( ADCs ) and the stochasticity is mainly induced by the NVM devices . According to the Kirchhoffs law , the VMM result is indicated as the summarization of the currents , which is an analog output . It is necessary to use the ADC to convert the analog currents to digital voltages for data transferring . The function of ADC is similar as the activation quantization in the network . The stochasticity of the NVM device is due to the intrinsic physical mechanism Zhao et al . ( 2017 ) ; Lin et al . ( 2018 ) . The random movement of the particles in the device makes the conductance varied . The output current will be different even applying the same voltage . The stochasticity of the device is usually simulated as a non-ideal factor that makes the network perform worse Prezioso et al . ( 2015 ) ; Ambrogio et al . ( 2018 ) ; Tang et al . ( 2017 ) . In this work , we proposed a training scheme that utilizes the stochasticity to improve the performance of the neuromorphic computing chips . 2 METHODOLOGY . 2.1 MODELING THE DEVICE STOCHASTICITY . There are several varieties of NVM devices including phase change memories Suri et al . ( 2012 ) , filamentary migrating oxide devices Wong & Salahuddin ( 2015 ) , ferroelectric tunnel junction synapses Chanthbouala et al . ( 2012 ) , and so on . The stochasticity of each types of devices is different due to the different intrinsic physical mechanisms . Without loss of generality , we used the Gaussian distribution to model the device stochasticity . Although the Gaussian distribution may not fit the stochasticity of each types of devices well , it has the basic characteristics , that is , a stable state and the farther away from the stable state , the probability is lower . The mean of the Gaussian distribution is the conductance value of the stable state . The variance of the Gaussian distribution is usually corresponded to the mean according to experimental results Zhao et al . ( 2017 ) . It is also hard to use a singular model the relations between the mean and the variance of various devices . For simplify and without loss of generality , we assumed that the standard deviation of the distribution is linearly and positively correlated to the mean . Furthermore , the real conductance of device is a positive value . Therefore , the conductances sampled from the distribution which are less than the given valueGmin will be cut off to Gmin , where Gmin > 0 denotes the minimum conductance that all devices could reach . The model of the devices stochasticity is as follow : Gs = max ( Gmin , G ′ ) , G′ ∼ N ( G0 , αG 2 0 ) , ( 1 ) where Gs denotes the stochastic conductance , G′ denotes the value sampled from the Gaussian distribution N ( µ , σ2 ) with the a mean of µ and a variance of σ2 , G0 denotes the conductance of the device when it is in the stable state , and α > 0 denotes the level of the devices stochasticity . 2.2 MODELING THE DEVICE FUZZINESS . The device fuzziness is a by-product of the writing process . Writing the conductance of each device in the crossbar is an essential step when using neuromorphic computing chip to realize an AI application . Before writing , we need to determine the target conductance of each device according to the weights of the neural network . That is what the mapping process do . Besides scaling the weights into the working range of devices conductance [ Glow , Ghigh ] , we used the difference of two devices conductances Gpos −Gneg to express one weight w , which can be positive or negative . In order to achieve higher energy efficiency , it is better to use lower conductances to express the same weight . Moreover , in order to make full use of the entire conductance working range , the mapping algorithm is as follow : Gpos = ( |w|+ w ) ( Ghigh −Glow ) 2maxw|w| +Glow ( 2 ) Gneg = ( |w| − w ) ( Ghigh −Glow ) 2maxw|w| +Glow ( 3 ) where |w| denotes the absolute value of a singular w , maxw |w| denotes the maximum of all the |w| , Gpos and Gneg are the target conductances that we want to write . However , the conductance can not be written accurately due to the stochasticity of the device and the fuzziness of the circuit . There is a variation when manipulate the conductance of the device and the measurement of the conductance is not accurate . The conductance value is obtained by dividing the read current by the applied voltage , which is obviously affected by the stochasticity of the device and the precision of the ADC . Therefore , we need a model to describe the fuzziness . Without loss of the generality , we also used the Gaussian distribution , which is as follow : Gf = max ( Gmin , G ′′ ) , G ′′ ∼ N ( Gtarget , β ) , ( 4 ) whereGf denotes fuzzy target conductance , G ′′ denotes the value sampled from the Gaussian distribution N ( µ , σ2 ) with the a mean of µ and a variance of σ2 , Gtarget denotes the target conductance determined in the mapping process , and β denotes the level of devices fuzziness , which is correspond to the level of the devices stochasticity , the precision of the ADC , and the writing strategy . For the sake of simplicity , we assumed β a constant in this work . 2.3 THE UNCERTAINTY ADAPTATION TRAINING SCHEME . If we program a well-trained DNN into a neuromorphic computing chip directly , the uncertainty will make the performance of the DNN worse , such as decrease the classification accuracy ( Tang et al . ( 2017 ) , Fig . 2 ) . However , the decrease of accuracy can be alleviated by using the proposed uncertainty adaptation training scheme ( UATS ) , and in some cases the accuracy will improved . The core idea of UATS is to tell the uncertainty to the neural networks during training process and guide them to learn how to deal with this situation . The stochasticity model is introduced in every feed forward ( FF ) process . When a weight w participates in the calculation of the FF process , we use a sample of random variable ws to do the calculation instead of w. ws is defined as : ws = maxw |w| Ghigh −Glow ( Gps −Gns ) ( 5 ) Both Gps and Gns are obtained by the stochasticity model ( 1 ) , and the conductances of the stable states Gp0 and Gn0 are calculated by ( 2 ) and ( 3 ) according to w , respectively . Actually , the ws can be approximated as ( 6 ) if G ′ is significantly larger than Gmin . ws ∼ N ( w , αw2 + c ) ( 6 ) where c = 2α ( Glow·maxw|w| Ghigh−Glow ) 2 . The fuzziness model is introduced during the training process . After every k epochs of training , every weight w are replaced by a sample of random variable wf , which is defined as : wf = maxw |w| Ghigh −Glow ( Gp −Gn ) ( 7 ) Both Gpf and Gnf are obtained by the fuzziness model ( 4 ) , and the target conductances Gptarget and Gntarget are calculated by ( 2 ) and ( 3 ) according to w , respectively . Actually , the wf can be approximated as ( 8 ) if G ′′ is significantly larger than Gmin . wf ∼ N ( w , β maxw |w| Ghigh −Glow ) , ( 8 ) Besides making the network train in an uncertain way , we also tried to teach it a better way to evaluate the networks performance when there is uncertainty , that is , how to calculate the loss function . The loss function is not calculated by the output of one FF process , but the average output of n FF processes with the same input batch .

Summary: This paper proposes a way of training neural nets on analog-circuit based chips, which are cursed with uncertainties. Such uncertainties are deeply rooted in the way neural nets are implemented on such chips. Take [c = a x b] as an example. In order to perform this operation, one can set the electric potential to a and the conductance to b and c will be the output current. The problem here is that we cannot set the conductance precisely, which often encodes the weights of a neural net. This implies we cannot precisely program a neural net into these chips. This paper proposes to train a neural net with the presence of such noise, by treating weight as a random variable during training. The experimental results based on simulation suggest this is a much better strategy than programming a neural net into chips imprecisely.