nils-herrmann/clean_scirepeval huggingface dataset:
E?cient Temporal Processing with Biolog-ically Realistic Dynamic Synapses

Synapses play a central role in neural computation: the strengths of synaptic connections determine the function of a neural circuit. In conventional models of computation, synaptic strength is assumed to be a static quantity that changes only on the slow timescale of learning. In biological systems, however, synaptic strength undergoes dynamic modulation on rapid timescales through mechanisms such as short term facilitation and depression. Here we describe a general model of computation that exploits dynamic synapses, and use a backpropagation-like algorithm to adjust the synaptic parameters. We show that such gradient descent suffices to approximate a given quadratic filter by a rather small neural system with dynamic synapses. We also compare our network model to artificial neural networks designed for time series processing. Our numerical results are complemented by theoretical analyses which show that even with just a single hidden layer such networks can approximate a surprisingly large class of nonlinear filters: all filters that can be characterized by Volterra series. This result is robust with regard to various changes in the model for synaptic dynamics.
Neural Systems as Nonlinear Filters

Experimental data show that biological synapses behave quite differently from the symbolic synapses in all common artificial neural network models. Biological synapses are dynamic; their weight changes on a short timescale by several hundred percent in dependence of the past input to the synapse. In this article we address the question how this inherent synaptic dynamics (which should not be confused with long term learning) affects the computational power of a neural network. In particular, we analyze computations on temporal and spatiotemporal patterns, and we give a complete mathematical characterization of all filters that can be approximated by feedforward neural networks with dynamic synapses. It turns out that even with just a single hidden layer, such networks can approximate a very rich class of nonlinear filters: all filters that can be characterized by Volterra series. This result is robust with regard to various changes in the model for synaptic dynamics. Our characterization result provides for all nonlinear filters that are approximable by Volterra series a new complexity hierarchy related to the cost of implementing such filters in neural systems.
Foundations for a Circuit Complexity Theory of Sensory Processing

We introduce total wire length as salient complexity measure for an analysis of the circuit complexity of sensory processing in biological neural systems and neuromorphic engineering. This new complexity measure is applied to a set of basic computational problems that apparently need to be solved by circuits for translation- and scale-invariant sensory processing. We exhibit new circuit design strategies for these new benchmark functions that can be implemented within realistic complexity bounds, in particular with linear or almost linear total wire length.
Processing of Time Series by Neural Circuits with Biologically Realistic Synaptic Dynamics

Experimental data show that biological synapses behave quite differently from the symbolic synapses in common artificial neural network models. Biological synapses are dynamic, i.e., their "weight" changes on a short time scale by several hundred percent in dependence of the past input to the synapse. In this article we explore the consequences that these synaptic dynamics entail for the computational power of feedforward neural networks. We show that gradient descent suffices to approximate a given (quadratic) filter by a rather small neural system with dynamic synapses. We also compare our network model to artificial neural networks designed for time series processing. Our numerical results are complemented by theoretical analysis which show that even with just a single hidden layer such networks can approximate a surprisingly large large class of nonlinear filters: all filters that can be characterized by Volterra series. This result is robust with regard to various changes in the model for synaptic dynamics.
On the Computational Power of Winner-Take-All

This article initiates a rigorous theoretical analysis of the computational power of circuits that employ modules for computing winner-take-all. Computational models that involve competitive stages have so far been neglected in computational complexity theory, although they are widely used in computational brain models, artificial neural networks, and analog VLSI. Our theoretical analysis shows that winner-take-all is a surprisingly powerful computational module in comparison with threshold gates (also referred to as McCulloch-Pitts neurons) and sigmoidal gates. We prove an optimal quadratic lower bound for computing winner-takeall in any feedforward circuit consisting of threshold gates. In addition we show that arbitrary continuous functions can be approximated by circuits employing a single soft winner-take-all gate as their only nonlinear operation. Our theoretical analysis also provides answers to two basic questions raised by neurophysiologists in view of the well-known asymmetry between excitatory and inhibitory connections in cortical circuits: how much computational power of neural networks is lost if only positive weights are employed in weighted sums and how much adaptive capability is lost if only the positive weights are subject to plasticity.
Finding the Key to a Synapse

Experimental data have shown that synapses are heterogeneous: different synapses respond with different sequences of amplitudes of postsynaptic responses to the same spike train. Neither the role of synaptic dynamics itself nor the role of the heterogeneity of synaptic dynamics for computations in neural circuits is well understood. We present in this article methods that make it feasible to compute for a given synapse with known synaptic parameters the spike train that is optimally fitted to the synapse, for example in the sense that it produces the largest sum of postsynaptic responses. To our surprise we find that most of these optimally fitted spike trains match common firing patterns of specific types of neurons that are discussed in the literature.
A Simple Model for Neural Computation with Firing Rates and Firing Correlations

A simple extension of standard neural network models is introduced which provides a model for neural computations that involve both firing rates and firing correlations. Such an extension appears to be useful since it has been shown that firing correlations play a significant computational role in many biological neural systems. Standard neural network models are only suitable for describing neural computations in terms of firing rates. The resulting extended neural network models are still relatively simple, so that their computational power can be analysed theoretically. We prove rigorous separation results, which show that the use of firing correlations in addition to firing rates can drastically increase the computational power of a neural network. Furthermore, one of our separation results also throws new light on a question that involves just standard neural network models: we prove that the gap between the computational power of high-order and first-order neural nets is substantially larger than shown previously.
A Model for Fast Analog Computation Based on Unreliable Synapses

We investigate through theoretical analysis and computer simulations the consequences of unreliable synapses for fast analog computations in networks of spiking neurons, with analog variables encoded by the current firing activities of pools of spiking neurons. Our results suggest a possible functional role for the well-established unreliability of synaptic transmission on the network level. We also investigate computations on time series and Hebbian learning in this context of space-rate coding in networks of spiking neurons with unreliable synapses.
On Computations with Pulses

We explore the computational power of formal models for computation with pulses. Such models are motivated by realistic models for biological neurons and by related new types of VLSI (“pulse stream VLSI”). In preceding work it was shown that the computational power of formal models for computation with pulses is quite high if the pulses arriving at a computational unit have an approximately linearly rising or linearly decreasing initial segment. This property is satisfied by common models for biological neurons. On the other hand, several implementations of pulse stream VLSI employ pulses that are approximately piecewise constant (i.e., step functions). In this article we investigate the relevance of the shape of pulses in formal models for computation with pulses. The results show that the computational power drops significantly if one replaces pulses with linearly rising or decreasing initial segments by piecewise constant pulses. We provide an exact characterization of the latter model in terms of a weak version of a random access machine (RAM). We also compare the language recognition capability of a recurrent version of this model with that of deterministic finite automata and Turing machines.
2 Computing with Spiking Neurons

In the preceding chapter a number of mathematical models for spiking neurons were introduced. Spiking neurons differ in essential aspects from the familiar computational units of common neural network models, such as McCulloch-Pitts neurons or sigmoidal gates. Therefore the question arises how one can compute with spiking neurons, or with related computational units in electronic hardware whose input and output consists of trains of pulses. Furthermore the question arises how the computational power of networks of such units relates to that of common reference models, such as threshold circuits or multi-layer perceptrons. Both of these questions will be addressed in this chapter.
Dynamic Stochastic Synapses as Computational Units

In most neural network models, synapses are treated as static weights that change only with the slow time scales of learning. It is well known, however, that synapses are highly dynamic and show use-dependent plasticity over a wide range of time scales. Moreover, synaptic transmission is an inherently stochastic process: a spike arriving at a presynaptic terminal triggers the release of a vesicle of neurotransmitter from a release site with a probability that can be much less than one. We consider a simple model for dynamic stochastic synapses that can easily be integrated into common models for networks of integrate-andfire neurons (spiking neurons). The parameters of this model have direct interpretations in terms of synaptic physiology. We investigate the consequences of the model for computing with individual spikes and demonstrate through rigorous theoretical results that the computational power of the network is increased through the use of dynamic synapses.
Neural Computation with Winner-Take-All as the Only Nonlinear Operation

Everybody "knows" that neural networks need more than a single layer of nonlinear units to compute interesting functions. We show that this is false if one employs winner-take-all as nonlinear unit: • Any boolean function can be computed by a single k-winner-take-all unit applied to weighted sums of the input variables. • Any continuous function can be approximated arbitrarily well by a single soft winner-take-all unit applied to weighted sums of the input variables. • Only positive weights are needed in these (linear) weighted sums. This may be of interest from the point of view of neurophysiology, since only 15% of the synapses in the cortex are inhibitory. In addition it is widely believed that there are special microcircuits in the cortex that compute winner-take-all. • Our results support the view that winner-take-all is a very useful basic computational unit in Neural VLSI: □ it is wellknown that winner-take-all of n input variables can be computed very efficiently with 2n transistors (and a total wire length and area that is linear in n) in analog VLSI [Lazzaro et at., 1989] □ we show that winner-take-all is not just useful for special purpose computations, but may serve as the only nonlinear unit for neural circuits with universal computational power □ we show that any multi-layer perceptron needs quadratically in n many gates to compute winner-take-all for n input variables, hence winner-take-all provides a substantially more powerful computational unit than a perceptron (at about the same cost of implementation in analog VLSI). Complete proofs and further details to these results can be found in [Maass, 2000].
Fast analog computation in networks of spiking neurons using unreliable synapses

We investigate through theoretical analysis and computer simulations the consequences of unreliable synapses for fast analog computations in networks of spiking neurons, with analog variables encoded by the ring activities of pools of spiking neurons. Our results suggest that the known unreliability of synaptic transmission may be viewed as a useful tool for analog computing, rather than as a \bug" in neuronal hardware. We also investigate computations on analog time series encoded by the ring activities of pools of spiking neurons.
Analog Neural Nets with Gaussian or Other Common Noise Distributions Cannot Recognize Arbitrary Regular Languages

We consider recurrent analog neural nets where the output of each gate is subject to gaussian noise or any other common noise distribution that is nonzero on a sufficiently large part of the state-space. We show that many regular languages cannot be recognized by networks of this type, and we give a precise characterization of languages that can be recognized. This result implies severe constraints on possibilities for constructing recurrent analog neural nets that are robust against realistic types of analog noise. On the other hand, we present a method for constructing feedfor-ward analog neural nets that are robust with regard to analog noise of this type.
A Precise Characterization of the Class of Languages Recognized by Neural Nets under Gaussian and Other Common Noise Distributions

We consider recurrent analog neural nets where each gate is subject to Gaussian noise, or any other common noise distribution whose probability density function is nonzero on a large set. We show that many regular languages cannot be recognized by networks of this type, for example the language {w e {O, I}*| w begins with O}, and we give a precise characterization of those languages which can be recognized. This result implies severe constraints on possibilities for constructing recurrent analog neural nets that are robust against realistic types of analog noise. On the other hand we present a method for constructing feed forward analog neural nets that are robust with regard to analog noise of this type.
MOSES: an example for the modelling of spatial services by means of route description system

In the last few years in cognitive science there has been a growing interest in the connection between visual perception and natual language. The question of interest is: How ca we discuss what we see? With this question in mind in this article we will look at the area of incremental route descriptions. Here, a speaker step-by-step presents the relevant rout information in a 3D-environment. The speaker must adjust his/her descriptions to the currently visible objects. Two major questions arise in this context: 1. How is visually obtained informaiton used in natural language generation? and 2. How are the modalities coordniated? We will present a computational framework for the interaction of visual perception and natural language descriptions which integrates several processes and representations. Especially discussed is the interaction between the spatial representation and the presentation representation used for natural language descriptions. We have implemented a prototypical version of the proposed model, called MOSES.
On the Complexity of Computing and Learning with Networks of Spiking Neurons

In a network of spiking neurons a new set of parameters becomes relevant which has no counterpart in traditional neural network models: the time that a pulse needs to travel through a connection between two neurons (henceforth called \delay" of a connection). It is known that these delays are tuned in biological neural systems through a variety of mechanisms. We investigate the VC-dimension of networks of spiking neurons where the delays are viewed as \programmable parameters" and we prove tight bounds for this VC-dimension. Thus we get quantitative estimates for the diversity of functions that a network with xed architecture can compute with diierent settings of its delays. It turns out that a network of spiking neurons with k adjustable delays is able to compute a much richer class of Boolean functions than a threshold circuit with k adjustable weights. The results also yield bounds for the number of training examples that an algorithm needs for tuning the delays of a network of spiking neurons. Results about the computational complexity of such algorithms are also given.
Emulation of Hopfield networks with spiking neurons in temporal coding

A theoretical model for analog computation with temporal coding is introduced and tested through simulations in GENESIS. It turns out that the use of multiple synapses yields very noise robust mechanisms for analog computations with temporal coding in networks of detailed compartmental neuron models. One arrives in this way at a method for emulating arbitrary Hopfield nets with spiking neurons in temporal coding, yielding new models for associative recall of spatio-temporal firing patterns. A corresponding layered architecture yields a refinement of the synfire-chain model that can assume a fairly large set of different firing patterns for different inputs.
On the Effect of Analog Noise in Discrete-Time Analog Computations

We introduce a model for analog computation with discrete time in the presence of analog noise that is flexible enough to cover the most important concrete cases, such as noisy analog neural nets and networks of spiking neurons. This model subsumes the classical model for digital computation in the presence of noise. We show that the presence of arbitrarily small amounts of analog noise reduces the power of analog computational models to that of finite automata, and we also prove a new type of upper bound for the VC-dimension of computational models with analog noise.
Efficient Learning With Virtual Threshold Gates

We reduce learning simple geometric concept classes to learning disjunctions over exponentially many variables. We then apply an online algorithm called Winnow whose number of prediction mistakes grows only logarithmically with the number of variables. The hypotheses of Winnow are linear threshold functions with one weight per variable. We find ways to keep the exponentially many weights of Winnow implicitly so that the time for the algorithm to compute a prediction and update its “virtual” weights is polynomial. Our method can be used to learnd-dimensional axis-parallel boxes whendis variable and unions ofd-dimensional axis-parallel boxes whendis constant. The worst-case number of mistakes of our algorithms for the above classes is optimal to within a constant factor, and our algorithms inherit the noise robustness of Winnow. We think that other online algorithms with multiplicative weight updates whose loss bounds grow logarithmically with the dimension are amenable to our methods.
Bounds for the Computational Power and Learning Complexity of Analog Neural Nets

It is shown that high-order feedforward neural nets of constant depth with piecewise-polynomial activation functions and arbitrary real weights can be simulated for Boolean inputs and outputs by neural nets of a somewhat larger size and depth with Heaviside gates and weights from {-1, 0, 1}. This provides the first known upper bound for the computational power of the former type of neural nets. It is also shown that in the case of first-order nets with piecewise-linear activation functions one can replace arbitrary real weights by rational numbers with polynomially many bits without changing the Boolean function that is computed by the neural net. In order to prove these results, we introduce two new methods for reducing nonlinear problems about weights in multilayer neural nets to linear problems for a transformed set of parameters. These transformed parameters can be interpreted as weights in a somewhat larger neural net. 
As another application of our new proof technique we show that neural nets with piecewise-polynomial activation functions and a constant number of analog inputs are probably approximately correct (PAC) learnable (in Valiant's model for PAC learning [ (1984), pp. 1134--1142]).
On the complexity of learning for a spiking neuron (extended abstract)

Spiking neurons are models for the computational units in biological neural systems where information is considered to be encoded mainly in the temporal patterns of their activity. They provide a way of analyzing neural computation that is not captured by the traditional neuron models such as sigmoidal and threshold gates (or “Perceptrons”). We introduce a simple model of a spiking neuron that, in addition to the weights that model the plasticity of synaptic strength, also has variable transmission delays between neurons as programmable parameters. For coding of input and output values two modes are taken into account: binary coding for the Boolean and analog coding for the real-valued domain. We investigate the complexity of learning for a single spiking neuron within the framework of PAC-learnability. With regard to sample complexity, we prove that the VC-dimension is O(nlogn) and, hence, strictly larger than that of a threshold gate. In particular, the lower bound holds for binary coding and even if all weights are kept fixed. The upper bound is valid for the case of analog coding if weights and delays are programmable. With regard to computational complexity, we show that there is no polynomial-time PAClearning algorithm, unless RP = NP, for a quite *Address: Institute for Theoretical Computer Science, Technische Universitgt Graz, Klosterwiesgasse 32/2, A-8010 Graz, Austria.  {maass,mschmitt}Qigi.tu-graz.ac.at, http://www.cis.tu-graz.ac.at/igi/ pennisioll to make di&l/hnrd copies ofall or pa11 ofthis mated for perso,,al or &qsroom use is granted without Ike provided chat lhe copia me not made or distributed for profit or commercial advantage. 11~ COPY right notice, the title of the puhlicntiou and it< date appear. ad notice is given thar copyright is hy pemkion of the ACM, Inc. TO copy othemk. to republish, to post on servers or to redistribute lo list%, requires specific permission and/or fee COLT 97 Nashville, Tennesee. USA Copy+1 1997 ACM 0-89791-891-6/97t7..S3.TO restricted spiking neuron that is only slightly more powerful than a Boolean threshold gate: The consistency problem for a spiking neuron using binary coding and programmable delays from (0, l} is NP-complete. This holds even if all weights are kept fixed. The results demonstrate that temporal coding has a surprisingly large impact on the complexity of learning for single neurons.