mtybilly/Instruct2DS huggingface dataset:
A Rigorous Analysis of Linsker-type Hebbian Learning
Authors: J. Feng, H. Pan, V. P. Roychowdhury
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We propose a novel rigorous approach for the analysis of Linsker's  unsupervised Hebbian learning network. The behavior of this  model is determined by the underlying nonlinear dynamics which  are parameterized by a set of parameters originating from the Heb(cid:173) bian rule and the arbor density of the synapses. These parameters  determine the presence or absence of a specific receptive field (also  referred to as a 'connection pattern') as a saturated fixed point  attractor of the model. In this paper, we perform a qualitative  analysis of the underlying nonlinear dynamics over the parameter  space, determine the effects of the system parameters on the emer(cid:173) gence of various receptive fields, and predict precisely within which  parameter regime the network will have the potential to develop  a specially designated connection pattern. In particular, this ap(cid:173) proach exposes, for the first time, the crucial role played by the  synaptic density functions, and provides a complete precise picture  of the parameter space that defines the relationships among the  different receptive fields. Our theoretical predictions are confirmed  by numerical simulations. 
320 
lian/eng Feng, H. Pan, V. P. Roychowdhury
Finding Structure in Reinforcement Learning
Authors: Sebastian Thrun, Anton Schwartz
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Reinforcement learning addresses  the problem of learning to select actions in order to  maximize one's performance in unknown environments.  To scale reinforcement learning  to complex real-world tasks, such as typically studied in AI, one must ultimately be able  to discover the structure in the world, in order to abstract away the myriad of details and  to operate in more tractable problem spaces.  This paper presents the SKILLS algorithm.  SKILLS discovers skills, which are partially  defined action policies that arise in the context of multiple, related tasks.  Skills collapse  whole action sequences into single operators.  They are learned by minimizing the com(cid:173) pactness of action policies, using a description length argument on their representation.  Empirical results  in  simple grid  navigation tasks  illustrate the successful  discovery  of  structure in reinforcement learning.
An Input Output HMM Architecture
Authors: Yoshua Bengio, Paolo Frasconi
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  introduce  a  recurrent  architecture  having  a  modular structure  and we formulate a training procedure based on the EM  algorithm.  The resulting model has similarities to hidden  Markov models, but  supports  recurrent  networks  processing style and allows  to exploit  the supervised  learning paradigm while using maximum likelihood  estimation.
Grouping Components of Three-Dimensional Moving Objects in Area MST of Visual Cortex
Authors: Richard S. Zemel, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Many  cells  in  the  dorsal  part  of  the  medial  superior  temporal  (MST)  area  of  visual  cortex  respond  selectively  to  spiral  flow  patterns-specific  combinations of expansion/ contraction  and  ro(cid:173) tation  motions.  Previous  investigators  have  suggested  that  these  cells  may represent  self-motion.  Spiral patterns  can  also  be  gener(cid:173) ated by the relative motion of the observer  and a particular object.  An  MST  cell  may  then  account  for  some  portion  of the  complex  flow  field,  and  the  set  of active  cells  could  encode  the  entire  flow;  in  this  manner,  MST  effectively  segments  moving  objects.  Such  a  grouping operation  is  essential  in  interpreting  scenes  containing  several  independent  moving objects  and  observer  motion.  We  de(cid:173) scribe  a  model  based  on  the  hypothesis  that  the  selective  tuning  of MST  cells  reflects  the grouping of object  components  undergo(cid:173) ing  coherent  motion.  Inputs  to  the  model  were  generated  from  sequences  of ray-traced  images that simulated realistic  motion sit(cid:173) uations,  combining observer  motion, eye  movements, and indepen(cid:173) dent  object  motion.  The  input  representation  was  modeled  after  response  properties of neurons in area MT,  which provides the pri(cid:173) mary input to area MST.  After  applying an  unsupervised  learning  algorithm,  the  units  became  tuned  to  patterns  signaling  coherent  motion.  The results  match many of the  known  properties  of MST  cells  and  are  consistent  with  recent  studies  indicating  that  these  cells  process  3-D  object  motion information. 
166 
Richard S.  Zemel,  Terrence  J.  Sejnowski
Higher Order Statistical Decorrelation without Information Loss
Authors: Gustavo Deco, Wilfried Brauer
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A neural network learning paradigm based on information theory is pro(cid:173) posed as a way to perform in an unsupervised fashion, redundancy  reduction among the elements of the output layer without loss of infor(cid:173) mation from the sensory input. The model developed performs nonlin(cid:173) ear decorrelation up to higher orders of the cumulant tensors and results  in probabilistic ally independent components of the output layer. This  means that we don't need to assume Gaussian distribution neither at the  input nor at the output. The theory presented is related to the unsuper(cid:173) vised-learning theory of Barlow, which proposes redundancy reduction  as the goal of cognition. When nonlinear units are used nonlinear princi(cid:173) pal component analysis is obtained. In this case nonlinear manifolds can  be reduced to minimum dimension manifolds. If such units are used the  network performs a generalized principal component analysis in the  sense that non-Gaussian distributions can be linearly decorrelated and  higher orders of the correlation tensors are also taken into account. The  basic structure of the architecture involves a general transfOlmation that  is volume conserving and therefore the entropy, yielding a map without  loss of infoIIDation. Minimization of the mutual infoIIDation among the  output neurons eliminates the redundancy between the outputs and  results in statistical decorrelation of the extracted features. This is  known as factorialleaming. 
248 
Gustavo Deco, Wilfried Brauer
Sample Size Requirements for Feedforward Neural Networks
Authors: Michael J. Turmon, Terrence L. Fine
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We estimate the number of training samples required to ensure that  the performance of a neural network on its training data matches  that obtained when fresh data is applied to the network. Existing  estimates are higher by orders of magnitude than practice indicates.  This work seeks to narrow the gap between theory and practice by  transforming the problem into determining the distribution of the  supremum of a random field in the space of weight vectors, which  in turn is attacked by application of a recent technique called the  Poisson clumping heuristic. 
1 
INTRODUCTION AND KNOWN RESULTS 
We investigate the tradeofi"s among network complexity, training set size, and sta(cid:173) tistical performance of feedforward neural networks so as to allow a reasoned choice  of network architecture in the face of limited training data. Nets are functions  7](x; w), parameterized by their weight vector w E W ~ Rd , which take as input  points x E Rk. For classifiers, network output is restricted to {a, 1} while for fore(cid:173) casting it may be any real number. The architecture of all nets under consideration  is N, whose complexity may be gauged by its Vapnik-Chervonenkis (VC) dimension  v, the size of the largest set of inputs the architecture can classify in any desired way  ('shatter'). Nets 7] EN are chosen on the basis of a training set T = {(Xi, YiHr=l.  These n samples are i.i.d. according to an unknown probability law P. Performance  of a network is measured by the mean-squared error
Generalisation in Feedforward Networks
Authors: Adam Kowalczyk, Herman L. Ferrá
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  discuss  a  model  of consistent  learning  with  an  additional  re(cid:173) striction  on  the  probability  distribution  of training  samples,  the  target concept  and hypothesis class.  We  show  that the model pro(cid:173) vides  a  significant  improvement  on  the  upper  bounds  of sample  complexity,  i.e.  the  minimal number  of random  training samples  allowing  a  selection  of the  hypothesis  with  a  predefined  accuracy  and  confidence.  Further,  we  show  that  the  model  has  the  poten(cid:173) tial  for  providing  a  finite  sample  complexity  even  in  the  case  of  infinite  VC-dimension  as  well  as  for  a  sample  complexity  below  VC-dimension.  This is  achieved  by  linking sample  complexity  to  an  "average"  number  of implement able dichotomies of a  training  sample  rather  than  the  maximal  size  of a  shattered  sample,  i.e.  VC-dimension.
The Use of Dynamic Writing Information in a Connectionist On-Line Cursive Handwriting Recognition System
Authors: Stefan Manke, Michael Finke, Alex Waibel
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In this paper we present NPen ++, a connectionist system for  writer independent, large vocabulary on-line cursive handwriting  recognition. This system combines a robust input representation,  which preserves the dynamic writing information, with a neural  network architecture, a so called Multi-State Time Delay Neural  Network (MS-TDNN), which integrates rec.ognition and segmen(cid:173) tation in a single framework. Our preprocessing transforms the  original coordinate sequence into a (still temporal) sequence offea(cid:173) ture vectors, which combine strictly local features, like curvature  or writing direction, with a bitmap-like representation of the co(cid:173) ordinate's proximity. The MS-TDNN architecture is well suited  for handling temporal sequences as provided by this input rep(cid:173) resentation. Our system is tested both on writer dependent and  writer independent tasks with vocabulary sizes ranging from 400  up to 20,000 words. For example, on a 20,000 word vocabulary we  achieve word recognition rates up to 88.9% (writer dependent) and  84.1 % (writer independent) without using any language models.
Capacity and Information Efficiency of a Brain-like Associative Net
Authors: Bruce Graham, David Willshaw
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We have determined the capacity and information efficiency of an  associative net configured in a brain-like way with partial connec(cid:173) tivity and noisy input cues. Recall theory was used to calculate  the capacity when pattern recall is achieved using a winners-take(cid:173) all strategy. Transforming the dendritic sum according to input  activity and unit usage can greatly increase the capacity of the  associative net under these conditions. For moderately sparse pat(cid:173) terns, maximum information efficiency is achieved with very low  connectivity levels (~ 10%). This corresponds to the level of con(cid:173) nectivity commonly seen in the brain and invites speculation that  the brain is connected in the most information efficient way.
SARDNET: A Self-Organizing Feature Map for Sequences
Authors: Daniel L. James, Risto Miikkulainen
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  self-organizing  neural  network  for  sequence  classification  called  SARDNET is described  and analyzed experimentally.  SARDNET  extends the Kohonen  Feature Map architecture  with activation re(cid:173) tention  and  decay  in  order  to  create  unique  distributed  response  patterns for different sequences.  SARDNET yields extremely dense  yet descriptive representations of sequential input in very few  train(cid:173) ing  iterations.  The  network  has  proven  successful  on  mapping ar(cid:173) bitrary sequences  of binary and real numbers,  as well  as  phonemic  representations  of  English  words.  Potential  applications  include  isolated  spoken  word  recognition  and  cognitive  science  models  of  sequence  processing.
Deterministic Annealing Variant of the EM Algorithm
Authors: Naonori Ueda, Ryohei Nakano
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  deterministic annealing variant of the EM  algorithm  for  maximum likelihood  parameter  estimation  problems.  In  our  approach,  the  EM  process  is  reformulated  as the  problem of min(cid:173) imizing  the  thermodynamic free  energy  by  using  the  principle  of  maximum  entropy and  statistical mechanics  analogy.  Unlike simu(cid:173) lated  annealing approaches,  this  minimization is  deterministically  performed.  Moreover,  the  derived  algorithm,  unlike  the  conven(cid:173) tional EM  algorithm, can obtain better estimates free of the initial  parameter values.
A Non-linear Information Maximisation Algorithm that Performs Blind Separation
Authors: Anthony J. Bell, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A new learning algorithm is derived which performs online stochas(cid:173) tic gradient ascent in the mutual information between outputs and  inputs  of a  network.  In  the  absence  of a  priori knowledge  about  the  'signal'  and  'noise'  components  of the  input,  propagation  of  information depends  on  calibrating network  non-linearities to  the  detailed  higher-order  moments of the  input  density functions.  By  incidentally  minimising  mutual  information  between  outputs,  as  well  as  maximising  their  individual  entropies,  the  network  'fac(cid:173) torises'  the  input  into  independent  components.  As  an  example  application,  we  have  achieved  near-perfect  separation of ten  digi(cid:173) tally mixed speech signals.  Our simulations lead us to believe  that  our network performs better at blind separation than the Herault(cid:173) J utten network,  reflecting the fact that it is derived rigorously from  the  mutual information objective. 
468 
Anthony J.  Bell,  Terrence  J.  Sejnowski
Pulsestream Synapses with Non-Volatile Analogue Amorphous-Silicon Memories
Authors: A. J. Holmes, Alan F. Murray, Stephen Churcher, J. Hajto, M. J. Rose
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper  presents  results  from  the  first  use  of neural  networks  for  the real-time feedback  control  of high  temperature plasmas in  a  tokamak fusion  experiment.  The tokamak is  currently  the  prin(cid:173) cipal  experimental  device  for  research  into  the  magnetic  confine(cid:173) ment  approach  to  controlled  fusion.  In  the  tokamak,  hydrogen  plasmas,  at  temperatures  of  up  to  100  Million  K,  are  confined  by  strong  magnetic  fields.  Accurate  control  of the  position  and  shape  of the plasma boundary  requires  real-time feedback  control  of the magnetic field  structure on  a  time-scale of a  few  tens of mi(cid:173) croseconds.  Software simulations have demonstrated that a  neural  network  approach  can  give  significantly  better  performance  than  the linear technique currently  used  on  most tokamak experiments.  The practical  application of the neural  network  approach  requires  high-speed  hardware,  for  which  a  fully  parallel  implementation of  the  multilayer perceptron,  using  a  hybrid  of digital and  analogue  technology,  has been  developed.
Dynamic Cell Structures
Authors: Jörg Bruske, Gerald Sommer
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Dynamic Cell Structures (DCS) represent a family  of artificial  neural  architectures  suited  both  for  unsupervised  and  supervised  learning.  They belong to the recently [Martinetz94] introduced class of Topology  Representing  Networks  (TRN)  which  build  perlectly topology  pre(cid:173) serving feature maps. DCS empI'oy a modified Kohonen learning rule  in conjunction with competitive Hebbian learning. The Kohonen type  learning rule serves to adjust the synaptic weight vectors while Hebbian  learning  establishes  a  dynamic  lateral  connection  structure  between  the units reflecting the topology of the feature manifold. In case of super(cid:173) vised learning, i.e. function approximation, each neural unit implements  a Radial Basis Function, and an  additional layer of linear output units  adjusts according to a delta-rule. DCS is the first RBF-based approxima(cid:173) tion  scheme attempting to concurrently learn and  utilize a perfectly to(cid:173) pology preserving map for improved performance.  Simulations on a selection of CMU-Benchmarks indicate that the DCS  idea applied to the Growing Cell Structure algorithm [Fritzke93] leads  to an  efficient and elegant algorithm  that can  beat conventional  models  on similar tasks.
Single Transistor Learning Synapses
Authors: Paul E. Hasler, Chris Diorio, Bradley A. Minch, Carver Mead
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe single-transistor silicon synapses that compute, learn,  and  provide  non-volatile  memory retention.  The single  transistor  synapses  simultaneously  perform long  term  weight  storage,  com(cid:173) pute the product of the input and the weight value, and update the  weight value according to a Hebbian or a backpropagation learning  rule.  Memory  is  accomplished  via  charge  storage  on  polysilicon  floating  gates, providing long-term  retention without refresh.  The  synapses efficiently use the physics of silicon to perform weight  up(cid:173) dates; the weight value is  increased using tunneling and the weight  value  decreases  using  hot  electron  injection.  The  small  size  and  low  power operation of single transistor synapses allows  the devel(cid:173) opment  of  dense  synaptic  arrays.  We  describe  the  design,  fabri(cid:173) cation,  characterization,  and  modeling  of an  array  of single  tran(cid:173) sistor  synapses.  When  the steady state source  current  is  used  as  the representation of the weight  value,  both the incrementing and  decrementing functions  are  proportional to a  power of the source  current.  The synaptic  array  was  fabricated  in  the standard  21'm  double - poly,  analog process available from  MOSIS.
Comparing the prediction accuracy of artificial neural networks and other statistical models for breast cancer survival
Authors: Harry B. Burke, David B. Rosen, Philip H. Goodman
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The  TNM  staging  system  has  been  used  since  the  early  1960's  to  predict  breast  cancer  patient  outcome.  In  an  attempt  to  in(cid:173) crease  prognostic  accuracy,  many putative prognostic factors  have  been  identified.  Because  the  TNM  stage  model  can  not  accom(cid:173) modate  these  new  factors,  the  proliferation  of factors  in  breast  cancer  has  lead  to  clinical  confusion.  What  is  required  is  a  new  computerized  prognostic system  that  can test  putative prognostic  factors  and  integrate  the  predictive  factors  with  the  TNM  vari(cid:173) ables  in order to increase  prognostic  accuracy.  Using  the area un(cid:173) der  the curve of the receiver  operating characteristic,  we  compare  the  accuracy  of the  following  predictive  models  in  terms  of five  year  breast cancer-specific  survival:  pTNM staging system,  princi(cid:173) pal component analysis,  classification and regression  trees,  logistic  regression,  cascade  correlation neural network,  conjugate gradient  descent  neural,  probabilistic neural network,  and backpropagation  neural network.  Several statistical models are significantly more ac-
Learning direction in global motion: two classes of psychophysically-motivated models
Authors: V. Sundareswaran, Lucia M. Vaina
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Perceptual learning is  defined  as  fast  improvement  in  performance and  retention  of the  learned  ability  over  a  period  of time.  In a  set  of psy(cid:173) chophysical experiments  we  demonstrated  that  perceptual learning oc(cid:173) curs for the discrimination of direction in stochastic motion stimuli.  Here  we  model  this  learning  using  two  approaches:  a clustering  model  that  learns  to  accommodate  the motion  noise,  and an averaging  model  that  learns to  ignore the noise.  Simulations of the models show  performance  similar to the psychophysical results.
On-line Learning of Dichotomies
Authors: N. Barkai, H. S. Seung, H. Sompolinsky
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The performance of on-line algorithms for learning dichotomies is studied. In on-line learn(cid:173) ing, the number of examples P is equivalent to the learning time, since each example is  presented only once. The learning curve, or generalization error as a function of P, depends  on the schedule at which the learning rate is lowered. For a target that is a perceptron rule,  the learning curve of the perceptron algorithm can decrease as fast as p- 1 , if the sched(cid:173) ule is optimized. If the target is not realizable by a perceptron, the perceptron algorithm  does not generally converge to the solution with lowest generalization error. For the case  of unrealizability due to a simple output noise, we propose a new on-line algorithm for a  perceptron yielding a learning curve that can approach the optimal generalization error as  fast as p-l/2. We then generalize the perceptron algorithm to any class of thresholded  smooth functions learning a target from that class. For "well-behaved" input distributions,  if this algorithm converges to the optimal solution, its learning curve can decrease as fast  as p-l.
Asymptotics of Gradient-based Neural Network Training Algorithms
Authors: Sayandev Mukherjee, Terrence L. Fine
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We study the asymptotic properties of the sequence of iterates of  weight-vector estimates obtained by training a multilayer feed for(cid:173) ward neural network with a basic gradient-descent method using  a fixed learning constant and no batch-processing. In the one(cid:173) dimensional case, an exact analysis establishes the existence of a  limiting distribution that is not Gaussian in general. For the gen(cid:173) eral case and small learning constant, a linearization approximation  permits the application of results from the theory of random ma(cid:173) trices to again establish the existence of a limiting distribution.  We study the first few moments of this distribution to compare  and contrast the results of our analysis with those of techniques of  stochastic approximation.
Convergence Properties of the K-Means Algorithms
Authors: Léon Bottou, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper studies the convergence properties of the well known  K-Means clustering algorithm. The K-Means algorithm can be de(cid:173) scribed either as a gradient descent algorithm or by slightly extend(cid:173) ing the mathematics of the EM algorithm to this hard threshold  case. We show that the K-Means algorithm actually minimizes the  quantization error using the very fast Newton algorithm.
Using Voice Transformations to Create Additional Training Talkers for Word Spotting
Authors: Eric I. Chang, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Speech  recognizers  provide  good  performance  for  most  users  but  the  error rate often  increases dramatically  for a small  percentage of talkers  who are "different" from  those talkers used for training.  One expensive  solution to  this problem is  to  gather more training data in an attempt to  sample these outlier users. A second solution, explored in this paper,  is  to artificially enlarge the number of training talkers by transforming the  speech of existing training talkers. This approach is similar to enlarging  the training set for  OCR digit recognition  by  warping the training digit  images,  but  is  more  difficult  because  continuous  speech  has  a  much  larger number of dimensions  (e.g.  linguistic,  phonetic,  style,  temporal,  spectral) that differ across talkers. We explored the use of simple linear  spectral warping to enlarge a 48-talker training data base used for  word  spotting.  The  average  detection  rate  overall  was  increased  by  2.9  percentage  points  (from  68.3%  to  71.2%)  for  male  speakers  and  2.5  percentage  points  (from  64.8%  to  67.3%)  for  female  speakers.  This  increase is  small but similar to  that obtained by  doubling the amount of  training data.
Forward dynamic models in human motor control: Psychophysical evidence
Authors: Daniel M. Wolpert, Zoubin Ghahramani, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Based  on  computational  principles,  with  as  yet  no  direct  experi(cid:173) mental  validation,  it  has  been  proposed  that  the  central  nervous  system  (CNS)  uses  an internal model to simulate the  dynamic be(cid:173) havior of the motor system in planning, control and learning (Sut(cid:173) ton  and  Barto,  1981;  Ito,  1984;  Kawato  et  aI.,  1987;  Jordan  and  Rumelhart,  1992;  Miall et aI.,  1993).  We  present  experimental re(cid:173) sults  and simulations based on a  novel  approach  that investigates  the temporal propagation of errors  in the sensorimotor integration  process.  Our results  provide  direct  support for  the  existence  of an  internal model.
Direction Selectivity In Primary Visual Cortex Using Massive Intracortical Connections
Authors: Humbert Suarez, Christof Koch, Rodney Douglas
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Almost all models of orientation and direction selectivity in visual  cortex are based on feedforward connection schemes, where genicu(cid:173) late input provides all excitation to both pyramidal and inhibitory  neurons. The latter neurons then suppress the response of the for(cid:173) mer for non-optimal stimuli. However, anatomical studies show  that up to 90 % of the excitatory synaptic input onto any corti(cid:173) cal cell is provided by other cortical cells. The massive excitatory  feedback nature of cortical circuits is embedded in the canonical  microcircuit of Douglas &. Martin (1991). We here investigate ana(cid:173) lytically and through biologically realistic simulations the function(cid:173) ing of a detailed model of this circuitry, operating in a hysteretic  mode. In the model, weak geniculate input is dramatically ampli(cid:173) fied by intracortical excitation, while inhibition has a dual role: (i)  to prevent the early geniculate-induced excitation in the null di(cid:173) rection and (ii) to restrain excitation and ensure that the neurons  fire only when the stimulus is in their receptive-field. Among the 
4 
Humbert Suarez, Christo! Koch, Rodney Douglas 
insights gained are the possibility that hysteresis underlies visual  cortical function, paralleling proposals for short-term memory, and  strong limitations on linearity tests that use gratings. Properties  of visual cortical neurons are compared in detail to this model and  to a classical model of direction selectivity that does not include  excitatory corti co-cortical connections. The model explain a num(cid:173) ber of puzzling features of direction-selective simple cells, includ(cid:173) ing the small somatic input conductance changes that have been  measured experimentally during stimulation in the null direction.  The model also allows us to understand why the velocity-response  curve of area 17 neurons is different from that of their LG N affer(cid:173) ents, and the origin of expansive and compressive nonlinearities in  the contrast-response curve of striate cortical neurons.
Bayesian Query Construction for Neural Network Models
Authors: Gerhard Paass, Jörg Kindermann
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: If data collection is costly, there is much to be gained by actively se(cid:173) lecting particularly informative data points in a sequential  way.  In  a  Bayesian decision-theoretic framework we  develop  a  query selec(cid:173) tion  criterion  which  explicitly takes  into account  the intended  use  of the model predictions.  By  Markov Chain Monte Carlo methods  the  necessary  quantities  can  be  approximated to  a  desired  preci(cid:173) sion.  As  the  number  of data  points  grows,  the  model  complexity  is  modified  by  a  Bayesian  model  selection  strategy.  The  proper(cid:173) ties of two  versions  of the criterion  ate demonstrated in numerical  experiments.
A Silicon Axon
Authors: Bradley A. Minch, Paul E. Hasler, Chris Diorio, Carver Mead
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  silicon  model of an  axon  which  shows  promise  as  a  building  block for  pulse-based  neural  computations involving cor(cid:173) relations of pulses  across  both space  and time.  The  circuit shares  a  number  of features  with  its  biological  counterpart  including  an  excitation  threshold,  a  brief refractory  period  after  pulse  comple(cid:173) tion, pulse amplitude restoration, and pulse width restoration.  We  provide  a simple explanation of circuit operation and present  data  from  a  chip fabricated  in  a standard  2Jlm  CMOS  process  through  the MOS  Implementation Service  (MOSIS).  We  emphasize the ne(cid:173) cessity of the restoration of the width of the pulse in time for stable  propagation in  axons.
Plasticity-Mediated Competitive Learning
Authors: Nicol N. Schraudolph, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Differentiation between the nodes of a competitive learning net(cid:173) work is conventionally achieved through competition on the ba(cid:173) sis of neural activity.  Simple inhibitory mechanisms are  limited  to  sparse  representations,  while  decorrelation  and  factorization  schemes that support distributed representations are computation(cid:173) ally unattractive.  By letting neural plasticity mediate the compet(cid:173) itive interaction instead, we obtain diffuse, nonadaptive alterna(cid:173) tives for fully distributed representations.  We  use this technique  to  Simplify and improve our binary information gain optimiza(cid:173) tion algorithm for feature extraction (Schraudolph and Sejnowski,  1993); the same approach could be used to improve other learning  algorithms.
Patterns of damage in neural networks: The effects of lesion area, shape and number
Authors: Eytan Ruppin, James A. Reggia
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Current understanding of the effects of damage on neural networks  is  rudimentary, even  though such  understanding  could lead to im(cid:173) portant insights concerning neurological and psychiatric disorders.  Motivated  by  this  consideration,  we  present  a  simple  analytical  framework for  estimating the functional  damage resulting from fo(cid:173) cal  structural  lesions  to  a  neural  network.  The  effects  of focal  le(cid:173) sions of varying area,  shape and number on the retrieval capacities  of a spatially-organized associative memory.  Although our analyti(cid:173) cal results are based on some approximations, they correspond well  with simulation results.  This study sheds light on  some important  features  characterizing  the  clinical  manifestations of multi-infarct  dementia, including the strong  association between  the number of  infarcts and the prevalence of dementia after stroke,  and the 'mul(cid:173) tiplicative' interaction that has been  postulated  to occur  between  Alzheimer's disease  and multi-infarct dementia. 
*Dr.  Reggia  is  also  with  the  Department  of Neurology  and  the Institute  of Advanced 
Computer Studies  at the University  of Maryland. 
36 
Eytan  Ruppin,  James A.  Reggia
A Study of Parallel Perturbative Gradient Descent
Authors: D. Lippe, Joshua Alspector
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  have  continued  our  study  of a  parallel  perturbative  learning  method  [Alspector  et al.,  1993]  and implications for  its implemen(cid:173) tation in analog VLSI. Our new results indicate that, in most cases,  a single parallel perturbation (per pattern presentation) of the func(cid:173) tion  parameters  (weights in a  neural network)  is  theoretically  the  best  course.  This  is  not  true,  however,  for  certain  problems  and  may  not  generally  be  true  when  faced  with  issues  of implemen(cid:173) tation  such  as limited  precision.  In  these  cases,  multiple  parallel  perturbations may be best as indicated in our previous results.
A Neural Model of Delusions and Hallucinations in Schizophrenia
Authors: Eytan Ruppin, James A. Reggia, David Horn
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  implement and study a computational model of Stevens'  [19921  theory  of the  pathogenesis  of schizophrenia.  This  theory  hypoth(cid:173) esizes  that  the  onset  of schizophrenia  is  associated  with  reactive  synaptic regeneration occurring in brain regions receiving degener(cid:173) ating temporal lobe  projections.  Concentrating on  one  such  area,  the  frontal  cortex,  we  model  a  frontal  module  as  an  associative  memory neural  network  whose  input synapses  represent  incoming  temporal  projections.  We  analyze  how,  in  the  face  of weakened  external input projections, compensatory strengthening of internal  synaptic connections and increased  noise levels can maintain mem(cid:173) ory  capacities  (which  are  generally  preserved  in  schizophrenia) .  However,  These  compensatory  changes  adversely  lead  to  sponta(cid:173) neous,  biased  retrieval  of stored  memories,  which  corresponds  to  the occurrence  of schizophrenic  delusions  and hallucinations with(cid:173) out  any  apparent  external  trigger,  and  for  their  tendency  to  con(cid:173) centrate on just few  central themes.  Our  results explain why  these  symptoms tend  to wane  as  schizophrenia progresses,  and why  de(cid:173) layed  therapeutical intervention leads to a  much slower  response. 
150 
Eytan Ruppin,  James  A.  Reggia,  David Hom
Correlation and Interpolation Networks for Real-time Expression Analysis/Synthesis
Authors: Trevor Darrell, Irfan A. Essa, Alex Pentland
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe  a  framework  for  real-time  tracking  of facial  expressions  that  uses  neurally-inspired  correlation  and  interpolation  methods.  A  distributed view-based representation is used to characterize facial state,  and is  computed using a  replicated correlation network.  The ensemble  response of the set of view correlation scores is input to a network based  interpolation method, which maps perceptual state to motor control states  for  a  simulated  3-D  face  model.  Activation levels  of the  motor  state  correspond to muscle activations in an  anatomically derived model.  By  integrating fast and robust 2-D processing with 3-D models, we obtain a  system that is able to quickly track and interpret complex facial motions  in real-time.
Neural Network Ensembles, Cross Validation, and Active Learning
Authors: Anders Krogh, Jesper Vedelsby
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Learning  of continuous  valued  functions  using  neural  network  en(cid:173) sembles  (committees) can give  improved accuracy,  reliable estima(cid:173) tion of the generalization error,  and active learning.  The  ambiguity  is  defined as the variation of the output of ensemble members aver(cid:173) aged  over  unlabeled  data, so  it quantifies the  disagreement  among  the networks.  It is  discussed  how  to use the ambiguity in combina(cid:173) tion with cross-validation to give a reliable estimate of the ensemble  generalization error, and how this type of ensemble cross-validation  can sometimes improve performance.  It is  shown  how  to estimate  the optimal weights of the ensemble members using unlabeled  data.  By a generalization of query  by  committee, it is finally shown how  the ambiguity can be used to select new training data to be labeled  in an active learning scheme.
A model of the hippocampus combining self-organization and associative memory function
Authors: Michael E. Hasselmo, Eric Schnell, Joshua Berke, Edi Barkai
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A model of the hippocampus is presented which forms rapid self -orga(cid:173) nized representations of input arriving via the perforant path, performs  recall of previous associations in region CA3, and performs comparison  of this recall with afferent input in region CA 1.  This comparison drives  feedback regulation of cholinergic modulation to set appropriate  dynamics for learning of new representations in region CA3 and CA 1.  The network responds to novel patterns with increased cholinergic mod(cid:173) ulation, allowing storage of new self-organized representations, but  responds to familiar patterns with a decrease in acetylcholine,  allowing  recall based on previous representations.  This requires selectivity of the  cholinergic suppression of synaptic transmission in stratum radiatum of  regions CA3 and CAl, which has been demonstrated experimentally.
Glove-TalkII: Mapping Hand Gestures to Speech Using Neural Networks
Authors: Sidney Fels, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Glove-TaikII is  a  system which  translates  hand gestures  to speech  through  an  adaptive interface.  Hand  gestures  are  mapped  contin(cid:173) uously  to  10  control  parameters  of a  parallel formant  speech  syn(cid:173) thesizer.  The mapping allows the hand to act  as  an  artificial  vocal  tract  that  produces  speech  in  real  time.  This  gives  an  unlimited  vocabulary  in addition to direct  control of fundamental  frequency  and  volume.  Currently,  the  best  version  of Glove-TalkII  uses  sev(cid:173) eral  input  devices  (including  a  CyberGlove,  a  ContactGlove,  a  3- space  tracker,  and a foot-pedal),  a  parallel formant speech  synthe(cid:173) sizer  and  3 neural  networks.  The gesture-to-speech  task  is  divided  into  vowel  and  consonant  production  by  using  a  gating  network  to weight  the outputs of a  vowel  and  a  consonant  neural  network.  The  gating  network  and  the  consonant  network  are  trained  with  examples  from  the  user.  The  vowel  network  implements  a  fixed,  user-defined  relationship  between  hand-position  and  vowel  sound  and does  not require any training examples from the user.  Volume,  fundamental  frequency  and  stop  consonants  are  produced  with  a  fixed  mapping from  the  input devices.  One subject  has  trained  to  speak  intelligibly with Glove-TalkII. He  speaks slowly with speech  quality  similar  to a  text-to-speech  synthesizer  but  with  far  more  natural-sounding pitch  variations. 
844 
S.  Sidney  Fe Is,  Geoffrey  Hinton
Learning Saccadic Eye Movements Using Multiscale Spatial Filters
Authors: Rajesh P. N. Rao, Dana H. Ballard
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  describe  a  framework  for  learning  saccadic  eye  movements  using  a  photometric representation of target points  in natural scenes.  The rep(cid:173) resentation takes the form of a high-dimensional vector comprised of the  responses  of spatial filters  at different  orientations and  scales.  We  first  demonstrate the use  of this  response  vector in  the task of locating pre(cid:173) viously foveated  points in a  scene and subsequently use  this property in  a  multisaccade strategy to derive  an adaptive motor map for  delivering  accurate saccades.
A Charge-Based CMOS Parallel Analog Vector Quantizer
Authors: Gert Cauwenberghs, Volnei Pedroni
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We present an analog VLSI chip for parallel analog vector quantiza(cid:173) tion. The MOSIS 2.0 J..Lm double-poly CMOS Tiny chip contains an  array of 16 x 16 charge-based distance estimation cells, implementing a  mean absolute difference (MAD) metric operating on a 16-input analog  vector field and 16 analog template vectors. The distance cell includ(cid:173) ing dynamic template storage measures 60 x 78 J..Lm2 • Additionally,  the chip features a winner-take-all (WTA) output circuit of linear com(cid:173) plexity, with global positive feedback for fast and decisive settling of a  single winner output. Experimental results on the complete 16 x 16 VQ  system demonstrate correct operation with 34 dB analog input dynamic  range and 3 J..Lsec cycle time at 0.7 mW power dissipation.
Boosting the Performance of RBF Networks with Dynamic Decay Adjustment
Authors: Michael R. Berthold, Jay Diamond
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Radial  Basis  Function  (RBF)  Networks,  also  known  as  networks  of locally-tuned processing units  (see  [6])  are well  known for  their  ease  of use.  Most  algorithms  used  to  train  these  types  of  net(cid:173) works,  however,  require a  fixed  architecture, in  which  the number  of units  in  the  hidden  layer  must  be  determined  before  training  starts.  The RCE training algorithm, introduced by  Reilly,  Cooper  and Elbaum  (see  [8]),  and its  probabilistic extension,  the P-RCE  algorithm,  take advantage of a  growing structure in  which  hidden  units are only introduced when  necessary.  The nature of these al(cid:173) gorithms allows training to reach stability much faster  than is  the  case  for  gradient-descent  based  methods.  Unfortunately  P-RCE  networks do not adjust the standard deviation of their prototypes  individually, using only one global value for  this parameter.  This  paper introduces the Dynamic Decay Adjustment  (DDA)  al(cid:173) gorithm  which  utilizes  the  constructive  nature  of the  P-RCE  al(cid:173) gorithm together with independent adaptation of each prototype's  decay factor.  In addition, this radial adjustment is  class dependent  and  distinguishes  between  different  neighbours.  It is  shown  that  networks  trained  with  the  presented  algorithm  perform  substan(cid:173) tially better than common RBF networks. 
522 
Michael  R.  Berthold,  Jay  Diamond
An Alternative Model for Mixtures of Experts
Authors: Lei Xu, Michael I. Jordan, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We propose an alternative model for mixtures of experts which uses  a  different  parametric form  for  the gating network.  The  modified  model is  trained by the EM  algorithm.  In comparison with earlier  models-trained by either EM  or gradient ascent-there is no need  to  select  a  learning  stepsize.  We  report  simulation experiments  which  show  that  the  new  architecture  yields  faster  convergence.  We  also  apply the new  model to two  problem domains:  piecewise  nonlinear function approximation and the combination of multiple  previously  trained classifiers.
Catastrophic Interference in Human Motor Learning
Authors: Tom Brashers-Krug, Reza Shadmehr, Emanuel Todorov
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Biological sensorimotor systems are not static maps that transform  input  (sensory  information)  into  output  (motor  behavior).  Evi(cid:173) dence  from  many  lines  of research  suggests  that  their  representa(cid:173) tions are plastic, experience-dependent entities.  While this plastic(cid:173) ity is  essential for  flexible  behavior, it presents  the nervous system  with difficult organizational challenges.  If the sensorimotor system  adapts itself to perform well  under one set of circumstances,  will it  then perform poorly when placed  in an environment with different  demands  (negative  transfer)?  Will  a  later  experience-dependent  change  undo  the benefits  of previous  learning  (catastrophic  inter(cid:173) ference)?  We  explore  the first  question  in  a separate paper in  this  volume  (Shadmehr et  al.  1995).  Here  we  present  psychophysical  and computational results that explore the question of catastrophic  interference  in the context of a  dynamic motor learning task.  Un(cid:173) der  some conditions, subjects show  evidence  of catastrophic  inter(cid:173) ference.  Under  other  conditions,  however,  subjects  appear  to  be  immune to  its  effects.  These  results  suggest  that  motor  learning  can  undergo  a  process  of consolidation.  Modular neural  networks  are  well  suited  for  the  demands of learning multiple input/output  mappings.  By  incorporating the notion of fast- and slow-changing  connections  into  a  modular architecture,  we  were  able  to  account  for  the psychophysical results. 
20 
Tom  Brashers-Krug,  Reza Shadmelzr,  Emanuel Todorov
New Algorithms for 2D and 3D Point Matching: Pose Estimation and Correspondence
Authors: Steven Gold, Chien-Ping Lu, Anand Rangarajan, Suguna Pappu, Eric Mjolsness
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  fundamental  open  problem  in  computer  vision-determining  pose  and  correspondence  between  two  sets  of  points  in  space(cid:173) is  solved with a novel, robust  and easily implementable algorithm.  The  technique  works  on  noisy  point sets  that  may be  of unequal  sizes  and  may  differ  by  non-rigid  transformations.  A  2D  varia(cid:173) tion  calculates  the  pose  between  point  sets  related  by  an  affine  transformation-translation, rotation, scale and shear.  A 3D to 3D  variation calculates translation and rotation.  An objective describ(cid:173) ing  the  problem is  derived  from  Mean field  theory.  The objective  is  minimized with clocked  (EM-like)  dynamics.  Experiments with  both  handwritten  and  synthetic  data provide  empirical  evidence  for  the method.