mtybilly/Instruct2DS huggingface dataset:
Statistics of Natural Images: Scaling in the Woods
Authors: Daniel L. Ruderman, William Bialek
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  order  to  best  understand  a  visual  system  one  should  attempt  to characterize  the natural images it processes.  We  gather  images  from the woods and find that these scenes possess an ensemble scale  invariance.  Further,  they  are  highly  non-Gaussian,  and  this  non(cid:173) Gaussian  character  cannot  be  removed  through local  linear filter(cid:173) ing.  We find  that including a simple  "gain control"  nonlinearity in  the filtering  process  makes the filter  output quite Gaussian, mean(cid:173) ing information is maximized at fixed  channel variance.  Finally, we  use  the measured power spectrum to place  an upper bound on the  information conveyed about natural scenes by an array of receptors.
Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach
Authors: Justin A. Boyan, Michael L. Littman
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This  paper  describes  the  Q-routing  algorithm for  packet  routing,  in  which  a  reinforcement  learning  module  is  embedded  into  each  node  of a  switching  network.  Only  local  communication  is  used  by each node to keep  accurate statistics on which routing decisions  lead  to  minimal  delivery  times.  In  simple  experiments  involving  a  36-node,  irregularly  connected  network,  Q-routing  proves  supe(cid:173) rior  to  a  nonadaptive  algorithm  based  on  precomputed  shortest  paths  and  is  able  to route  efficiently  even  when  critical  aspects  of  the  simulation,  such  as  the  network  load,  are  allowed  to  vary  dy(cid:173) namically.  The  paper  concludes  with  a  discussion  of the  tradeoff  between  discovering  shortcuts  and  maintaining stable policies.
A Massively-Parallel SIMD Processor for Neural Network and Machine Vision Applications
Authors: Michael A. Glover, W. Thomas Miller III
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper describes the MM32k, a massively-parallel SIMD com(cid:173) puter which is easy to program, high in performance, low in cost  and effective for implementing highly parallel neural network ar(cid:173) chitectures. The MM32k has 32768 bit serial processing elements,  each of which has 512 bits of memory, and all of which are inter(cid:173) connected by a switching network. The entire system resides on  a single PC-AT compatible card. It is programmed from the host  computer using a C++ language class library which abstracts the  parallel processor in terms of fast arithmetic operators for vectors  of variable precision integers.
How to Choose an Activation Function
Authors: H. N. Mhaskar, C. A.. Micchelli
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We study the complexity problem in artificial feedforward neural networks  designed to approximate real valued functions of several real variables; i.e.,  we estimate the number of neurons in a network required to ensure a given  degree of approximation to every function in a given function class. We  indicate how to construct networks with the indicated number of neurons  evaluating standard activation functions. Our general theorem shows that  the smoother the activation function, the better the rate of approximation.
Efficient Simulation of Biological Neural Networks on Massively Parallel Supercomputers with Hypercube Architecture
Authors: Ernst Niebur, Dean Brettle
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present  a  neural  network  simulation  which  we  implemented  on  the  massively  parallel  Connection  Machine  2.  In  contrast  to  previous  work,  this simulator is  based on biologically realistic neu(cid:173) rons  with nontrivial single-cell  dynamics,  high  connectivity with a  structure  modelled in  agreement  with  biological data,  and preser(cid:173) vation of the temporal dynamics of spike interactions.  We simulate  neural networks of 16,384 neurons  coupled by about 1000 synapses  per neuron, and estimate the performance for  much larger systems.  Communication between neurons  is  identified as the computation(cid:173) ally  most  demanding task  and we  present a  novel method to over(cid:173) come this bottleneck.  The simulator has already been used to study  the primary visual system of the  cat.
Convergence of Stochastic Iterative Dynamic Programming Algorithms
Authors: Tommi Jaakkola, Michael I. Jordan, Satinder P. Singh
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Increasing attention has recently  been  paid to algorithms based on  dynamic programming (DP) due  to the suitability of DP for  learn(cid:173) ing  problems  involving control.  In  stochastic  environments  where  the  system  being  controlled  is  only  incompletely  known,  however,  a  unifying theoretical  account  of these  methods has  been  missing.  In  this  paper  we  relate  DP-based  learning  algorithms to  the  pow(cid:173) erful  techniques of stochastic approximation via a  new convergence  theorem,  enabling us  to establish  a  class  of convergent  algorithms  to which  both TD(")  and Q-Iearning belong.
High Performance Neural Net Simulation on a Multiprocessor System with "Intelligent" Communication
Authors: Urs A. Müller, Michael Kocheisen, Anton Gunzinger
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The performance requirements in experimental research on arti(cid:173) ficial neural nets often exceed the capability of workstations and  PCs by a great amount. But speed is not the only requirement.  Flexibility and implementation time for new algorithms are usually  of equal importance. This paper describes the simulation of neural  nets on the MUSIC parallel supercomputer, a system that shows a  good balance between the three issues and therefore made many  research projects possible that were unthinkable before. (MUSIC  stands for Multiprocessor System with Intelligent Communication) 
1 Overview of the MUSIC System 
The goal of the MUSIC project was to build a fast parallel system and to use it in  real-world applications like neural net simulations, image processing or simulations  in chemistry and physics [1, 2]. The system should be flexible, simple to program  and the realization time should be short enough to not have an obsolete system by  the time it is finished. Therefore, the fastest available standard components were  used. The key idea of the architecture is to support the collection and redistribution  of complete data blocks by a simple, efficient and autonomously working commu(cid:173) nication network realized in hardware. Instead of considering where to send data  and where from to receive data, each processing element determines which part of  a (virtual) data block it has produced and which other part of the same data block  it wants to receive for the continuation of the algorithm.
Non-Linear Statistical Analysis and Self-Organizing Hebbian Networks
Authors: Jonathan L. Shapiro, Adam Prügel-Bennett
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Neurons learning under an unsupervised  Hebbian learning rule can  perform a nonlinear generalization of principal component analysis.  This relationship between  nonlinear PCA  and nonlinear neurons is  reviewed.  The stable fixed  points of the neuron learning dynamics  correspond  to  the  maxima of the  statist,ic  optimized  under  non(cid:173) linear  PCA.  However,  in  order  to predict.  what  the  neuron  learns,  knowledge  of the  basins  of attractions  of the  neuron  dynamics  is  required.  Here  the  correspondence  between  nonlinear  PCA  and  neural  networks  breaks  down.  This  is  shown  for  a  simple  model.  Methods  of statistical  mechanics  can  be  used  to  find  the  optima  of the objective function  of non-linear PCA. This determines  what  the neurons  can learn.  In order to find  how  the solutions are parti(cid:173) tioned amoung the neurons, however, one must solve the dynamics.
Locally Adaptive Nearest Neighbor Algorithms
Authors: Dietrich Wettschereck, Thomas G. Dietterich
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Four versions of a  k-nearest  neighbor algorithm with locally adap(cid:173) tive  k  are  introduced  and  compared  to  the  basic  k-nearest  neigh(cid:173) bor algorithm (kNN).  Locally adaptive kNN  algorithms choose the  value of k  that should be used  to classify a query by consulting the  results  of cross-validation computations in  the local  neighborhood  of the query.  Local kNN  methods are shown  to perform similar to  kNN in experiments with twelve commonly used data sets.  Encour(cid:173) aging  results  in  three  constructed  tasks  show  that  local  methods  can  significantly  outperform  kNN  in  specific  applications.  Local  methods  can  be  recommended  for  on-line  learning  and  for  appli(cid:173) cations  where  different  regions  of the  input  space  are  covered  by  patterns solving different  sub-tasks.
A Comparative Study of a Modified Bumptree Neural Network with Radial Basis Function Networks and the Standard Multi Layer Perceptron
Authors: Richard T. J. Bostock, Alan J. Harget
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Bumptrees  are  geometric  data  structures  introduced  by  Omohundro  (1991)  to  provide  efficient  access  to  a  collection  of functions  on  a  Euclidean space of interest.  We describe a modified bumptree structure  that has been employed as  a neural network classifier, and compare its  performance on  several classification  tasks against that of radial basis  function networks and the standard mutIi-Iayer perceptron.
Combined Neural Networks for Time Series Analysis
Authors: Iris Ginzburg, David Horn
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  propose  a  method  for  improving  the  performance  of any  net(cid:173) work  designed  to predict  the next value of a  time series.  Vve  advo(cid:173) cate analyzing the deviations of the network's predictions from the  data in the training set.  This can be carried out by a secondary net(cid:173) work  trained  on  the  time series  of these  residuals.  The  combined  system  of  the  two  networks  is  viewed  as  the  new  predictor.  We  demonstrate  the  simplicity  and  success  of this  method,  by  apply(cid:173) ing it to the sunspots data.  The small corrections of the secondary  network  can  be  regarded  as  resulting  from  a  Taylor  expansion  of  a  complex  network  which  includes  the  combined system.  \Te  find  that  the  complex network  is  more  difficult  to  train  and  performs  worse  than the two-step  procedure of the  combined system.
Illumination-Invariant Face Recognition with a Contrast Sensitive Silicon Retina
Authors: Joachim M. Buhmann, Martin Lades, Frank Eeckman
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Changes in lighting conditions strongly effect the performance and reli(cid:173) ability of computer vision systems. We report face recognition results  under drastically changing lighting conditions for a computer vision sys(cid:173) tem which concurrently uses a contrast sensitive silicon retina and a con(cid:173) ventional, gain controlled CCO camera. For both input devices the face  recognition system employs an elastic matching algorithm with wavelet  based features to classify unknown faces. To assess the effect of analog  on -chip preprocessing by the silicon retina the CCO images have been  "digitally preprocessed" with a bandpass filter to adjust the power spec(cid:173) trum. The silicon retina with its ability to adjust sensitivity increases  the recognition rate up to 50 percent. These comparative experiments  demonstrate that preprocessing with an analog VLSI silicon retina gen(cid:173) erates image data enriched with object-constant features.
Address Block Location with a Neural Net System
Authors: Hans Peter Graf, Eric Cosatto
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We developed a system for finding address blocks on mail pieces that can  process four  images per second. Besides  locating the  address block,  our  system also determines the writing style, handwritten or machine printed, and  moreover,  it measures  the  skew  angle  of the  text  lines  and  cleans noisy  images.  A  layout  analysis  of all  the  elements  present  in  the  image  is  performed in order to distinguish drawings and dirt from text and to separate  text of advertisement from that of the destination address.  A  speed  of more  than  four  images per second  is obtained on a  modular  hardware platform, containing a board with two of the NET32K neural net  chips,  a  SP ARC2  processor  board,  and  a  board  with  2  digital  signal  processors. The system has been tested with more than 100,000 images. Its  performance depends on the quality of the images, and lies between 85%  correct location in very noisy images to over 98% in cleaner images.
Assessing the Quality of Learned Local Models
Authors: Stefan Schaal, Christopher G. Atkeson
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: An  approach  is  presented  to  learning  high  dimensional functions  in  the  case  where  the  learning  algorithm  can  affect  the  generation  of new  data.  A  local  modeling algorithm, locally weighted regression, is used to represent the learned  function.  Architectural parameters of the approach, such as distance metrics, are  also localized and become a function of the query point instead of being global.  Statistical tests are given for  when a local model is good enough and sampling  should be moved to a new area. Our methods explicitly deal with the case where  prediction accuracy requirements exist during exploration: By gradually shifting  a "center of exploration" and controlling the speed of the  shift with local pre(cid:173) diction accuracy, a goal-directed exploration of state space takes place along the  fringes  of the current data support until  the task goal is achieved. We illustrate  this  approach  with  simulation results  and results  from  a real  robot learning  a  complex juggling task.
Synchronization, oscillations, and 1/f noise in networks of spiking neurons
Authors: Martin Stemmler, Marius Usher, Christof Koch, Zeev Olami
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We investigate a model for neural activity that generates long range  temporal correlations, 1/ f  noise, and oscillations in global activity.  The model  consists  of a  two-dimensional sheet  of leaky  integrate(cid:173) and-fire  neurons  with feedback  connectivity consisting of local ex(cid:173) citation  and  surround  inhibition.  Each  neuron  is  independently  driven  by  homogeneous  external  noise.  Spontaneous  symmetry  breaking occurs,  resulting in  the formation of "hotspots"  of activ(cid:173) ity in  the  network.  These  localized  patterns  of excitation  appear  as  clusters  that coalesce,  disintegrate, or fluctuate  in size  while si(cid:173) multaneously moving in a random walk constrained by the interac(cid:173) tion  with other  clusters.  The emergent  cross-correlation functions  have  a  dual  structure,  with  a  sharp  peak  around  zero  on  top  of  a  much broader  hill .  The  power  spectrum  associated  with  single  units  shows a  1/ f  decay for  small frequencies  and is flat  at higher  frequencies,  while the power spectrum of the spiking activity aver(cid:173) aged over many cells-equivalent to the local field  potential-shows  no  1/ f  decay  but a prominent peak  around 40  Hz.
A Hodgkin-Huxley Type Neuron Model That Learns Slow Non-Spike Oscillation
Authors: Kenji Doya, Allen I. Selverston, Peter F. Rowat
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: A  gradient  descent  algorithm  for  parameter  estimation  which  is  similar to those used for  continuous-time recurrent neural networks  was  derived for  Hodgkin-Huxley type  neuron models.  Using mem(cid:173) brane  potential  trajectories  as  targets,  the  parameters  (maximal  conductances,  thresholds and slopes of activation curves, time con(cid:173) stants)  were  successfully estimated.  The  algorithm was  applied  to  modeling  slow  non-spike  oscillation  of an  identified  neuron  in  the  lobster stomatogastric ganglion.  A model with three ionic  currents  was  trained  with  experimental  data.  It revealed  a  novel  role  of  A-current for  slow oscillation below -50  mY.
A Comparison of Dynamic Reposing and Tangent Distance for Drug Activity Prediction
Authors: Thomas G. Dietterich, Ajay N. Jain, Richard H. Lathrop, Tomás Lozano-Pérez
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  drug  activity  prediction  (as  in  handwritten  character  recogni(cid:173) tion), the features extracted to describe a training example depend  on  the  pose  (location, orientation, etc.)  of the example.  In  hand(cid:173) written  character  recognition,  one  of the  best  techniques  for  ad(cid:173) dressing  this  problem  is  the  tangent  distance  method  of Simard,  LeCun  and Denker  (1993).  Jain, et al.  (1993a;  1993b) introduce  a  new  technique-dynamic reposing-that also  addresses  this  prob(cid:173) lem.  Dynamic reposing iteratively learns a neural network and then  reposes  the  examples in  an effort  to  maximize  the  predicted  out(cid:173) put values.  New  models are trained and new  poses computed until  models and poses converge.  This paper compares dynamic reposing  to the  tangent  distance  method on the  task of predicting  the bio(cid:173) logical  activity  of musk  compounds.  In  a  20-fold cross-validation,
Hoo Optimality Criteria for LMS and Backpropagation
Authors: Babak Hassibi, Ali H. Sayed, Thomas Kailath
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  have  recently  shown that the  widely  known  LMS  algorithm is  an H OO  optimal estimator.  The H OO  criterion  has been introduced,  initially in  the  control  theory  literature,  as  a  means to ensure  ro(cid:173) bust  performance  in  the  face  of model  uncertainties  and  lack  of  statistical  information on  the  exogenous  signals.  We  extend  here  our  analysis  to  the  nonlinear  setting  often  encountered  in  neural  networks,  and show  that the  backpropagation  algorithm is  locally  H OO  optimal.  This fact  provides  a  theoretical justification  of the  widely  observed  excellent  robustness  properties  of the  LMS  and  backpropagation algorithms.  We further discuss  some implications  of these  results.
Classifying Hand Gestures with a View-Based Distributed Representation
Authors: Trevor J. Darrell, Alex P. Pentland
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We present a method for learning, tracking, and recognizing human hand  gestures recorded by a conventional CCD camera without any special  gloves or other sensors. A view-based representation is used to model  aspects of the hand relevant to the trained gestures, and is found using an  unsupervised clustering technique. We use normalized correlation net(cid:173) works, with dynamic time warping in the temporal domain, as a distance  function for unsupervised clustering. Views are computed separably for  space and time dimensions; the distributed response of the combination  of these units characterizes the input data with a low dimensional repre(cid:173) sentation. A supervised classification stage uses labeled outputs of the  spatio-temporal units as training data. Our system can correctly classify  gestures in real time with a low-cost image processing accelerator.
Amplifying and Linearizing Apical Synaptic Inputs to Cortical Pyramidal Cells
Authors: Öjvind Bernander, Christof Koch, Rodney J. Douglas
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Intradendritic  electrophysiological recordings  reveal  a  bewildering  repertoire  of complex  electrical  spikes  and  plateaus  that  are  dif(cid:173) ficult  to  reconcile  with  conventional notions  of neuronal function.  In  this paper  we  argue  that such  dendritic events are just an exu(cid:173) berant expression of a more important mechanism - a proportional  current amplifier whose primary task is to offset electrotonic losses.  Using the example of functionally important synaptic inputs to the  superficial  layers of an  anatomically and  electrophysiologically re(cid:173) constructed  layer  5 pyramidal neuron,  we  derive  and  simulate the  properties of conductances  that linearize and amplify distal synap(cid:173) tic  input  current  in  a  graded  manner.  The  amplification  depends  on a potassium conductance in the apical tuft and calcium conduc(cid:173) tances  in  the  apical trunk. 
·To whom  all  correspondence  should  be addressed.
Decoding Cursive Scripts
Authors: Yoram Singer, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Online cursive handwriting recognition  is  currently one of the most  intriguing challenges in  pattern recognition.  This study  presents  a  novel  approach  to this  problem which  is  composed  of two  comple(cid:173) mentary  phases.  The first  is  dynamic encoding  of the  writing  tra(cid:173) jectory into a  compact sequence  of discrete  motor control symbols.  In this compact representation we largely remove the redundancy of  the script,  while  preserving  most of its intelligible components.  In  the second  phase these  control sequences  are used  to train adaptive  probabilistic acyclic automata (PAA) for the important ingredients  of the writing  trajectories,  e.g.  letters.  We  present  a  new  and effi(cid:173) cient  learning algorithm for such stochastic automata, and demon(cid:173) strate  its  utility  for  spotting  and  segmentation of cursive  scripts.  Our  experiments  show  that  over  90%  of  the  letters  are  correctly  spotted  and  identified,  prior to  any  higher  level  language  model.  Moreover,  both  the  training  and  recognition  algorithms  are  very  efficient  compared to other  modeling methods, and the models are  'on-line' adaptable to other  writers  and  styles.
Foraging in an Uncertain Environment Using Predictive Hebbian Learning
Authors: P. Read Montague, Peter Dayan, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Survival  is  enhanced  by  an  ability  to  predict  the  availability  of food,  the  likelihood of predators,  and  the  presence  of mates.  We  present  a  concrete model that uses diffuse neurotransmitter systems to implement  a  predictive version  of a  Hebb  learning  rule  embedded  in  a  neural  ar(cid:173) chitecture based  on  anatomical  and  physiological studies on  bees.  The  model captured the strategies seen in the behavior of bees and a number of  other animals when foraging in an uncertain environment.  The predictive  model  suggests a unified  way  in which  neuromodulatory influences can  be used to bias actions and control synaptic plasticity. 
Successful predictions enhance adaptive behavior by allowing organisms to prepare for fu(cid:173) ture actions, rewards, or punishments.  Moreover, it is possible to improve upon behavioral  choices  if the consequences  of executing different actions can  be  reliably predicted.  Al(cid:173) though classical and instrumental conditioning results from the psychological literature [1]  demonstrate that the vertebrate brain is capable of reliable prediction, how these predictions  are computed in brains is not yet known. 
The  brains  of  vertebrates  and  invertebrates  possess  small  nuclei  which  project  axons  throughout large  expanses  of target  tissue  and  deliver  various  neurotransmitters  such  as  dopamine, norepinephrine, and acetylcholine [4].  The activity in these systems may report  on reinforcing stimuli in the world or may reflect an expectation of future reward [5, 6,7,8]. 
*Division of Neuroscience, Baylor College of Medicine, Houston, TX 77030
Analysis of Short Term Memories for Neural Networks
Authors: Jose C. Principe, Hui-H. Hsu, Jyh-Ming Kuo
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Short term memory is indispensable for the processing of time  varying information with artificial neural networks. In this paper a  model for linear memories is presented, and ways to include  memories in connectionist topologies are discussed. A comparison  is drawn among different memory types, with indication of what is  the salient characteristic of each memory model.
Credit Assignment through Time: Alternatives to Backpropagation
Authors: Yoshua Bengio, Paolo Frasconi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Learning  to  recognize  or  predict  sequences  using  long-term  con(cid:173) text  has  many  applications.  However,  practical  and  theoretical  problems  are  found  in  training  recurrent  neural  networks  to  per(cid:173) form tasks in which input/output dependencies span long intervals.  Starting from a mathematical analysis of the  problem, we  consider  and compare alternative algorithms and  architectures  on  tasks for  which the span of the input/output dependencies can be controlled.  Results  on the new  algorithms show  performance qualitatively su(cid:173) perior  to that obtained with backpropagation.
Analyzing Cross-Connected Networks
Authors: Thomas R. Shultz, Jeffrey L. Elman
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Jeffrey  L.  Elman 
Center for Research on Language  Department of Cognitive Science 
University of California at San Diego 
LaJolla, CA 92093-0126 U.S.A. 
elman@crl.ucsd.edu 
The non-linear complexities of neural networks make network solutions  difficult to understand. Sanger's contribution analysis is here extended to  the  analysis  of  networks  automatically  generated  by  the  cascade(cid:173) correlation  learning  algorithm.  Because  such  networks  have  cross  connections  that supersede hidden  layers, standard analyses of hidden  unit activation patterns are insufficient. A contribution is defined as the  product of an output weight and the associated activation on the sending  unit, whether that sending unit is an input or a hidden unit,  multiplied  by  the  sign  of  the  output  target  for  the  current  input  pattern.  Intercorrelations  among contributions,  as  gleaned from  the  matrix  of  contributions  x  input  patterns,  can  be  subjected  to  principal  components analysis (PCA) to extract the main features of variation in  the  contributions.  Such  an  analysis  is  applied  to  three  problems,  continuous XOR,  arithmetic comparison, and distinguishing  between  two interlocking spirals. In all  three cases, this technique yields useful  insights  into  network  solutions  that  are  consistent  across  several  networks.
Implementing Intelligence on Silicon Using Neuron-Like Functional MOS Transistors
Authors: Tadashi Shibata, Koji Kotani, Takeo Yamashita, Hiroshi Ishii, Hideo Kosaka, Tadahiro Ohmi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  will  present  the  implementation  of  intelligent  electronic  circuits  realized  for  the  first  time  using  a new  functional  device  called  Neuron  MOS  Transistor  (neuMOS  or vMOS  in  short)  simulating  the  behavior  of biological  neurons  at  a  single  transistor  level.  Search  for  the  most  resembling  data  in  the  memory  cell  array,  for  instance,  can  be  automatically  software  manipulation.  Soft Hardware,  which  we named,  can  arbitrarily  change  its  logic  function  in  real  time  by  external  control  signals  without  any  hardware  modification.  Implementation  of a  neural  network  equipped  with  an on-chip  self-learning  capability  is also  described.  Through  the  studies  of  vMOS  intelligent  circuit  implementation,  we  noticed  an  interesting  similarity  in  the  architectures  of vMOS  logic  circuitry  and  biological  systems. 
carried  out  on  hardware  without  any
Learning Complex Boolean Functions: Algorithms and Applications
Authors: Arlindo L. Oliveira, Alberto Sangiovanni-Vincentelli
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The most commonly used neural network models are not well suited  to direct digital implementations because each node needs to per(cid:173) form a large number of operations between floating point values.  Fortunately, the ability to learn from examples and to generalize is  not restricted to networks ofthis type. Indeed, networks where each  node implements a simple Boolean function (Boolean networks) can  be designed in such a way as to exhibit similar properties. Two  algorithms that generate Boolean networks from examples are pre(cid:173) sented. The results show that these algorithms generalize very  well in a class of problems that accept compact Boolean network  descriptions. The techniques described are general and can be ap(cid:173) plied to tasks that are not known to have that characteristic. Two  examples of applications are presented: image reconstruction and  hand-written character recognition.
A Unified Gradient-Descent/Clustering Architecture for Finite State Machine Induction
Authors: Sreerupa Das, Michael Mozer
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Although  recurrent  neural  nets  have  been  moderately  successful  in  learning  to  emulate  finite-state  machines  (FSMs),  the  continu(cid:173) ous  internal  state  dynamics  of a  neural  net  are  not  well  matched  to  the  discrete  behavior  of an  FSM.  We  describe  an  architecture,  called  DOLCE, that allows discrete  states to evolve in a net as learn(cid:173) ing  progresses.  DOLCE  consists  of a  standard  recurrent  neural  net  trained  by  gradient  descent  and  an  adaptive  clustering  technique  that quantizes  the state space.  DOLCE  is  based  on the assumption  that a  finite  set  of discrete  internal states  is  required  for  the  task,  and that the actual network state belongs  to this  set  but has been  corrupted by noise  due  to inaccuracy in the weights.  DOLCE  learns  to  recover  the discrete  state with  maximum a  posteriori  probabil(cid:173) ity  from  the  noisy  state.  Simulations  show  that  DOLCE  leads  to  a  significant  improvement in  generalization  performance  over  earlier  neural  net  approaches  to FSM  induction.
Convergence of Indirect Adaptive Asynchronous Value Iteration Algorithms
Authors: Vijaykumar Gullapalli, Andrew G. Barto
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Reinforcement Learning methods based on approximating dynamic  programming  (DP)  are  receiving  increased  attention  due  to  their  utility  in  forming  reactive  control  policies  for  systems  embedded  in  dynamic  environments.  Environments  are  usually  modeled  as  controlled  Markov  processes,  but  when  the  environment  model  is  not known a priori,  adaptive methods are necessary.  Adaptive con(cid:173) trol methods are often classified  as being direct  or indirect.  Direct  methods directly adapt the control policy from experience,  whereas  indirect  methods adapt a  model of the controlled process and com(cid:173) pute  control  policies  based  on  the  latest  model.  Our  focus  is  on  indirect  adaptive  DP-based  methods  in  this  paper.  We  present  a  convergence  result  for  indirect  adaptive  asynchronous  value  itera(cid:173) tion algorithms for the case in which a look-up table is used to store  the  value  function.  Our  result  implies  convergence  of several  ex(cid:173) isting reinforcement  learning algorithms such as adaptive real-time  dynamic programming (ARTDP)  (Barto,  Bradtke, &  Singh,  1993)  and prioritized  sweeping  (Moore  &  Atkeson,  1993).  Although  the  emphasis  of researchers  studying DP-based reinforcement  learning  has been on direct  adaptive methods such as Q-Learning (Watkins,  1989)  and  methods  using  TD  algorithms  (Sutton,  1988),  it is  not  clear that these direct methods are preferable in practice to indirect  methods such as those  analyzed  in  this  paper.
Fast Pruning Using Principal Components
Authors: Asriel U. Levin, Todd K. Leen, John E. Moody
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present a new  algorithm for  eliminating excess parameters and  improving  network  generalization  after  supervised  training.  The  method,  "Principal Components Pruning (PCP)", is based on prin(cid:173) cipal component analysis of the node activations of successive layers  of the network.  It is  simple,  cheap to implement,  and effective.  It  requires  no  network  retraining,  and  does  not  involve  calculating  the full  Hessian of the cost function.  Only the weight and the node  activity  correlation  matrices for  each  layer of nodes  are  required.  We demonstrate the efficacy of the method on a regression problem  using  polynomial  basis functions,  and on an  economic time  series  prediction problem using a  two-layer, feedforward  network.
Hidden Markov Models for Human Genes
Authors: Pierre Baldi, Søren Brunak, Yves Chauvin, Jacob Engelbrecht, Anders Krogh
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Human  genes  are  not  continuous  but  rather  consist  of short  cod(cid:173) ing  regions  (exons)  interspersed  with  highly  variable  non-coding  regions  (introns).  We apply HMMs  to the problem of modeling ex(cid:173) ons,  introns and  detecting splice  sites  in  the  human  genome.  Our  most  interesting result  so  far  is  the  detection of particular oscilla(cid:173) tory patterns, with a minimal period ofroughly 10 nucleotides, that  seem to be characteristic of exon  regions  and  may have significant  biological  implications. 
• and  Division of Biology,  California Institute of Technology.  t and  Department of Psychology,  Stanford  University.
Postal Address Block Location Using a Convolutional Locator Network
Authors: Ralph Wolf, John C. Platt
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This  paper  describes  the  use  of  a  convolutional  neural  network  to perform address block location on machine-printed mail pieces.  Locating the address block is a difficult object recognition problem  because  there is  often a  large  amount of extraneous printing on  a  mail piece and because address blocks vary dramatically in size and  shape.  We  used  a  convolutional locator  network  with four  outputs,  each  trained  to find  a  different  corner  of the  address  block.  A  simple  set of rules was used to generate ABL candidates from the network  output.  The system performs very well:  when allowed five guesses,  the network will tightly bound the address delivery information in  98.2% of the cases.
An Analog VLSI Saccadic Eye Movement System
Authors: Timothy K. Horiuchi, Brooks Bishofberger, Christof Koch
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  an effort  to understand saccadic eye  movements and their rela(cid:173) tion  to visual  attention and other forms  of eye  movements,  we  - in  collaboration with  a  number of other laboratories - are  carry(cid:173) ing out a  large-scale effort  to design  and build a  complete primate  oculomotor  system  using  analog  CMOS  VLSI  technology.  Using  this technology, a low power, compact, multi-chip system has been  built  which  works in  real-time  using  real-world  visual  inputs.  We  describe  in  this paper  the  performance  of an early  version  of such  a  system  including  a  1-D  array of photoreceptors  mimicking  the  retina, a circuit computing the mean location of activity represent(cid:173) ing  the  superior  colliculus,  a  saccadic  burst  generator,  and  a  one  degree-of-freedom  rotational  platform  which  models  the  dynamic  properties  of the  primate oculomotor plant.
A Computational Model for Cursive Handwriting Based on the Minimization Principle
Authors: Yasuhiro Wada, Yasuharu Koike, Eric Vatikiotis-Bateson, Mitsuo Kawato
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  propose a  trajectory  planning  and  control  theory  for  continuous  movements  such  as  connected  cursive  handwriting  and  continuous  natural  speech.  Its  hardware  is  based  on  our  previously  proposed  forward-inverse-relaxation  neural  network  (Wada  & Kawato,  1993).  Computationally,  its  optimization  principle  is  the  minimum  torque(cid:173) change criterion.  Regarding  the  representation level, hard constraints  satisfied by a trajectory are represented as a set of via-points extracted  from  a  handwritten  character.  Accordingly,  we  propose a  via-point  estimation  algorithm  that  estimates  via-points  by  repeating  the  trajectory formation of a character and the via-point extraction from  the  character.  In  experiments,  good  quantitative  agreement  is  found  between  human  handwriting data and the trajectories generated by  the  theory.  Finally,  we  propose  a  recognition  schema  based  on  the  movement  generation.  We  show  a  result  in  which  the  recognition  schema is applied to the handwritten character recognition and can be  extended to the phoneme timing estimation of natural speech.
Discontinuous Generalization in Large Committee Machines
Authors: H. Schwarze, J. Hertz
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The problem of learning from examples in multilayer networks is  studied within the framework of statistical mechanics. Using the  replica formalism we calculate the average generalization error of a  fully connected committee machine in the limit of a large number  of hidden units. If the number of training examples is proportional  to the number of inputs in the network, the generalization error  as a function of the training set size approaches a finite value. If  the number of training examples is proportional to the number of  weights in the network we find first-order phase transitions with a  discontinuous drop in the generalization error for both binary and  continuous weights.
Inverse Dynamics of Speech Motor Control
Authors: Makoto Hirayama, Eric Vatikiotis-Bateson, Mitsuo Kawato
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Progress ha.s been made in comput.ational implementation of speech  production  based  on  physiological  dat.a.  An  inverse  dynamics  model  of  the  speech  articulator's  l1111sculo-skeletal  system.  which  is  the  mapping from  art.iculator  t.rajectories  to  e\ectromyogl'aphic  (EMG) signals,  was  modeled  using  the acquired  forward  dynamics  model  and  temporal  (smoot.hness  of EMG  activation)  and  range  constraints.  This inverse dynamics model allows  the use of a  faster  speech  mot.or control scheme, which  can  be applied to phoneme-to(cid:173) speech  synthesis via musclo-skeletal system  dynamics, or  to future  use in speech  recognition.  The forward acoustic model, which is  the  mapping from  articulator  trajectories  t.o  the  acoustic  parameters,  was  improved  by  adding  velocity  and  voicing  information  inputs  to distinguish  acollst.ic  paramet.er  differences  caused  by  changes  in  source  characterist.ics.
Learning Temporal Dependencies in Connectionist Speech Recognition
Authors: Steve Renals, Mike Hochberg, Tony Robinson
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Hybrid connectionistfHMM systems model time both using a Markov  chain and through properties of a connectionist network. In this paper,  we discuss the nature of the time dependence currently employed in our  systems using recurrent networks (RNs) and feed-forward multi-layer  perceptrons (MLPs). In particular, we introduce local recurrences into a  MLP to produce an enhanced input representation. This is in the form  of an adaptive gamma filter and incorporates an automatic approach for  learning temporal dependencies. We have experimented on a speaker(cid:173) independent phone recognition task using the TIMIT database. Results  using the gamma filtered input representation have shown improvement  over the baseline MLP system. Improvements have also been obtained  through merging the baseline and gamma filter models.
Odor Processing in the Bee: A Preliminary Study of the Role of Central Input to the Antennal Lobe
Authors: Christiane Linster, David Marsan, Claudine Masson, Michel Kerszberg
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Based  on  precise  anatomical  data  of the  bee's  olfactory  system,  we  propose an investigation of the possible mechanisms of modulation and  control between the two levels of olfactory  information processing:  the  antennallobe glomeruli  and  the  mushroom  bodies.  We use  simplified  neurons,  but realistic  architecture.  As  a  first  conclusion,  we  postulate  that the feature extraction performed by the antennallobe (glomeruli and  interneurons)  necessitates  central  input from  the  mushroom bodies for  fine  tuning. The central input thus  facilitates  the evolution from fuzzy  olfactory images in the glomerular layer towards more focussed  images  upon odor presentation.
Surface Learning with Applications to Lipreading
Authors: Christoph Bregler, Stephen M. Omohundro
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Most connectionist research has focused on learning mappings from  one space to another (eg.  classification and regression).  This paper  introduces  the  more  general  task  of learning  constraint  surfaces.  It describes  a  simple  but  powerful  architecture  for  learning  and  manipulating  nonlinear  surfaces  from  data.  We  demonstrate  the  technique  on low dimensional synthetic surfaces  and compare it to  nearest  neighbor  approaches.  We  then  show  its  utility in learning  the space of lip images in a system for improving speech recognition  by lip  reading.  This learned  surface  is  used  to improve the visual  tracking  performance during recognition. 
1  Surface Learning 
Mappings  are  an  appropriate representation for  systems  whose  variables  naturally  decompose into "inputs"  and  "outputs)).  To use  a learned mapping, the input vari(cid:173) ables must be known and error-free and a single output value must be estimated for  each input.  Many tasks in vision, robotics,  and control must maintain relationships  between  variables  which  don't  naturally  decompose  in  this  way.  Instead,  there  is  a  nonlinear  constraint  surface  on  which  the  values  of the  variables  are  jointly re(cid:173) stricted to lie.  We propose a representation for such surfaces  which supports a wide  range of queries  and which  can  be  naturally learned from  data. 
The simplest queries are "completion queries)).  In these queries, the values of certain  variables  are  specified  and  the  values  (or  constraints  on  the  values)  of remaining
Figure of Merit Training for Detection and Spotting
Authors: Eric I. Chang, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Spotting tasks require detection of target patterns from a background of  richly varied non-target inputs. The performance measure of interest for  these  tasks,  called  the figure  of merit (FOM),  is  the  detection  rate  for  target  patterns  when  the  false  alarm  rate  is  in  an  acceptable  range.  A  new approach to training spotters is presented which computes the FOM  gradient  for  each  input pattern  and  then  directly  maximizes  the  FOM  using  b ackpropagati on.  This  eliminates  the  need  for  thresholds  during  training. It also  uses  network resources  to model  Bayesian a posteriori  probability  functions  accurately  only  for  patterns  which  have  a  significant effect on  the  detection accuracy over the  false  alarm rate of  interest.  FOM  training  increased  detection  accuracy  by  5  percentage  points for a hybrid radial basis function  (RBF) - hidden Markov model  (HMM) wordspotter on the credit-card speech corpus.
Autoencoders, Minimum Description Length and Helmholtz Free Energy
Authors: Geoffrey E. Hinton, Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: An autoencoder network uses a set of recognition weights to convert an  input vector into a code vector.  It then uses a set of generative weights to  convert the code vector into an approximate reconstruction of the input  vector.  We derive an objective function for training autoencoders based  on the  Minimum  Description Length  (MDL)  principle.  The aim  is  to  minimize the information required to describe both the code vector and  the reconstruction error.  We  show  that this information  is  minimized  by choosing code vectors stochastically according to a Boltzmann distri(cid:173) bution, where the generative weights define the energy of each possible  code vector given  the input vector.  Unfortunately,  if the code vectors  use distributed representations, it is exponentially expensive to compute  this Boltzmann distribution because it involves all possible code vectors.  We show that the recognition weights of an autoencoder can  be used to  compute an approximation to the Boltzmann distribution and that this ap(cid:173) proximation gives an upper bound on the description length.  Even when  this bound is poor,  it can  be used  as a  Lyapunov function for learning  both the generative and the recognition weights.  We  demonstrate that  this approach can be used to learn factorial codes.