mtybilly/Instruct2DS huggingface dataset:
An Analog VLSI Model of Central Pattern Generation in the Leech
Authors: Micah S. Siegel
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: I  detail  the  design  and construction  of an  analog  VLSI  model  of the  neural system responsible for swimming behaviors of the leech.  Why  the  leech?  The  biological  network  is  small  and  relatively  well  understood,  and  the  silicon  model  can  therefore  span  three  levels  of  organization in  the leech nervous system (neuron, ganglion, system); it  represents  one of the first comprehensive models of leech  swimming  operating in  real-time.  The circuit employs  biophysically  motivated  analog neurons networked to form  multiple biologically inspired silicon  ganglia.  These  ganglia  are  coupled  using  known  interganglionic  connections.  Thus  the  model  retains  the  flavor  of  its  biological  counterpart, and  though  simplified,  the output of the  silicon circuit is  similar to  the output of the leech swim central pattern generator.  The  model operates on the same time- and spatial-scale as the leech nervous  system  and  will  provide an  excellent platform  with  which  to  explore  real-time  adaptive  locomotion  in  the  leech  and  other  "simple"  invertebrate nervous systems.
Non-Linear Statistical Analysis and Self-Organizing Hebbian Networks
Authors: Adam Prügel-Bennett Jonathan L. Shapiro
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Neurons learning under an unsupervised  Hebbian learning rule can  perform a nonlinear generalization of principal component analysis.  This relationship between  nonlinear PCA  and nonlinear neurons is  reviewed.  The stable fixed  points of the neuron learning dynamics  correspond  to  the  maxima of the  statist,ic  optimized  under  non(cid:173) linear  PCA.  However,  in  order  to predict.  what  the  neuron  learns,  knowledge  of the  basins  of attractions  of the  neuron  dynamics  is  required.  Here  the  correspondence  between  nonlinear  PCA  and  neural  networks  breaks  down.  This  is  shown  for  a  simple  model.  Methods  of statistical  mechanics  can  be  used  to  find  the  optima  of the objective function  of non-linear PCA. This determines  what  the neurons  can learn.  In order to find  how  the solutions are parti(cid:173) tioned amoung the neurons, however, one must solve the dynamics.
Locally Adaptive Nearest Neighbor Algorithms
Authors: Thomas G. Dietterich Dietrich Wettschereck
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Four versions of a  k-nearest  neighbor algorithm with locally adap(cid:173) tive  k  are  introduced  and  compared  to  the  basic  k-nearest  neigh(cid:173) bor algorithm (kNN).  Locally adaptive kNN  algorithms choose the  value of k  that should be used  to classify a query by consulting the  results  of cross-validation computations in  the local  neighborhood  of the query.  Local kNN  methods are shown  to perform similar to  kNN in experiments with twelve commonly used data sets.  Encour(cid:173) aging  results  in  three  constructed  tasks  show  that  local  methods  can  significantly  outperform  kNN  in  specific  applications.  Local  methods  can  be  recommended  for  on-line  learning  and  for  appli(cid:173) cations  where  different  regions  of the  input  space  are  covered  by  patterns solving different  sub-tasks.
Bayesian Backprop in Action: Pruning, Committees, Error Bars and an Application to Spectroscopy
Authors: Hans Henrik Thodberg
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: MacKay's Bayesian framework for  backpropagation is  conceptually  appealing  as  well  as  practical.  It automatically adjusts the  weight  decay  parameters  during  training,  and  computes  the  evidence  for  each  trained  network.  The  evidence  is  proportional  to  our  belief  in  the  model.  The  networks  with  highest  evidence  turn  out  to  generalise well.  In this paper, the framework is extended to pruned  nets,  leading  to  an  Ockham  Factor  for  "tuning  the  architecture  to  the  data".  A  committee  of networks,  selected  by  their  high  evidence,  is  a  natural  Bayesian  construction.  The  evidence  of a  committee is computed.  The framework is illustrated on real-world  data from  a  near  infrared  spectrometer  used  to determine  the fat  content  in  minced  meat.  Error  bars  are  computed,  including  the  contribution from the  dissent  of the committee members. 
1  THE OCKHAM  FACTOR 
William  of  Ockham's  (1285-1349)  principle  of economy  in  explanations,  can  be  formulated as follows: 
If several theories  account  for  a phenomenon  we  should  prefer the  simplest  which  describes  the  data  sufficiently  well.
Solvable Models of Artificial Neural Networks
Authors: Sumio Watanabe
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Solvable  models of nonlinear learning machines  are  proposed,  and  learning in artificial neural networks is  studied based on the theory  of  ordinary  differential  equations.  A  learning  algorithm  is  con(cid:173) structed,  by  which  the  optimal  parameter  can  be  found  without  any recursive procedure.  The solvable models enable us  to analyze  the reason  why  experimental results  by  the error backpropagation  often contradict the statistical learning theory.
A Comparative Study of a Modified Bumptree Neural Network with Radial Basis Function Networks and the Standard Multi Layer Perceptron
Authors: Alan J. Harget Richard T. J. Bostock
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Bumptrees  are  geometric  data  structures  introduced  by  Omohundro  (1991)  to  provide  efficient  access  to  a  collection  of functions  on  a  Euclidean space of interest.  We describe a modified bumptree structure  that has been employed as  a neural network classifier, and compare its  performance on  several classification  tasks against that of radial basis  function networks and the standard mutIi-Iayer perceptron.
Combined Neural Networks for Time Series Analysis
Authors: David Horn Iris Ginzburg
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  propose  a  method  for  improving  the  performance  of any  net(cid:173) work  designed  to predict  the next value of a  time series.  Vve  advo(cid:173) cate analyzing the deviations of the network's predictions from the  data in the training set.  This can be carried out by a secondary net(cid:173) work  trained  on  the  time series  of these  residuals.  The  combined  system  of  the  two  networks  is  viewed  as  the  new  predictor.  We  demonstrate  the  simplicity  and  success  of this  method,  by  apply(cid:173) ing it to the sunspots data.  The small corrections of the secondary  network  can  be  regarded  as  resulting  from  a  Taylor  expansion  of  a  complex  network  which  includes  the  combined system.  \Te  find  that  the  complex network  is  more  difficult  to  train  and  performs  worse  than the two-step  procedure of the  combined system.
Illumination-Invariant Face Recognition with a Contrast Sensitive Silicon Retina
Authors: Martin Lades, Frank Eeckman Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Changes in lighting conditions strongly effect the performance and reli(cid:173) ability of computer vision systems. We report face recognition results  under drastically changing lighting conditions for a computer vision sys(cid:173) tem which concurrently uses a contrast sensitive silicon retina and a con(cid:173) ventional, gain controlled CCO camera. For both input devices the face  recognition system employs an elastic matching algorithm with wavelet  based features to classify unknown faces. To assess the effect of analog  on -chip preprocessing by the silicon retina the CCO images have been  "digitally preprocessed" with a bandpass filter to adjust the power spec(cid:173) trum. The silicon retina with its ability to adjust sensitivity increases  the recognition rate up to 50 percent. These comparative experiments  demonstrate that preprocessing with an analog VLSI silicon retina gen(cid:173) erates image data enriched with object-constant features.
Address Block Location with a Neural Net System
Authors: Eric Cosatto Hans Peter Graf
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We developed a system for finding address blocks on mail pieces that can  process four  images per second. Besides  locating the  address block,  our  system also determines the writing style, handwritten or machine printed, and  moreover,  it measures  the  skew  angle  of the  text  lines  and  cleans noisy  images.  A  layout  analysis  of all  the  elements  present  in  the  image  is  performed in order to distinguish drawings and dirt from text and to separate  text of advertisement from that of the destination address.  A  speed  of more  than  four  images per second  is obtained on a  modular  hardware platform, containing a board with two of the NET32K neural net  chips,  a  SP ARC2  processor  board,  and  a  board  with  2  digital  signal  processors. The system has been tested with more than 100,000 images. Its  performance depends on the quality of the images, and lies between 85%  correct location in very noisy images to over 98% in cleaner images.
Assessing the Quality of Learned Local Models
Authors: Christopher G. Atkeson Stefan Schaal
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: An  approach  is  presented  to  learning  high  dimensional functions  in  the  case  where  the  learning  algorithm  can  affect  the  generation  of new  data.  A  local  modeling algorithm, locally weighted regression, is used to represent the learned  function.  Architectural parameters of the approach, such as distance metrics, are  also localized and become a function of the query point instead of being global.  Statistical tests are given for  when a local model is good enough and sampling  should be moved to a new area. Our methods explicitly deal with the case where  prediction accuracy requirements exist during exploration: By gradually shifting  a "center of exploration" and controlling the speed of the  shift with local pre(cid:173) diction accuracy, a goal-directed exploration of state space takes place along the  fringes  of the current data support until  the task goal is achieved. We illustrate  this  approach  with  simulation results  and results  from  a real  robot learning  a  complex juggling task.
Synchronization, oscillations, and 1/f noise in networks of spiking neurons
Authors: Marius Usher, Christof Koch, Zeev Olami Martin Stemmler
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We investigate a model for neural activity that generates long range  temporal correlations, 1/ f  noise, and oscillations in global activity.  The model  consists  of a  two-dimensional sheet  of leaky  integrate(cid:173) and-fire  neurons  with feedback  connectivity consisting of local ex(cid:173) citation  and  surround  inhibition.  Each  neuron  is  independently  driven  by  homogeneous  external  noise.  Spontaneous  symmetry  breaking occurs,  resulting in  the formation of "hotspots"  of activ(cid:173) ity in  the  network.  These  localized  patterns  of excitation  appear  as  clusters  that coalesce,  disintegrate, or fluctuate  in size  while si(cid:173) multaneously moving in a random walk constrained by the interac(cid:173) tion  with other  clusters.  The emergent  cross-correlation functions  have  a  dual  structure,  with  a  sharp  peak  around  zero  on  top  of  a  much broader  hill .  The  power  spectrum  associated  with  single  units  shows a  1/ f  decay for  small frequencies  and is flat  at higher  frequencies,  while the power spectrum of the spiking activity aver(cid:173) aged over many cells-equivalent to the local field  potential-shows  no  1/ f  decay  but a prominent peak  around 40  Hz.
Unsupervised Learning of Mixtures of Multiple Causes in Binary Data
Authors: Eric Saund
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper presents a formulation for unsupervised learning of clus(cid:173) ters  reflecting  multiple causal structure  in  binary data.  Unlike the  standard  mixture  model,  a  multiple cause  model  accounts  for  ob(cid:173) served data by combining assertions from many hidden causes, each  of which can pertain to varying degree  to any subset of the observ(cid:173) able dimensions.  A crucial  issue is  the  mixing-function for  combin(cid:173) ing  beliefs  from  different  cluster-centers  in  order  to generate  data  reconstructions whose errors are minimized both during recognition  and  learning.  We  demonstrate a  weakness  inherent  to  the popular  weighted  sum followed  by sigmoid squashing,  and  offer  an alterna(cid:173) tive form  of the  nonlinearity.  Results are  presented  demonstrating  the  algorithm's  ability  successfully  to  discover  coherent  multiple  causal  representat.ions  of noisy  test  data and  in  images of printed  characters.
A Hodgkin-Huxley Type Neuron Model That Learns Slow Non-Spike Oscillation
Authors: Allen I. Selverston, Peter F. Rowat Kenji Doya
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: A  gradient  descent  algorithm  for  parameter  estimation  which  is  similar to those used for  continuous-time recurrent neural networks  was  derived for  Hodgkin-Huxley type  neuron models.  Using mem(cid:173) brane  potential  trajectories  as  targets,  the  parameters  (maximal  conductances,  thresholds and slopes of activation curves, time con(cid:173) stants)  were  successfully estimated.  The  algorithm was  applied  to  modeling  slow  non-spike  oscillation  of an  identified  neuron  in  the  lobster stomatogastric ganglion.  A model with three ionic  currents  was  trained  with  experimental  data.  It revealed  a  novel  role  of  A-current for  slow oscillation below -50  mY.
A Comparison of Dynamic Reposing and Tangent Distance for Drug Activity Prediction
Authors: Ajay N. Jain, Richard H. Lathrop, Tomás Lozano-Pérez Thomas G. Dietterich
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  drug  activity  prediction  (as  in  handwritten  character  recogni(cid:173) tion), the features extracted to describe a training example depend  on  the  pose  (location, orientation, etc.)  of the example.  In  hand(cid:173) written  character  recognition,  one  of the  best  techniques  for  ad(cid:173) dressing  this  problem  is  the  tangent  distance  method  of Simard,  LeCun  and Denker  (1993).  Jain, et al.  (1993a;  1993b) introduce  a  new  technique-dynamic reposing-that also  addresses  this  prob(cid:173) lem.  Dynamic reposing iteratively learns a neural network and then  reposes  the  examples in  an effort  to  maximize  the  predicted  out(cid:173) put values.  New  models are trained and new  poses computed until  models and poses converge.  This paper compares dynamic reposing  to the  tangent  distance  method on the  task of predicting  the bio(cid:173) logical  activity  of musk  compounds.  In  a  20-fold cross-validation,
Hoo Optimality Criteria for LMS and Backpropagation
Authors: Ali H. Sayed, Thomas Kailath Babak Hassibi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  have  recently  shown that the  widely  known  LMS  algorithm is  an H OO  optimal estimator.  The H OO  criterion  has been introduced,  initially in  the  control  theory  literature,  as  a  means to ensure  ro(cid:173) bust  performance  in  the  face  of model  uncertainties  and  lack  of  statistical  information on  the  exogenous  signals.  We  extend  here  our  analysis  to  the  nonlinear  setting  often  encountered  in  neural  networks,  and show  that the  backpropagation  algorithm is  locally  H OO  optimal.  This fact  provides  a  theoretical justification  of the  widely  observed  excellent  robustness  properties  of the  LMS  and  backpropagation algorithms.  We further discuss  some implications  of these  results.
Classifying Hand Gestures with a View-Based Distributed Representation
Authors: Alex P. Pentland Trevor J. Darrell
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We present a method for learning, tracking, and recognizing human hand  gestures recorded by a conventional CCD camera without any special  gloves or other sensors. A view-based representation is used to model  aspects of the hand relevant to the trained gestures, and is found using an  unsupervised clustering technique. We use normalized correlation net(cid:173) works, with dynamic time warping in the temporal domain, as a distance  function for unsupervised clustering. Views are computed separably for  space and time dimensions; the distributed response of the combination  of these units characterizes the input data with a low dimensional repre(cid:173) sentation. A supervised classification stage uses labeled outputs of the  spatio-temporal units as training data. Our system can correctly classify  gestures in real time with a low-cost image processing accelerator.
Optimal Unsupervised Motor Learning Predicts the Internal Representation of Barn Owl Head Movements
Authors: Terence D. Sanger
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: (Masino and Knudsen 1990) showed some remarkable results which  suggest  that head motion in the barn owl  is  controlled by  distinct  circuits coding for the horizontal and vertical components of move(cid:173) ment.  This implies the existence of a set of orthogonal internal co(cid:173) ordinates that are related to meaningful coordinates of the external  world.  No  coherent  computational  theory  has  yet  been  proposed  to explain this finding.  I  have proposed a simple model which pro(cid:173) vides a framework for  a  theory of low-level motor learning.  I show  that  the  theory  predicts  the  observed  microstimulation results  in  the  barn  owl.  The  model  rests  on  the  concept  of  "Optimal  U n(cid:173) supervised  Motor  Learning",  which  provides  a  set  of criteria  that  predict  optimal internal  representations.  I  describe  two  iterative  Neural  Network  algorithms  which  find  the  optimal  solution  and  demonstrate  possible  mechanisms for  the  development  of internal  representations  in animals.
Amplifying and Linearizing Apical Synaptic Inputs to Cortical Pyramidal Cells
Authors: Christof Koch, Rodney J. Douglas Öjvind Bernander
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Intradendritic  electrophysiological recordings  reveal  a  bewildering  repertoire  of complex  electrical  spikes  and  plateaus  that  are  dif(cid:173) ficult  to  reconcile  with  conventional notions  of neuronal function.  In  this paper  we  argue  that such  dendritic events are just an exu(cid:173) berant expression of a more important mechanism - a proportional  current amplifier whose primary task is to offset electrotonic losses.  Using the example of functionally important synaptic inputs to the  superficial  layers of an  anatomically and  electrophysiologically re(cid:173) constructed  layer  5 pyramidal neuron,  we  derive  and  simulate the  properties of conductances  that linearize and amplify distal synap(cid:173) tic  input  current  in  a  graded  manner.  The  amplification  depends  on a potassium conductance in the apical tuft and calcium conduc(cid:173) tances  in  the  apical trunk. 
·To whom  all  correspondence  should  be addressed.
Decoding Cursive Scripts
Authors: Naftali Tishby Yoram Singer
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Online cursive handwriting recognition  is  currently one of the most  intriguing challenges in  pattern recognition.  This study  presents  a  novel  approach  to this  problem which  is  composed  of two  comple(cid:173) mentary  phases.  The first  is  dynamic encoding  of the  writing  tra(cid:173) jectory into a  compact sequence  of discrete  motor control symbols.  In this compact representation we largely remove the redundancy of  the script,  while  preserving  most of its intelligible components.  In  the second  phase these  control sequences  are used  to train adaptive  probabilistic acyclic automata (PAA) for the important ingredients  of the writing  trajectories,  e.g.  letters.  We  present  a  new  and effi(cid:173) cient  learning algorithm for such stochastic automata, and demon(cid:173) strate  its  utility  for  spotting  and  segmentation of cursive  scripts.  Our  experiments  show  that  over  90%  of  the  letters  are  correctly  spotted  and  identified,  prior to  any  higher  level  language  model.  Moreover,  both  the  training  and  recognition  algorithms  are  very  efficient  compared to other  modeling methods, and the models are  'on-line' adaptable to other  writers  and  styles.
Foraging in an Uncertain Environment Using Predictive Hebbian Learning
Authors: Peter Dayan, Terrence J. Sejnowski P. Read Montague
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Survival  is  enhanced  by  an  ability  to  predict  the  availability  of food,  the  likelihood of predators,  and  the  presence  of mates.  We  present  a  concrete model that uses diffuse neurotransmitter systems to implement  a  predictive version  of a  Hebb  learning  rule  embedded  in  a  neural  ar(cid:173) chitecture based  on  anatomical  and  physiological studies on  bees.  The  model captured the strategies seen in the behavior of bees and a number of  other animals when foraging in an uncertain environment.  The predictive  model  suggests a unified  way  in which  neuromodulatory influences can  be used to bias actions and control synaptic plasticity. 
Successful predictions enhance adaptive behavior by allowing organisms to prepare for fu(cid:173) ture actions, rewards, or punishments.  Moreover, it is possible to improve upon behavioral  choices  if the consequences  of executing different actions can  be  reliably predicted.  Al(cid:173) though classical and instrumental conditioning results from the psychological literature [1]  demonstrate that the vertebrate brain is capable of reliable prediction, how these predictions  are computed in brains is not yet known. 
The  brains  of  vertebrates  and  invertebrates  possess  small  nuclei  which  project  axons  throughout large  expanses  of target  tissue  and  deliver  various  neurotransmitters  such  as  dopamine, norepinephrine, and acetylcholine [4].  The activity in these systems may report  on reinforcing stimuli in the world or may reflect an expectation of future reward [5, 6,7,8]. 
*Division of Neuroscience, Baylor College of Medicine, Houston, TX 77030
Analysis of Short Term Memories for Neural Networks
Authors: Hui-H. Hsu, Jyh-Ming Kuo Jose C. Principe
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Short term memory is indispensable for the processing of time  varying information with artificial neural networks. In this paper a  model for linear memories is presented, and ways to include  memories in connectionist topologies are discussed. A comparison  is drawn among different memory types, with indication of what is  the salient characteristic of each memory model.
Credit Assignment through Time: Alternatives to Backpropagation
Authors: Paolo Frasconi Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Learning  to  recognize  or  predict  sequences  using  long-term  con(cid:173) text  has  many  applications.  However,  practical  and  theoretical  problems  are  found  in  training  recurrent  neural  networks  to  per(cid:173) form tasks in which input/output dependencies span long intervals.  Starting from a mathematical analysis of the  problem, we  consider  and compare alternative algorithms and  architectures  on  tasks for  which the span of the input/output dependencies can be controlled.  Results  on the new  algorithms show  performance qualitatively su(cid:173) perior  to that obtained with backpropagation.
Analyzing Cross-Connected Networks
Authors: Jeffrey L. Elman Thomas R. Shultz
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Jeffrey  L.  Elman 
Center for Research on Language  Department of Cognitive Science 
University of California at San Diego 
LaJolla, CA 92093-0126 U.S.A. 
elman@crl.ucsd.edu 
The non-linear complexities of neural networks make network solutions  difficult to understand. Sanger's contribution analysis is here extended to  the  analysis  of  networks  automatically  generated  by  the  cascade(cid:173) correlation  learning  algorithm.  Because  such  networks  have  cross  connections  that supersede hidden  layers, standard analyses of hidden  unit activation patterns are insufficient. A contribution is defined as the  product of an output weight and the associated activation on the sending  unit, whether that sending unit is an input or a hidden unit,  multiplied  by  the  sign  of  the  output  target  for  the  current  input  pattern.  Intercorrelations  among contributions,  as  gleaned from  the  matrix  of  contributions  x  input  patterns,  can  be  subjected  to  principal  components analysis (PCA) to extract the main features of variation in  the  contributions.  Such  an  analysis  is  applied  to  three  problems,  continuous XOR,  arithmetic comparison, and distinguishing  between  two interlocking spirals. In all  three cases, this technique yields useful  insights  into  network  solutions  that  are  consistent  across  several  networks.
Recognition-based Segmentation of On-Line Cursive Handwriting
Authors: Nicholas S. Flann
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper introduces a new recognition-based segmentation ap(cid:173) proach to recognizing on-line cursive handwriting from a database  of 10,000 English words. The original input stream of z, y pen coor(cid:173) dinates is encoded as a sequence of uniform stroke descriptions that  are processed by six feed-forward neural-networks, each designed  to recognize letters of different sizes. Words are then recognized by  performing best-first search over the space of all possible segmen(cid:173) tations. Results demonstrate that the method is effective at both  writer dependent recognition (1.7% to 15.5% error rate) and writer  independent recognition (5.2% to 31.1% error rate).
Implementing Intelligence on Silicon Using Neuron-Like Functional MOS Transistors
Authors: Koji Kotani, Takeo Yamashita, Hiroshi Ishii, Hideo Kosaka, Tadahiro Ohmi Tadashi Shibata
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  will  present  the  implementation  of  intelligent  electronic  circuits  realized  for  the  first  time  using  a new  functional  device  called  Neuron  MOS  Transistor  (neuMOS  or vMOS  in  short)  simulating  the  behavior  of biological  neurons  at  a  single  transistor  level.  Search  for  the  most  resembling  data  in  the  memory  cell  array,  for  instance,  can  be  automatically  software  manipulation.  Soft Hardware,  which  we named,  can  arbitrarily  change  its  logic  function  in  real  time  by  external  control  signals  without  any  hardware  modification.  Implementation  of a  neural  network  equipped  with  an on-chip  self-learning  capability  is also  described.  Through  the  studies  of  vMOS  intelligent  circuit  implementation,  we  noticed  an  interesting  similarity  in  the  architectures  of vMOS  logic  circuitry  and  biological  systems. 
carried  out  on  hardware  without  any
Probabilistic Anomaly Detection in Dynamic Systems
Authors: Padhraic Smyth
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This  paper  describes  probabilistic  methods  for  novelty  detection  when  using  pattern  recognition  methods  for  fault  monitoring  of  dynamic  systems.  The problem of novelty  detection  is  particular(cid:173) ly  acute  when  prior  knowledge  and  training  data only  allow  one  to  construct  an  incomplete  classification  model.  Allowance  must  be  made  in  model  design  so  that  the  classifier  will  be  robust  to  data generated  by  classes  not  included  in  the  training  phase.  For  diagnosis  applications  one  practical approach  is  to  construct  both  an  input  density  model  and  a  discriminative  class  model.  Using  Bayes'  rule  and  prior  estimates  of the  relative  likelihood  of  data  of known and unknown origin the resulting classification equations  are  straightforward.  The  paper  describes  the  application  of this  method  in  the  context  of hidden  Markov  models  for  online  fault  monitoring of large  ground  antennas for  spacecraft  tracking,  with  particular  application  to  the  detection  of  transient  behaviour  of  unknown  origin. 
1  PROBLEM BACKGROUND 
Conventional control-theoretic models for  fault  detection typically rely on an accu(cid:173) rate model ofthe plant being monitored (Patton, Frank, and Clark, 1989).  However,  in  practice  it  common  that  no such  model  exists  for  complex  non-linear  systems.  The  large  ground  antennas  used  by  JPL's  Deep  Space  Network  (DSN)  to  track
Learning Complex Boolean Functions: Algorithms and Applications
Authors: Alberto Sangiovanni-Vincentelli Arlindo L. Oliveira
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The most commonly used neural network models are not well suited  to direct digital implementations because each node needs to per(cid:173) form a large number of operations between floating point values.  Fortunately, the ability to learn from examples and to generalize is  not restricted to networks ofthis type. Indeed, networks where each  node implements a simple Boolean function (Boolean networks) can  be designed in such a way as to exhibit similar properties. Two  algorithms that generate Boolean networks from examples are pre(cid:173) sented. The results show that these algorithms generalize very  well in a class of problems that accept compact Boolean network  descriptions. The techniques described are general and can be ap(cid:173) plied to tasks that are not known to have that characteristic. Two  examples of applications are presented: image reconstruction and  hand-written character recognition.
A Unified Gradient-Descent/Clustering Architecture for Finite State Machine Induction
Authors: Michael Mozer Sreerupa Das
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Although  recurrent  neural  nets  have  been  moderately  successful  in  learning  to  emulate  finite-state  machines  (FSMs),  the  continu(cid:173) ous  internal  state  dynamics  of a  neural  net  are  not  well  matched  to  the  discrete  behavior  of an  FSM.  We  describe  an  architecture,  called  DOLCE, that allows discrete  states to evolve in a net as learn(cid:173) ing  progresses.  DOLCE  consists  of a  standard  recurrent  neural  net  trained  by  gradient  descent  and  an  adaptive  clustering  technique  that quantizes  the state space.  DOLCE  is  based  on the assumption  that a  finite  set  of discrete  internal states  is  required  for  the  task,  and that the actual network state belongs  to this  set  but has been  corrupted by noise  due  to inaccuracy in the weights.  DOLCE  learns  to  recover  the discrete  state with  maximum a  posteriori  probabil(cid:173) ity  from  the  noisy  state.  Simulations  show  that  DOLCE  leads  to  a  significant  improvement in  generalization  performance  over  earlier  neural  net  approaches  to FSM  induction.
Convergence of Indirect Adaptive Asynchronous Value Iteration Algorithms
Authors: Andrew G. Barto Vijaykumar Gullapalli
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Reinforcement Learning methods based on approximating dynamic  programming  (DP)  are  receiving  increased  attention  due  to  their  utility  in  forming  reactive  control  policies  for  systems  embedded  in  dynamic  environments.  Environments  are  usually  modeled  as  controlled  Markov  processes,  but  when  the  environment  model  is  not known a priori,  adaptive methods are necessary.  Adaptive con(cid:173) trol methods are often classified  as being direct  or indirect.  Direct  methods directly adapt the control policy from experience,  whereas  indirect  methods adapt a  model of the controlled process and com(cid:173) pute  control  policies  based  on  the  latest  model.  Our  focus  is  on  indirect  adaptive  DP-based  methods  in  this  paper.  We  present  a  convergence  result  for  indirect  adaptive  asynchronous  value  itera(cid:173) tion algorithms for the case in which a look-up table is used to store  the  value  function.  Our  result  implies  convergence  of several  ex(cid:173) isting reinforcement  learning algorithms such as adaptive real-time  dynamic programming (ARTDP)  (Barto,  Bradtke, &  Singh,  1993)  and prioritized  sweeping  (Moore  &  Atkeson,  1993).  Although  the  emphasis  of researchers  studying DP-based reinforcement  learning  has been on direct  adaptive methods such as Q-Learning (Watkins,  1989)  and  methods  using  TD  algorithms  (Sutton,  1988),  it is  not  clear that these direct methods are preferable in practice to indirect  methods such as those  analyzed  in  this  paper.
Fast Pruning Using Principal Components
Authors: Todd K. Leen, John E. Moody Asriel U. Levin
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present a new  algorithm for  eliminating excess parameters and  improving  network  generalization  after  supervised  training.  The  method,  "Principal Components Pruning (PCP)", is based on prin(cid:173) cipal component analysis of the node activations of successive layers  of the network.  It is  simple,  cheap to implement,  and effective.  It  requires  no  network  retraining,  and  does  not  involve  calculating  the full  Hessian of the cost function.  Only the weight and the node  activity  correlation  matrices for  each  layer of nodes  are  required.  We demonstrate the efficacy of the method on a regression problem  using  polynomial  basis functions,  and on an  economic time  series  prediction problem using a  two-layer, feedforward  network.
Hidden Markov Models for Human Genes
Authors: Søren Brunak, Yves Chauvin, Jacob Engelbrecht, Anders Krogh Pierre Baldi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Human  genes  are  not  continuous  but  rather  consist  of short  cod(cid:173) ing  regions  (exons)  interspersed  with  highly  variable  non-coding  regions  (introns).  We apply HMMs  to the problem of modeling ex(cid:173) ons,  introns and  detecting splice  sites  in  the  human  genome.  Our  most  interesting result  so  far  is  the  detection of particular oscilla(cid:173) tory patterns, with a minimal period ofroughly 10 nucleotides, that  seem to be characteristic of exon  regions  and  may have significant  biological  implications. 
• and  Division of Biology,  California Institute of Technology.  t and  Department of Psychology,  Stanford  University.
Postal Address Block Location Using a Convolutional Locator Network
Authors: John C. Platt Ralph Wolf
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This  paper  describes  the  use  of  a  convolutional  neural  network  to perform address block location on machine-printed mail pieces.  Locating the address block is a difficult object recognition problem  because  there is  often a  large  amount of extraneous printing on  a  mail piece and because address blocks vary dramatically in size and  shape.  We  used  a  convolutional locator  network  with four  outputs,  each  trained  to find  a  different  corner  of the  address  block.  A  simple  set of rules was used to generate ABL candidates from the network  output.  The system performs very well:  when allowed five guesses,  the network will tightly bound the address delivery information in  98.2% of the cases.
An Analog VLSI Saccadic Eye Movement System
Authors: Brooks Bishofberger, Christof Koch Timothy K. Horiuchi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  an effort  to understand saccadic eye  movements and their rela(cid:173) tion  to visual  attention and other forms  of eye  movements,  we  - in  collaboration with  a  number of other laboratories - are  carry(cid:173) ing out a  large-scale effort  to design  and build a  complete primate  oculomotor  system  using  analog  CMOS  VLSI  technology.  Using  this technology, a low power, compact, multi-chip system has been  built  which  works in  real-time  using  real-world  visual  inputs.  We  describe  in  this paper  the  performance  of an early  version  of such  a  system  including  a  1-D  array of photoreceptors  mimicking  the  retina, a circuit computing the mean location of activity represent(cid:173) ing  the  superior  colliculus,  a  saccadic  burst  generator,  and  a  one  degree-of-freedom  rotational  platform  which  models  the  dynamic  properties  of the  primate oculomotor plant.
An Optimization Method of Layered Neural Networks based on the Modified Information Criterion
Authors: Sumio Watanabe
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper proposes a practical optimization method for layered  neural networks, by which the optimal model and parameter can  be found simultaneously. 'i\Te modify the conventional information  criterion into a differentiable function of parameters, and then, min(cid:173) imize it, while controlling it back to the ordinary form. Effective(cid:173) ness of this method is discussed theoretically and experimentally.
A Computational Model for Cursive Handwriting Based on the Minimization Principle
Authors: Yasuharu Koike, Eric Vatikiotis-Bateson, Mitsuo Kawato Yasuhiro Wada
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  propose a  trajectory  planning  and  control  theory  for  continuous  movements  such  as  connected  cursive  handwriting  and  continuous  natural  speech.  Its  hardware  is  based  on  our  previously  proposed  forward-inverse-relaxation  neural  network  (Wada  & Kawato,  1993).  Computationally,  its  optimization  principle  is  the  minimum  torque(cid:173) change criterion.  Regarding  the  representation level, hard constraints  satisfied by a trajectory are represented as a set of via-points extracted  from  a  handwritten  character.  Accordingly,  we  propose a  via-point  estimation  algorithm  that  estimates  via-points  by  repeating  the  trajectory formation of a character and the via-point extraction from  the  character.  In  experiments,  good  quantitative  agreement  is  found  between  human  handwriting data and the trajectories generated by  the  theory.  Finally,  we  propose  a  recognition  schema  based  on  the  movement  generation.  We  show  a  result  in  which  the  recognition  schema is applied to the handwritten character recognition and can be  extended to the phoneme timing estimation of natural speech.
Discontinuous Generalization in Large Committee Machines
Authors: J. Hertz H. Schwarze
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The problem of learning from examples in multilayer networks is  studied within the framework of statistical mechanics. Using the  replica formalism we calculate the average generalization error of a  fully connected committee machine in the limit of a large number  of hidden units. If the number of training examples is proportional  to the number of inputs in the network, the generalization error  as a function of the training set size approaches a finite value. If  the number of training examples is proportional to the number of  weights in the network we find first-order phase transitions with a  discontinuous drop in the generalization error for both binary and  continuous weights.
Inverse Dynamics of Speech Motor Control
Authors: Eric Vatikiotis-Bateson, Mitsuo Kawato Makoto Hirayama
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Progress ha.s been made in comput.ational implementation of speech  production  based  on  physiological  dat.a.  An  inverse  dynamics  model  of  the  speech  articulator's  l1111sculo-skeletal  system.  which  is  the  mapping from  art.iculator  t.rajectories  to  e\ectromyogl'aphic  (EMG) signals,  was  modeled  using  the acquired  forward  dynamics  model  and  temporal  (smoot.hness  of EMG  activation)  and  range  constraints.  This inverse dynamics model allows  the use of a  faster  speech  mot.or control scheme, which  can  be applied to phoneme-to(cid:173) speech  synthesis via musclo-skeletal system  dynamics, or  to future  use in speech  recognition.  The forward acoustic model, which is  the  mapping from  articulator  trajectories  t.o  the  acoustic  parameters,  was  improved  by  adding  velocity  and  voicing  information  inputs  to distinguish  acollst.ic  paramet.er  differences  caused  by  changes  in  source  characterist.ics.
Learning Temporal Dependencies in Connectionist Speech Recognition
Authors: Mike Hochberg, Tony Robinson Steve Renals
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Hybrid connectionistfHMM systems model time both using a Markov  chain and through properties of a connectionist network. In this paper,  we discuss the nature of the time dependence currently employed in our  systems using recurrent networks (RNs) and feed-forward multi-layer  perceptrons (MLPs). In particular, we introduce local recurrences into a  MLP to produce an enhanced input representation. This is in the form  of an adaptive gamma filter and incorporates an automatic approach for  learning temporal dependencies. We have experimented on a speaker(cid:173) independent phone recognition task using the TIMIT database. Results  using the gamma filtered input representation have shown improvement  over the baseline MLP system. Improvements have also been obtained  through merging the baseline and gamma filter models.
Odor Processing in the Bee: A Preliminary Study of the Role of Central Input to the Antennal Lobe
Authors: David Marsan, Claudine Masson, Michel Kerszberg Christiane Linster
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Based  on  precise  anatomical  data  of the  bee's  olfactory  system,  we  propose an investigation of the possible mechanisms of modulation and  control between the two levels of olfactory  information processing:  the  antennallobe glomeruli  and  the  mushroom  bodies.  We use  simplified  neurons,  but realistic  architecture.  As  a  first  conclusion,  we  postulate  that the feature extraction performed by the antennallobe (glomeruli and  interneurons)  necessitates  central  input from  the  mushroom bodies for  fine  tuning. The central input thus  facilitates  the evolution from fuzzy  olfactory images in the glomerular layer towards more focussed  images  upon odor presentation.
Surface Learning with Applications to Lipreading
Authors: Stephen M. Omohundro Christoph Bregler
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Most connectionist research has focused on learning mappings from  one space to another (eg.  classification and regression).  This paper  introduces  the  more  general  task  of learning  constraint  surfaces.  It describes  a  simple  but  powerful  architecture  for  learning  and  manipulating  nonlinear  surfaces  from  data.  We  demonstrate  the  technique  on low dimensional synthetic surfaces  and compare it to  nearest  neighbor  approaches.  We  then  show  its  utility in learning  the space of lip images in a system for improving speech recognition  by lip  reading.  This learned  surface  is  used  to improve the visual  tracking  performance during recognition. 
1  Surface Learning 
Mappings  are  an  appropriate representation for  systems  whose  variables  naturally  decompose into "inputs"  and  "outputs)).  To use  a learned mapping, the input vari(cid:173) ables must be known and error-free and a single output value must be estimated for  each input.  Many tasks in vision, robotics,  and control must maintain relationships  between  variables  which  don't  naturally  decompose  in  this  way.  Instead,  there  is  a  nonlinear  constraint  surface  on  which  the  values  of the  variables  are  jointly re(cid:173) stricted to lie.  We propose a representation for such surfaces  which supports a wide  range of queries  and which  can  be  naturally learned from  data. 
The simplest queries are "completion queries)).  In these queries, the values of certain  variables  are  specified  and  the  values  (or  constraints  on  the  values)  of remaining
Figure of Merit Training for Detection and Spotting
Authors: Richard P Lippmann Eric I. Chang
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Spotting tasks require detection of target patterns from a background of  richly varied non-target inputs. The performance measure of interest for  these  tasks,  called  the figure  of merit (FOM),  is  the  detection  rate  for  target  patterns  when  the  false  alarm  rate  is  in  an  acceptable  range.  A  new approach to training spotters is presented which computes the FOM  gradient  for  each  input pattern  and  then  directly  maximizes  the  FOM  using  b ackpropagati on.  This  eliminates  the  need  for  thresholds  during  training. It also  uses  network resources  to model  Bayesian a posteriori  probability  functions  accurately  only  for  patterns  which  have  a  significant effect on  the  detection accuracy over the  false  alarm rate of  interest.  FOM  training  increased  detection  accuracy  by  5  percentage  points for a hybrid radial basis function  (RBF) - hidden Markov model  (HMM) wordspotter on the credit-card speech corpus.