mtybilly/Instruct2DS huggingface dataset:
Autoencoders, Minimum Description Length and Helmholtz Free Energy
Authors: Richard S. Zemel Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: An autoencoder network uses a set of recognition weights to convert an  input vector into a code vector.  It then uses a set of generative weights to  convert the code vector into an approximate reconstruction of the input  vector.  We derive an objective function for training autoencoders based  on the  Minimum  Description Length  (MDL)  principle.  The aim  is  to  minimize the information required to describe both the code vector and  the reconstruction error.  We  show  that this information  is  minimized  by choosing code vectors stochastically according to a Boltzmann distri(cid:173) bution, where the generative weights define the energy of each possible  code vector given  the input vector.  Unfortunately,  if the code vectors  use distributed representations, it is exponentially expensive to compute  this Boltzmann distribution because it involves all possible code vectors.  We show that the recognition weights of an autoencoder can  be used to  compute an approximation to the Boltzmann distribution and that this ap(cid:173) proximation gives an upper bound on the description length.  Even when  this bound is poor,  it can  be used  as a  Lyapunov function for learning  both the generative and the recognition weights.  We  demonstrate that  this approach can be used to learn factorial codes.
Dual Mechanisms for Neural Binding and Segmentation
Authors: Leif H. Finkel Paul Sajda
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  propose  that  the  binding  and  segmentation  of visual  features  is  mediated  by  two  complementary  mechanisms;  a  low  resolu(cid:173) tion,  spatial-based,  resource-free  process  and  a  high  resolution,  temporal-based,  resource-limited  process.  In the visual  cortex,  the  former  depends  upon  the orderly  topographic organization in  stri(cid:173) ate  and  extrastriate  areas  while  the  latter  may  be  related  to  ob(cid:173) served  temporal  relationships  between  neuronal  activities .  Com(cid:173) puter  simulations  illustrate  the  role  the  two  mechanisms  play  in  figure/ ground discrimination,  depth-from-occlusion,  and the vivid(cid:173) ness  of perceptual completion. 
1  COMPLEMENTARY BINDING  MECHANISMS 
The  "binding  problem"  is  a  classic  problem  in  computational  neuroscience  which  considers how  neuronal activities are grouped to create mental representations.  For  the  case  of visual  processing,  the  binding  of neuronal  activities  requires  a  mecha(cid:173) nism  for  selectively  grouping fragmented  visual  features  in  order  to  construct  the  coherent representations (i.e.  objects)  which we  perceive.  In this paper we  argue for  the  existence  of two  complementary  mechanisms  for  neural  binding,  and  we  show  how  such mechanisms  may operate  in  the  constructiO:l  of intermediate-level visual  representations.
The Statistical Mechanics of k-Satisfaction
Authors: Géza Györgyi, Naftali Tishby, Lidror Troyansky Scott Kirkpatrick
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The  satisfiability of random  CNF  formulae  with  precisely  k  vari(cid:173) ables per clause ("k-SAT") is a popular testbed for the performance  of search  algorithms.  Formulae have  M  clauses  from  N  variables,  randomly  negated,  keeping  the  ratio  a  =  M / N  fixed .  For  k  =  2,  this  model  has  been  proven  to  have  a  sharp  threshold  at  a  = 1  between  formulae which  are almost aways satisfiable and formulae  which  are  almost never  satisfiable  as  N  --jo  00 .  Computer  experi(cid:173) ments for  k  = 2,  3,  4,  5  and  6,  (carried  out  in  collaboration  with  B.  Selman  of ATT Bell  Labs). show  similar threshold  behavior for  each  value  of k.  Finite-size  scaling,  a  theory  of the  critical  point  phenomena used  in statistical physics, is shown  to characterize the  size  dependence  near  the  threshold.  Annealed  and  replica-based  mean field  theories  give a  good account  of the results. 
"Permanent  address:  IBM  TJ Watson  Research  Center,  Yorktown  Heights,  NY 10598  USA.  (kirk@watson.ibm.com)  Portions  of  this  work  were  done  while  visiting  the  Salk  Institute,  with support from  the McDonnell-Pew  Foundation.
Lipreading by neural networks: Visual preprocessing, learning, and sensory integration
Authors: K. Venkatesh Prasad, David G. Stork, Marcus Hennecke Gregory J. Wolff
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We have developed visual preprocessing algorithms for extracting  phonologically relevant features from the grayscale video image of  a speaker, to provide speaker-independent inputs for an automat(cid:173) ic lipreading ("speechreading") system. Visual features such as  mouth open/closed, tongue visible/not-visible, teeth visible/not(cid:173) visible, and several shape descriptors of the mouth and its motion  are all rapidly computable in a manner quite insensitive to lighting  conditions. We formed a hybrid speechreading system consisting  of two time delay neural networks (video and acoustic) and inte(cid:173) grated their responses by means of independent opinion pooling  - the Bayesian optimal method given conditional independence,  which seems to hold for our data. This hybrid system had an er(cid:173) ror rate 25% lower than that of the acoustic subsystem alone on a  five-utterance speaker-independent task, indicating that video can  be used to improve speech recognition.
Dynamic Modulation of Neurons and Networks
Authors: Eve Marder
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Biological neurons have a variety of intrinsic properties because of the  large number of voltage dependent currents  that control their activity.  Neuromodulatory substances modify both the balance of conductances  that determine intrinsic properties and  the strength of synapses.  These  mechanisms alter circuit dynamics,  and suggest that functional circuits  exist only in the modulatory environment in which  they operate.
Training Neural Networks with Deficient Data
Authors: Subutai Ahmad, Ralph Neuneier Volker Tresp
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  analyze how  data with uncertain or missing input features  can  be  incorporated  into  the  training  of a  neural  network.  The  gen(cid:173) eral  solution requires  a  weighted  integration over  the unknown  or  uncertain  input although  computationally cheaper  closed-form so(cid:173) lutions  can  be  found  for  certain  Gaussian  Basis  Function  (GBF)  networks.  We also discuss  cases  in which heuristical solutions such  as substituting the mean of an unknown input can  be harmful.
When will a Genetic Algorithm Outperform Hill Climbing
Authors: John H. Holland, Stephanie Forrest Melanie Mitchell
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We analyze a simple hill-climbing algorithm (RMHC) that was pre(cid:173) viously shown to outperform a genetic algorithm (GA) on a simple  "Royal  Road"  function.  We  then  analyze  an  "idealized"  genetic  algorithm (IGA)  that is  significantly faster  than RMHC  and that  gives  a lower  bound for  GA speed.  We  identify the features  of the  IGA that give rise  to this speedup,  and discuss  how  these features  can be incorporated into a real GA.
Unsupervised Parallel Feature Extraction from First Principles
Authors: Reiner Lenz Mats Österberg
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We describe a number of learning rules that can be used to train un(cid:173) supervised  parallel feature  extraction systems.  The learning  rules  are  derived  using  gradient  ascent  of a  quality  function.  We  con(cid:173) sider  a  number of quality functions  that are  rational functions  of  higher  order  moments of the  extracted  feature  values.  We  show  that  one  system  learns  the  principle  components  of  the  correla(cid:173) tion matrix.  Principal component analysis systems are usually not  optimal feature  extractors  for  classification.  Therefore  we  design  quality functions which produce feature vectors that support unsu(cid:173) pervised  classification.  The properties of the  different systems are  compared  with  the  help  of different  artificially  designed  datasets  and a  database consisting of all Munsell color spectra.
Classification of Electroencephalogram using Artificial Neural Networks
Authors: D S C So, A Sergejew A C Tsoi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In this paper, we will consider the problem of classifying electroencephalo(cid:173) gram (EEG) signals of normal subjects, and subjects suffering from psychi(cid:173) atric disorder, e.g., obsessive compulsive disorder, schizophrenia, using a  class of artificial neural networks, viz., multi-layer perceptron. It is shown  that the multilayer perceptron is capable of classifying unseen test EEG  signals to a high degree of accuracy.
Comparison Training for a Rescheduling Problem in Neural Networks
Authors: Martine de Gerlache Didier Keymeulen
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Airline  companies  usually  schedule  their flights  and  crews  well  in  advance  to optimize  their  crew  pools  activities.  Many  events such  as  flight  delays or  the  absence  of a  member require  the  crew  pool  rescheduling team to change the initial schedule  (rescheduling).  In  this paper, we  show  that the neural network comparison paradigm  applied  to  the  backgammon  game  by  Tesauro  (Tesauro  and  Se(cid:173) jnowski,  1989)  can  also  be applied  to the rescheduling  problem of  an  aircrew  pool.  Indeed  both  problems  correspond  to  choosing  the best solut.ion  from a set of possible ones without ranking them  (called here best choice problem).  The paper explains from a math(cid:173) ematical point of view  the architecture and the learning strategy of  the backpropagation neural network used for  the best  choice  prob(cid:173) lem.  We  also  show  how  the learning  phase  of the  network  can  be  accelerated.  Finally  we  apply  the  neural  network  model  to some  real rescheduling  problems for  the  Belgian  Airline  (Sabena).
Bounds on the complexity of recurrent neural network implementations of finite state machines
Authors: Don R. Hush Bill G. Horne
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In this paper the efficiency of recurrent neural network implementa(cid:173) tions of m-state finite state machines will be explored. Specifically,  it will be shown that the node complexity for the unrestricted case  can be bounded above by 0 ( fo) . It will also be shown that the  node complexity is 0 (y'm log m) when the weights and thresholds  are restricted to the set {-I, I}, and 0 (m) when the fan-in is re(cid:173) stricted to two. Matching lower bounds will be provided for each  of these upper bounds assuming that the state of the FSM can be  encoded in a subset of the nodes of size rlog m 1.
Classification of Multi-Spectral Pixels by the Binary Diamond Neural Network
Authors: Yehuda Salu
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: A new neural network, the Binary Diamond, is presented and its use  as  a classifier is  demonstrated and evaluated. The network  is  of the  feed-forward  type.  It learns  from  examples  in  the 'one shot'  mode,  and recruits new neurons as needed. It was tested on the problem of  pixel classification, and performed well.  Possible applications of the  network in associative memories are outlined. 
1 
INTRODUCTION:  CLASSIFICATION  BY  CLUES 
Classification  is  a  process  by  which  an  item  is  assigned  to  a  class.  Classification  is  widely  used  in  the  animal  kingdom.  Identifying  an  item  as  food  is  classification.  Assigning  words  to  objects,  actions,  feelings,  and  situations  is  classification.  The  purpose  of  this  work  is  to  introduce  a  new  neural  network,  the  Binary  Diamond,  which  can  be  used  as  a  general  purpose  classification  tool.  The  design  and  operational  mode  of  the  Binary  Diamond  are  influenced  by  observations  of  the  underlying  mechanisms that take place in  human classification  processes. 
An item to be classified consists of basic features. Any arbitrary combination of basic  features will be called a clue. Generally, an item will consist of many clues. Clues  are  related not only to  the items which  contain them, but  also  to the classes.  Each class,  that resides in the memory, has a list of clues which are associated with it. These clues
Optimal Brain Surgeon: Extensions and performance comparisons
Authors: David G. Stork, Gregory Wolff Babak Hassibi
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: a  second-order 
We  extend  Optimal  Brain  Surgeon  (OBS)  - to  allow  for  general  error  mea(cid:173) method  for  pruning  networks  - sures, and explore a reduced computational and storage implemen(cid:173) tation  via  a  dominant  eigenspace  decomposition.  Simulations  on  nonlinear,  noisy  pattern  classification  problems  reveal  that  OBS  does  lead  to  improved  generalization,  and  performs  favorably  in  comparison with Optimal Brain Damage (OBD).  We  find  that the  required  retraining  steps  in  OBD  may  lead  to inferior  generaliza(cid:173) tion, a result that can be interpreted as due to injecting noise back  into the system.  A common technique is to stop training of a large  network at the minimum validation error.  We  found  that the test  error  could  be  reduced  even  further  by  means  of  OBS  (but  not  OBD)  pruning.  Our  results justify the t  ~ 0  approximation  used  in  OBS  and  indicate  why  retraining  in  a  highly  pruned  network  may lead to inferior  performance.
Agnostic PAC-Learning of Functions on Analog Neural Nets
Authors: Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: There  exist  a  number  of negative  results  ([J),  [BR),  [KV])  about 
learning on neural nets in Valiant's model [V)  for  probably approx(cid:173)
imately correct  learning ("PAC-learning").  These  negative results 
are  based  on an  asymptotic analysis  where  one  lets  the  number of 
nodes  in the neural net go to infinit.y.  Hence  this analysis is less ad(cid:173)
equate for  the investigation of learning on a  small fixed  neural  net. 
with relatively few  analog inputs  (e.g.  the principal components of 
some sensory  data).  The latter  type of learning problem gives  rise 
to a different kind of asymptotic question:  Can the true error of the 
neural  net be brought arbitrarily close  to  that of a  neural  net  with 
"optimal"  weights through sufficiently long training?  In this paper 
we  employ some new  arguments ill  order  to give  a  positive answer 
to this question in Haussler's rather realistic refinement of Valiant's 
model  for  PAC-learning  ([H),  [KSS)).  In  this  more realistic  model 
no  a-priori  assumptions are  required  about  the  "learning  target" , 
noise is  permitted in the  training data, and the inputs and outputs 
are  not  restricted  to  boolean  values.  As  a  special  case  our  result 
implies one of the first  positive results about learning on multi-layer 
neural  net.s  in  Valiant's original  PAC-learning model.  At  the  end 
of this  paper  we  will  describe  an  efficient  parallel implementation 
of this new  learning algorit.hm.
Mixtures of Controllers for Jump Linear and Non-Linear Plants
Authors: Steven J. Nowlan Timothy W. Cacciatore
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We describe an extension to the Mixture of Experts architecture for  modelling and  controlling dynamical systems which  exhibit  multi(cid:173) ple modes of behavior.  This extension is based on a Markov process  model,  and suggests  a  recurrent  network for  gating a  set  of linear  or  non-linear controllers.  The new  architecture  is  demonstrated  to  be  capable  of learning  effective  control  strategies  for  jump  linear  and  non-linear plants with multiple modes of behavior.
A Hybrid Radial Basis Function Neurocomputer and Its Applications
Authors: Paul M. Chau, Raoul Tawel, Bjorn Lambrigtsen, Mark Plutowski Steven S. Watkins
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: A neurocomputer was  implemented using radial basis functions  and a  combination  of  analog  and  digital  VLSI  circuits.  The  hybrid  system  uses  custom  analog  circuits  for  the  input  layer  and  a  digital  signal  processing board for the hidden and output layers. The system combines  the  advantages of both analog and digital circuits. featuring low power  consumption while minimizing overall system error. The analog circuits  have been fabricated and tested, the  system has been built,  and several  applications  have  been  executed  on  the  system.  One  application  provides significantly better results for  a remote sensing problem than  have been previously obtained using conventional methods.
Resolving motion ambiguities
Authors: D. Geiger K. I. Diamantaras
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We address the problem of optical flow reconstruction and in par(cid:173) ticular the problem of resolving ambiguities near edges. They oc(cid:173) cur due to (i) the aperture problem and (ii) the occlusion problem,  where pixels on both sides of an intensity edge are assigned the same  velocity estimates (and confidence). However, these measurements  are correct for just one side of the edge (the non occluded one).  Our approach is to introduce an uncertamty field with respect to  the estimates and confidence measures. We note that the confi(cid:173) dence measures are large at intensity edges and larger at the con(cid:173) vex sides of the edges, i.e. inside corners, than at the concave side.  We resolve the ambiguities through local interactions via coupled  Markov random fields (MRF) . The result is the detection of motion  for regions of images with large global convexity.
Developing Population Codes by Minimizing Description Length
Authors: Geoffrey E. Hinton Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The Minimum Description Length principle (MDL) can be used to  train the hidden units of a neural network to extract a representa(cid:173) tion that is cheap to describe but nonetheless allows the input to  be reconstructed accurately. We show how MDL can be used to  develop highly redundant population codes. Each hidden unit has  a location in a low-dimensional implicit space. If the hidden unit  activities form a bump of a standard shape in this space, they can  be cheaply encoded by the center ofthis bump. So the weights from  the input units to the hidden units in an autoencoder are trained  to make the activities form a standard bump. The coordinates of  the hidden units in the implicit space are also learned, thus allow(cid:173) ing flexibility, as the network develops a discontinuous topography  when presented with different input classes. Population-coding in  a space other than the input enables a network to extract nonlinear  higher-order properties of the inputs. 
Most existing unsupervised learning algorithms can be understood using the Min(cid:173) imum Description Length (MDL) principle (Rissanen, 1989). Given an ensemble  of input vectors, the aim of the learning algorithm is to find a method of coding  each input vector that minimizes the total cost, in bits, of communicating the input  vectors to a receiver. There are three terms in the total description length: 
• The code-cost is the number of bits required to communicate the code 
that the algorithm assigns to each input vector.
Neural Network Definitions of Highly Predictable Protein Secondary Structure Classes
Authors: Evan Steeg, Robert Farber Alan Lapedes
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We use two co-evolving neural networks to determine new classes  of protein secondary structure which are significantly more pre(cid:173) dictable from local amino sequence than the conventional secondary  structure classification. Accurate prediction of the conventional  secondary structure classes: alpha helix, beta strand, and coil, from  primary sequence has long been an important problem in compu(cid:173) tational molecular biology. Neural networks have been a popular  method to attempt to predict these conventional secondary struc(cid:173) ture classes. Accuracy has been disappointingly low. The algo(cid:173) rithm presented here uses neural networks to similtaneously exam(cid:173) ine both sequence and structure data, and to evolve new classes  of secondary structure that can be predicted from sequence with  significantly higher accuracy than the conventional classes. These  new classes have both similarities to, and differences with the con(cid:173) ventional alpha helix, beta strand and coil.
Coupled Dynamics of Fast Neurons and Slow Interactions
Authors: R. W. Penney, D. Sherrington A.C.C. Coolen
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: A simple model of coupled dynamics of fast neurons and slow inter(cid:173) actions, modelling self-organization in recurrent neural networks,  leads naturally to an effective statistical mechanics characterized  by a partition function which is an average over a replicated system.  This is reminiscent of the replica trick used to study spin-glasses,  but with the difference that the number of replicas has a physi(cid:173) cal meaning as the ratio of two temperatures and can be varied  throughout the whole range of real values. The model has inter(cid:173) esting phase consequences as a function of varying this ratio and  external stimuli, and can be extended to a range of other models. 
1 A SIMPLE MODEL WITH FAST DYNAMIC 
NEURONS AND SLOW DYNAMIC INTERACTIONS 
As the basic archetypal model we consider a system of Ising spin neurons (J'i E  {-I, I}, i E {I, ... , N}, interacting via continuous-valued symmetric interactions,  Iij, which themselves evolve in response to the states of the neurons. The neurons  are taken to have a stochastic field-alignment dynamics which is fast compared with  the evolution rate of the interactions hj, such that on the time-scale of Iii-dynamics  the neurons are effectively in equilibrium according to a Boltzmann distribution,
Using Local Trajectory Optimizers to Speed Up Global Optimization in Dynamic Programming
Authors: Christopher G. Atkeson
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Dynamic programming provides a methodology to develop planners  and  controllers  for  nonlinear systems.  However,  general  dynamic  programming is  computationally intractable.  We  have  developed  procedures that allow more complex planning and control problems  to  be  solved.  We  use  second  order  local  trajectory  optimization  to  generate  locally  optimal  plans  and  local  models  of  the  value  function  and its derivatives.  We  maintain global consistency of the  local  models  of the  value  function,  guaranteeing  that  our  locally  optimal plans are actually globally optimal, up  to the resolution of  our search  procedures. 
Learning to  do the right thing at each  instant in situations that evolve over  time is  difficult,  as  the future  cost  of actions chosen  now  may not be obvious immediately,  and  may only  become  clear  with  time.  Value  functions  are  a  representational  tool  that  makes  the  consequences  of actions  explicit.  Value  functions  are  difficult  to  learn  directly,  but they  can  be built up from  learned  models of the dynamics of the  world  and  the  cost  function.  This  paper focuses  on  how  fast  optimizers  that  only  produce  locally optimal answers  can  playa useful  role  in  speeding  up  the  process  of computing or learning a globally optimal value function.  Consider  a  system  with  dynamics  Xk+l  = f(xk, Uk)  and  a  cost  function  L(Xk, Uk),
Connectionist Modeling and Parallel Architectures
Authors: Ah Chung Tsoi Joachim Diederich
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The  introduction  of  specialized  hardware  platforms  for  connectionist  modeling 
("connectionist  supercomputer")  has  created a number  of research  topics.  Some of 
these issues are controversial, e.g.  the efficient implementation of incremental learn(cid:173)
ing techniques, the need for the dynamic reconfiguration of networks and possible 
programming environments for these machines.
Structural and Behavioral Evolution of Recurrent Networks
Authors: Peter J. Angeline, Jordan B. Pollack Gregory M. Saunders
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper introduces GNARL, an evolutionary program which induces  recurrent neural networks that are structurally unconstrained. In contrast  to constructive and destructive algorithms, GNARL employs a popula(cid:173) tion of networks and uses a fitness  function's  unsupervised feedback to  guide  search  through  network  space.  Annealing  is  used  in  generating  both  gaussian  weight  changes  and  structural  modifications.  Applying  GNARL  to  a complex  search and collection task demonstrates  that the  system is capable of inducing networks with complex internal dynamics.
Two-Dimensional Object Localization by Coarse-to-Fine Correlation Matching
Authors: Eric Mjolsness Chien-Ping Lu
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present  a  Mean  Field  Theory  method  for  locating  two(cid:173) dimensional  objects  that  have  undergone  rigid  transformations.  The  resulting  algorithm  is  a  form  of  coarse-to-fine  correlation  matching.  We first  consider  problems of matching synthetic point  data,  and  derive  a  point matching objective function.  A  tractable  line segment matching objective function is  derived by considering  each  line segment  as  a  dense  collection of points,  and approximat(cid:173) ing it by a sum of Gaussians.  The algorithm is tested on real images  from  which line segments are extracted  and  matched.
WATTLE: A Trainable Gain Analogue VLSI Neural Network
Authors: Marwan Jabri Richard Coggins
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: This paper describes a low power analogue VLSI neural network  called Wattle. Wattle is a 10:6:4 three layer perceptron with multi(cid:173) plying DAC synapses and on chip switched capacitor neurons fabri(cid:173) cated in 1.2um CMOS. The on chip neurons facillitate variable gain  per neuron and lower energy/connection than for previous designs.  The intended application of this chip is Intra Cardiac Electrogram  classification as part of an implantable pacemaker / defibrillator sys(cid:173) tem. Measurements of t.he chip indicate that 10pJ per connection  is achievable as part of an integrated system. Wattle has been suc(cid:173) cessfully trained in loop on parity 4 and ICEG morphology classi(cid:173) fication problems.
Bayesian Backpropagation Over I-O Functions Rather Than Weights
Authors: David H. Wolpert
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The conventional Bayesian justification of backprop is that it finds the  MAP weight vector. As this paper shows, to find the MAP i-o function  instead one must add a correction tenn to backprop. That tenn biases one  towards i-o functions with small description lengths, and in particular fa(cid:173) vors (some kinds of) feature-selection, pruning, and weight-sharing.
Fast Non-Linear Dimension Reduction
Authors: Todd K. Leen Nanda Kambhatla
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present  a  fast  algorithm  for  non-linear  dimension  reduction.  The  algorithm  builds  a  local  linear model  of the data by  merging  PCA  with  clustering  based  on  a  new  distortion  measure.  Exper(cid:173) iments  with  speech  and  image  data indicate  that  the  local  linear  algorithm produces encodings with lower distortion than those built  by five  layer auto-associative networks.  The local linear algorithm  is  also more than an order of magnitude faster  to train.
Observability of Neural Network Behavior
Authors: Fernanda Botelho Max Garzon
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We prove that except possibly for small exceptional sets, discrete(cid:173) time analog neural nets are globally observable, i.e. all their cor(cid:173) rupted pseudo-orbits on computer simulations actually reflect the  true dynamical behavior of the network. Locally finite discrete  (boolean) neural networks are observable without exception.
Optimal Signalling in Attractor Neural Networks
Authors: Eytan Ruppin Isaac Meilijson
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: In  [Meilijson  and  Ruppin,  1993]  we  presented  a  methodological  framework  describing  the  two-iteration  performance  of Hopfield(cid:173) like  attractor  neural  networks  with  history-dependent,  Bayesian  dynamics.  We  now  extend  this  analysis in  a number of directions:  input  patterns  applied  to  small subsets  of neurons,  general  con(cid:173) nectivity  architectures  and  more efficient  use  of history.  We  show  that the optimal signal (activation) function has a slanted sigmQidal  shape, and provide an intuitive account of activation functions with  a non-monotone shape.  This function endows the model with some  properties  characteristic of cortical  neurons' firing.
Neural Network Exploration Using Optimal Experiment Design
Authors: David A. Cohn
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Consider  the  problem of learning input/output mappings through  exploration,  e.g.  learning  the kinematics or  dynamics of a  robotic  manipulator.  If actions  are  expensive  and  computation is  cheap,  then  we  should  explore  by  selecting  a  trajectory  through  the  in(cid:173) put  space  which  gives  us  the  most  amount  of information in  the  fewest  number of steps.  I discuss how  results from the field  of opti(cid:173) mal experiment design may be used to guide such exploration, and  demonstrate its use  on a  simple kinematics problem.
Directional Hearing by the Mauthner System
Authors: Robert C. Eaton Audrey L. Guzik
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We provide a computational description of the function of the Mau(cid:173) thner  system.  This  is  the  brainstem  circuit  which  initiates  fast(cid:173) start  escapes  in  teleost  fish  in  response  to  sounds.  Our  simula(cid:173) tions,  using  back propagation in  a  realistically  constrained  feedfor(cid:173) ward  network,  have generated hypotheses  which  are directly inter(cid:173) pretable  in  terms  of the  activity  of the  auditory  nerve  fibers,  the  principle  cells  of the  system  and  their  associated  inhibitory  neu(cid:173) rons.
Asynchronous Dynamics of Continuous Time Neural Networks
Authors: Qingnan Li, Edward K. Blum Xin Wang
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Motivated by mathematical modeling, analog implementation and  distributed simulation of neural networks, we present a definition of  asynchronous  dynamics  of general  CT dynamical systems  defined  by  ordinary  differential equations,  based  on  notions of local  times  and  communication  times.  We  provide  some  preliminary  results  on  globally  asymptotical  convergence  of  asynchronous  dynamics  for  contractive  and  monotone CT  dynamical systems.  When  ap(cid:173) plying  the  results  to  neural  networks,  we  obtain  some  conditions  that ensure  additive-type neural  networks  to be  asynchronizable.
Development of Orientation and Ocular Dominance Columns in Infant Macaques
Authors: Lynne Kiorpes, Gary G. Blasdel Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Maps of orientation preference and ocular dominance were recorded  optically from the cortices of 5 infant macaque monkeys, ranging in  age from 3.5 to 14 weeks.  In agreement with previous observations,  we  found  that basic features  of orientation and ocular  dominance  maps, as well  as correlations between them, are present  and robust  by  3.5  weeks  of age.  We  did  observe  changes  in  the  strength  of  ocular dominance signals, as  well  as  in the spacing of ocular dom(cid:173) inance bands, both of which  increased steadily between  3.5 and  14  weeks  of age.  The  latter  finding  suggests  that  the  adult  spacing  of ocular dominance bands depends on cortical growth in  neonatal  animals.  Since  we  found  no  corresponding  increase  in  the spacing  of orientation  preferences,  however,  there  is  a  possibility  that  the  orientation preferences  of some cells  change as  the cortical surface  expands.  Since  correlations  between  the  patterns  of orientation  selectivity  and ocular dominance  are  present  at  an  age,  when  the  visual system is still immature, it seems more likely that  their  de(cid:173) velopment may be an innate process and may not require extensive  visual experience.
Counting function theorem for multi-layer networks
Authors: Adam Kowalczyk
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We show that a randomly selected N-tuple x of points ofRn with  probability> 0 is such that any multi-layer percept ron with the  first hidden layer composed of hi threshold logic units can imple- ment exactly 2 2:~~~ ( Nil) different dichotomies of x. If N > hin  then such a perceptron must have all units of the first hidden layer  fully connected to inputs. This implies the maximal capacities (in  the sense of Cover) of 2n input patterns per hidden unit and 2 input  patterns per synaptic weight of such networks (both capacities are  achieved by networks with single hidden layer and are the same as  for a single neuron). Comparing these results with recent estimates  of VC-dimension we find that in contrast to the single neuron case,  for sufficiently large nand hl, the VC-dimension exceeds Cover's  capacity.
Supervised Learning with Growing Cell Structures
Authors: Bernd Fritzke
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We  present  a  new  incremental  radial  basis  function  network  suit(cid:173) able  for  classification  and  regression  problems.  Center  positions  are  continuously  updated  through  soft  competitive learning.  The  width  of  the  radial  basis  functions  is  derived  from  the  distance  to  topological  neighbors.  During  the  training  the  observed  error  is  accumulated  locally  and  used  to  determine  where  to  insert  the  next  unit.  This  leads  (in  case  of classification  problems)  to  the  placement  of units  near  class  borders  rather  than  near  frequency  peaks as is done by  most existing methods.  The resulting networks  need  few  training epochs  and seem  to generalize  very  well.  This is  demonstrated  by  examples.
A Learning Analog Neural Network Chip with Continuous-Time Recurrent Dynamics
Authors: Gert Cauwenberghs
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We present experimental results on supervised learning of dynam(cid:173) ical features  in  an  analog  VLSI  neural  network  chip.  The recur(cid:173) rent network, containing six continuous-time analog neurons and 42  free parameters (connection strengths and thresholds), is trained to  generate time-varying outputs approximating given periodic signals  presented to the network.  The chip implements a stochastic pertur(cid:173) bative algorithm,  which  observes the error gradient along random  directions in the parameter space for  error-descent learning.  In ad(cid:173) dition  to  the  integrated  learning functions  and  the  generation  of  pseudo-random  perturbations,  the chip  provides  for  teacher forc(cid:173) ing and long-term storage of the volatile parameters.  The network  learns  a  1  kHz  circular trajectory in  100  sec.  The  chip  occupies  2mm  x 2mm  in a  2JLm  CMOS  process, and dissipates  1.2  m W.
Learning Classification with Unlabeled Data
Authors: Virginia R. de Sa
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: One of the advantages of supervised learning is that the final  error met(cid:173) ric is available during training.  For classifiers, the algorithm can directly  reduce  the  number  of misclassifications  on  the  training  set.  Unfortu(cid:173) nately, when modeling human learning or constructing classifiers for au(cid:173) tonomous  robots,  supervisory  labels  are  often  not available  or  too  ex(cid:173) pensive.  In this paper we show that we can  substitute for  the labels by  making use of structure between the pattern distributions to different sen(cid:173) sory modalities.  We show that minimizing the disagreement between the  outputs of networks processing patterns from these different modalities is  a sensible approximation to minimizing the number of misclassifications  in each modality, and leads to similar results.  Using the Peterson-Barney  vowel  dataset we show  that the  algorithm performs well  in finding  ap(cid:173) propriate placement for the codebook vectors particularly when the con(cid:173) fuseable classes are different for the two modalities.
Temporal Difference Learning of Position Evaluation in the Game of Go
Authors: Peter Dayan, Terrence J. Sejnowski Nicol N. Schraudolph
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The  game of Go has a high branching factor that defeats the tree  search  approach  used  in  computer  chess,  and  long-range  spa(cid:173) tiotemporal interactions that make position evaluation extremely  difficult.  Development of conventional Go programs is hampered  by their  knowledge-intensive  nature.  We  demonstrate a  viable  alternative by training networks to evaluate Go positions via tem(cid:173) poral difference (TD)  learning.  Our approach  is  based  on network architectures  that reflect  the  spatial organization of both  input and  reinforcement  signals on  the  Go  board,  and  training  protocols  that  provide  exposure  to  competent (though unlabelled)  play.  These  techniques yield  far  better performance than undifferentiated networks trained by self(cid:173) play alone.  A network with less than 500 weights learned within  3,000 games of 9x9 Go a position evaluation function that enables  a primitive one-ply search to defeat a commercial Go program at  a low playing level.
Recovering a Feed-Forward Net From Its Output
Authors: Scott Markel Charles Fefferman
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: We study feed-forward  nets with arbitrarily many layers,  using the stan(cid:173) dard  sigmoid,  tanh  x.  Aside  from  technicalities,  our  theorems  are:  1.  Complete knowledge of the output of a neural net for  arbitrary inputs  uniquely specifies  the architecture,  weights and thresholds;  and 2.  There  are  only  finitely  many  critical  points  on  the  error  surface  for  a  generic  training problem. 
Neural  nets  were  originally  introduced  as  highly  simplified  models  of the  nervous  system.  Today  they  are  widely  used  in  technology  and  studied  theoretically  by  scientists  from  several  disciplines.  However,  they  remain little understood.  Mathematically, a  (feed-forward)  neural  net  consists  of: 
(1)  A  finite  sequence  of positive integers  (Do, D 1 ,  ... ,  D£);  (2)  A family of real numbers (wJ d  defined for 1 :5 e 5:  L,  1 5:  j  5:  D l ,  1 5:  k :5  Dl-l ; 
and 
(3)  A family of real  numbers  (OJ)  defined  for  15: f 5:  L,  15: j  5:  Dl. 
The sequence  (Do,  D 1 ,  .. "  DL )  is  called the  architecture of the neural net, while the  W]k  are called  weights and  the  OJ  thresholds.  Neural  nets  are used  to compute non-linear  maps from  }R.N  to  }R.M  by  the following  construction.  vVe  begin by fixing a nonlinear function 0-( x)  of one  variable.  Analogy  with  the  nervous  system  suggests  that  we  take  o-(x)  asymptotic  to  constants  as  x  tends  to ±oo;  a  standard  choice,  which  we  adopt  throughout  this  paper,  is  o-(.r)  = 

Alternate address:  Dept.  of Mathematics. Princeton University, Princeton, NJ  08544-1000.
Optimal Stochastic Search and Adaptive Momentum
Authors: Genevieve B. Orr Todd K. Leen
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Stochastic  optimization  algorithms  typically  use  learning  rate  schedules  that  behave asymptotically  as  J.t(t)  = J.to/t.  The ensem(cid:173) ble dynamics  (Leen and Moody,  1993) for such algorithms provides  an easy path to results  on  mean squared weight  error and asymp(cid:173) totic  normality.  We  apply  this  approach  to  stochastic  gradient  algorithms with momentum.  We  show that  at late times,  learning  is  governed  by  an  effective learning rate  J.tejJ  = J.to/(l  - f3)  where  f3  is  the  momentum  parameter.  We  describe  the  behavior  of the  asymptotic  weight  error  and  give  conditions  on  J.tejJ  that  insure  optimal  convergence speed.  Finally,  we  use  the  results to develop  an adaptive form of momentum that achieves optimal convergence  speed  independent of J.to.
The Parti-Game Algorithm for Variable Resolution Reinforcement Learning in Multidimensional State-Spaces
Authors: Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Parti-game is  a  new  algorithm for  learning  from  delayed  rewards  in  high  dimensional  real-valued  state-spaces.  In  high  dimensions  it  is  essential  that  learning  does  not  explore  or  plan  over  state  space  uniformly.  Part i-game maintains a  decision-tree  partitioning  of state-space  and  applies  game-theory  and  computational geom(cid:173) etry  techniques  to efficiently  and reactively  concentrate high  reso(cid:173) lution only on  critical  areas.  Many  simulated  problems have  been  tested,  ranging  from  2-dimensional to  9-dimensional state-spaces,  including  mazes,  path  planning, non-linear  dynamics,  and  uncurl(cid:173) ing snake  robots  in restricted  spaces.  In  all  cases,  a  good  solution  is  found  in less  than twenty trials  and  a few  minutes. 
1  REINFORCEMENT LEARNING 
Reinforcement  learning  [Samuel,  1959,  Sutton,  1984,  Watkins,  1989,  Barto  et  al.,  1991] is a promising method for control systems to program and improve themselves.  This paper addresses  its biggest stumbling block:  the curse  of dimensionality  [Bell(cid:173) man, 1957], in which costs increase exponentially with the number of state variables. 
Some earlier work [Simons  et  al.,  1982, Moore,  1991, Chapman and Kaelbling,  1991,  Dayan  and  Hinton,  1993]  has  considered  recursively  partitioning state-space  while  learning  from  delayed  rewards.  The  new  ideas  in  the  parti-game  algorithm  in-
Stability and Observability
Authors: Fernanda Botelho Max H. Garzon
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: he theme was the effect of perturbations of the defining parameters of a neural net(cid:173)
work due to:  1)  mea"urement" (particularly with analog networks); 2)  di"cretization 
due to a)  digital implementation of analog nets; b)  bounded-precision implementa(cid:173)
tion  of digital networks;  or c)  inaccurate evaluation of the  transfer function(s};  3) 
noise in or incomplete input and/or output of the net or individual cells  (particu(cid:173)
larly with analog networks).
The Role of MT Neuron Receptive Field Surrounds in Computing Object Shape from Velocity Fields
Authors: T. D. Albright G. T. Buracas
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: The goal of this work was to investigate the role of primate  MT  neurons  in  solving  the  structure  from  motion  (SFM)  problem.  Three  types  of  receptive  field  (RF)  surrounds  found in area MT neurons (K.Tanaka et al.,1986; Allman et  al.,1985) correspond, as our analysis suggests, to the oth,  pt  and 2nd  order fuzzy  space-differential operators.  The large  surround/center  both  differentiation  of  smooth  velocity  fields  and  discontinuity  detection  at  boundaries  of  objects.  The  model  is  in  agreement  with  recent  psychophysical  data  on  surface  interpolation involvement  in  SFM.  We  suggest  that  area  MT  partially  segregates  information  about  object  shape  from  information  about  spatial  relations  necessary  for  navigation and manipulation.
Estimating analogical similarity by dot-products of Holographic Reduced Representations
Authors: Tony A. Plate
Published in: Advances in Neural Information Processing Systems 6  (NIPS 1993) (Main Conference Track), 1993

Abstract: Models of analog retrieval require a computationally cheap method of  estimating similarity between a probe and the candidates in a large pool  of memory items. The vector dot-product operation would be ideal for  this purpose if it were possible to encode complex structures as vector  representations in such a way that the superficial similarity of vector  representations reflected underlying structural similarity. This paper de(cid:173) scribes how such an encoding is provided by Holographic Reduced Rep(cid:173) resentations (HRRs), which are a method for encoding nested relational  structures as fixed-width distributed representations. The conditions un(cid:173) der which structural similarity is reflected in the dot-product rankings of  HRRs are discussed.