mtybilly/Instruct2DS huggingface dataset:
The Ni1000: High Speed Parallel VLSI for Implementing Multilayer Perceptrons
Authors: Leon N. Cooper Michael P. Perrone
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In  this paper  we  present  a  new  version  of the standard  multilayer  perceptron  (MLP) algorithm for  the state-of-the-art in neural net(cid:173) work VLSI implementations:  the Intel Ni1000.  This new version of  the  MLP  uses  a fundamental property of high dimensional spaces  which  allows  the  12-norm  to  be  accurately  approximated  by  the  It -norm.  This  approach  enables  the  standard  MLP  to  utilize  the  parallel architecture of the Ni1000 to achieve on the order of 40000,  256-dimensional classifications per second. 
1  The Intel NilOOO  VLSI Chip 
The Nestor/Intel radial basis function  neural chip (Ni1000) contains the equivalent  of  1024  256-dimensional  artificial  digital  neurons  and can  perform  at  least  40000  classifications  per  second  [Sullivan, 1993].  To attain  this  great  speed,  the  Ni1000  was  designed  to  calculate  "city  block"  distances  (Le.  the  II-norm)  and  thus  to  avoid the large  number  of multiplication units that  would be required  to calculate  Euclidean dot  products in  parallel.  Each  neuron calculates the city block distance  between  its stored weights and the current  input: 
neuron  activity = L IWi  - :eil 
(1)  where  w,  is  the  neuron's stored  weight  for  the  ith  input  and  :ei  is  the  ith  input.  Thus the Nil000 is ideally suited to perform both the RCE [Reillyet al.,  1982]  and 
748 
Michael P.  Perrone.  Leon N.  Cooper 
PRCE  [Scofield et al.,  1987]  algorithms or  any of the  other  commonly used  radial  basis function  (RBF) algorithms.  However,  dot products are central in the calcula(cid:173) tions performed by most neural network algorithms (e.g.  MLP, Cascade Correlation,  etc.).  Furthermore, for  high dimensional data, the dot product becomes the compu(cid:173) tation bottleneck (i.e.  most ofthe network's time is spent calculating dot products).  If the  dot  product  can  not  be  performed in  parallel  there  will  be  little  advantage  using  the  NilOOO  for  such  algorithms.  In  this  paper,  we  address  this  problem  by  showing  that  we  can  extend  the  NilOOO  to  many of the  standard  neural  network  algorithms by  representing  the  Euclidean  dot  product  as  a  function  of Euclidean  norms and by then using a  city block norm approximation to the  Euclidean norm.  Section  2,  introduces  the  approximate dot  productj  Section  3  describes  the  City  Block  MLP  which  uses  the  approximate dot  productj  and  Section  4  presents  ex(cid:173) periments which demonstrate that the City Block  MLP performs well on the  NIST  OCR data and on human face  recognition data. 
2  Approximate  Dot  Product 
Consider  the following approximation [Perrone,  1993]: 
1
Model of a Biological Neuron as a Temporal Neural Network
Authors: Edward W. Kairiss Sean D. Murphy
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A biological neuron can be viewed as a device that maps a multidimen(cid:173) sional  temporal  event signal  (dendritic  postsynaptic  activations)  into a  unidimensional  temporal  event  signal  (action  potentials).  We  have  designed  a  network,  the  Spatio-Temporal  Event  Mapping  (STEM)  architecture, which can learn to perform this mapping for arbitrary bio(cid:173) physical  models  of  neurons.  Such  a  network  appropriately  trained,  called a STEM cell, can be used in place of a conventional compartmen(cid:173) tal  model  in  simulations  where only the  transfer function  is  important,  such  as  network  simulations.  The  STEM  cell  offers  advantages  over  compartmental  models in  terms of computational efficiency, analytical  tractabili1ty,  and as a framework  for  VLSI  implementations of biologi(cid:173) cal neurons.
Non-linear Prediction of Acoustic Vectors Using Hierarchical Mixtures of Experts
Authors: Anthony J. Robinson Steve R. Waterhouse
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In this paper we consider speech coding as a problem of speech  modelling. In particular, prediction of parameterised speech over  short time segments is performed using the Hierarchical Mixture of  Experts (HME) (Jordan & Jacobs 1994). The HME gives two ad(cid:173) vantages over traditional non-linear function approximators such  as the Multi-Layer Percept ron (MLP); a statistical understand(cid:173) ing of the operation of the predictor and provision of information  about the performance of the predictor in the form of likelihood  information and local error bars. These two issues are examined  on both toy and real world problems of regression and time series  prediction. In the speech coding context, we extend the principle  of combining local predictions via the HME to a Vector Quantiza(cid:173) tion scheme in which fixed local codebooks are combined on-line  for each observation.
JPMAX: Learning to Recognize Moving Objects as a Model-fitting Problem
Authors: Suzanna Becker
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Unsupervised learning procedures have been successful at low-level  feature  extraction  and  preprocessing of raw  sensor  data.  So  far,  however,  they  have  had  limited  success  in  learning  higher-order  representations, e.g.,  of objects in  visual images.  A promising ap(cid:173) proach  is  to  maximize  some  measure  of  agreement  between  the  outputs of two groups of units which receive inputs physically sep(cid:173) arated in  space, time or modality,  as  in  (Becker and Hinton,  1992;  Becker, 1993; de Sa,  1993).  Using the same approach, a much sim(cid:173) pler  learning  procedure is  proposed  here  which  discovers  features  in a single-layer network consisting of several populations of units,  and  can  be  applied  to  multi-layer  networks  trained  one  layer  at  a  time.  When  trained with  this  algorithm  on  image  sequences  of  moving geometric objects a two-layer network can learn to perform  accurate position-invariant object classification. 
1  LEARNING  COHERENT CLASSIFICATIONS 
A powerful constraint in  sensory data is  coherence over time,  in  space,  and  across  different sensory modalities.  An  unsupervised learning procedure which can capital(cid:173) ize on these constraints may be able to explain much of perceptual self-organization  in  the mammalian brain.  The problem is to derive an appropriate cost function for  unsupervised  learning which  will  capture coherence constraints in  sensory  signals;  we  would  also  like  it  to  be  applicable  to  multi-layer  nets  to  train  hidden  as  well  as  output  layers.  Our  ultimate  goal  is  for  the  network  to  discover  natural object  classes  based on these coherence assumptions. 
934
The Electrotonic Transformation: a Tool for Relating Neuronal Form to Function
Authors: Kenneth Y. Tsai, Brenda J. Claiborne, Thomas H. Brown Nicholas T Carnevale
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The spatial distribution and time course of electrical signals in neurons  have important theoretical and practical consequences. Because it is  difficult to infer how neuronal form affects electrical signaling, we  have developed a quantitative yet intuitive approach to the analysis of  electrotonus. This approach transforms the architecture of the cell  from anatomical to electrotonic space, using the logarithm of voltage  attenuation as the distance metric. We describe the theory behind this  approach and illustrate its use.
Dynamic Modelling of Chaotic Time Series with Neural Networks
Authors: Jyh-Ming Kuo Jose C. Principe
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The auditory system of the barn owl contains several spatial maps.  In young barn owls raised with optical prisms over their eyes, these  auditory maps are shifted to stay in register with the visual map,  suggesting that the visual input imposes a frame of reference on  the auditory maps. However, the optic tectum, the first site of  convergence of visual with auditory information, is not the site of  plasticity for the shift of the auditory maps; the plasticity occurs  instead in the inferior colliculus, which contains an auditory map  and projects into the optic tectum. We explored a model of the owl  remapping in which a global reinforcement signal whose delivery is  controlled by visual foveation. A hebb learning rule gated by rein(cid:173) forcement learned to appropriately adjust auditory maps. In addi(cid:173) tion, reinforcement learning preferentially adjusted the weights in  the inferior colliculus, as in the owl brain, even though the weights  were allowed to change throughout the auditory system. This ob(cid:173) servation raises the possibility that the site of learning does not  have to be genetically specified, but could be determined by how  the learning procedure interacts with the network architecture. 
126 
Alexandre Pouget, Cedric Deffayet, Te"ence J. Sejnowski 
c:::======:::::» •
Learning with Product Units
Authors: C. Lee Giles, Bill G. Horne, Marwan A. Jabri Laurens R. Leerink
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The  TNM  staging  system  has  been  used  since  the  early  1960's  to  predict  breast  cancer  patient  outcome.  In  an  attempt  to  in(cid:173) crease  prognostic  accuracy,  many putative prognostic factors  have  been  identified.  Because  the  TNM  stage  model  can  not  accom(cid:173) modate  these  new  factors,  the  proliferation  of factors  in  breast  cancer  has  lead  to  clinical  confusion.  What  is  required  is  a  new  computerized  prognostic system  that  can test  putative prognostic  factors  and  integrate  the  predictive  factors  with  the  TNM  vari(cid:173) ables  in order to increase  prognostic  accuracy.  Using  the area un(cid:173) der  the curve of the receiver  operating characteristic,  we  compare  the  accuracy  of the  following  predictive  models  in  terms  of five  year  breast cancer-specific  survival:  pTNM staging system,  princi(cid:173) pal component analysis,  classification and regression  trees,  logistic  regression,  cascade  correlation neural network,  conjugate gradient  descent  neural,  probabilistic neural network,  and backpropagation  neural network.  Several statistical models are significantly more ac-
Efficient Methods for Dealing with Missing Data in Supervised Learning
Authors: Ralph Neuneier, Subutai Ahmad Volker Tresp
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We present efficient algorithms for dealing with the problem of mis(cid:173) sing inputs (incomplete feature  vectors) during training and recall.  Our approach is based on the approximation of the input data dis(cid:173) tribution using  Parzen  windows.  For  recall,  we  obtain closed form  solutions for arbitrary feedforward networks.  For training, we show  how  the  backpropagation  step  for  an  incomplete  pattern  can  be  approximated by  a  weighted  averaged backpropagation step.  The  complexity  of the  solutions for  training  and  recall  is  independent  of the number of missing features.  We verify our theoretical results  using one classification and one regression  problem.
Predictive Coding with Neural Nets: Application to Text Compression
Authors: Stefan Heil Jürgen Schmidhuber
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: To compress text files,  a neural predictor network  P  is used to ap(cid:173) proximate the conditional probability distribution of possible  "next  characters",  given  n  previous  characters.  P's outputs are fed  into  standard coding algorithms that generate short codes for characters  with  high  predicted  probability and  long  codes  for  highly  unpre(cid:173) dictable  characters.  Tested  on  short  German  newspaper  articles,  our method outperforms widely used  Lempel-Ziv algorithms (used  in  UNIX  functions such  as  "compress"  and  "gzip").
Computational Structure of coordinate transformations: A generalization study
Authors: Daniel M. Wolpert, Michael I. Jordan Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: One  of the fundamental properties  that both neural  networks  and  the  central  nervous  system share is  the  ability to learn  and gener(cid:173) alize  from examples.  While this  property  has  been  studied  exten(cid:173) sively  in  the  neural  network  literature  it  has  not  been  thoroughly  explored in human perceptual and motor learning.  We have chosen  a  coordinate  transformation  system-the  visuomotor  map  which  transforms visual coordinates into motor coordinates-to study the  generalization effects  of learning new  input-output  pairs.  Using  a  paradigm of computer  controlled  altered  visual  feedback,  we  have  studied  the  generalization  of the  visuomotor  map  subsequent  to  both local  and context-dependent  remappings.  A  local  remapping  of one  or  two  input-output  pairs  induced  a  significant  global,  yet  decaying,  change  in  the  visuomotor map, suggesting  a  representa(cid:173) tion for  the  map composed of units  with large functional  receptive  fields.  Our study of context-dependent  remappings indicated that  a  single  point  in  visual  space  can  be  mapped to  two different  fin(cid:173) ger  locations  depending  on  a  context  variable-the starting  point  of  the  movement.  Furthermore,  as  the  context  is  varied  there  is  a  gradual  shift  between  the  two  remappings,  consistent  with  two  visuomotor  modules  being  learned  and  gated  smoothly  with  the  context.
Recognizing Handwritten Digits Using Mixtures of Linear Models
Authors: Michael Revow, Peter Dayan Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We construct a mixture of locally linear generative models of a col(cid:173) lection of pixel-based images of digits, and use them for  recogni(cid:173) tion. Different models of a given digit are used to capture different  styles of writing, and new images are classified by evaluating their  log-likelihoods under each model. We use an EM-based algorithm  in which the M-step is computationally straightforward principal  components analysis (PCA).  Incorporating tangent-plane informa(cid:173) tion [12]  about expected local deformations only requires adding  tangent vectors into the sample covariance matrices for  the PCA,  and it demonstrably improves performance.
A Critical Comparison of Models for Orientation and Ocular Dominance Columns in the Striate Cortex
Authors: K. Obermayer, K. Schulten E. Erwin
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: More than ten of the most prominent models for the structure  and for the activity dependent formation of orientation and ocu(cid:173) lar dominance columns in the striate cort(>x have been evaluated.  We implemented those models on parallel machines, we extensively  explored parameter space, and we quantitatively compared model  predictions with experimental data which were recorded optically  from macaque striate cortex.  In our contribution we present a summary of our results to date.  Briefly, we find that (i) despite apparent differences, many models  are based on similar principles and, consequently, make similar pre(cid:173) dictions, (ii) certain "pattern models" as well as the developmental  "correlation-based learning" models disagree with the experimen(cid:173) tal data, and (iii) of the models we have investigated, "competitive  Hebbian" models and the recent model of Swindale provide the  best match with experimental data. 
1 Models and Data 
The models for the formation and structure of orientation and ocular dominance  columns which we have investigated are summarized in table 1. Models fall into  two categories: "Pattern models" whose aim is to achieve a concise description of  the observed patterns and "developmental models" which are focussed on the pro-
94 
E. Erwin, K. Obermayer, K. Schulten 
Class  Pattern  Models Models
Classifying with Gaussian Mixtures and Clusters
Authors: Todd K. Leen Nanda Kambhatla
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In this paper, we derive classifiers which are winner-take-all (WTA)  approximations  to  a  Bayes  classifier  with  Gaussian  mixtures  for  class conditional densities.  The derived classifiers include clustering  based algorithms like LVQ  and k-Means.  We propose a  constrained  rank  Gaussian  mixtures model and derive a WTA algorithm for  it.  Our experiments with two speech classification tasks indicate that  the constrained rank model and the WTA approximations improve  the performance over the unconstrained models.
Anatomical origin and computational role of diversity in the response properties of cortical neurons
Authors: Shimon Edelman, Rafael Malach Kalanit Grill Spector
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: The  maximization of  diversity  of neuronal  response  properties  has  been  recently  suggested  as  an  organizing  principle  for  the  formation  of such  prominent features of the functional architecture of the brain as the corti(cid:173) cal columns and the associated patchy projection patterns (Malach,  1994).  We show that (1) maximal diversity is attained when the ratio of dendritic  and  axonal  arbor  sizes  is  equal  to  one,  as  found  in  many  cortical  areas  and  across  species  (Lund  et al.,  1993;  Malach,  1994),  and  (2)  that maxi(cid:173) mization of diversity leads  to  better  performance  in  systems of receptive  fields  implementing steerable/shiftable  filters,  and  in  matching spatially  distributed  signals,  a  problem that  arises  in  many high-level  visual tasks. 
1  Anatomical substrate for  sampling diversity 
A  fundamental  feature  of  cortical  architecture  is  its  columnar  organization,  mani(cid:173) fested  in  the tendency  of neurons  with similar properties to be organized  in  columns  that run perpendicular to the cortical surface.  This organization of the cortex was ini(cid:173) tially discovered  by  physiological experiments  (Mouncastle,  1957;  Hubel  and  Wiesel,  1962),  and  subsequently  confirmed  with  the  demonstration  of histologically  defined  columns.  Tracing  experiments  have  shown  that  axonal  projections  throughout  the  cerebral  cortex tend to be organized in  vertically  aligned  clusters  or patches.  In  par(cid:173) ticular,  intrinsic  horizontal  connections  linking neighboring  cortical sites,  which  may  extend  up  to  2 - 3  mm,  have  a  striking tendency  to  arborize  selectively  in preferred  sites,  forming  distinct  axonal patches  200 - 300  J.lm  in  diameter. 
Recently,  it  has  been  observed  that  the  size  of  these  patches  matches  closely  the  average  diameter  of  individual  dendritic  arbors  of  upper-layer  pyramidal  cells 
118 
Kalanit Grill Spector,  Shimon  Edelman, Rafael Malach
Synchrony and Desynchrony in Neural Oscillator Networks
Authors: David Terman Deliang Wang
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: An novel class of locally excitatory, globally inhibitory oscillator  networks is proposed. The model of each oscillator corresponds to a  standard relaxation oscillator with two time scales. The network  exhibits a mechanism of selective gating, whereby an oscillator  jumping up to its active phase rapidly recruits the oscillators stimulated  by the same pattern, while preventing others from jumping up. We  show analytically that with the selective gating mechanism the network  rapidly achieves both synchronization within blocks of oscillators that  are stimulated by connected regions and desynchronization between  different blocks. Computer simulations demonstrate the network's  promising ability for segmenting multiple input patterns in real time.  This model lays a physical foundation for the oscillatory correlation  theory of feature binding, and may provide an effective computational  framework for scene segmentation and figure/ground segregation.
A Computational Model of Prefrontal Cortex Function
Authors: Jonathan D. Cohen, David Servan-Schreiber Todd S. Braver
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Accumulating  data  from  neurophysiology  and  neuropsychology  have  suggested  two  information processing roles  for  prefrontal cor(cid:173) tex  (PFC):  1)  short-term  active  memory;  and  2)  inhibition.  We  present  a  new  behavioral  task  and  a  computational model  which  were  developed  in  parallel.  The  task  was  developed  to probe  both  of these  prefrontal  functions  simultaneously,  and  produces  a  rich  set  of behavioral  data  that  act  as  constraints  on  the  model.  The  model is  implemented in continuous-time, thus providing a  natural  framework  in  which  to study  the  temporal dynamics of processing  in the task.  We show how the model can be used to examine the be(cid:173) havioral  consequences of neuromodulation in PFC . Specifically, we  use  the model to make novel and testable predictions regarding the  behavioral performance of schizophrenics,  who  are hypothesized  to  suffer  from  reduced  dopaminergic tone  in  this brain  area.
Combining Estimators Using Non-Constant Weighting Functions
Authors: Michiaki Taniguchi Volker Tresp
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This paper  discusses  the  linearly  weighted combination of estima(cid:173) tors  in which  the weighting functions  are  dependent  on the input .  We  show  that  the  weighting  functions  can  be  derived  either  by  evaluating  the  input  dependent  variance  of each  estimator  or  by  estimating how  likely  it is  that a given  estimator has  seen  data in  the  region  of the input space  close  to the input pattern.  The lat(cid:173) ter  solution  is  closely  related  to  the  mixture of experts  approach  and we  show how  learning rules  for  the mixture of experts  can  be  derived from the theory  about learning with missing features.  The  presented  approaches  are  modular  since  the  weighting  functions  can  easily  be  modified  (no  retraining)  if more  estimators  are  ad(cid:173) ded.  Furthermore, it is  easy  to incorporate estimators which  were  not  derived from  data such as  expert systems or  algorithms.
Stochastic Dynamics of Three-State Neural Networks
Authors: Jack D. Cowan Toru Ohira
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  here  an  analysis  of  the  stochastic  neurodynamics  of  a  neural  network  composed  of  three-state  neurons  described  by  a  master equation.  An  outer-product representation of the mas(cid:173) ter  equation  is  employed.  In  this  representation,  an  extension  of  the  analysis  from  two  to  three-state neurons is  easily  performed.  We  apply  this  formalism  with  approximation  schemes  to  a  sim(cid:173) ple three-state network and compare the results with Monte Carlo  simulations.
On the Computational Utility of Consciousness
Authors: Michael Mozer Donald W. Mathis
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  propose  a  computational  framework  for  understanding  and  modeling  human consciousness.  This  framework  integrates  many  existing theoretical perspectives,  yet is sufficiently concrete to allow  simulation experiments.  We do not attempt to explain qualia (sub(cid:173) jective  experience),  but  instead  ask  what  differences  exist  within  the cognitive information processing  system  when a  person is  con(cid:173) scious  of mentally-represented  information versus  when that infor(cid:173) mation is  unconscious.  The central idea we explore is  that the con(cid:173) tents  of consciousness  correspond  to  temporally  persistent  states  in a  network of computational modules.  Three simulations are de(cid:173) scribed  illustrating  that  the  behavior  of  persistent  states  in  the  models  corresponds  roughly  to  the  behavior  of  conscious  states  people  experience when  performing similar tasks.  Our simulations  show that periodic  settling to persistent  (i.e.,  conscious)  states im(cid:173) proves  performance  by cleaning  up inaccuracies  and noise,  forcing  decisions,  and helping  keep the system on track  toward a  solution.
Ocular Dominance and Patterned Lateral Connections in a Self-Organizing Model of the Primary Visual Cortex
Authors: Risto Miikkulainen Joseph Sirosh
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A neural network model for the self-organization of ocular dominance and  lateral connections from binocular input is presented.  The self-organizing  process results in a network where (1) afferent weights of each neuron or(cid:173) ganize into smooth hill-shaped receptive fields primarily on one of the reti(cid:173) nas, (2) neurons with common eye preference form connected, intertwined  patches, and (3) lateral connections primarily link regions of the same eye  preference.  Similar self-organization of cortical  structures has  been  ob(cid:173) served  experimentally in strabismic kittens.  The model  shows how  pat(cid:173) terned lateral connections in  the cortex may  develop based on correlated  activity and explains why lateral connection patterns follow receptive field  properties such as ocular dominance.
Effects of Noise on Convergence and Generalization in Recurrent Networks
Authors: Bill G. Horne, C. Lee Giles Kam Jim
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We introduce and study methods of inserting synaptic noise into  dynamically-driven recurrent neural networks and show that ap(cid:173) plying a controlled amount of noise during training may improve  convergence and generalization. In addition, we analyze the effects  of each noise parameter (additive vs. multiplicative, cumulative vs.  non-cumulative, per time step vs. per string) and predict that best  overall performance can be achieved by injecting additive noise at  each time step. Extensive simulations on learning the dual parity  grammar from temporal strings substantiate these predictions.
An Integrated Architecture of Adaptive Neural Network Control for Dynamic Systems
Authors: Robert L. Tokar, Brain D. McVey Ke Liu
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In  this  study,  an  integrated  neural  network  control  architecture  for  nonlinear  dynamic  systems  is  presented.  Most of the recent emphasis in  the neural network control field has no error feedback as the  control  input,  which  rises  the  lack  of adaptation  problem.  The  integrated  architecture  in  this  paper  combines  feed  forward  control  and error  feedback  adaptive  control  using  neural  networks.  The  paper  reveals the  different  internal  functionality  of these  two  kinds of neural  network  controllers  for  certain  input  styles,  e.g.,  state  feedback  and error feedback.  With  error  feedback,  neural  network  controllers  learn  the  slopes  or  the  gains  with  respect  to  the  error  feedback,  producing  an  error  driven  adaptive  control  systems.  The  results  demonstrate  that  the  two  kinds  of control  scheme  can  be  combined  to  realize  their individual  advantages.  Testing  with disturbances added to  the  plant  shows  good  tracking  and adaptation with the integrated neural control architecture.
Implementation of Neural Hardware with the Neural VLSI of URAN in Applications with Reduced Representations
Authors: Ki-Chul Kim, Hwang-Soo Lee Il Song Han
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: implement  Korean 
This  paper  describes  a  way  of  neural  hardware  implementation  with  the  analog-digital  mixed  mode  neural  chip.  The  full  custom  neural  VLSI  of  Universally  Reconstructible  Artificial  Neural  network (URAN)  is  used  system.  A  to  multi-layer  perceptron  with  is  trained  successfully  under  the  limited  accuracy  in  computations.  The  network  with  a  large  frame  input  layer  is  tested  to  recognize  spoken  korean  words  at  a  forward  retrieval.  Multichip  hardware  module  is  suggested  with  eight  chips  or  more  for  the  extended  performance  and  capacity.
Estimating Conditional Probability Densities for Periodic Variables
Authors: Claire Legleye Chris M. Bishop
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Most of the common techniques for estimating conditional prob(cid:173) ability densities are inappropriate for applications involving peri(cid:173) odic variables. In this paper we introduce three novel techniques  for tackling such problems, and investigate their performance us(cid:173) ing synthetic data. We then apply these techniques to the problem  of extracting the distribution of wind vector directions from radar  scatterometer data gathered by a remote-sensing satellite.
Analysis of Unstandardized Contributions in Cross Connected Networks
Authors: Yuriko Oshima-Takane, Yoshio Takane Thomas R. Shultz
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Understanding  knowledge  representations  in  neural  nets  has  been  a  difficult  problem.  Principal  components  analysis  (PCA)  of  contributions (products of sending activations and connection weights)  has yielded valuable insights into knowledge representations, but much  of this work has focused on the correlation matrix of contributions. The  present work  shows  that analyzing  the  variance-covariance matrix  of  contributions yields more valid  insights by taking account of weights.
A Rigorous Analysis of Linsker-type Hebbian Learning
Authors: H. Pan, V. P. Roychowdhury J. Feng
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We propose a novel rigorous approach for the analysis of Linsker's  unsupervised Hebbian learning network. The behavior of this  model is determined by the underlying nonlinear dynamics which  are parameterized by a set of parameters originating from the Heb(cid:173) bian rule and the arbor density of the synapses. These parameters  determine the presence or absence of a specific receptive field (also  referred to as a 'connection pattern') as a saturated fixed point  attractor of the model. In this paper, we perform a qualitative  analysis of the underlying nonlinear dynamics over the parameter  space, determine the effects of the system parameters on the emer(cid:173) gence of various receptive fields, and predict precisely within which  parameter regime the network will have the potential to develop  a specially designated connection pattern. In particular, this ap(cid:173) proach exposes, for the first time, the crucial role played by the  synaptic density functions, and provides a complete precise picture  of the parameter space that defines the relationships among the  different receptive fields. Our theoretical predictions are confirmed  by numerical simulations. 
320 
lian/eng Feng, H. Pan, V. P. Roychowdhury
Associative Decorrelation Dynamics: A Theory of Self-Organization and Optimization in Feedback Networks
Authors: Dawei W. Dong
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper  outlines  a  dynamic  theory  of  development  and  adap(cid:173) tation  in  neural  networks  with  feedback  connections.  Given  in(cid:173) put ensemble,  the  connections  change in strength according  to  an  associative  learning  rule  and  approach  a  stable  state  where  the  neuronal  outputs  are  decorrelated .  We  apply  this  theory  to  pri(cid:173) mary  visual  cortex and  examine  the implications of the  dynamical  decorrelation  of the  activities  of orientation  selective  cells  by  the  intracortical connections.  The theory gives a  unified  and quantita(cid:173) tive explanation of the  psychophysical experiments on  orientation  contrast and orientation adaptation.  Using only one parameter, we  achieve  good  agreements  between  the  theoretical  predictions  and  the experimental data.
Visual Speech Recognition with Stochastic Networks
Authors: Javier R. Movellan
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: This  paper presents  ongoing  work  on  a  speaker  independent  visual  speech recognition system. The work presented here builds on  previous  research  efforts  in  this  area  and  explores  the  potential  use  of simple  hidden  Markov  models  for  limited  vocabulary,  speaker  independent  visual  speech recognition.  The  task  at  hand  is  recognition  of the  first  four  English  digits,  a  task  with  possible  applications  in  car-phone  images  were  modeled  as  mixtures  of  independent  dialing.  The  Gaussian  distributions,  and  the  temporal  dependencies  were  captured  with standard left-to-right hidden Markov  models.  The  results  indicate  that  simple  hidden  Markov  models  may  be  used  to  successfully  recognize relatively unprocessed image sequences. The system  achieved  performance  levels  equivalent  to  untrained  humans  when  asked  to  recognize the fIrst four English digits.
Finding Structure in Reinforcement Learning
Authors: Anton Schwartz Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Reinforcement learning addresses  the problem of learning to select actions in order to  maximize one's performance in unknown environments.  To scale reinforcement learning  to complex real-world tasks, such as typically studied in AI, one must ultimately be able  to discover the structure in the world, in order to abstract away the myriad of details and  to operate in more tractable problem spaces.  This paper presents the SKILLS algorithm.  SKILLS discovers skills, which are partially  defined action policies that arise in the context of multiple, related tasks.  Skills collapse  whole action sequences into single operators.  They are learned by minimizing the com(cid:173) pactness of action policies, using a description length argument on their representation.  Empirical results  in  simple grid  navigation tasks  illustrate the successful  discovery  of  structure in reinforcement learning.
From Data Distributions to Regularization in Invariant Learning
Authors: Todd K. Leen
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Ideally pattern recognition machines provide constant output when  the inputs are transformed under a group 9 of desired invariances.  These invariances can be achieved  by  enhancing the training data  to include examples of inputs transformed by elements of g,  while  leaving  the  corresponding  targets  unchanged.  Alternatively  the  cost  function  for  training  can  include  a  regularization  term  that  penalizes changes in the output when the input is  transformed un(cid:173) der the group. 
This paper relates the two approaches, showing precisely the sense  in  which  the regularized  cost function  approximates the result  of  adding  transformed (or distorted)  examples  to  the  training data.  The cost function for the enhanced training set is equivalent to the  sum of the original  cost function plus  a  regularizer.  For  unbiased  models,  the regularizer reduces  to the intuitively obvious choice - a  term that penalizes changes  in  the output when  the inputs are  transformed under  the  group.  For  infinitesimal  transformations,  the coefficient of the regularization term reduces to the variance of  the distortions introduced into the training data.  This correspon(cid:173) dence provides a simple bridge between the two  approaches.
An Input Output HMM Architecture
Authors: Paolo Frasconi Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  introduce  a  recurrent  architecture  having  a  modular structure  and we formulate a training procedure based on the EM  algorithm.  The resulting model has similarities to hidden  Markov models, but  supports  recurrent  networks  processing style and allows  to exploit  the supervised  learning paradigm while using maximum likelihood  estimation.
Grouping Components of Three-Dimensional Moving Objects in Area MST of Visual Cortex
Authors: Terrence J. Sejnowski Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: Many  cells  in  the  dorsal  part  of  the  medial  superior  temporal  (MST)  area  of  visual  cortex  respond  selectively  to  spiral  flow  patterns-specific  combinations of expansion/ contraction  and  ro(cid:173) tation  motions.  Previous  investigators  have  suggested  that  these  cells  may represent  self-motion.  Spiral patterns  can  also  be  gener(cid:173) ated by the relative motion of the observer  and a particular object.  An  MST  cell  may  then  account  for  some  portion  of the  complex  flow  field,  and  the  set  of active  cells  could  encode  the  entire  flow;  in  this  manner,  MST  effectively  segments  moving  objects.  Such  a  grouping operation  is  essential  in  interpreting  scenes  containing  several  independent  moving objects  and  observer  motion.  We  de(cid:173) scribe  a  model  based  on  the  hypothesis  that  the  selective  tuning  of MST  cells  reflects  the grouping of object  components  undergo(cid:173) ing  coherent  motion.  Inputs  to  the  model  were  generated  from  sequences  of ray-traced  images that simulated realistic  motion sit(cid:173) uations,  combining observer  motion, eye  movements, and indepen(cid:173) dent  object  motion.  The  input  representation  was  modeled  after  response  properties of neurons in area MT,  which provides the pri(cid:173) mary input to area MST.  After  applying an  unsupervised  learning  algorithm,  the  units  became  tuned  to  patterns  signaling  coherent  motion.  The results  match many of the  known  properties  of MST  cells  and  are  consistent  with  recent  studies  indicating  that  these  cells  process  3-D  object  motion information. 
166 
Richard S.  Zemel,  Terrence  J.  Sejnowski
Higher Order Statistical Decorrelation without Information Loss
Authors: Wilfried Brauer Gustavo Deco
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A neural network learning paradigm based on information theory is pro(cid:173) posed as a way to perform in an unsupervised fashion, redundancy  reduction among the elements of the output layer without loss of infor(cid:173) mation from the sensory input. The model developed performs nonlin(cid:173) ear decorrelation up to higher orders of the cumulant tensors and results  in probabilistic ally independent components of the output layer. This  means that we don't need to assume Gaussian distribution neither at the  input nor at the output. The theory presented is related to the unsuper(cid:173) vised-learning theory of Barlow, which proposes redundancy reduction  as the goal of cognition. When nonlinear units are used nonlinear princi(cid:173) pal component analysis is obtained. In this case nonlinear manifolds can  be reduced to minimum dimension manifolds. If such units are used the  network performs a generalized principal component analysis in the  sense that non-Gaussian distributions can be linearly decorrelated and  higher orders of the correlation tensors are also taken into account. The  basic structure of the architecture involves a general transfOlmation that  is volume conserving and therefore the entropy, yielding a map without  loss of infoIIDation. Minimization of the mutual infoIIDation among the  output neurons eliminates the redundancy between the outputs and  results in statistical decorrelation of the extracted features. This is  known as factorialleaming. 
248 
Gustavo Deco, Wilfried Brauer
Sample Size Requirements for Feedforward Neural Networks
Authors: Terrence L. Fine Michael J. Turmon
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We estimate the number of training samples required to ensure that  the performance of a neural network on its training data matches  that obtained when fresh data is applied to the network. Existing  estimates are higher by orders of magnitude than practice indicates.  This work seeks to narrow the gap between theory and practice by  transforming the problem into determining the distribution of the  supremum of a random field in the space of weight vectors, which  in turn is attacked by application of a recent technique called the  Poisson clumping heuristic. 
1 
INTRODUCTION AND KNOWN RESULTS 
We investigate the tradeofi"s among network complexity, training set size, and sta(cid:173) tistical performance of feedforward neural networks so as to allow a reasoned choice  of network architecture in the face of limited training data. Nets are functions  7](x; w), parameterized by their weight vector w E W ~ Rd , which take as input  points x E Rk. For classifiers, network output is restricted to {a, 1} while for fore(cid:173) casting it may be any real number. The architecture of all nets under consideration  is N, whose complexity may be gauged by its Vapnik-Chervonenkis (VC) dimension  v, the size of the largest set of inputs the architecture can classify in any desired way  ('shatter'). Nets 7] EN are chosen on the basis of a training set T = {(Xi, YiHr=l.  These n samples are i.i.d. according to an unknown probability law P. Performance  of a network is measured by the mean-squared error
Generalisation in Feedforward Networks
Authors: Herman L. Ferrá Adam Kowalczyk
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  discuss  a  model  of consistent  learning  with  an  additional  re(cid:173) striction  on  the  probability  distribution  of training  samples,  the  target concept  and hypothesis class.  We  show  that the model pro(cid:173) vides  a  significant  improvement  on  the  upper  bounds  of sample  complexity,  i.e.  the  minimal number  of random  training samples  allowing  a  selection  of the  hypothesis  with  a  predefined  accuracy  and  confidence.  Further,  we  show  that  the  model  has  the  poten(cid:173) tial  for  providing  a  finite  sample  complexity  even  in  the  case  of  infinite  VC-dimension  as  well  as  for  a  sample  complexity  below  VC-dimension.  This is  achieved  by  linking sample  complexity  to  an  "average"  number  of implement able dichotomies of a  training  sample  rather  than  the  maximal  size  of a  shattered  sample,  i.e.  VC-dimension.
The Use of Dynamic Writing Information in a Connectionist On-Line Cursive Handwriting Recognition System
Authors: Michael Finke, Alex Waibel Stefan Manke
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: In this paper we present NPen ++, a connectionist system for  writer independent, large vocabulary on-line cursive handwriting  recognition. This system combines a robust input representation,  which preserves the dynamic writing information, with a neural  network architecture, a so called Multi-State Time Delay Neural  Network (MS-TDNN), which integrates rec.ognition and segmen(cid:173) tation in a single framework. Our preprocessing transforms the  original coordinate sequence into a (still temporal) sequence offea(cid:173) ture vectors, which combine strictly local features, like curvature  or writing direction, with a bitmap-like representation of the co(cid:173) ordinate's proximity. The MS-TDNN architecture is well suited  for handling temporal sequences as provided by this input rep(cid:173) resentation. Our system is tested both on writer dependent and  writer independent tasks with vocabulary sizes ranging from 400  up to 20,000 words. For example, on a 20,000 word vocabulary we  achieve word recognition rates up to 88.9% (writer dependent) and  84.1 % (writer independent) without using any language models.
Capacity and Information Efficiency of a Brain-like Associative Net
Authors: David Willshaw Bruce Graham
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We have determined the capacity and information efficiency of an  associative net configured in a brain-like way with partial connec(cid:173) tivity and noisy input cues. Recall theory was used to calculate  the capacity when pattern recall is achieved using a winners-take(cid:173) all strategy. Transforming the dendritic sum according to input  activity and unit usage can greatly increase the capacity of the  associative net under these conditions. For moderately sparse pat(cid:173) terns, maximum information efficiency is achieved with very low  connectivity levels (~ 10%). This corresponds to the level of con(cid:173) nectivity commonly seen in the brain and invites speculation that  the brain is connected in the most information efficient way.
SARDNET: A Self-Organizing Feature Map for Sequences
Authors: Risto Miikkulainen Daniel L. James
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: A  self-organizing  neural  network  for  sequence  classification  called  SARDNET is described  and analyzed experimentally.  SARDNET  extends the Kohonen  Feature Map architecture  with activation re(cid:173) tention  and  decay  in  order  to  create  unique  distributed  response  patterns for different sequences.  SARDNET yields extremely dense  yet descriptive representations of sequential input in very few  train(cid:173) ing  iterations.  The  network  has  proven  successful  on  mapping ar(cid:173) bitrary sequences  of binary and real numbers,  as well  as  phonemic  representations  of  English  words.  Potential  applications  include  isolated  spoken  word  recognition  and  cognitive  science  models  of  sequence  processing.
Deterministic Annealing Variant of the EM Algorithm
Authors: Ryohei Nakano Naonori Ueda
Published in: Advances in Neural Information Processing Systems 7  (NIPS 1994) (Main Conference Track), 1994

Abstract: We  present  a  deterministic annealing variant of the EM  algorithm  for  maximum likelihood  parameter  estimation  problems.  In  our  approach,  the  EM  process  is  reformulated  as the  problem of min(cid:173) imizing  the  thermodynamic free  energy  by  using  the  principle  of  maximum  entropy and  statistical mechanics  analogy.  Unlike simu(cid:173) lated  annealing approaches,  this  minimization is  deterministically  performed.  Moreover,  the  derived  algorithm,  unlike  the  conven(cid:173) tional EM  algorithm, can obtain better estimates free of the initial  parameter values.