mtybilly/Instruct2DS huggingface dataset:
Adaptive Mixture of Probabilistic Transducers
Authors: Yoram Singer
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  introduce and  analyze  a  mixture  model  for  supervised  learning  of  probabilistic transducers.  We  devise an  online learning algorithm  that  efficiently infers the structure and estimates the parameters of each model  in the mixture. Theoretical analysis and comparative simulations indicate  that the learning algorithm tracks the best model from an arbitrarily large  (possibly infinite) pool of models.  We also present an application of the  model for inducing a noun phrase recognizer.
SEEMORE: A View-Based Approach to 3-D Object Recognition Using Multiple Visual Cues
Authors: Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  neurally-inspired  visual  object  recognition  system  is  described  called  SEEMORE,  whose  goal  is  to  identify  common objects  from  a  large  known  set-independent  of  3-D  viewiag  angle,  distance,  and  non-rigid  distortion.  SEEMORE's  database  consists  of 100  ob(cid:173) jects  that  are  rigid  (shovel),  non-rigid  (telephone  cord),  articu(cid:173) lated  (book), statistical (shrubbery),  and complex (photographs of  scenes).  Recognition results  were  obtained using a  set of 102 color  and shape feature channels within a simple feedforward network ar(cid:173) chitecture.  In  response  to  a  test  set  of 600  novel  test  views  (6  of  each object)  presented individually in color video images, SEEMORE  identified  the  object correctly  97% of the time (chance is 1%)  using  a  nearest  neighbor  classifier.  Similar levels  of performance  were  obtained for  the  subset  of 15  non-rigid objects.  Generalization be(cid:173) havior reveals emergence  of striking natural category structure  not  explicit in  the  input feature  dimensions.
Using Pairs of Data-Points to Define Splits for Decision Trees
Authors: Michael Revow Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Conventional binary classification trees  such  as  CART either split  the data using axis-aligned hyperplanes or  they perform a  compu(cid:173) tationally expensive search  in the continuous space of hyperplanes  with unrestricted orientations.  We show that the limitations of the  former  can  be overcome  without resorting to the latter.  For every  pair of training data-points, there is one hyperplane that is orthog(cid:173) onal  to the line joining the data-points and  bisects  this line.  Such  hyperplanes  are  plausible  candidates  for  splits.  In  a  comparison  on  a suite of 12  datasets  we  found  that this  method of generating  candidate splits outperformed  the standard  methods,  particularly  when  the training sets were  small.
Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks
Authors: Sara A. Solla David Saad
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  consider  the  problem  of on-line  gradient  descent  learning for  general  two-layer  neural  networks.  An  analytic  solution  is  pre(cid:173) sented  and used  to investigate the  role of the learning  rate in con(cid:173) trolling the evolution and  convergence  of the learning process. 
Learning in layered neural networks refers  to the modification of internal parameters  {J} which specify the strength of the interneuron couplings, so  as  to bring the map  fJ  implemented  by  the  network  as  close  as  possible  to  a  desired  map  1.  The  degree  of success  is  monitored  through  the  generalization  error,  a  measure  of the  dissimilarity between  fJ  and 1.  Consider  maps from  an  N-dimensional  input  space e onto  a  scalar  (,  as  arise  in  the formulation of classification  and  regression  tasks.  Two-layer  networks  with  an  arbitrary  number of hidden  units  have  been  shown  to  be  universal  approximators  [1]  for  such  N-to-one  dimensional  maps.  Information about  the  desired  map i  is  provided  through  independent  examples (e, (1'),  with  (I'  =  i(e) for  all  p .  The  examples are used  to train  a student  network with  N  input units,  K  hidden  units,  and  a  single  linear  output  unit;  the  target  map  i  is  defined  through  a  teacher  network  of  similar  architecture  except  for  the  number  M  of hidden  units.  We  investigate  the  emergence  of generalization  ability  in  an  on-line  learning scenario  [2],  in  which  the  couplings  are  modified  after  the  presentation  of each  example so  as  to minimize the corresponding error.  The resulting changes in {J} are described  as  a  dynamical evolution; the  number of examples plays the role  of time.  In  this  paper  we  limit our  discussion  to  the  case  of the  soft-committee  machine  [2],  in  which  all  the  hidden  units  are  connected  to  the  output  unit  with  positive  couplings  of unit  strength,  and  only  the  input-to-hidden  couplings  are  adaptive. 
*D.Saad@aston.ac.uk  tOn leave  from  AT&T  Bell  Laboratories,  Holmdel,  NJ  07733,  USA 
Dynamics of On-line Gradient Descent Learning for Multilayer Neural Networks 
303 
Consider  the  student  network:  hidden  unit  i  receives  information from  input  unit  r  through  the weight  hr, and  its activation under  presentation of an input pattern  ~ =  (6,· .. ,~N) is  Xi  =  J i  .~,  with  J i  =  (hl, ... ,JiN)  defined  as  the  vector  of  incoming weights  onto the  i-th  hidden  unit.  The output of the student  network  is  a(J,~) =  L:~l  9  (Ji . ~), where  9  is  the  activation function  of the  hidden  units,  taken here  to be the error function g(x) ==  erf(x/V2), and J  ==  {Jdl
Geometry of Early Stopping in Linear Networks
Authors: Robert H. Dodier
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A theory of early stopping as applied to linear models is presented.  The  backpropagation  learning  algorithm  is  modeled  as  gradient  descent  in  continuous time.  Given  a  training set and  a  validation  set,  all  weight  vectors  found  by  early  stopping must  lie  on  a  cer(cid:173) tain quadric surface, usually an ellipsoid.  Given a training set and  a  candidate early stopping weight  vector,  all  validation  sets  have  least-squares weights lying on a certain plane.  This latter fact  can  be exploited  to estimate  the  probability  of stopping  at any  given  point along the trajectory from the initial weight vector to the least(cid:173) squares weights  derived from  the training set, and to estimate the  probability  that  training  goes  on  indefinitely.  The  prospects  for  extending this theory to nonlinear models are discussed.
A Unified Learning Scheme: Bayesian-Kullback Ying-Yang Machine
Authors: Lei Xu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  Bayesian-Kullback  learning scheme,  called Ying-Yang  Machine,  is  proposed  based on  the  two  complement but equivalent Bayesian  representations  for  joint  density  and  their  Kullback  divergence.  Not  only  the  scheme  unifies  existing  major supervised  and  unsu(cid:173) pervised  learnings,  including  the  classical  maximum likelihood  or  least  square learning,  the  maximum information preservation,  the  EM  & em algorithm and information geometry, the recent  popular  Helmholtz  machine,  as  well  as  other  learning  methods  with  new  variants  and  new  results;  but  also  the  scheme  provides  a  number  of new  learning models.
Using Feedforward Neural Networks to Monitor Alertness from Changes in EEG Correlation and Coherence
Authors: Tzyy-Ping Jung, Terrence J. Sejnowski Scott Makeig
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  report  here  that  changes  in  the  normalized  electroencephalo(cid:173) graphic  (EEG)  cross-spectrum  can  be  used  in  conjunction  with  feedforward  neural networks to monitor changes in alertness of op(cid:173) erators  continuously  and  in  near-real  time.  Previously,  we  have  shown  that EEG spectral  amplitudes covary with changes in alert(cid:173) ness  as indexed by changes in behavioral error rate on an auditory  detection task [6,4].  Here, we report for the first time that increases  in the frequency  of detection errors  in  this task are  also  accompa(cid:173) nied  by  patterns of increased  and  decreased  spectral  coherence  in  several  frequency  bands  and  EEG  channel  pairs.  Relationships  between  EEG  coherence  and  performance vary  between  subjects,  but within subjects,  their topographic and spectral profiles appear  stable  from  session  to  session.  Changes  in  alertness  also  covary  with  changes  in  correlations  among  EEG  waveforms  recorded  at  different  scalp  sites,  and  neural  networks  can  also  estimate  alert(cid:173) ness  from  correlation  changes  in  spontaneous  and  unobtrusively(cid:173) recorded  EEG signals.
A Dynamical Model of Context Dependencies for the Vestibulo-Ocular Reflex
Authors: Terrence J. Sejnowski Olivier J. M. D. Coenen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The vestibulo-ocular reflex  (VOR)  stabilizes  images  on  the  retina during  rapid  head motions.  The gain of the VOR (the ratio of eye to head rotation velocity)  is  typically around -1  when the eyes are focused on a distant target.  However, to  stabilize images accurately, the VOR gain must vary  with context (eye position,  eye vergence and head  translation).  We  first  describe  a  kinematic  model of the  VOR which relies solely on sensory information available from the semicircular  canals (head rotation), the otoliths (head translation), and neural correlates of eye  position and vergence angle.  We then propose a dynamical model and compare it  to the eye velocity responses measured in monkeys. The dynamical model repro(cid:173) duces the observed amplitude and time course of the modulation of the VOR and  suggests one way to combine the required neural signals within the cerebellum and  the brain stem.  It also makes predictions for the responses of neurons to multiple  inputs (head rotation and translation, eye position, etc.) in the oculomotor system.
Constructive Algorithms for Hierarchical Mixtures of Experts
Authors: Anthony J. Robinson Steve R. Waterhouse
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  present  two  additions  to  the  hierarchical  mixture  of experts  (HME)  architecture.  By  applying a  likelihood splitting criteria to  each expert in the HME we  "grow" the tree adaptively during train(cid:173) ing.  Secondly,  by considering only the most probable path through  the tree we  may "prune"  branches away, either temporarily, or per(cid:173) manently  if they  become  redundant.  We  demonstrate  results  for  the  growing  and  path  pruning  algorithms  which  show  significant  speed  ups  and  more efficient  use  of parameters over  the standard  fixed  structure  in  discriminating  between  two  interlocking spirals  and classifying 8-bit parity patterns.
Discovering Structure in Continuous Variables Using Bayesian Networks
Authors: Volker Tresp Reimar Hofmann
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  study  Bayesian  networks  for  continuous  variables  using  non(cid:173) linear  conditional  density  estimators.  We  demonstrate  that  use(cid:173) ful  structures  can  be extracted  from  a  data set  in  a  self-organized  way and we  present sampling techniques for  belief update based on  Markov  blanket conditional density models.
Temporal coding in the sub-millisecond range: Model of barn owl auditory pathway
Authors: Wulfram Gerstner, J. Leo van Hemmen, Hermann Wagner Richard Kempter
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Binaural  coincidence  detection  is  essential  for  the  localization  of  external  sounds  and  requires  auditory signal  processing  with  high  temporal precision.  We present an integrate-and-fire model of spike  processing in the auditory pathway of the barn owl.  It is shown that  a temporal precision in the microsecond range can be achieved with  neuronal  time constants  which  are  at least  one  magnitude longer.  An  important  feature  of our  model  is  an  unsupervised  Hebbian  learning rule which leads to a  temporal fine  tuning of the neuronal  connections. 
·email:  kempter.wgerst.lvh@physik.tu-muenchen.de 
Temporal Coding in  the Submillisecond Range:  Model of Bam Owl Auditory Pathway
Optimizing Cortical Mappings
Authors: Steven Finch, Terrence J. Sejnowski Geoffrey J. Goodhill
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: "Topographic"  mappings occur frequently  in the brain.  A  pop(cid:173) ular approach to understanding the structure of such mappings  is  to map points representing input features  in a  space of a  few  dimensions  to points in a  2 dimensional space using some self(cid:173) organizing  algorithm.  We  argue  that a  more general  approach  may be useful,  where similarities between features  are  not con(cid:173) strained to be geometric distances, and the objective function for  topographic matching is chosen explicitly rather than being spec(cid:173) ified implicitly by the self-organizing algorithm.  We  investigate  analytically an example of this more general approach applied to  the  structure of interdigitated mappings,  such as the pattern of  ocular dominance columns in primary visual cortex.
Experiments with Neural Networks for Real Time Implementation of Control
Authors: Michael Dale, Herman L. Ferrá, Adam Kowalczyk Peter K. Campbell
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper  describes  a  neural  network  based  controller  for  allocating  capacity in a telecommunications network.  This system was proposed in  order  to  overcome  a  "real  time"  response  constraint.  Two  basic  architectures are evaluated:  1) a feedforward network-heuristic and; 2) a  feedforward  network-recurrent  network.  These  architectures  are  compared against a linear programming (LP) optimiser as  a benchmark.  This LP optimiser was  also used as  a teacher to label  the  data samples  for  the feedforward  neural  network training  algorithm.  It is  found  that  the systems  are  able to provide a traffic  throughput  of 99%  and  95%,  respectively,  of  the  throughput  obtained  by  the  linear  programming  solution. Once trained, the neural network based solutions are found in a  fraction of the time required by the LP optimiser.
Cholinergic suppression of transmission may allow combined associative memory function and self-organization in the neocortex
Authors: Milos Cekic Michael E. Hasselmo
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Selective  suppression  of  transmission  at  feedback  synapses  during  learning  is  proposed  as  a  mechanism  for  combining  associative feed(cid:173) back  with  self-organization  of feed forward  synapses.  Experimental  data  demonstrates  cholinergic  suppression  of synaptic  transmission  in  layer I (feedback synapses), and a lack of suppression in layer IV (feed(cid:173) forward synapses).  A network with this feature uses local rules to learn  mappings  which  are not  linearly  separable.  During  learning,  sensory  stimuli  and  desired  response  are  simultaneously  presented  as  input.  Feedforward connections  form  self-organized representations  of input,  while suppressed feedback connections learn  the  transpose of feedfor(cid:173) ward connectivity. During recall, suppression is removed, sensory input  activates  the  self-organized  representation,  and  activity  generates  the  learned response.
Dynamics of Attention as Near Saddle-Node Bifurcation Behavior
Authors: Kenji Doya Hiroyuki Nakahara
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  consideration  of attention  as  a  means  for  goal-directed  behav(cid:173) ior in non-stationary environments,  we  argue that the dynamics of  attention  should  satisfy  two  opposing  demands:  long-term  main(cid:173) tenance  and  quick  transition.  These  two  characteristics  are  con(cid:173) tradictory  within  the linear domain.  We  propose  the near saddle(cid:173) node  bifurcation behavior of a  sigmoidal unit with self-connection  as  a  candidate of dynamical mechanism that satisfies both of these  demands.  We  further  show  in  simulations  of  the  'bug-eat-food'  tasks  that  the  near  saddle-node  bifurcation  behavior  of recurrent  networks  can  emerge  as  a functional  property  for  survival in  non(cid:173) stationary environments.
Softassign versus Softmax: Benchmarks in Combinatorial Optimization
Authors: Anand Rangarajan Steven Gold
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  new  technique,  termed  soft assign,  is  applied  for  the  first  time  to  two  classic  combinatorial  optimization  problems,  the  travel(cid:173) ing  salesman  problem  and  graph  partitioning.  Soft assign ,  which  has emerged from  the recurrent  neural  network/statistical physics  framework, enforces  two-way  (assignment) constraints without the  use  of penalty  terms  in  the  energy  functions.  The  soft assign  can  also  be  generalized  from  two-way  winner-take-all  constraints  to  multiple membership constraints which are required for graph par(cid:173) titioning.  The  soft assign  technique  is  compared  to  the  softmax  (Potts  glass).  Within  the  statistical  physics  framework,  softmax  and a penalty term has been a widely used method for enforcing the  two-way constraints common within many combinatorial optimiza(cid:173) tion  problems.  The  benchmarks  present  evidence  that  soft assign  has clear advantages in accuracy, speed,  parallelizabilityand algo(cid:173) rithmic simplicity over softmax and a penalty term in optimization  problems with two-way constraints.
From Isolation to Cooperation: An Alternative View of a System of Experts
Authors: Christopher G. Atkeson Stefan Schaal
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We introduce a constructive,  incremental learning system for regression  problems that models data by means of locally linear experts.  In contrast  to  other approaches,  the  experts  are  trained  independently  and  do  not  compete for data during learning.  Only when a prediction for  a query  is  required  do  the  experts  cooperate  by  blending  their  individual  predic(cid:173) tions.  Each expert is trained by  minimizing  a penalized local cross vali(cid:173) dation error using second order methods. In this way, an expert is able to  find a local distance metric by adjusting the size and  shape of the recep(cid:173) tive field in which its predictions are valid, and also to detect relevant in(cid:173) put features  by  adjusting  its bias  on  the  importance of individual input  dimensions. We derive asymptotic results for our method. In a variety of  simulations the properties of the algorithm are demonstrated with respect  to  interference,  learning  speed,  prediction  accuracy,  feature  detection,  and task oriented incremental learning.
Fast Learning by Bounding Likelihoods in Sigmoid Type Belief Networks
Authors: Lawrence K. Saul, Michael I. Jordan Tommi Jaakkola
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Sigmoid  type  belief networks,  a  class  of probabilistic  neural  net(cid:173) works,  provide  a  natural  framework  for  compactly  representing  probabilistic  information  in  a  variety  of unsupervised  and super(cid:173) vised  learning  problems.  Often  the  parameters used  in  these  net(cid:173) works  need  to  be  learned  from examples.  Unfortunately,  estimat(cid:173) ing  the  parameters  via  exact  probabilistic  calculations  (i.e,  the  EM-algorithm)  is  intractable  even  for  networks  with  fairly  small  numbers of hidden  units.  We  propose  to  avoid the  infeasibility of  the  E step  by  bounding likelihoods instead of computing them ex(cid:173) actly.  We  introduce extended  and  complementary representations  for  these  networks  and  show  that  the  estimation  of the  network  parameters  can  be  made fast  (reduced  to  quadratic optimization)  by  performing the estimation in either of the  alternative domains.  The  complementary networks  can  be  used  for  continuous  density  estimation as  well.
Neural Control for Nonlinear Dynamic Systems
Authors: Anuradha M. Annaswamy Ssu-Hsin Yu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A neural network based approach is presented for controlling two distinct  types of nonlinear systems.  The first  corresponds to  nonlinear systems  with  parametric  uncertainties  where  the  parameters occur  nonlinearly.  The second corresponds to  systems  for  which  stabilizing control  struc(cid:173) tures cannot be determined.  The proposed neural controllers are shown  to result in closed-loop system stability under certain conditions.
Sample Complexity for Learning Recurrent Perceptron Mappings
Authors: Eduardo D. Sontag Bhaskar DasGupta
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Recurrent  perceptron  classifiers  generalize the classical  perceptron  model.  They take into account  those correlations and dependences  among input  coordinates  which  arise  from  linear  digital  filtering.  This paper provides tight bounds on sample complexity associated  to the fitting of such  models to experimental data.
Is Learning The n-th Thing Any Easier Than Learning The First?
Authors: Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This paper investigates learning in  a lifelong context.  Lifelong learning  addresses  situations  in  which  a  learner  faces  a  whole  stream  of learn(cid:173) ing tasks.  Such scenarios provide the opportunity to transfer knowledge  across multiple learning tasks, in order to generalize more accurately from  less  training data.  In  this paper,  several  different approaches  to lifelong  learning  are  described,  and  applied in  an  object recognition domain.  It  is  shown  that  across  the  board,  lifelong learning  approaches  generalize  consistently  more  accurately  from  less  training data,  by  their  ability  to  transfer knowledge across learning tasks.
Parallel analog VLSI architectures for computation of heading direction and time-to-contact
Authors: Jörg Kramer, Christof Koch Giacomo Indiveri
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  describe  two  parallel analog VLSI  architectures  that integrate  optical flow  data obtained from  arrays of elementary velocity  sen(cid:173) sors to estimate heading direction and time-to-contact.  For heading  direction  computation,  we  performed  simulations to  evaluate  the  most important qualitative properties of the optical flow  field  and  determine the best  functional operators for  the  implementation of  the  architecture.  For  time-to-contact we  exploited  the  divergence  theorem  to integrate  data from  all  velocity  sensors  present  in the  architecture  and average out  possible errors.
A Dynamical Systems Approach for a Learnable Autonomous Robot
Authors: Naohiro Fukumura Jun Tani
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper discusses  how  a  robot  can  learn  goal-directed  naviga(cid:173) tion  tasks  using  local  sensory  inputs.  The  emphasis  is  that  such  learning  tasks  could  be  formulated  as  an  embedding  problem  of  dynamical  systems:  desired  trajectories  in  a  task space  should  be  embedded  into an  adequate  sensory-based internal  state  space  so  that an unique mapping from the internal state space to the motor  command could  be established.  The  paper shows that a  recurrent  neural network suffices in self-organizing such an adequate internal  state space  from  the temporal  sensory input.  In  our experiments,  using  a  real  robot  with  a  laser  range  sensor,  the  robot  navigated  robustly  by achieving dynamical coherence with  the environment.  It was  also  shown  that  such  coherence  becomes  structurally  sta(cid:173) ble  as  the  global  attractor is  self-organized  in the coupling of the  internal and the environmental dynamics. 
1
Optimization Principles for the Neural Code
Authors: Michael DeWeese
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Recent  experiments  show  that  the  neural  codes  at  work  in  a  wide  range of creatures share some common features.  At first sight, these  observations seem unrelated.  However,  we show  that these  features  arise naturally in  a linear filtered  threshold crossing  (LFTC) model  when we set the threshold to maximize the transmitted information.  This maximization process  requires  neural  adaptation  to  not only  the  DC  signal  level,  as  in  conventional  light  and  dark  adaptation,  but also to the statistical structure of the signal and noise distribu(cid:173) tions.  We  also  present  a  new  approach for  calculating the  mutual  information between  a  neuron's  output  spike  train and any aspect  of its  input signal  which  does not require  reconstruction  of the  in(cid:173) put  signal.  This formulation  is  valid  provided  the  correlations  in  the  spike  train are  small, and we  provide  a  procedure  for  checking  this  assumption.  This paper is  based  on joint work  (DeWeese  [1],  1995).  Preliminary  results  from  the  LFTC  model  appeared  in  a  previous  proceedings  (DeWeese  [2],  1995),  and the  conclusions  we  reached  at that time have been reaffirmed by further  analysis of the  model.
A Framework for Non-rigid Matching and Correspondence
Authors: Steven Gold, Anand Rangarajan Suguna Pappu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Matching feature point sets lies at the core of many approaches to  object recognition. We present a framework for non-rigid match(cid:173) ing that begins with a skeleton module, affine point matching,  and then integrates multiple features to improve correspondence  and develops an object representation based on spatial regions to  model local transformations. The algorithm for feature matching  iteratively updates the transformation parameters and the corre(cid:173) spondence solution, each in turn. The affine mapping is solved in  closed form, which permits its use for data of any dimension. The  correspondence is set via a method for two-way constraint satisfac(cid:173) tion, called softassign, which has recently emerged from the neural  network/statistical physics realm. The complexity of the non-rigid  matching algorithm with multiple features is the same as that of  the affine point matching algorithm. Results for synthetic and real  world data are provided for point sets in 2D and 3D, and for 2D  data with multiple types of features and parts.
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Authors: Yoshua Bengio Salah El Hihi
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  have already shown  that extracting long-term dependencies  from  se(cid:173) quential  data is  difficult,  both for  determimstic dynamical systems  such  as  recurrent  networks,  and  probabilistic  models  such  as  hidden  Markov  models  (HMMs)  or input/output hidden  Markov  models  (IOHMMs).  In  practice,  to  avoid  this  problem,  researchers  have  used  domain  specific  a-priori  knowledge  to give  meaning  to  the hidden or  state variables rep(cid:173) resenting  past context.  In  this  paper,  we  propose  to use  a  more general  type  of a-priori  knowledge,  namely  that  the  temporal  dependencIes  are  structured  hierarchically.  This implies that  long-term  dependencies  are  represented  by  variables with  a long time scale.  This principle is  applied  to a recurrent network which includes delays and multiple time scales.  Ex(cid:173) periments confirm the advantages of such  structures.  A similar approach  is  proposed for  HMMs  and IOHMMs.
Finite State Automata that Recurrent Cascade-Correlation Cannot Represent
Authors: Stefan C. Kremer
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper  relates  the  computational  power  of Fahlman' s  Recurrent  Cascade Correlation (RCC) architecture to that of fInite  state automata  (FSA).  While some recurrent networks are FSA equivalent, RCC is  not.  The paper presents a theoretical analysis of the  RCC architecture in the  form of a proof describing a large class of FSA which cannot be realized  by  RCC.
Memory-based Stochastic Optimization
Authors: Jeff G. Schneider Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper  we  introduce  new  algorithms for  optimizing  noisy  plants in which each experiment is  very expensive.  The algorithms  build a global non-linear model of the expected output at the same  time as using Bayesian linear regression analysis of locally weighted  polynomial models.  The local  model answers  queries  about  confi(cid:173) dence,  noise,  gradient  and  Hessians,  and  use  them  to  make auto(cid:173) mated decisions similar to those made by a practitioner of Response  Surface  Methodology.  The global  and  local  models are  combined  naturally  as  a  locally  weighted  regression.  We  examine the  ques(cid:173) tion of whether the global model can really  help optimization, and  we  extend  it  to  the  case  of time-varying functions.  We  compare  the new algorithms with a  highly tuned higher-order stochastic op(cid:173) timization algorithm on  randomly-generated functions  and  a  sim(cid:173) ulated  manufacturing  task.  We  note  significant  improvements in  total regret , time to converge, and final  solution quality.
Handwritten Word Recognition using Contextual Hybrid Radial Basis Function Network/Hidden Markov Models
Authors: Michel Gilloux, Manuel Leroux Bernard Lemarié
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A hybrid and contextual radial basis function networklhidden Markov  model  off-line  handwritten  word  recognition  system  is  presented.  The  task  assigned  to  the radial  basis  function  networks  is  the  estimation of  emission probabilities associated to Markov states. The model is contex(cid:173) tual because the estimation of emission probabilities takes  into account  the left context of the current image segment as represented by its pred(cid:173) ecessor in the sequence. The new system does not outperform the previ(cid:173) ous system without context but acts differently.
Universal Approximation and Learning of Trajectories Using Oscillators
Authors: Kurt Hornik Pierre Baldi
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Natural  and  artificial  neural  circuits  must  be  capable  of travers(cid:173) ing  specific  state  space  trajectories.  A  natural  approach  to  this  problem  is  to  learn  the  relevant  trajectories  from  examples.  Un(cid:173) fortunately,  gradient  descent  learning  of complex  trajectories  in  amorphous  networks  is  unsuccessful.  We  suggest  a  possible  ap(cid:173) proach  where  trajectories  are  realized  by  combining simple oscil(cid:173) lators,  in  various  modular ways.  We  contrast  two  regimes  of fast  and slow oscillations.  In all cases,  we show that banks of oscillators  with bounded frequencies have universal approximation properties.  Open  questions  are also discussed  briefly. 
1 
INTRODUCTION:  TRAJECTORY LEARNING 
The  design  of artificial  neural  systems,  in  robotics  applications  and  others,  often  leads to the problem of constructing a recurrent neural network capable of producing  a particular trajectory, in the state space of its visible units.  Throughout evolution,  biological neural systems,  such  as  central  pattern generators,  have  also  been faced  with  similar  challenges.  A  natural  approach  to  tackle  this  problem  is  to  try  to  "learn"  the  desired  trajectory,  for  instance  through  a  process  of trial  and  error  and  subsequent  optimization.  Unfortunately,  gradient descent  learning of complex  trajectories  in  amorphous  networks  is  unsuccessful.  Here,  we  suggest  a  possible  approach where  trajectories are realized,  in a  modular and hierarchical fashion,  by  combining simple oscillators.  In particular,  we  show  that  banks of oscillators have  universal approximation properties. 

Also  with the Jet Propulsion  Laboratory,  California  Institute of Technology. 

452 
P.  BALDI,  K.  HORNIK 
To  begin  with,  we  can restrict  ourselves  to the simple case  of a  network  with one!  visible  linear  unit  and consider  the  problem of adjusting  the  network  parameters  in  a  way  that  the output unit  activity  u(t)  is  equal  to a  target function  I(t),  over  an interval of time [0, T].  The hidden units of the network  may be non-linear and  satisfy,  for  instance, one of the usual  neural  network charging equations such  as 
dUi 
dt  = - Ti  + L..JjWij/jUj(t - Tij),
Correlated Neuronal Response: Time Scales and Mechanisms
Authors: Ehud Zohary, Christof Koch Wyeth Bair
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  have  analyzed the relationship between  correlated spike  count  and the peak in the cross-correlation of spike trains for  pairs of si(cid:173) multaneously recorded neurons  from  a  previous  study of area MT  in  the  macaque  monkey  (Zohary  et  al.,  1994).  We  conclude  that  common  input,  responsible  for  creating peaks  on  the order of ten  milliseconds  wide  in  the  spike  train  cross-correlograms  (CCGs),  is  also  responsible  for  creating  the  correlation  in  spike  count  ob(cid:173) served  at  the  two  second  time  scale  of  the  trial.  We  argue  that  both common excitation and  inhibition  may  play significant  roles  in establishing this  correlation.
Worst-case Loss Bounds for Single Neurons
Authors: Jyrki Kivinen, Manfred K. Warmuth David P. Helmbold
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  analyze  and  compare  the  well-known  Gradient  Descent  algo(cid:173) rithm  and  a  new  algorithm,  called  the  Exponentiated  Gradient  algorithm, for  training  a  single neuron  with  an  arbitrary  transfer  function .  Both  algorithms are  easily  generalized  to  larger  neural  networks,  and the  generalization  of Gradient  Descent  is  the stan(cid:173) dard  back-propagation  algorithm.  In  this  paper  we  prove  worst(cid:173) case  loss  bounds  for  both  algorithms  in  the  single  neuron  case.  Since  local  minima make  it  difficult  to  prove  worst-case  bounds  for  gradient-based  algorithms,  we  must  use  a  loss  function  that  prevents  the  formation of spurious  local  minima.  We  define  such  a  matching  loss  function  for  any  strictly  increasing  differentiable  transfer  function  and  prove  worst-case  loss  bound  for  any  such  transfer  function  and  its  corresponding  matching loss.  For  exam(cid:173) ple,  the  matching loss  for  the  identity function  is  the  square  loss  and  the matching loss  for  the logistic sigmoid is  the entropic loss.  The different  structure  of the bounds for  the  two  algorithms indi(cid:173) cates that the new  algorithm out-performs Gradient Descent  when  the inputs contain a  large number of irrelevant components. 
310 
D. P.  HELMBOLD, J.  KIVINEN, M.  K.  WARMUTH
Stock Selection via Nonlinear Multi-Factor Models
Authors: Asriel E. Levin
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper discusses  the use  of multilayer feed forward  neural  net(cid:173) works  for  predicting  a stock's  excess  return  based  on  its  exposure  to various technical  and fundamental factors.  To demonstrate the  effectiveness  of the approach  a  hedged  portfolio which  consists  of  equally  capitalized long and short  positions is  constructed  and  its  historical  returns  are  benchmarked  against T-bill  returns  and  the  S&P500  index.
The Geometry of Eye Rotations and Listing's Law
Authors: Tamar Flash Amir A. Handzel
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  analyse  the  geometry  of  eye  rotations,  and  in  particular  saccades,  using  basic  Lie  group  theory  and  differential  geome(cid:173) try.  Various  parameterizations  of  rotations  are  related  through  a  unifying  mathematical treatment,  and  transformations  between  co-ordinate  systems  are  computed  using  the  Campbell-Baker(cid:173) Hausdorff  formula.  Next,  we  describe  Listing's  law  by  means  of  the  Lie  algebra  so(3).  This  enables  us  to  demonstrate  a  direct  connection  to  Donders'  law, by  showing that  eye  orientations  are  restricted  to the quotient space  80(3)/80(2).  The latter is  equiv(cid:173) alent to the sphere S2, which is exactly the space of gaze directions.  Our analysis provides  a  mathematical framework  for  studying the  oculomotor system  and  could  also  be  extended  to  investigate  the  geometry of mUlti-joint arm movements.
When is an Integrate-and-fire Neuron like a Poisson Neuron?
Authors: Anthony M. Zador Charles F. Stevens
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In the  Poisson neuron  model, the output is  a  rate-modulated Pois(cid:173) son  process  (Snyder  and  Miller,  1991);  the  time  varying  rate  pa(cid:173) rameter  ret)  is  an  instantaneous  function  G[.]  of  the  stimulus,  ret)  =  G[s(t)].  In  a  Poisson  neuron,  then,  ret)  gives  the  instan(cid:173) taneous firing  rate-the instantaneous probability of firing  at  any  instant t-and the output is  a  stochastic function  of the input.  In  part because of its great simplicity, this model is  widely used  (usu(cid:173) ally with  the addition of a  refractory  period), especially  in  in  vivo  single unit electrophysiological studies,  where  set)  is usually taken  to be  the value of some sensory  stimulus.  In the  integrate-and-fire  neuron model, by  contrast, the output is a filtered and thresholded  function  of the input:  the input is  passed  through a  low-pass filter  (determined by the membrane time constant T)  and integrated un(cid:173) til the membrane potential vet)  reaches threshold 8,  at which  point  vet)  is reset  to its initial value.  By contrast with the Poisson model,  in the integrate-and-fire model the ouput is a deterministic function  of the input.  Although the integrate-and-fire model is  a caricature  of real  neural  dynamics,  it  captures  many of the  qualitative  fea(cid:173) tures,  and  is  often  used  as  a  starting point for  conceptualizing the  biophysical behavior of single neurons.  Here we show how a slightly  modified  Poisson model can  be  derived  from  the integrate-and-fire  model with noisy inputs  yet)  =  set) + net).  In  the modified model,  the  transfer  function  G[.] is  a  sigmoid  (erf)  whose  shape  is  deter(cid:173) mined  by  the  noise  variance  /T~.  Understanding  the  equivalence  between  the  dominant  in  vivo  and  in  vitro  simple neuron  models  may help  forge  links between  the  two levels.
Examples of learning curves from a modified VC-formalism
Authors: Jacek Szymanski, Peter L. Bartlett, Robert C. Williamson Adam Kowalczyk
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  examine  the  issue  of evaluation of model  specific  parameters  in  a  modified VC-formalism.  Two examples are analyzed:  the 2-dimensional  homogeneous  perceptron  and  the  I-dimensional  higher  order  neuron.  Both models are solved theoretically, and their learning curves are com(cid:173) pared against  true learning  curves.  It is  shown  that the  formalism  has  the  potential  to  generate  a  variety  of learning  curves,  including  ones  displaying ''phase transitions."
Using Unlabeled Data for Supervised Learning
Authors: Geoffrey G. Towell
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Many classification problems have the property that the only costly  part of obtaining examples  is  the class  label.  This  paper suggests  a  simple  method  for  using  distribution  information  contained  in  unlabeled  examples  to augment  labeled  examples  in  a  supervised  training framework.  Empirical  tests  show  that  the  technique  de(cid:173) scribed  in  this  paper  can  significantly  improve  the  accuracy  of  a  supervised  learner  when  the  learner  is  well  below  its  asymptotic  accuracy level.
Implementation Issues in the Fourier Transform Algorithm
Authors: Sigal Sahar Yishay Mansour
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The  Fourier  transform of boolean  functions  has  come  to  play  an  important role in proving many important learnability results.  We  aim to demonstrate that the  Fourier transform techniques  are also  a  useful  and  practical  algorithm in  addition  to  being  a  powerful  theoretical  tool.  We  describe  the more prominent changes  we  have  introduced  to  the  algorithm,  ones  that  were  crucial  and  without  which  the  performance  of  the  algorithm  would  severely  deterio(cid:173) rate.  One of the benefits we  present  is  the confidence  level for each  prediction  which  measures  the  likelihood the prediction is  correct.
Classifying Facial Action
Authors: Paul A. Viola, Terrence J. Sejnowski, Beatrice A. Golomb, Jan Larsen, Joseph C. Hager, Paul Ekman Marian Stewart Bartlett
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The Facial Action Coding System, (FACS),  devised by Ekman and  Friesen (1978), provides an objective meanS for measuring the facial  muscle  contractions  involved in  a  facial  expression.  In  this  paper,  we  approach automated facial expression  analysis by detecting and  classifying  facial  actions.  We  generated  a  database  of over  1100  image sequences  of 24  subjects  performing over  150 distinct facial  actions  or  action  combinations.  We  compare  three  different  ap(cid:173) proaches  to classifying the facial  actions  in  these  images:  Holistic  spatial analysis based on principal components of graylevel images;  explicit measurement of local image features  such  as wrinkles;  and  template matching with  motion flow  fields.  On  a  dataset contain(cid:173) ing six individual actions and 20 subjects, these methods had 89%,  57%, and 85% performances respectively for generalization to novel  subjects.  When combined,  performance improved to 92%.
Gradient and Hamiltonian Dynamics Applied to Learning in Neural Networks
Authors: Chaouki T. Abdallah, Gregory L. Heileman James W. Howse
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The  process  of machine  learning  can  be  considered  in  two  stages:  model  selection  and parameter estimation.  In this  paper a  technique  is  presented  for constructing dynamical systems with desired qualitative properties.  The  approach  is  based  on the fact  that  an n-dimensional  nonlinear  dynamical  system can be decomposed into one gradient and  (n - 1)  Hamiltonian sys(cid:173) tems.  Thus, the model  selection stage consists of choosing the gradient and  Hamiltonian portions appropriately so that a certain behavior is obtainable.  To estimate the parameters, a stably convergent learning rule  is  presented.  This algorithm has been proven to converge to the desired system trajectory  for  all  initial  conditions  and system inputs.  This technique can be used to  design  neural network models  which  are guaranteed to solve the trajectory  learning problem.
A New Learning Algorithm for Blind Signal Separation
Authors: Andrzej Cichocki, Howard Hua Yang Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A new on-line learning algorithm which minimizes a  statistical de(cid:173) pendency among outputs is  derived for  blind separation  of mixed  signals.  The  dependency  is  measured  by  the  average  mutual  in(cid:173) formation  (MI)  of the outputs.  The source signals and the mixing  matrix  are  unknown  except  for  the  number  of  the  sources.  The  Gram-Charlier  expansion  instead  of the  Edgeworth  expansion  is  used  in  evaluating  the  MI.  The  natural gradient  approach is  used  to minimize the MI. A novel activation function is proposed for the  on-line learning algorithm  which  has  an equivariant property and  is easily implemented on a  neural network like model.  The validity  of the new learning algorithm are verified by computer simulations.
Adaptive Back-Propagation in On-Line Learning of Multilayer Networks
Authors: David Saad Ansgar H. L. West
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: An  adaptive back-propagation algorithm is  studied  and compared  with  gradient  descent  (standard  back-propagation)  for  on-line  learning  in  two-layer  neural  networks  with  an  arbitrary  number  of hidden  units.  Within  a  statistical  mechanics  framework ,  both  numerical  studies  and  a  rigorous  analysis show  that  the  adaptive  back-propagation method results in faster  training by breaking the  symmetry between  hidden  units  more efficiently  and by  providing  faster  convergence to optimal generalization than gradient descent.