mtybilly/Instruct2DS huggingface dataset:
Extracting Tree-Structured Representations of Trained Networks
Authors: Jude W. Shavlik Mark Craven
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  significant  limitation  of  neural  networks  is  that  the  represen(cid:173) tations  they  learn  are  usually  incomprehensible  to  humans.  We  present a  novel algorithm , TREPAN,  for extracting comprehensible,  symbolic representations from  trained neural  networks.  Our algo(cid:173) rithm uses  queries to induce a  decision tree that approximates the  concept  represented by  a  given  network.  Our experiments demon(cid:173) strate that TREPAN is  able to produce decision trees that maintain  a high level of fidelity to their respective networks while being com(cid:173) prehensible  and  accurate.  Unlike  previous  work  in  this  area,  our  algorithm is  general in its applicability and scales well  to large net(cid:173) works  and problems with  high-dimensional input spaces.
Improving Committee Diagnosis with Resampling Techniques
Authors: Paul W. Munro, Howard R. Doyle Bambang Parmanto
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Central to the performance improvement of a committee relative to  individual networks is the error correlation between networks in the  committee.  We  investigated  methods  of achieving  error  indepen(cid:173) dence between the networks by training the networks with different  resampling sets  from  the  original training set.  The  methods were  tested  on the sinwave artificial task and the real-world problems of  hepatoma (liver cancer)  and breast  cancer  diagnoses.
The Capacity of a Bump
Authors: Gary William Flake
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Recently, several researchers have reported encouraging experimental re(cid:173) sults when using Gaussian or bump-like activation functions in multilayer  perceptrons.  Networks of this type usually require fewer  hidden layers  and  units and often learn much  faster  than typical  sigmoidal networks.  To  explain these  results we consider a hyper-ridge network,  which is a  simple perceptron with no hidden units and a rid¥e activation function. If  we are interested in partitioningp points in d dimensions into two classes  then in the limit as d approaches infinity the capacity of a hyper-ridge and  a perceptron is identical.  However, we show that for p  ~ d, which is the  usual case in practice, the ratio of hyper-ridge to perceptron dichotomies  approaches pl2(d + 1).
Boosting Decision Trees
Authors: Corinna Cortes Harris Drucker
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We introduce a constructive,  incremental learning system for regression  problems that models data by means of locally linear experts.  In contrast  to  other approaches,  the  experts  are  trained  independently  and  do  not  compete for data during learning.  Only when a prediction for  a query  is  required  do  the  experts  cooperate  by  blending  their  individual  predic(cid:173) tions.  Each expert is trained by  minimizing  a penalized local cross vali(cid:173) dation error using second order methods. In this way, an expert is able to  find a local distance metric by adjusting the size and  shape of the recep(cid:173) tive field in which its predictions are valid, and also to detect relevant in(cid:173) put features  by  adjusting  its bias  on  the  importance of individual input  dimensions. We derive asymptotic results for our method. In a variety of  simulations the properties of the algorithm are demonstrated with respect  to  interference,  learning  speed,  prediction  accuracy,  feature  detection,  and task oriented incremental learning.
Some results on convergent unlearning algorithm
Authors: Irina B. Shuvalova Serguei A. Semenov
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper  we  consider  probabilities of different  asymptotics of  convergent  unlearning  algorithm for  the  Hopfield-type  neural  net(cid:173) work  (Plakhov  &  Semenov,  1994)  treating  the  case  of unbiased  random  patterns.  We  show  also  that  failed  unlearning  results  in  total memory breakdown.
Forward-backward retraining of recurrent neural networks
Authors: Anthony J. Robinson Andrew W. Senior
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper  describes  the  training  of  a  recurrent  neural  network  as  the  letter  posterior  probability estimator for  a  hidden  Markov  model,  off-line  handwriting recognition  system.  The network  esti(cid:173) mates  posterior  distributions for  each  of a  series  of frames  repre(cid:173) senting  sections  of a  handwritten  word.  The  supervised  training  algorithm,  backpropagation through time,  requires  target outputs  to  be  provided for  each  frame.  Three  methods for  deriving  these  targets  are  presented.  A  novel  method  based  upon  the  forward(cid:173) backward  algorithm is  found  to  result  in  the  recognizer  with  the  lowest  error  rate.
KODAK lMAGELINK™ OCR Alphanumeric Handprint Module
Authors: Christopher W. Thrasher Alexander Shustorovich
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: This  paper  describes  the  Kodak  Imageliok TM  OCR  alphanumeric  handprint  module.  There  are  two  neural  network  algorithms  at  its  cme:  the  first  network  is  trained  to  find  individual  characters  in  an  alphamuneric  field,  while  the  second one perfmns the  classification.  Both  networks  were  trained  on  Gabor  projections  of  the  ociginal  pixel  images,  which  resulted  in higher  recognition  rates  and  greater  noise  immunity.  Compared  its  purely  numeric  counterpart  (Shusurovich  and  Thrasher,  1995),  this  version  of the  system  has  a  significant  applicatim  specific  postprocessing  module.  The  system  has  been  implemented in specialized parallel hardware,  which allows  it to run at 80 char/sec/board.  It has been installed  at the Driver and  Vehicle  Licensing  Agency  (DVLA)  in  the  United  Kingdom.  and  its  overall  success  rate  exceeds  96%  (character  level  without  rejects).  which  translates  into  85%  field  rate.  If approximately  20%  of the  fields  are  rejected.  the  system  achieves  99.8%  character  and  99.5%  field success rate. 
to
Predictive Q-Routing: A Memory-based Reinforcement Learning Approach to Adaptive Traffic Control
Authors: Dit-Yan Yeung Samuel P. M. Choi
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper,  we  propose  a  memory-based  Q-Iearning  algorithm  called  predictive  Q-routing  (PQ-routing)  for  adaptive  traffic  con(cid:173) trol.  We attempt to address two problems encountered in Q-routing  (Boyan  &  Littman,  1994),  namely,  the inability to fine-tune  rout(cid:173) ing policies under  low  network  load and  the  inability to learn  new  optimal  policies  under  decreasing  load  conditions.  Unlike  other  memory-based  reinforcement  learning  algorithms  in  which  mem(cid:173) ory  is  used  to  keep  past  experiences  to  increase  learning  speed,  PQ-routing  keeps  the  best  experiences  learned  and  reuses  them  by  predicting  the  traffic  trend.  The  effectiveness  of  PQ-routing  has been  verified  under various network  topologies and traffic con(cid:173) ditions.  Simulation  results  show  that  PQ-routing  is  superior  to  Q-routing in terms of both learning speed and  adaptability.
Family Discovery
Authors: Stephen M. Omohundro
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: "Family discovery"  is the task of learning the dimension and struc(cid:173) ture  of a  parameterized  family  of  stochastic  models.  It is  espe(cid:173) cially appropriate when the training examples are partitioned into  "episodes"  of samples  drawn  from  a  single  parameter  value.  We  present  three  family  discovery  algorithms  based on  surface  learn(cid:173) ing and show that they significantly improve performance over two  alternatives on a  parameterized classification task.
Modeling Interactions of the Rat's Place and Head Direction Systems
Authors: David S. Touretzky A. David Redish
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  have  developed  a  computational  theory  of rodent  navigation  that  includes analogs of the place cell system, the head direction system,  and  path integration.  In this paper we present simulation results showing how  interactions between the place and head direction systems can account for  recent observations about hippocampal  place cell  responses to doubling  and/or rotation of cue cards in a cylindrical arena (Sharp et at.,  1990). 
Rodents have multiple internal representations of their relationship to their environment.  They  have,  for example,  a representation of their location (place cells in  the hippocampal  formation,  see  Muller  et  at.,  1991),  and  a  location-independent representation  of their  heading  (head  direction  cells  in  the  postsubiculum and  the  anterior thalamic  nuclei,  see  Taube et at.,  1990; Taube,  1995).  If these  representations  are  to  be  used  for  navigation, they  must  be  aligned  consistently  whenever the animal reenters a familiar environment.  This process was examined in a set  of experiments by Sharp et at.  (1990). 
1  The Sharp et al., 1990 experiment 
Rats spent multiple sessions finding food scattered randomly on the floor of a black cylin(cid:173) drical arena with a white cue card along the wall subtending 90°  of arc.  The animals were  not disoriented before entering the arena, and they always entered at the same location:  the  northwest corner.  See  Figure 3a.  Hippocampal  place  fields  were  mapped  by  single-cell  recording.  A variety of probe trials were then  introduced.  When  an  identical second  cue 
62 
A. D. REDISH, D. S. TOURETZKY
Empirical Entropy Manipulation for Real-World Problems
Authors: Nicol N. Schraudolph, Terrence J. Sejnowski Paul A. Viola
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: No finite sample is sufficient to determine the density, and therefore  the entropy, of a signal directly.  Some assumption about either the  functional form of the density or about its smoothness is necessary.  Both amount to a prior over the space of possible density functions.  By  far  the  most common approach  is  to  assume  that  the  density  has a  parametric form. 
By  contrast  we  derive  a  differential  learning  rule  called  EMMA  that  optimizes  entropy  by  way  of kernel  density  estimation.  En(cid:173) tropy  and  its  derivative  can  then  be  calculated  by  sampling from  this density  estimate.  The resulting  parameter update rule  is  sur(cid:173) prisingly simple and efficient. 
We  will  show  how  EMMA  can  be  used  to detect  and  correct  cor(cid:173) ruption  in  magnetic  resonance  images  (MRI).  This  application  is  beyond  the scope  of existing parametric entropy  models.
A Neural Network Model of 3-D Lightness Perception
Authors: William D. Ross Luiz Pessoa
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  neural  network  model  of 3-D  lightness  perception  is  presented  which  builds  upon  the  FACADE  Theory  Boundary  Contour  Sys(cid:173) tem/Feature Contour  System  of Grossberg  and  colleagues.  Early  ratio encoding by  retinal ganglion neurons  as  well  as psychophysi(cid:173) cal  results on constancy  across  different  backgrounds (background  constancy)  are used to provide functional constraints to the theory  and  suggest  a  contrast  negation  hypothesis  which  states that ratio  measures  between  coplanar  regions  are  given  more  weight  in  the  determination  of lightness  of the  respective  regions.  Simulations  of the  model  address  data on  lightness  perception,  including  the  coplanar ratio hypothesis,  the  Benary cross,  and White's illusion.
Adaptive Retina with Center-Surround Receptive Field
Authors: Kwabena Boahen Shih-Chii Liu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Both vertebrate and invertebrate retinas are highly efficient  in ex(cid:173) tracting contrast independent of the background intensity over five  or  more  decades.  This  efficiency  has  been  rendered  possible  by  the adaptation of the DC operating point to the background inten(cid:173) sity while maintaining high gain  transient responses.  The center(cid:173) surround properties of the retina allows the system  to extract in(cid:173) formation at the edges in the image.  This silicon retina models the  adaptation properties of the receptors and the antagonistic center(cid:173) surround properties of the laminar cells  of the invertebrate retina  and the outer-plexiform layer of the vertebrate retina.  We also illus(cid:173) trate the spatio-temporal responses of the silicon retina on moving  bars.  The  chip  has  59x64  pixels  on  a  6.9x6.8mm2  die  and  it  is  fabricated in 2 J-tm  n-well  technology.
Does the Wake-sleep Algorithm Produce Good Density Estimators?
Authors: Geoffrey E. Hinton, Peter Dayan Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The wake-sleep algorithm (Hinton, Dayan, Frey and Neal 1995) is a rel(cid:173) atively efficient method of fitting  a multilayer stochastic generative  model to high-dimensional data. In addition to  the top-down connec(cid:173) tions in the generative model, it makes use of bottom-up connections for  approximating the probability distribution over the hidden units given  the data, and it trains these bottom-up connections using a simple delta  rule. We use a variety of synthetic and real data sets to compare the per(cid:173) formance of the wake-sleep algorithm with Monte Carlo and mean field  methods for fitting  the same generative model and also compare it with  other models that are less powerful but easier to fit.
EM Optimization of Latent-Variable Density Models
Authors: Markus Svensén, Christopher K. I. Williams Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: There is  currently  considerable interest  in developing general non(cid:173) linear  density  models  based  on  latent,  or  hidden,  variables.  Such  models have the ability to discover the presence of a relatively small  number of underlying  'causes'  which,  acting  in  combination,  give  rise  to the apparent complexity of the observed  data set.  Unfortu(cid:173) nately, to train such models generally requires large computational  effort.  In this paper we  introduce a  novel  latent variable algorithm  which retains the general non-linear capabilities of previous models  but  which  uses  a  training procedure  based  on  the  EM  algorithm.  We  demonstrate  the  performance  of the  model  on  a  toy  problem  and on data from flow  diagnostics for  a  multi-phase oil  pipeline.
Modern Analytic Techniques to Solve the Dynamics of Recurrent Neural Networks
Authors: S. N. Laughton, D. Sherrington A.C.C. Coolen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  describe the use of modern analytical techniques in  solving the  dynamics  of  symmetric  and  nonsymmetric  recurrent  neural  net(cid:173) works  near saturation.  These explicitly take into account the cor(cid:173) relations between  the  post-synaptic  potentials,  and  thereby  allow  for  a  reliable prediction of transients.
A Realizable Learning Task which Exhibits Overfitting
Authors: Siegfried Bös
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper we  examine  a  perceptron  learning  task.  The  task  is  realizable  since  it  is  provided  by  another  perceptron  with  identi(cid:173) cal  architecture.  Both  perceptrons  have  nonlinear  sigmoid  output  functions.  The gain  of the output function  determines the level  of  nonlinearity  of the  learning  task.  It is  observed  that  a  high  level  of nonlinearity leads to overfitting. We  give an explanation for  this  rather  surprising  observation  and  develop  a  method  to  avoid  the  overfitting.  This  method  has  two  possible  interpretations,  one  is  learning with noise,  the other cross-validated early stopping. 
1  Learning  Rules  from  Examples 
The  property  which  makes feedforward  neural  nets  interesting for  many  practical  applications  is  their  ability  to approximate functions,  which  are  given  only  by ex(cid:173) amples.  Feed-forward  networks  with  at  least  one  hidden  layer  of nonlinear  units  are  able  to  approximate  each  continuous function  on  a  N-dimensional  hypercube  arbitrarily  well.  While  the  existence  of neural  function  approximators  is  already  established, there is  still a lack of knowledge about their practical realizations. Also  major problems, which complicate a good realization, like overfitting, need  a better  understanding. 
In  this  work  we  study  overfitting in  a  one-layer  percept ron  model.  The  model  allows a good theoretical description while it exhibits already a qualitatively similar  behavior as  the multilayer  perceptron. 
A  one-layer  perceptron  has  N  input  units  and  one  output  unit.  Between  input  and output it  has one layer of adjustable weights Wi, (i  =  1, ... ,N). The output  z  is  a possibly nonlinear function  of the weighted sum of inputs Xi,  i.e. 
z  =  g(h) ,  with
Simulation of a Thalamocortical Circuit for Computing Directional Heading in the Rat
Authors: Hugh T. Blair
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Several  regions  of the  rat  brain  contain  neurons  known  as  head-direc(cid:173) tion celis,  which encode the animal's directional heading during spatial  navigation.  This  paper presents  a  biophysical  model of head-direction  cell  acti vity,  which  suggests  that  a  thalamocortical  circuit  might com(cid:173) pute the  rat's  head  direction  by  integrating  the  angular  velocity  of the  head over time.  The model was implemented using the neural simulator  NEURON, and makes testable predictions about the structure and func(cid:173) tion of the rat head-direction circuit. 
1  HEAD-DIRECTION CELLS  As a rat navigates through space, neurons called head-direction celis encode the animal's  directional  heading in the horizontal plane (Ranck,  1984; Taube, Muller, &  Ranck,  1990).  Head-direction cells have been  recorded in  several  brain areas, including the postsubicu(cid:173) lum (Ranck,  1984) and anterior thalamus (Taube,  1995).  A  variety of theories  have pro(cid:173) posed  that  head-direction  cells  might  play  an  important  role  in  spatial  learning  and  navigation  (Brown  &  Sharp,  1995;  Burgess,  Recce,  &  O'Keefe,  1994;  McNaughton,  Knierim, &  Wilson,  1995; Wan, Touretzky, &  Redish,  1994; Zhang,  1995). 
1.1  BASIC FIRING PROPERTIES 
A head-direction cell fires action potentials only when the rat's head is facing in a particu(cid:173) lar direction with respect to the static surrounding environment, regardless of the animal's  location  within  that environment.  Head-direction cells are not influenced  by the position  of the rat's head with respect to its body, they are only influenced by the direction of the 
*Also at the Yale Neuroengineering and Neuroscience Center (NNC), 5 Science Park North, New
Learning Fine Motion by Markov Mixtures of Experts
Authors: Michael I. Jordan Marina Meila
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Compliant control is a standard method for performing fine  manip(cid:173) ulation tasks, like grasping and assembly, but it requires estimation  of the state of contact  (s.o.c.)  between  the robot  arm and  the  ob(cid:173) jects  involved.  Here  we  present  a  method to learn  a  model of the  movement from  measured  data.  The  method  requires  little or  no  prior  knowledge  and  the  resulting  model  explicitly  estimates  the  s.o.c.  The  current  s.o.c.  is  viewed  as  the  hidden  state  variable  of  a  discrete  HMM.  The  control  dependent  transition  probabilities  between  states  are  modeled  as  parametrized functions  of the mea(cid:173) surement.  We  show  that  their  parameters  can  be  estimated  from  measurements at the same time as the parameters of the movement  in  each  s.o.c.  The learning algorithm is  a  variant of the EM  proce(cid:173) dure.  The  E step  is  computed exactly ; solving  the  M step  exactly  is  not possible  in general.  Here,  gradient ascent  is  used  to produce  an increase  in  likelihood .
Stable LInear Approximations to Dynamic Programming for Stochastic Control Problems with Local Transitions
Authors: John N. Tsitsiklis Benjamin Van Roy
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  consider  the  solution  to  large  stochastic  control  problems  by  means of methods that rely on compact representations and a vari(cid:173) ant of the value iteration algorithm to compute approximate cost(cid:173) to-go functions.  While  such methods are known  to be unstable in  general, we identify a new class of problems for  which convergence,  as  well  as  graceful  error  bounds,  are  guaranteed.  This  class  in(cid:173) volves linear parameterizations of the cost-to- go function  together  with  an assumption  that the  dynamic  programming operator is  a  contraction  with  respect  to  the  Euclidean  norm  when  applied  to  functions  in  the  parameterized  class.  We  provide  a  special  case  where  this  assumption  is  satisfied,  which  relies  on  the  locality of  transitions in a  state space.  Other cases will  be discussed  in a  full  length version of this paper.
Generalisation of A Class of Continuous Neural Networks
Authors: Jieyu Zhao John Shawe-Taylor
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  propose  a  way of using  boolean circuits  to perform real valued  computation  in  a  way  that  naturally  extends  their  boolean  func(cid:173) tionality.  The  functionality  of multiple  fan  in  threshold  gates  in  this  model  is  shown  to  mimic  that  of a  hardware implementation  of continuous Neural Networks.  A Vapnik-Chervonenkis dimension  and  sample  size  analysis  for  the  systems  is  performed  giving  best  known  sample sizes  for  a  real  valued  Neural Network.  Experimen(cid:173) tal results confirm the conclusion that the sample sizes required for  the  networks  are significantly  smaller than for  sigmoidal networks.
Visual gesture-based robot guidance with a modular neural system
Authors: Andrea Drees, Helge Ritter Enno Littmann
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We report on the development of the modular neural system  "SEE(cid:173) EAGLE"  for  the  visual  guidance  of robot  pick-and-place  actions.  Several  neural  networks  are  integrated  to  a  single system  that vi(cid:173) sually  recognizes  human hand  pointing gestures  from  stereo  pairs  of color  video  images.  The output of the  hand recognition stage is  processed  by  a  set  of color-sensitive  neural  networks  to  determine  the cartesian location of the target object that is  referenced  by the  pointing gesture.  Finally, this information is  used  to guide a robot  to grab  the  target  object  and  put  it  at  another  location  that  can  be specified  by a second pointing gesture.  The accuracy of the cur(cid:173) rent  system allows to identify the location of the referenced  target  object to an accuracy of 1 cm  in  a  workspace  area of 50x50 cm.  In  our  current  environment,  this  is  sufficient  to  pick  and  place  arbi(cid:173) trarily positioned target objects within the workspace.  The system  consists  of neural  networks  that  perform  the  tasks  of image seg(cid:173) mentation, estimation of hand  location, estimation of 3D-pointing  direction, object recognition, and necessary  coordinate transforms.  Drawing heavily on the use of learning algorithms, the functions of  all  network modules were  created from  data examples only.
Symplectic Nonlinear Component Analysis
Authors: Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Statistically independent features  can be extracted by finding a fac(cid:173) torial representation of a signal distribution.  Principal Component  Analysis  (PCA)  accomplishes  this  for  linear  correlated  and  Gaus(cid:173) sian  distributed signals.  Independent  Component Analysis  (ICA),  formalized  by  Comon (1994),  extracts  features  in  the  case  of lin(cid:173) ear  statistical  dependent  but  not  necessarily  Gaussian  distributed  signals.  Nonlinear  Component Analysis finally should find  a  facto(cid:173) rial  representation  for  nonlinear  statistical  dependent  distributed  signals.  This  paper  proposes  for  this  task  a  novel  feed-forward,  information  conserving,  nonlinear  map  - the  explicit  symplectic  transformations.  It also solves the problem of non-Gaussian output  distributions  by  considering  single  coordinate  higher  order  statis(cid:173) tics.
Independent Component Analysis of Electroencephalographic Data
Authors: Anthony J. Bell, Tzyy-Ping Jung, Terrence J. Sejnowski Scott Makeig
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Because of the distance between the skull and brain and their differ(cid:173) ent resistivities,  electroencephalographic (EEG) data collected from  any  point  on  the  human scalp  includes  activity  generated  within  a  large  brain area.  This spatial smearing of EEG  data by  volume  conduction  does  not involve significant time delays,  however,  sug(cid:173) gesting that the Independent  Component  Analysis (ICA) algorithm  of Bell and Sejnowski [1]  is suitable for  performing blind source sep(cid:173) aration on EEG data.  The ICA algorithm separates the problem of  source  identification from  that of source  localization.  First  results  of applying the ICA  algorithm to  EEG and  event-related potential  (ERP)  data  collected  during  a  sustained  auditory  detection  task  show:  (1)  ICA training is insensitive to different  random seeds.  (2)  ICA may be used to segregate obvious artifactual EEG components  (line and muscle noise, eye movements) from other sources.  (3) ICA  is  capable of isolating overlapping  EEG  phenomena,  including  al(cid:173) pha and theta bursts and spatially-separable ERP components, to  separate  ICA  channels.  (4)  N onstationarities  in  EEG  and  behav(cid:173) ioral state can be  tracked using ICA  via changes  in  the  amount of  residual  correlation  between  ICA-filtered  output channels. 
146 
S. MAKEIG, A. l . BELL, T.-P. lUNG, T. l. SEJNOWSKI
Competence Acquisition in an Autonomous Mobile Robot using Hardware Neural Techniques
Authors: Alan F. Murray Geoffrey B. Jackson
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper  we  examine  the  practical  use  of  hardware  neural  networks  in  an  autonomous  mobile  robot.  We  have  developed  a  hardware  neural  system  based  around  a  custom  VLSI  chip,  EP(cid:173) SILON  III,  designed  specifically  for  embedded  hardware  neural  applications.  We  present  here  a  demonstration  application  of an  autonomous mobile robot that highlights the flexibility of this sys(cid:173) tem.  This robot gains basic mobility competence in very few  train(cid:173) ing epochs  using  an  "instinct-rule"  training methodology.
Gaussian Processes for Regression
Authors: Carl Edward Rasmussen Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: The Bayesian analysis of neural networks is difficult because a sim(cid:173) ple  prior  over  weights  implies  a  complex  prior  distribution  over  functions .  In this paper we investigate the use of Gaussian process  priors  over  functions,  which  permit  the  predictive  Bayesian  anal(cid:173) ysis  for  fixed  values  of hyperparameters  to  be  carried  out exactly  using matrix operations.  Two methods, using optimization and av(cid:173) eraging (via Hybrid Monte Carlo) over  hyperparameters have been  tested  on  a  number  of  challenging  problems  and  have  produced  excellent  results.
Reorganisation of Somatosensory Cortex after Tactile Training
Authors: John G. Taylor Rasmus S. Petersen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Topographic maps in  primary areas of mammalian cerebral cortex reor(cid:173) ganise as  a result of behavioural  training.  The nature  of this  reorgani(cid:173) sation  seems  consistent  with  the  behaviour  of competitive  neural  net(cid:173) works,  as  has  been  demonstrated  in  the  past  by  computer  simulation.  We model tactile training on the hand representation in primate somato(cid:173) sensory  cortex,  using  the  Neural  Field  Theory  of Amari  and  his  col(cid:173) leagues.  Expressions for  changes in  both receptive field  size  and  mag(cid:173) nification factor are derived,  which  are consistent with  owl  monkey ex(cid:173) periments and make a prediction which goes beyond them.
Prediction of Beta Sheets in Proteins
Authors: Soren Kamaric Riis Anders Krogh
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Most current methods for prediction of protein secondary structure  use a small window of the protein sequence to predict the structure  of the central amino acid.  We describe a new method for prediction  of the  non-local  structure  called  ,8-sheet,  which  consists  of two  or  more  ,8-strands  that  are  connected  by  hydrogen  bonds.  Since,8- strands  are  often  widely separated in the  protein  chain, a network  with two windows is introduced.  After training on a set of proteins  the network  predicts  the sheets  well,  but there are many false  pos(cid:173) itives.  By  using  a  global energy  function  the ,8-sheet  prediction is  combined with a  local prediction of the three secondary structures  a-helix, ,8-strand and coil.  The energy function is minimized using  simulated annealing to give  a  final  prediction.
Active Learning in Multilayer Perceptrons
Authors: Kenji Fukumizu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We propose an active learning method with  hidden-unit reduction.  which is  devised specially for multilayer perceptrons (MLP). First,  we  review  our active  learning  method,  and  point  out  that  many  Fisher-information-based  methods  applied  to  MLP  have a  critical  problem:  the  information  matrix  may  be  singular.  To  solve  this  problem, we  derive the singularity condition of an information ma(cid:173) trix, and  propose an active learning technique that is applicable to  MLP.  Its effectiveness is  verified  through experiments.
A model of transparent motion and non-transparent motion aftereffects
Authors: Alexander Grunewald
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  model  of  human  motion  perception  is  presented.  The  model  contains two stages of direction selective units.  The first stage con(cid:173) tains  broadly  tuned  units,  while  the  second  stage  contains  units  that are  narrowly  tuned.  The  model  accounts  for  the  motion  af(cid:173) tereffect  through  adapting  units  at  the  first  stage  and  inhibitory  interactions at the second stage.  The model explains how two pop(cid:173) ulations of dots moving in slightly different directions are perceived  as  a  single  population  moving  in  the  direction of the  vector  sum,  and how two populations moving in strongly different directions are  perceived as  transparent motion.  The model also explains why the  motion aftereffect in both cases appears as non-transparent motion.
A Practical Monte Carlo Implementation of Bayesian Learning
Authors: Carl Edward Rasmussen
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  practical  method  for  Bayesian  training  of feed-forward  neural  networks  using  sophisticated  Monte  Carlo  methods  is  presented  and evaluated.  In reasonably small amounts of computer time this  approach  outperforms  other  state-of-the-art  methods  on  5  data(cid:173) limited tasks  from real world  domains.
A Smoothing Regularizer for Recurrent Neural Networks
Authors: John E. Moody Lizhong Wu
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We derive a smoothing regularizer for recurrent network models by  requiring robustness in prediction performance to perturbations of  the training data.  The regularizer can be viewed  as  a  generaliza(cid:173) tion of the first order Tikhonov stabilizer to dynamic models.  The  closed-form  expression  of the  regularizer  covers  both time-lagged  and  simultaneous recurrent  nets,  with  feedforward  nets  and  one(cid:173) layer linear nets as special cases.  We  have successfully tested this  regularizer in a  number of case studies and found that it performs  better than standard quadratic weight  decay. 
1
REMAP: Recursive Estimation and Maximization of A Posteriori Probabilities - Application to Transition-Based Connectionist Speech Recognition
Authors: Hervé Bourlard, Nelson Morgan Yochai Konig
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: In  this  paper,  we  introduce  REMAP,  an  approach for  the training  and estimation of posterior probabilities using a recursive algorithm  that is  reminiscent of the EM-based  Forward-Backward  (Liporace  1982)  algorithm  for  the  estimation  of sequence  likelihoods.  Al(cid:173) though  very  general,  the  method  is  developed  in  the  context  of a  statistical  model for  transition-based speech  recognition  using  Ar(cid:173) tificial  Neural  Networks  (ANN)  to  generate  probabilities for  Hid(cid:173) den  Markov  Models  (HMMs).  In  the  new  approach,  we  use  local  conditional posterior probabilities of transitions to estimate global  posterior  probabilities of word  sequences.  Although  we  still  use  ANNs  to  estimate  posterior  probabilities,  the  network  is  trained  with targets that are themselves estimates of local posterior proba(cid:173) bilities.  An  initial experimental result shows a significant decrease  in error-rate in  comparison to a  baseline system.
Harmony Networks Do Not Work
Authors: René Gourley
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Harmony  networks  have  been  proposed  as  a  means by  which  con(cid:173) nectionist  models can perform symbolic computation.  Indeed,  pro(cid:173) ponents claim that a harmony network can be built that constructs  parse trees for strings in  a context free  language.  This paper shows  that  harmony  networks  do  not  work  in  the  following  sense:  they  construct  many outputs that are  not valid  parse  trees. 
In order to show  that the notion of systematicity is compatible with connectionism,  Paul  Smolensky,  Geraldine  Legendre  and  Yoshiro  Miyata  (Smolensky,  Legendre,  and  Miyata  1992;  Smolen sky  1993;  Smolen sky,  Legendre,  and  Miyata  1994)  pro(cid:173) posed a mechanism, "Harmony Theory,"  by which connectionist models purportedly  perform  structure  sensitive  operations  without  implementing classical  algorithms.  Harmony theory  describes  a  "harmony network"  which,  in the course of reaching a  stable equilibrium, apparently computes parse trees  that are valid according to the  rules  of a  particular context-free grammar. 
Harmony  networks  consist  of four  major  components  which  will  be  explained  in  detail in  Section  1.  The four  components are, 
Tensor Representation:  A means to interpret the activation vector of a  connec(cid:173)
tionist system as  a  parse  tree for  a string in a  context-free  language. 
Harmony:  A  function  that  maps all  possible  parse trees  to the  non-positive inte(cid:173)
gers so that a  parse tree is  valid  if and only if its harmony is  zero. 
Energy:  A  function  that  maps  the  set  of activation  vectors  to  the  real  numbers 
and which  is  minimized by  certain  connectionist  networks!. 
Recursive Construction:  A system for  determining  the weight  matrix of a  con(cid:173)
nectionist  network  so  that if its  activation  vector  is  interpreted  as  a  parse 
1 Smolensky,  Legendre and Miyata use the term  "harmony"  to refer to both energy and  harmony.  To distinguish  between them, we  will  use the term that is often used  to describe  the Lyapunov  function  of dynamic  systems,  "energy"  (see for  example  Golden  1986).
Learning Sparse Perceptrons
Authors: Mark Craven Jeffrey C. Jackson
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  introduce  a  new  algorithm  designed  to  learn  sparse  percep(cid:173) trons over input representations which include high-order features.  Our  algorithm,  which  is  based  on a  hypothesis-boosting  method,  is  able to PAC-learn a  relatively  natural class  of target concepts.  Moreover, the algorithm appears to work well in practice:  on a set  of three  problem  domains,  the algorithm  produces  classifiers  that  utilize  small  numbers  of features  yet  exhibit  good  generalization  performance.  Perhaps most  importantly,  our algorithm generates  concept descriptions that are easy for  humans to understand. 
1
The Role of Activity in Synaptic Competition at the Neuromuscular Junction
Authors: David J. Willshaw Samuel R. H. Joseph
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: An  extended  version  of the  dual  constraint  model  of motor  end(cid:173) plate morphogenesis is  presented that includes activity dependent  and  independent  competition.  It is  supported by  a  wide  range  of  recent neurophysiological evidence that indicates a strong relation(cid:173) ship  between  synaptic  efficacy  and  survival.  The  computational  model is  justified  at the molecular level  and its predictions match  the developmental and regenerative behaviour of real synapses.
Explorations with the Dynamic Wave Model
Authors: Jeffrey L. Elman Thomas P. Rebotier
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Following  Shrager  and  Johnson  (1995)  we  study  growth  of  logi(cid:173) cal  function  complexity  in  a  network  swept  by  two  overlapping  waves:  one  of pruning, and  the  other of Hebbian  reinforcement  of  connections.  Results  indicate  a  significant  spatial  gradient  in  the  appearance  of  both  linearly  separable  and  non  linearly  separable  functions of the two inputs of the network; the n.l.s.  cells are  much  sparser  and  their slope of appearance  is  sensitive  to  parameters  in  a  highly  non-linear  way.
Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding
Authors: Richard S. Sutton
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: On large  problems,  reinforcement  learning  systems  must  use  parame(cid:173) terized function approximators such as neural networks in order to gen(cid:173) eralize  between similar  situations and actions.  In these cases  there are  no strong theoretical results on the accuracy  of convergence,  and com(cid:173) putational results  have  been  mixed.  In  particular,  Boyan and  Moore  reported at last year's meeting a series of negative results in attempting  to apply dynamic programming together with function approximation  to simple control problems with continuous state spaces.  In this paper,  we present positive results for all the control tasks they attempted, and  for  one that is significantly larger.  The most important differences  are  that  we  used  sparse-coarse-coded  function  approximators  (CMACs)  whereas  they used mostly global function  approximators, and that we  learned  online  whereas  they  learned  offline.  Boyan  and  Moore  and  others  have  suggested  that  the  problems  they  encountered  could  be  solved  by  using  actual  outcomes  ("rollouts"),  as  in  classical  Monte  Carlo  methods,  and as in the TD().)  algorithm when). =  1.  However,  in our experiments  this always resulted in substantially poorer perfor(cid:173) mance.  We  conclude  that  reinforcement  learning  can  work  robustly  in  conjunction  with  function  approximators,  and  that  there  is  little  justification at present for  avoiding the case  of general  ).. 
1  Reinforcement  Learning and  Function  Approximation 
Reinforcement learning is a  broad class of optimal control methods based on estimating  value functions  from  experience,  simulation,  or  search  (Barto, Bradtke  &;  Singh,  1995;  Sutton,  1988;  Watkins,  1989).  Many  of these  methods,  e.g.,  dynamic  programming  and  temporal-difference  learning,  build  their  estimates  in  part  on  the  basis  of other 
Generalization  in  Reinforcement Learning
A Multiscale Attentional Framework for Relaxation Neural Networks
Authors: Eric Mjolsness Dimitris I. Tsioutsias
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  investigate  the  optimization  of neural  networks  governed  by  general  objective functions.  Practical formulations of such  objec(cid:173) tives  are  notoriously  difficult  to solve;  a  common problem  is  the  poor  local extrema that  result  by  any  of the  applied  methods.  In  this paper, a novel framework is introduced for the solution oflarge(cid:173) scale  optimization problems.  It assumes  little about  the objective  function and can be applied to general nonlinear, non-convex func(cid:173) tions;  objectives  in thousand  of variables  are  thus efficiently  min(cid:173) imized  by  a  combination of techniques  - deterministic  annealing,  multiscale optimization, attention mechanisms and trust region op(cid:173) timization methods.
VLSI Model of Primate Visual Smooth Pursuit
Authors: Jan Van der Spiegel, Paul Mueller Ralph Etienne-Cummings
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: A  one  dimensional  model  of primate  smooth  pursuit  mechanism  has  been  implemented  in  2  11m  CMOS  VLSI.  The  model  consolidates  Robinson's  negative  feedback  model  with  Wyatt  and  Pola's  positive  feedback scheme, to produce a  smooth  pursuit  system  which  zero's  the  velocity  of a  target on  the  retina.  Furthermore,  the  system  uses  the  current eye  motion  as  a  predictor for  future  target  motion.  Analysis,  stability and biological correspondence of the system are discussed.  For  implementation  at  the  focal  plane,  a  local  correlation  based  visual  motion  detection  technique  is  used.  Velocity  measurements,  ranging  over 4 orders of magnitude with < 15%  variation, provides  the  input  to  the smooth pursuit system.  The  system  performed  successful  velocity  tracking for high contrast scenes.  Circuit design and performance of the  complete smooth pursuit system is presented.
Reinforcement Learning by Probability Matching
Authors: Michael I. Jordan Philip N. Sabes
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  present  a  new  algorithm  for  associative  reinforcement  learn(cid:173) ing.  The algorithm is based upon the idea of matching a network's  output probability with a probability distribution derived from the  environment's reward signal.  This Probability Matching algorithm  is shown  to perform faster  and  be  less  susceptible to local  minima  than  previously  existing  algorithms.  We  use  Probability  Match(cid:173) ing to train mixture of experts networks,  an architecture for  which  other reinforcement  learning rules fail  to converge reliably on even  simple  problems.  This  architecture  is  particularly  well  suited  for  our  algorithm as it can compute arbitrarily complex functions  yet  calculation of the output probability is  simple.
Generalized Learning Vector Quantization
Authors: Keiji Yamada Atsushi Sato
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  propose  a  new  learning  method,  "Generalized  Learning  Vec(cid:173) tor Quantization (GLVQ),"  in which reference vectors are updated  based on the steepest descent method in order to minimize the cost  function .  The  cost  function  is  determined  so  that  the  obtained  learning  rule  satisfies  the  convergence  condition.  We  prove  that  Kohonen's  rule  as  used  in  LVQ  does  not  satisfy  the  convergence  condition  and  thus  degrades  recognition  ability.  Experimental re(cid:173) sults  for  printed  Chinese  character recognition  reveal  that  GLVQ  is  superior to  LVQ  in  recognition  ability.
Bayesian Methods for Mixtures of Experts
Authors: David MacKay, Anthony J. Robinson Steve R. Waterhouse
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: We  present  a  Bayesian framework  for  inferring  the  parameters of  a  mixture of experts  model based  on  ensemble  learning  by  varia(cid:173) tional free energy minimisation.  The Bayesian approach avoids the  over-fitting and noise level under-estimation problems of traditional  maximum likelihood inference.  We  demonstrate these  methods on  artificial  problems and sunspot  time series  prediction.
High-Speed Airborne Particle Monitoring Using Artificial Neural Networks
Authors: Theo Sabisch, Paul Kaye, Laurence C. Dixon, Hamid Bolouri Alistair Ferguson
Published in: Advances in Neural Information Processing Systems 8  (NIPS 1995) (Main Conference Track), 1995

Abstract: Current environmental monitoring systems assume particles to be  spherical, and do not attempt to classify them. A laser-based sys(cid:173) tem developed at the University of Hertfordshire aims at classify(cid:173) ing airborne particles through the generation of two-dimensional  scattering profiles. The pedormances of template matching, and  two types of neural network (HyperNet and semi-linear units) are  compared for image classification. The neural network approach is  shown to be capable of comparable recognition pedormance, while  offering a number of advantages over template matching.