mtybilly/Instruct2DS huggingface dataset:
Bayesian Model Comparison by Monte Carlo Chaining
Authors: David Barber, Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The techniques of Bayesian inference  have been  applied with great  success to many problems in neural computing including evaluation  of regression  functions,  determination of error bars on predictions,  and  the  treatment of hyper-parameters.  However,  the  problem of  model comparison is a much more challenging one for which current  techniques  have significant limitations.  In this paper we  show  how  an  extended  form  of Markov  chain  Monte  Carlo,  called  chaining,  is  able  to provide effective  estimates of the relative probabilities of  different  models.  We  present  results  from  the  robot  arm problem  and  compare  them  with  the  corresponding  results  obtained  using  the standard  Gaussian approximation framework. 
1  Bayesian Model  Comparison 
In a Bayesian treatment of statistical inference, our state of knowledge of the values  of the parameters w  in a model M  is described in terms of a probability distribution  function.  Initially this  is  chosen  to be  some prior distribution p(wIM), which  can  be  combined with a  likelihood function  p( Dlw, M) using Bayes'  theorem  to give  a  posterior  distribution p(wID, M) in  the form
Bayesian Unsupervised Learning of Higher Order Structure
Authors: Michael S. Lewicki, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Multilayer architectures such  as  those used in Bayesian belief net(cid:173) works  and  Helmholtz  machines  provide  a  powerful framework  for  representing  and learning higher  order statistical relations  among  inputs.  Because  exact  probability  calculations  with  these  mod(cid:173) els  are often intractable, there is  much interest in finding  approxi(cid:173) mate algorithms.  We present an algorithm that efficiently discovers  higher  order structure using EM  and Gibbs  sampling.  The model  can be interpreted as a stochastic recurrent network in which ambi(cid:173) guity in lower-level states is resolved through feedback from higher  levels.  We demonstrate the performance of the algorithm on bench(cid:173) mark problems.
An Architectural Mechanism for Direction-tuned Cortical Simple Cells: The Role of Mutual Inhibition
Authors: Silvio P. Sabatini, Fabio Solari, Giacomo M. Bisio
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  linear  architectural  model  of cortical  simple cells  is  presented.  The  model  evidences  how  mutual  inhibition,  occurring  through  synaptic  coupling  functions  asymmetrically  distributed  in  space,  can be a possible basis for  a wide variety of spatio-temporal simple  cell  response properties, including direction selectivity and velocity  tuning.  While  spatial  asymmetries  are  included  explicitly  in  the  structure of the inhibitory interconnections, temporal asymmetries  originate  from  the  specific  mutual  inhibition  scheme  considered.  Extensive simulations supporting the model are reported.
Complex-Cell Responses Derived from Center-Surround Inputs: The Surprising Power of Intradendritic Computation
Authors: Bartlett W. Mel, Daniel L. Ruderman, Kevin A. Archie
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Biophysical  modeling  studies  have  previously  shown  that  cortical  pyramidal  cells  driven  by  strong  NMDA-type  synaptic  currents  and/or containing dendritic voltage-dependent Ca++ or Na+  chan(cid:173) nels, respond more strongly when synapses are activated in several  spatially  clustered  groups  of  optimal  size-in  comparison  to  the  same  number  of synapses  activated  diffusely  about  the  dendritic  arbor  [8]- The  nonlinear  intradendritic  interactions  giving  rise  to  this  "cluster sensitivity"  property are akin to a layer of virtual non(cid:173) linear "hidden units" in the dendrites, with implications for the cel(cid:173) lular basis of learning and memory  [7,  6],  and for  certain classes of  nonlinear sensory processing [8]- In the present study, we show that  a  single  neuron,  with  access  only  to excitatory inputs  from  unori(cid:173) ented ON- and OFF-center cells in the LGN, exhibits the principal  nonlinear response properties of a  "complex"  cell in primary visual  cortex,  namely orientation tuning  coupled with  translation invari(cid:173) ance  and  contrast  insensitivity_  We  conjecture  that  this  type  of  intradendritic processing could explain how complex cell responses  can persist in the absence of oriented simple cell input  [13]-
84 
B.  W.  Mel,  D.  L.  Ruderman and K.  A. Archie
A Constructive Learning Algorithm for Discriminant Tangent Models
Authors: Diego Sona, Alessandro Sperduti, Antonina Starita
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: (HSS)  developed  an  algo(cid:173)
To  reduce  the  computational complexity of classification systems  using  tangent  distance,  Hastie  et  al.  rithm  to  devise  rich  models  for  representing  large  subsets  of the  data  which  computes  automatically  the  "best"  associated  tan(cid:173) gent subspace.  Schwenk  &  Milgram proposed  a  discriminant mod(cid:173) ular classification system (Diabolo)  based on several autoassociative  multilayer perceptrons  which  use  tangent  distance  as  error  recon(cid:173) struction  measure.  We  propose  a  gradient  based  constructive  learning  algorithm for  building  a  tangent  subspace  model  with  discriminant  capabilities  which  combines  several  of the  the  advantages  of both  HSS  and  Diabolo:  devised  tangent  models  hold  discriminant  capabilities,  space  requirements  are  improved  with  respect  to  HSS  since  our  algorithm is discriminant and thus it needs fewer prototype models,  dimension of the tangent subspace is  determined automatically by  the  constructive algorithm, and our  algorithm is  able to learn new  transformations.
Effective Training of a Neural Network Character Classifier for Word Recognition
Authors: Larry S. Yaeger, Richard F. Lyon, Brandyn J. Webb
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  combined  an  artificial  neural  network  (ANN)  character  classifier with context-driven search over character segmentation, word  segmentation,  and  word  recognition  hypotheses  to  provide  robust  recognition  of hand-printed  English  text  in  new  models  of  Apple  Computer's Newton MessagePad.  We present some innovations in  the  training and use of ANNs al;  character classifiers for  word recognition,  including normalized output error, frequency balancing, error emphasis,  negative training, and stroke warping.  A recurring theme of reducing a  priori biases emerges and is discussed.
Second-order Learning Algorithm with Squared Penalty Term
Authors: Kazumi Saito, Ryohei Nakano
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This  paper compares three penalty terms  with respect to the effi(cid:173) ciency of supervised learning, by using first- and second-order learn(cid:173) ing algorithms.  Our experiments showed that for  a reasonably ade(cid:173) quate penalty factor,  the combination of the squared penalty term  and  the second-order  learning  algorithm  drastically improves  the  convergence  performance more than 20  times  over  the other  com(cid:173) binations, at the same time bringing about a better generalization  performance.
Multi-effect Decompositions for Financial Data Modeling
Authors: Lizhong Wu, John E. Moody
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: High  frequency  foreign  exchange  data  can  be  decomposed  into  three  components:  the  inventory effect component,  the  surprise infonnation  (news) component and the regular infonnation component. The presence  of the inventory effect and news can make analysis of trends due to  the  diffusion of infonnation (regular information component) difficult.  We propose a neural-net-based, independent component analysis to sep(cid:173) arate high frequency foreign exchange data into these three components.  Our empirical results show that our proposed multi-effect decomposition  can reveal the intrinsic price behavior.
Dynamic Features for Visual Speechreading: A Systematic Comparison
Authors: Michael S. Gray, Javier R. Movellan, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Humans  use  visual as  well  as  auditory speech signals to recognize  spoken words.  A variety of systems have been investigated for per(cid:173) forming  this  task.  The main  purpose of this  research  was  to sys(cid:173) tematically compare the performance of a range of dynamic visual  features  on  a  speechreading  task.  We  have  found  that  normal(cid:173) ization  of images  to eliminate  variation  due  to  translation,  scale,  and planar rotation  yielded  substantial  improvements  in  general(cid:173) ization performance regardless of the visual representation used.  In  addition,  the dynamic  information  in  the difference  between  suc(cid:173) cessive  frames  yielded  better performance than optical-flow  based  approaches, and compression by local low-pass filtering worked sur(cid:173) prisingly better than global principal components analysis  (PCA).  These results are examined and possible explanations are explored.
The Effect of Correlated Input Data on the Dynamics of Learning
Authors: Søren Halkjær, Ole Winther
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The convergence properties of the gradient descent  algorithm in the  case  of the  linear  perceptron  may  be  obtained  from  the  response  function.  We  derive  a general expression for  the  response  function  and  apply it to the  case  of data with simple input  correlations.  It  is  found  that  correlations severely  may slow  down  learning.  This  explains the success of PCA as a method for reducing training time.  Motivated by this finding  we  furthermore propose to transform the  input  data by  removing the mean across  input variables as  well  as  examples to decrease correlations.  Numerical findings for  a medical  classification  problem  are  in  fine  agreement  with  the  theoretical  results.
Spectroscopic Detection of Cervical Pre-Cancer through Radial Basis Function Networks
Authors: Kagan Tumer, Nirmala Ramanujam, Rebecca R. Richards-Kortum, Joydeep Ghosh
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  mortality  related  to cervical  cancer  can  be  substantially  re(cid:173) duced  through  early  detection  and  treatment.  However,  cur(cid:173) rent  detection  techniques,  such  as  Pap  smear  and  colposcopy,  fail  to  achieve  a  concurrently  high  sensitivity  and  specificity.  In  vivo  fluorescence  spectroscopy  is  a  technique  which  quickly,  non(cid:173) invasively and quantitatively probes the biochemical and morpho(cid:173) logical  changes that occur in pre-cancerous tissue.  RBF ensemble  algorithms based on such spectra provide automated, and near real(cid:173) time implementation of pre-cancer  detection in the  hands of non(cid:173) experts.  The  results  are  more  reliable,  direct  and  accurate  than  those achieved  by  either human experts or multivariate statistical  algorithms.
Radial Basis Function Networks and Complexity Regularization in Function Learning
Authors: Adam Krzyzak, Tamás Linder
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In this paper we apply the method of complexity regularization to de(cid:173) rive estimation bounds for nonlinear function estimation using a single hidden layer radial basis function network. Our approach differs from the previous complexity regularization neural network function learning schemes in that we operate with random covering numbers and 11 metric entropy, making it po~sibleto consider much broader families of activa(cid:173) tion functions, namely functions of bounded variation. Some constraints previously imposed on the network parameters are also eliminated this way. The network is trained by means of complexity regularization in(cid:173) volving empirical risk minimization. Bounds on the expected risk in tenns of the sample size are obtained for a large class of loss functions. Rates of convergence to the optimal loss are also derived.
Adaptively Growing Hierarchical Mixtures of Experts
Authors: Jürgen Fritsch, Michael Finke, Alex Waibel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We propose a novel approach to automatically growing and pruning  Hierarchical  Mixtures  of Experts.  The constructive  algorithm pro(cid:173) posed  here  enables  large  hierarchies  consisting  of several  hundred  experts  to be  trained effectively.  We  show  that HME's  trained  by  our  automatic  growing  procedure  yield  better  generalization  per(cid:173) formance  than  traditional  static  and  balanced  hierarchies.  Eval(cid:173) uation  of  the  algorithm  is  performed  (1)  on  vowel  classification  and  (2)  within  a  hybrid  version  of the  JANUS  r9]  speech  recog(cid:173) nition system  using a  subset  of the Switchboard large-vocabulary  speaker-independent  continuous speech  recognition database.
On-line Policy Improvement using Monte-Carlo Search
Authors: Gerald Tesauro, Gregory R. Galperin
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We present a Monte-Carlo simulation algorithm for  real-time policy  improvement  of an  adaptive  controller.  In  the  Monte-Carlo  sim(cid:173) ulation,  the  long-term expected  reward  of each  possible  action  is  statistically measured, using the initial policy to make decisions  in  each  step of the simulation.  The action maximizing the  measured  expected reward is then taken, resulting in an improved policy.  Our  algorithm is easily parallelizable and has been implemented on the  IBM  SP! and SP2 parallel-RISC supercomputers.  We  have  obtained  promising  initial  results  in  applying  this  algo(cid:173) rithm  to the  domain of backgammon.  Results  are  reported  for  a  wide  variety  of initial  policies,  ranging  from  a  random  policy  to  TD-Gammon, an extremely strong multi-layer neural  network.  In  each case,  the Monte-Carlo algorithm gives a substantial reduction,  by  as  much as  a  factor  of 5 or  more,  in  the error  rate  of the  base  players.  The  algorithm  is  also  potentially  useful  in  many  other  adaptive control applications in which it is possible to simulate the  environment.
Blind Separation of Delayed and Convolved Sources
Authors: Te-Won Lee, Anthony J. Bell, Russell H. Lambert
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  address  the  difficult  problem  of separating multiple  speakers  with  multiple microphones  in  a  real room.  We  combine the work  of Torkkola and  Amari,  Cichocki  and Yang,  to give  Natural  Gra(cid:173) dient information maximisation rules for recurrent (IIR)  networks,  blindly  adjusting  delays,  separating  and  deconvolving  mixed  sig(cid:173) nals.  While  they  work  well  on  simulated  data,  these  rules  fail  in  real  rooms  which  usually  involve  non-minimum phase  transfer  functions,  not-invertible using stable IIR filters.  An approach that  sidesteps this problem is to perform infomax on a feedforward archi(cid:173) tecture in the frequency domain  (Lambert 1996).  We  demonstrate  real-room separation of two natural signals using this approach.
A New Approach to Hybrid HMM/ANN Speech Recognition using Mutual Information Neural Networks
Authors: Gerhard Rigoll, Christoph Neukirchen
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper presents a new approach to speech recognition with hybrid  HMM/ANN  technology.  While  the  standard  approach  to  hybrid  HMMI ANN  systems  is  based  on  the  use  of  neural  networks  as  posterior probability estimators, the new approach is based on the use  of mutual information neural  networks trained with a special learning  algorithm  in  order to  maximize the  mutual  information  between  the  input classes of the network and its resulting sequence of firing output  neurons  during  training.  It is  shown  in  this  paper that  such  a  neural  network  is  an  optimal  neural  vector quantizer for  a  discrete  hidden  Markov  model  system  trained  on  Maximum  Likelihood  principles.  One  of the  main  advantages  of this  approach  is  the  fact,  that  such  neural  networks  can  be  easily  combined  with  HMM's  of  any  complexity  with context-dependent capabilities.  It is  shown  that  the  resulting  hybrid  system  achieves  very  high  recognition  rates,  which  are  now  already  on  the  same  level  as  the  best  conventional  HMM  systems  with  continuous  parameters,  and  the  capabilities  of  the  mutual information neural networks are not yet entirely exploited.
Competition Among Networks Improves Committee Performance
Authors: Paul W. Munro, Bambang Parmanto
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The separation of generalization error into two types, bias and variance  (Geman,  Bienenstock,  Doursat,  1992),  leads  to  the  notion  of error  reduction  by  averaging  over  a  "committee"  of classifiers  (Perrone,  1993).  Committee perfonnance decreases with both the average error of  the constituent classifiers and increases  with  the degree to  which the  misclassifications are correlated across the committee.  Here, a method  for  reducing  correlations  is  introduced,  that  uses  a  winner-take-all  procedure  similar  to  competitive  learning  to  drive  the  individual  networks  to  different  minima  in  weight  space  with  respect  to  the  training set, such that correlations in  generalization perfonnance will be  reduced, thereby reducing committee error.
Selective Integration: A Model for Disparity Estimation
Authors: Michael S. Gray, Alexandre Pouget, Richard S. Zemel, Steven J. Nowlan, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Local disparity information is often sparse and noisy, which creates  two conflicting demands when estimating disparity in an image re(cid:173) gion:  the need to spatially average to get an accurate estimate, and  the problem of not averaging over discontinuities.  We  have devel(cid:173) oped a  network model of disparity estimation based on disparity(cid:173) selective neurons, such as those found in the early stages of process(cid:173) ing  in  visual cortex.  The model  can accurately estimate multiple  disparities in a region, which may be caused by transparency or oc(cid:173) clusion,  in real images and random-dot stereograms.  The use of a  selection mechanism to selectively integrate reliable local disparity  estimates  results  in  superior  performance  compared  to  standard  back-propagation  and  cross-correlation  approaches.  In  addition,  the representations learned with this selection mechanism are con(cid:173) sistent  with  recent  neurophysiological  results  of  von  der  Heydt,  Zhou, Friedman, and Poggio [8]  for cells in cortical visual area V2.  Combining multi-scale biologically-plausible image processing with  the power of the mixture-of-experts learning algorithm represents  a  promising approach that yields both high  performance and new  insights into visual system function. 
Selective Integration: A Model for Disparity Estimation
The Neurothermostat: Predictive Optimal Control of Residential Heating Systems
Authors: Michael Mozer, Lucky Vidmar, Robert H. Dodier
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Neurothermostat  is  an  adaptive  controller  that  regulates  in(cid:173) door  air  temperature  in  a  residence  by  switching  a  furnace  on  or  off.  The  task  is  framed  as  an  optimal control  problem  in  which  both  comfort  and  energy  costs  are  considered  as  part of the  con(cid:173) trol  objective.  Because  the  consequences  of control  decisions  are  delayed  in  time, the  N eurothermostat  must  anticipate heating de(cid:173) mands with predictive models of occupancy  patterns and the ther(cid:173) mal response of the house and furnace.  Occupancy  pattern predic(cid:173) tion  is  achieved  by  a  hybrid  neural  net  /  look-up table.  The  Neu(cid:173) rothermostat  searches,  at  each  discrete  time  step,  for  a  decision  sequence  that  minimizes  the  expected  cost  over  a  fixed  planning  horizon.  The first  decision  in this sequence  is  taken,  and  this  pro(cid:173) cess  repeats.  Simulations of the  Neurothermostat  were  conducted  using  artificial  occupancy  data  in  which  regularity  was  systemat(cid:173) ically  varied,  as  well  as  occupancy  data from  an  actual  residence.  The Neurothermostat is  compared against three  conventional poli(cid:173) cies,  and  achieves  reliably  lower  costs.  This result  is  robust  to the  relative  weighting of comfort  and  energy  costs  and  the  degree  of  variability in the occupancy  patterns. 
For over a  quarter  century,  the home automation industry has  promised  to revolu(cid:173) tionize our lifestyle with the so-called  Smart House@  in which appliances,  lighting,  stereo,  video,  and  security  systems  are  integrated  under  computer  control.  How(cid:173) ever,  home automation has yet  to make significant inroads, at least in  part because  software  must be tailored  to the  home occupants.  Instead  of expecting  the occupants  to program their  homes or  to hire  someone  to  do  so,  one  would  ideally like  the  home  to  essentially  program  itself  by  observing  the lifestyle of the occupants.  This is the goal of the  Neural  Network  House  (Mozer  et  al.,  1995),  an  actual  residence  that  has  been  outfitted  with  over  75  sensors(cid:173) including temperature,  light, sound,  motion-and actua.tors  to control air heating,  water  heating,  lighting,  and  ventilation.  In  this  paper,  we  describe  one  research 
954 
M.  C.  Mozer. L.  Vidmar and R.  H.  Dodier 
project  within  the  house,  the  Neurothermostat,  that  learns  to  regulate  the  indoor  air temperature automatically by observing and detecting patterns in the occupants'  schedules  and  comfort  preferences.  We  focus  on  the  problem  of air  heating  with  a  whole-house  furnace,  but  the  same  approach  can  be  taken  with  alternative  or  multiple heating devices,  and to the  problems of cooling and ventilation. 
1  TEMPERATURE REGULATION AS  AN OPTIMAL
Softening Discrete Relaxation
Authors: Andrew M. Finch, Richard C. Wilson, Edwin R. Hancock
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes a  new framework for  relational graph match(cid:173) ing.  The starting point is  a recently reported Bayesian consistency  measure  which  gauges  structural  differences  using  Hamming  dis(cid:173) tance.  The main  contributions of the  work  are threefold.  Firstly,  we  demonstrate  how  the  discrete  components  of  the  cost  func(cid:173) tion  can  be  softened.  The  second  contribution  is  to  show  how  the  softened  cost  function  can  be  used  to  locate  matches  using  continuous non-linear optimisation.  Finally,  we  show how  the res(cid:173) ulting graph matching algorithm relates to the standard quadratic  assignment problem.
A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer
Authors: Tony Plate, Pierre Band, Joel Bert, John Grace
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Epidemiological  data  is  traditionally  analyzed  with  very  simple  techniques.  Flexible  models,  such  as  neural  networks,  have  the  potential to discover unanticipated features in the data.  However,  to be useful,  flexible  models  must have effective control on overfit(cid:173) ting.  This paper reports on a  comparative study of the predictive  quality of neural networks and other flexible models applied to real  and artificial epidemiological data.  The results  suggest that there  are no major unanticipated complex features  in the real data, and  also  demonstrate  that  MacKay's  [1995]  Bayesian  neural  network  methodology  provides effective  control on overfitting while  retain(cid:173) ing the ability  to discover complex features  in the artificial data.
An Apobayesian Relative of Winnow
Authors: Nick Littlestone, Chris Mesterharm
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  study  a  mistake-driven  variant  of an  on-line  Bayesian  learn(cid:173) ing  algorithm  (similar  to one  studied  by  Cesa-Bianchi,  Helmbold,  and Panizza [CHP96]).  This variant only updates its state (learns)  on trials in which it makes  a mistake.  The algorithm makes binary  classifications using a linear-threshold classifier and runs in time lin(cid:173) ear in  the  number of attributes seen by the learner.  We  have  been  able  to  show,  theoretically  and  in  simulations,  that  this  algorithm  performs well  under assumptions quite different from  those embod(cid:173) ied  in  the  prior of the  original  Bayesian algorithm.  It can handle  situations that  we  do  not  know  how  to handle  in  linear  time  with  Bayesian  algorithms.  We  expect  our  techniques  to  be  useful  in  deriving and analyzing other apobayesian algorithms.
LSTM can Solve Hard Long Time Lag Problems
Authors: Sepp Hochreiter, Jürgen Schmidhuber
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Standard recurrent nets cannot deal with long minimal time lags between relevant signals. Several recent NIPS papers propose alter(cid:173) native methods. We first show: problems used to promote various previous algorithms can be solved more quickly by random weight guessing than by the proposed algorithms. We then use LSTM, our own recent algorithm, to solve a hard problem that can neither be quickly solved by random search nor by any other recurrent net algorithm we are aware of.
1 TRIVIAL PREVIOUS LONG TIME LAG PROBLEMS
Traditional recurrent nets fail in case 'of long minimal time lags between input sig(cid:173) nals and corresponding error signals [7, 3]. Many recent papers propose alternative methods, e.g., [16, 12, 1,5,9]. For instance, Bengio et ale investigate methods such as simulated annealing, multi-grid random search, time-weighted pseudo-Newton optimization, and discrete error propagation [3]. They also propose. an EM ap(cid:173) proach [1]. Quite a few papers use variants of the "2-sequence problem" (and ('latch problem" ) to show the proposed algorithm's superiority, e.g. [3, 1, 5, 9]. Some pa(cid:173) pers also use the "parity problem", e.g., [3, 1]. Some of Tomita's [18] grammars are also often used as benchmark problems for recurrent nets [2, 19, 14, 11].
Trivial versus non-trivial tasks. By our definition, a "trivial" task is one that can be solved quickly by random search (RS) in weight space. RS works as follows: REPEAT randomly initialize the weights and test the resulting net on a training set UNTIL solution found.
488 Solutions to the XOR Problem
Authors: Frans Coetzee, Virginia L. Stonick
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  globally convergent homotopy method is  defined  that is capable  of sequentially producing large numbers of stationary points of the  multi-layer perceptron  mean-squared error  surface.  Using  this  al(cid:173) gorithm large subsets of the stationary points of two test problems  are  found.  It is  shown  empirically that  the  MLP  neural  network  appears  to  have  an  extreme  ratio  of saddle  points  compared  to  local  minima,  and  that  even  small neural  network  problems  have  extremely large numbers of solutions.
A Mixture of Experts Classifier with Learning Based on Both Labelled and Unlabelled Data
Authors: David J. Miller, Hasan S. Uyar
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  address  statistical  classifier  design  given  a  mixed  training  set  con(cid:173) sisting  of  a  small  labelled  feature  set  and  a  (generally  larger)  set  of  unlabelled features.  This situation  arises,  e.g., for  medical images,  where  although  training  features  may  be  plentiful,  expensive  expertise  is  re(cid:173) quired  to  extract  their  class  labels.  We  propose  a  classifier  structure  and learning  algorithm  that make effective  use of unlabelled data to im(cid:173) prove performance.  The learning  is  based  on  maximization  of the  total  data  likelihood,  i.e.  over  both  the  labelled  and  unlabelled  data  sub(cid:173) sets.  Two distinct EM learning  algorithms  are proposed,  differing  in  the  EM  formalism  applied  for  unlabelled  data.  The classifier,  based  on  a  joint probability model for features  and labels,  is  a  "mixture of experts"  structure that is  equivalent to the  radial  basis function  (RBF) classifier,  but unlike RBFs, is  amenable  to likelihood-based training.  The scope of  application  for  the  new  method  is  greatly  extended  by  the  observation  that test data, or any new data to classify, is in fact  additional,  unlabelled  data - thus,  a combined learning/classification operation - much akin  to  what  is  done  in  image  segmentation  - can  be  invoked  whenever  there  is  new  data to classify.  Experiments  with  data sets from  the  UC  Irvine  database  demonstrate  that  the  new  learning  algorithms  and  structure  achieve substantial performance  gains  over alternative  approaches.
Statistical Mechanics of the Mixture of Experts
Authors: Kukjin Kang, Jong-Hoon Oh
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We study generalization capability of the mixture of experts  learn(cid:173) ing  from  examples  generated  by  another  network  with  the  same  architecture.  When the number of examples is smaller than  a crit(cid:173) ical  value,  the  network  shows  a  symmetric  phase  where  the  role  of the experts  is  not specialized.  Upon  crossing  the critical  point,  the  system  undergoes  a  continuous  phase  transition  to  a  symme(cid:173) try  breaking  phase  where  the gating network  partitions  the  input  space effectively  and each expert is assigned  to an appropriate sub(cid:173) space.  We  also find  that the mixture of experts  with multiple level  of hierarchy shows  multiple phase transitions.
Reinforcement Learning for Mixed Open-loop and Closed-loop Control
Authors: Eric A. Hansen, Andrew G. Barto, Shlomo Zilberstein
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Closed-loop  control  relies  on  sensory  feedback  that  is  usually  as(cid:173) sumed  to  be  free .  But  if sensing  incurs  a  cost,  it  may  be  cost(cid:173) effective  to  take  sequences  of actions  in  open-loop  mode.  We  de(cid:173) scribe  a  reinforcement  learning  algorithm  that  learns  to  combine  open-loop  and  closed-loop  control  when  sensing  incurs  a  cost.  Al(cid:173) though  we  assume reliable  sensors,  use  of open-loop control means  that  actions  must  sometimes  be  taken  when  the  current  state  of  the  controlled  system  is  uncertain.  This  is  a  special  case  of the  hidden-state  problem  in  reinforcement  learning,  and  to  cope,  our  algorithm relies  on short-term memory.  The main result  of the pa(cid:173) per is a rule that significantly limits exploration of possible  memory  states  by  pruning  memory states  for  which  the  estimated  value  of  information is  greater than its cost.  We  prove that  this  rule  allows  convergence  to an optimal  policy.
Regression with Input-Dependent Noise: A Bayesian Treatment
Authors: Christopher M. Bishop, Cazhaow S. Quazaz
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  most  treatments  of the  regression  problem  it  is  assumed  that  the distribution of target data can be described by a  deterministic  function  of the inputs, together with additive Gaussian noise hav(cid:173) ing constant variance.  The use of maximum likelihood to train such  models then  corresponds to the minimization of a  sum-of-squares  error function.  In  many applications a more realistic model would  allow  the  noise  variance  itself to  depend  on  the  input  variables.  However, the use of maximum likelihood to train such models would  give highly biased  results.  In  this paper we  show  how  a  Bayesian  treatment  can  allow  for  an  input-dependent  variance  while  over(cid:173) coming the bias of maximum likelihood.
The Generalisation Cost of RAMnets
Authors: Richard Rohwer, Michal Morciniec
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Given unlimited computational resources,  it is  best  to use  a  crite(cid:173) rion of minimal expected generalisation error to select  a model and  determine its  parameters.  However,  it may be  worthwhile to sac(cid:173) rifice  some  generalisation  performance  for  higher  learning  speed.  A  method  for  quantifying sub-optimality is  set  out  here,  so  that  this  choice  can  be  made  intelligently.  Furthermore,  the  method  is  applicable  to  a  broad  class  of models,  including  the  ultra-fast  memory-based methods such  as  RAMnets.  This brings the added  benefit  of providing,  for  the  first  time,  the  means  to  analyse  the  generalisation  properties of such  models in a  Bayesian framework .
Multilayer Neural Networks: One or Two Hidden Layers?
Authors: Graham Brightwell, Claire Kenyon, Hélène Paugam-Moisy
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We study the number of hidden layers required by a multilayer neu(cid:173) ral network with threshold  units to compute a function  f  from n d  to {O, I}.  In dimension d  =  2,  Gibson  characterized  the functions  computable with just one hidden layer, under  the assumption that  there  is no  "multiple intersection  point"  and that f  is only defined  on a compact set.  We consider the restriction of f  to the neighbor(cid:173) hood of a  multiple intersection  point or of infinity,  and give  neces(cid:173) sary  and sufficient  conditions for  it  to  be  locally computable with  one  hidden  layer.  We  show  that  adding  these  conditions  to  Gib(cid:173) son's  assumptions  is  not  sufficient  to  ensure  global  computability  with one hidden layer,  by exhibiting a new  non-local configuration,  the  "critical cycle",  which  implies  that f  is  not  computable  with  one  hidden  layer.
Improving the Accuracy and Speed of Support Vector Machines
Authors: Christopher J. C. Burges, Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Support  Vector  Learning  Machines  (SVM)  are  finding  application  in  pattern  recognition,  regression  estimation , and  operator inver(cid:173) sion  for  ill-posed  problems.  Against  this  very  general  backdrop ,  any methods for  improving the  generalization  performance, or for  improving  the  speed  in  test  phase,  of SVMs  are  of increasing  in(cid:173) terest.  In this paper we  combine two such  techniques on  a  pattern  recognition problem. The method for improving generalization per(cid:173) formance  (the  "virtual support  vector"  method)  does  so  by  incor(cid:173) porating known invariances of the problem.  This method achieves  a  drop  in  the error  rate on  10,000 NIST  test  digit images of 1.4%  to  1.0%.  The method for  improving the speed  (the  "reduced  set"  method) does so by approximating the support vector decision sur(cid:173) face.  We  apply  this method to achieve  a  factor  of fifty  speedup  in  test phase over  the  virtual support vector machine.  The combined  approach  yields a  machine which  is  both  22  times faster  than  the  original machine, and which has better generalization performance,  achieving  1.1 % error.  The virtual support vector  method is  appli(cid:173) cable  to any SVM  problem  with  known  invariances.  The  reduced  set  method is  applicable to any support  vector machine.
An Adaptive WTA using Floating Gate Technology
Authors: W. Fritz Kruger, Paul E. Hasler, Bradley A. Minch, Christof Koch
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  designed,  fabricated,  and  tested  an  adaptive  Winner(cid:173) Take-All  (WTA)  circuit  based  upon  the  classic  WTA  of Lazzaro,  et  al  [IJ.  We  have  added  a  time  dimension  (adaptation)  to  this  circuit to make the input derivative an important factor in winner  selection.  To  accomplish  this,  we  have  modified  the  classic  WTA  circuit  by  adding floating  gate transistors which  slowly  null  their  inputs over time.  We  present a  simplified analysis and experimen(cid:173) tal data of this adaptive WTA fabricated in a standard CMOS 2f.tm  process. 
1  Winner-Take-All Circuits 
In a WTA network, each cell has one input and one output.  For any set of inputs, the  outputs will all be at zero except for the one which is from the cell with the maximum  input.  One way to accomplish this is by a global nonlinear inhibition coupled with a  self-excitation term [2J.  Each cell  inhibits all others while exciting itself;  thus a cell  with even a slightly greater input than the others will excite itself up to its maximal  state  and  inhibit  the  others  down  to  their  minimal  states.  The  WTA  function  is  important for  many classical neural  nets that involve  competitive  learning,  vector  quantization  and  feature  mapping.  The  classic  WTA  network  characterized  by  Lazzaro et.  al.  [IJ  is  an  elegant,  simple  circuit  that shares just one  common  line  among all  cells of the network to propagate the inhibition. 
Our motivation  to  add  adaptation  comes  from  the  idea of saliency  maps.  Picture  a  saliency  map  as  a  large  number  of cells  each  of which  encodes  an  analog  value 
An Adaptive wrA using Floating Gate Technology
Representation and Induction of Finite State Machines using Time-Delay Neural Networks
Authors: Daniel S. Clouse, C. Lee Giles, Bill G. Horne, Garrison W. Cottrell
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This work investigates the representational and inductive capabili(cid:173) ties of time-delay neural networks (TDNNs)  in general, and of two  subclasses of TDNN, those  with delays only on the inputs (IDNN),  and those which  include delays on hidden  units (HDNN) . Both ar(cid:173) chitectures are capable of representing  the same class of languages,  the  definite  memory machine (DMM)  languages, but the delays on  the  hidden  units  in  the  HDNN  helps  it  outperform  the  IDNN  on  problems composed of repeated  features  over short  time windows.
Probabilistic Interpretation of Population Codes
Authors: Richard S. Zemel, Peter Dayan, Alexandre Pouget
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  present  a  theoretical  framework  for  population  codes  which  generalizes  naturally  to  the  important  case  where  the  population  provides  information  about  a  whole  probability  distribution  over  an  underlying  quantity  rather  than  just  a  single  value.  We  use  the framework  to analyze two existing models,  and to suggest and  evaluate a  third model for  encoding such probability distributions.
Analog VLSI Circuits for Attention-Based, Visual Tracking
Authors: Timothy K. Horiuchi, Tonia G. Morris, Christof Koch, Stephen P. DeWeerth
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A one-dimensional visual tracking chip has been implemented us(cid:173) ing neuromorphic, analog VLSI techniques to model selective visual  attention in the control of saccadic and smooth pursuit eye move(cid:173) ments. The chip incorporates focal-plane processing to compute  image saliency and a winner-take-all circuit to select a feature for  tracking. The target position and direction of motion are reported  as the target moves across the array. We demonstrate its function(cid:173) ality in a closed-loop system which performs saccadic and smooth  pursuit tracking movements using a one-dimensional mechanical  eye.
Online Learning from Finite Training Sets: An Analytical Case Study
Authors: Peter Sollich, David Barber
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  analyse  online  learning  from  finite  training  sets  at  non(cid:173) infinitesimal  learning  rates  TJ.  By  an  extension  of statistical  me(cid:173) chanics  methods,  we  obtain  exact  results  for  the  time-dependent  generalization  error  of  a  linear  network  with  a  large  number  of  weights  N.  We  find,  for  example,  that  for  small training  sets  of  size  p  ~ N,  larger  learning rates  can  be  used  without compromis(cid:173) ing  asymptotic  generalization  performance  or  convergence  speed.  Encouragingly,  for  optimal  settings  of  TJ  (and,  less  importantly,  weight decay ,)  at given final  learning time, the generalization per(cid:173) formance of online learning is  essentially  as good as  that of offline  learning.
ARC-LH: A New Adaptive Resampling Algorithm for Improving ANN Classifiers
Authors: Friedrich Leisch, Kurt Hornik
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We introduce arc-Ih,  a new algorithm for improvement of ANN clas(cid:173) sifier  performance,  which  measures  the  importance of patterns  by  aggregated network output errors.  On several  artificial benchmark  problems,  this  algorithm compares favorably  with other  resample  and combine techniques.
Monotonicity Hints
Authors: Joseph Sill, Yaser S. Abu-Mostafa
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A hint is any piece of side information about the target function to  be  learned.  We  consider  the monotonicity hint,  which  states  that  the function  to be learned  is  monotonic in some or all of the input  variables.  The  application of mono tonicity  hints  is  demonstrated  on  two  real-world  problems- a  credit  card  application task,  and  a  problem in medical diagnosis.  A measure of the monotonicity error  of a candidate function is defined and an objective function for  the  enforcement  of monotonicity is  derived  from  Bayesian  principles.  We report experimental results which show that using monotonicity  hints leads to a statistically significant improvement in performance  on  both problems.
A Convergence Proof for the Softassign Quadratic Assignment Algorithm
Authors: Anand Rangarajan, Alan L. Yuille, Steven Gold, Eric Mjolsness
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  softassign  quadratic  assignment  algorithm  has  recently  emerged as an effective strategy for  a variety of optimization prob(cid:173) lems in pattern recognition and combinatorial optimization.  While  the  effectiveness  of the algorithm  was  demonstrated  in  thousands  of simulations,  there  was  no  known  proof of convergence.  Here,  we  provide a  proof of convergence for  the most general form of the  algorithm.
Size of Multilayer Networks for Exact Learning: Analytic Approach
Authors: André Elisseeff, Hélène Paugam-Moisy
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This  article  presents  a  new  result  about  the  size  of a  multilayer  neural network computing real outputs for exact learning of a finite  set of real samples.  The architecture of the network is feedforward,  with  one  hidden  layer  and several  outputs.  Starting from  a  fixed  training set,  we  consider  the  network  as  a  function  of its  weights.  We  derive,  for  a  wide family  of transfer  functions,  a  lower  and  an  upper  bound  on  the  number  of hidden  units  for  exact  learning,  given  the size  of the  dataset  and  the  dimensions of the  input and  output spaces.
Support Vector Regression Machines
Authors: Harris Drucker, Christopher J. C. Burges, Linda Kaufman, Alex J. Smola, Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  new  regression  technique  based  on  Vapnik's  concept  of  support  vectors  is  introduced.  We compare  support  vector  regression  (SVR)  with  a  committee regression  technique  (bagging)  based  on  regression  trees  and  ridge regression  done in  feature space.  On  the basis of these  experiments,  it  is  expected  that  SVR  will  have  advantages  in  high  dimensionality space because SVR optimization does not depend on the  dimensionality of the input space.
Dynamically Adaptable CMOS Winner-Take-All Neural Network
Authors: Kunihiko Iizuka, Masayuki Miyamoto, Hirofumi Matsui
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The major  problem  that  has  prevented  practical  application  of analog  neuro-LSIs  has  been  poor  accuracy  due  to  fluctuating  analog  device  characteristics  inherent  in  each  device  as  a  result  of  manufacturing.  This paper proposes a dynamic control architecture that  allows analog  silicon  neural  networks  to  compensate  for  the  fluctuating  device  characteristics  and  adapt  to  a  change  in  input  DC  level.  We  have  applied  this  architecture to  compensate  for  input  offset  voltages of an  analog  CMOS  WTA (Winner-Take-AlI)  chip  that we  have  fabricated.  Experimental data show the effectiveness of the architecture.