mtybilly/Instruct2DS huggingface dataset:
Spectroscopic Detection of Cervical Pre-Cancer through Radial Basis Function Networks
Authors: Nirmala Ramanujam, Rebecca R. Richards-Kortum, Joydeep Ghosh Kagan Tumer
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  mortality  related  to cervical  cancer  can  be  substantially  re(cid:173) duced  through  early  detection  and  treatment.  However,  cur(cid:173) rent  detection  techniques,  such  as  Pap  smear  and  colposcopy,  fail  to  achieve  a  concurrently  high  sensitivity  and  specificity.  In  vivo  fluorescence  spectroscopy  is  a  technique  which  quickly,  non(cid:173) invasively and quantitatively probes the biochemical and morpho(cid:173) logical  changes that occur in pre-cancerous tissue.  RBF ensemble  algorithms based on such spectra provide automated, and near real(cid:173) time implementation of pre-cancer  detection in the  hands of non(cid:173) experts.  The  results  are  more  reliable,  direct  and  accurate  than  those achieved  by  either human experts or multivariate statistical  algorithms.
Radial Basis Function Networks and Complexity Regularization in Function Learning
Authors: Tamás Linder Adam Krzyzak
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In this paper we apply the method of complexity regularization to de(cid:173) rive estimation bounds for nonlinear function estimation using a single hidden layer radial basis function network. Our approach differs from the previous complexity regularization neural network function learning schemes in that we operate with random covering numbers and 11 metric entropy, making it po~sibleto consider much broader families of activa(cid:173) tion functions, namely functions of bounded variation. Some constraints previously imposed on the network parameters are also eliminated this way. The network is trained by means of complexity regularization in(cid:173) volving empirical risk minimization. Bounds on the expected risk in tenns of the sample size are obtained for a large class of loss functions. Rates of convergence to the optimal loss are also derived.
Adaptively Growing Hierarchical Mixtures of Experts
Authors: Michael Finke, Alex Waibel Jürgen Fritsch
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We propose a novel approach to automatically growing and pruning  Hierarchical  Mixtures  of Experts.  The constructive  algorithm pro(cid:173) posed  here  enables  large  hierarchies  consisting  of several  hundred  experts  to be  trained effectively.  We  show  that HME's  trained  by  our  automatic  growing  procedure  yield  better  generalization  per(cid:173) formance  than  traditional  static  and  balanced  hierarchies.  Eval(cid:173) uation  of  the  algorithm  is  performed  (1)  on  vowel  classification  and  (2)  within  a  hybrid  version  of the  JANUS  r9]  speech  recog(cid:173) nition system  using a  subset  of the Switchboard large-vocabulary  speaker-independent  continuous speech  recognition database.
On-line Policy Improvement using Monte-Carlo Search
Authors: Gregory R. Galperin Gerald Tesauro
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We present a Monte-Carlo simulation algorithm for  real-time policy  improvement  of an  adaptive  controller.  In  the  Monte-Carlo  sim(cid:173) ulation,  the  long-term expected  reward  of each  possible  action  is  statistically measured, using the initial policy to make decisions  in  each  step of the simulation.  The action maximizing the  measured  expected reward is then taken, resulting in an improved policy.  Our  algorithm is easily parallelizable and has been implemented on the  IBM  SP! and SP2 parallel-RISC supercomputers.  We  have  obtained  promising  initial  results  in  applying  this  algo(cid:173) rithm  to the  domain of backgammon.  Results  are  reported  for  a  wide  variety  of initial  policies,  ranging  from  a  random  policy  to  TD-Gammon, an extremely strong multi-layer neural  network.  In  each case,  the Monte-Carlo algorithm gives a substantial reduction,  by  as  much as  a  factor  of 5 or  more,  in  the error  rate  of the  base  players.  The  algorithm  is  also  potentially  useful  in  many  other  adaptive control applications in which it is possible to simulate the  environment.
Blind Separation of Delayed and Convolved Sources
Authors: Anthony J. Bell, Russell H. Lambert Te-Won Lee
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  address  the  difficult  problem  of separating multiple  speakers  with  multiple microphones  in  a  real room.  We  combine the work  of Torkkola and  Amari,  Cichocki  and Yang,  to give  Natural  Gra(cid:173) dient information maximisation rules for recurrent (IIR)  networks,  blindly  adjusting  delays,  separating  and  deconvolving  mixed  sig(cid:173) nals.  While  they  work  well  on  simulated  data,  these  rules  fail  in  real  rooms  which  usually  involve  non-minimum phase  transfer  functions,  not-invertible using stable IIR filters.  An approach that  sidesteps this problem is to perform infomax on a feedforward archi(cid:173) tecture in the frequency domain  (Lambert 1996).  We  demonstrate  real-room separation of two natural signals using this approach.
A New Approach to Hybrid HMM/ANN Speech Recognition using Mutual Information Neural Networks
Authors: Christoph Neukirchen Gerhard Rigoll
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper presents a new approach to speech recognition with hybrid  HMM/ANN  technology.  While  the  standard  approach  to  hybrid  HMMI ANN  systems  is  based  on  the  use  of  neural  networks  as  posterior probability estimators, the new approach is based on the use  of mutual information neural  networks trained with a special learning  algorithm  in  order to  maximize the  mutual  information  between  the  input classes of the network and its resulting sequence of firing output  neurons  during  training.  It is  shown  in  this  paper that  such  a  neural  network  is  an  optimal  neural  vector quantizer for  a  discrete  hidden  Markov  model  system  trained  on  Maximum  Likelihood  principles.  One  of the  main  advantages  of this  approach  is  the  fact,  that  such  neural  networks  can  be  easily  combined  with  HMM's  of  any  complexity  with context-dependent capabilities.  It is  shown  that  the  resulting  hybrid  system  achieves  very  high  recognition  rates,  which  are  now  already  on  the  same  level  as  the  best  conventional  HMM  systems  with  continuous  parameters,  and  the  capabilities  of  the  mutual information neural networks are not yet entirely exploited.
Competition Among Networks Improves Committee Performance
Authors: Bambang Parmanto Paul W. Munro
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The separation of generalization error into two types, bias and variance  (Geman,  Bienenstock,  Doursat,  1992),  leads  to  the  notion  of error  reduction  by  averaging  over  a  "committee"  of classifiers  (Perrone,  1993).  Committee perfonnance decreases with both the average error of  the constituent classifiers and increases  with  the degree to  which the  misclassifications are correlated across the committee.  Here, a method  for  reducing  correlations  is  introduced,  that  uses  a  winner-take-all  procedure  similar  to  competitive  learning  to  drive  the  individual  networks  to  different  minima  in  weight  space  with  respect  to  the  training set, such that correlations in  generalization perfonnance will be  reduced, thereby reducing committee error.
Selective Integration: A Model for Disparity Estimation
Authors: Alexandre Pouget, Richard S. Zemel, Steven J. Nowlan, Terrence J. Sejnowski Michael S. Gray
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Local disparity information is often sparse and noisy, which creates  two conflicting demands when estimating disparity in an image re(cid:173) gion:  the need to spatially average to get an accurate estimate, and  the problem of not averaging over discontinuities.  We  have devel(cid:173) oped a  network model of disparity estimation based on disparity(cid:173) selective neurons, such as those found in the early stages of process(cid:173) ing  in  visual cortex.  The model  can accurately estimate multiple  disparities in a region, which may be caused by transparency or oc(cid:173) clusion,  in real images and random-dot stereograms.  The use of a  selection mechanism to selectively integrate reliable local disparity  estimates  results  in  superior  performance  compared  to  standard  back-propagation  and  cross-correlation  approaches.  In  addition,  the representations learned with this selection mechanism are con(cid:173) sistent  with  recent  neurophysiological  results  of  von  der  Heydt,  Zhou, Friedman, and Poggio [8]  for cells in cortical visual area V2.  Combining multi-scale biologically-plausible image processing with  the power of the mixture-of-experts learning algorithm represents  a  promising approach that yields both high  performance and new  insights into visual system function. 
Selective Integration: A Model for Disparity Estimation
The Neurothermostat: Predictive Optimal Control of Residential Heating Systems
Authors: Lucky Vidmar, Robert H. Dodier Michael Mozer
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  Neurothermostat  is  an  adaptive  controller  that  regulates  in(cid:173) door  air  temperature  in  a  residence  by  switching  a  furnace  on  or  off.  The  task  is  framed  as  an  optimal control  problem  in  which  both  comfort  and  energy  costs  are  considered  as  part of the  con(cid:173) trol  objective.  Because  the  consequences  of control  decisions  are  delayed  in  time, the  N eurothermostat  must  anticipate heating de(cid:173) mands with predictive models of occupancy  patterns and the ther(cid:173) mal response of the house and furnace.  Occupancy  pattern predic(cid:173) tion  is  achieved  by  a  hybrid  neural  net  /  look-up table.  The  Neu(cid:173) rothermostat  searches,  at  each  discrete  time  step,  for  a  decision  sequence  that  minimizes  the  expected  cost  over  a  fixed  planning  horizon.  The first  decision  in this sequence  is  taken,  and  this  pro(cid:173) cess  repeats.  Simulations of the  Neurothermostat  were  conducted  using  artificial  occupancy  data  in  which  regularity  was  systemat(cid:173) ically  varied,  as  well  as  occupancy  data from  an  actual  residence.  The Neurothermostat is  compared against three  conventional poli(cid:173) cies,  and  achieves  reliably  lower  costs.  This result  is  robust  to the  relative  weighting of comfort  and  energy  costs  and  the  degree  of  variability in the occupancy  patterns. 
For over a  quarter  century,  the home automation industry has  promised  to revolu(cid:173) tionize our lifestyle with the so-called  Smart House@  in which appliances,  lighting,  stereo,  video,  and  security  systems  are  integrated  under  computer  control.  How(cid:173) ever,  home automation has yet  to make significant inroads, at least in  part because  software  must be tailored  to the  home occupants.  Instead  of expecting  the occupants  to program their  homes or  to hire  someone  to  do  so,  one  would  ideally like  the  home  to  essentially  program  itself  by  observing  the lifestyle of the occupants.  This is the goal of the  Neural  Network  House  (Mozer  et  al.,  1995),  an  actual  residence  that  has  been  outfitted  with  over  75  sensors(cid:173) including temperature,  light, sound,  motion-and actua.tors  to control air heating,  water  heating,  lighting,  and  ventilation.  In  this  paper,  we  describe  one  research 
954 
M.  C.  Mozer. L.  Vidmar and R.  H.  Dodier 
project  within  the  house,  the  Neurothermostat,  that  learns  to  regulate  the  indoor  air temperature automatically by observing and detecting patterns in the occupants'  schedules  and  comfort  preferences.  We  focus  on  the  problem  of air  heating  with  a  whole-house  furnace,  but  the  same  approach  can  be  taken  with  alternative  or  multiple heating devices,  and to the  problems of cooling and ventilation. 
1  TEMPERATURE REGULATION AS  AN OPTIMAL
Softening Discrete Relaxation
Authors: Richard C. Wilson, Edwin R. Hancock Andrew M. Finch
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper describes a  new framework for  relational graph match(cid:173) ing.  The starting point is  a recently reported Bayesian consistency  measure  which  gauges  structural  differences  using  Hamming  dis(cid:173) tance.  The main  contributions of the  work  are threefold.  Firstly,  we  demonstrate  how  the  discrete  components  of  the  cost  func(cid:173) tion  can  be  softened.  The  second  contribution  is  to  show  how  the  softened  cost  function  can  be  used  to  locate  matches  using  continuous non-linear optimisation.  Finally,  we  show how  the res(cid:173) ulting graph matching algorithm relates to the standard quadratic  assignment problem.
A Comparison between Neural Networks and other Statistical Techniques for Modeling the Relationship between Tobacco and Alcohol and Cancer
Authors: Pierre Band, Joel Bert, John Grace Tony Plate
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Epidemiological  data  is  traditionally  analyzed  with  very  simple  techniques.  Flexible  models,  such  as  neural  networks,  have  the  potential to discover unanticipated features in the data.  However,  to be useful,  flexible  models  must have effective control on overfit(cid:173) ting.  This paper reports on a  comparative study of the predictive  quality of neural networks and other flexible models applied to real  and artificial epidemiological data.  The results  suggest that there  are no major unanticipated complex features  in the real data, and  also  demonstrate  that  MacKay's  [1995]  Bayesian  neural  network  methodology  provides effective  control on overfitting while  retain(cid:173) ing the ability  to discover complex features  in the artificial data.
An Apobayesian Relative of Winnow
Authors: Chris Mesterharm Nick Littlestone
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  study  a  mistake-driven  variant  of an  on-line  Bayesian  learn(cid:173) ing  algorithm  (similar  to one  studied  by  Cesa-Bianchi,  Helmbold,  and Panizza [CHP96]).  This variant only updates its state (learns)  on trials in which it makes  a mistake.  The algorithm makes binary  classifications using a linear-threshold classifier and runs in time lin(cid:173) ear in  the  number of attributes seen by the learner.  We  have  been  able  to  show,  theoretically  and  in  simulations,  that  this  algorithm  performs well  under assumptions quite different from  those embod(cid:173) ied  in  the  prior of the  original  Bayesian algorithm.  It can handle  situations that  we  do  not  know  how  to handle  in  linear  time  with  Bayesian  algorithms.  We  expect  our  techniques  to  be  useful  in  deriving and analyzing other apobayesian algorithms.
LSTM can Solve Hard Long Time Lag Problems
Authors: Jürgen Schmidhuber Sepp Hochreiter
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Standard recurrent nets cannot deal with long minimal time lags between relevant signals. Several recent NIPS papers propose alter(cid:173) native methods. We first show: problems used to promote various previous algorithms can be solved more quickly by random weight guessing than by the proposed algorithms. We then use LSTM, our own recent algorithm, to solve a hard problem that can neither be quickly solved by random search nor by any other recurrent net algorithm we are aware of.
1 TRIVIAL PREVIOUS LONG TIME LAG PROBLEMS
Traditional recurrent nets fail in case 'of long minimal time lags between input sig(cid:173) nals and corresponding error signals [7, 3]. Many recent papers propose alternative methods, e.g., [16, 12, 1,5,9]. For instance, Bengio et ale investigate methods such as simulated annealing, multi-grid random search, time-weighted pseudo-Newton optimization, and discrete error propagation [3]. They also propose. an EM ap(cid:173) proach [1]. Quite a few papers use variants of the "2-sequence problem" (and ('latch problem" ) to show the proposed algorithm's superiority, e.g. [3, 1, 5, 9]. Some pa(cid:173) pers also use the "parity problem", e.g., [3, 1]. Some of Tomita's [18] grammars are also often used as benchmark problems for recurrent nets [2, 19, 14, 11].
Trivial versus non-trivial tasks. By our definition, a "trivial" task is one that can be solved quickly by random search (RS) in weight space. RS works as follows: REPEAT randomly initialize the weights and test the resulting net on a training set UNTIL solution found.
488 Solutions to the XOR Problem
Authors: Virginia L. Stonick Frans Coetzee
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  globally convergent homotopy method is  defined  that is capable  of sequentially producing large numbers of stationary points of the  multi-layer perceptron  mean-squared error  surface.  Using  this  al(cid:173) gorithm large subsets of the stationary points of two test problems  are  found.  It is  shown  empirically that  the  MLP  neural  network  appears  to  have  an  extreme  ratio  of saddle  points  compared  to  local  minima,  and  that  even  small neural  network  problems  have  extremely large numbers of solutions.
A Mixture of Experts Classifier with Learning Based on Both Labelled and Unlabelled Data
Authors: Hasan S. Uyar David J. Miller
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  address  statistical  classifier  design  given  a  mixed  training  set  con(cid:173) sisting  of  a  small  labelled  feature  set  and  a  (generally  larger)  set  of  unlabelled features.  This situation  arises,  e.g., for  medical images,  where  although  training  features  may  be  plentiful,  expensive  expertise  is  re(cid:173) quired  to  extract  their  class  labels.  We  propose  a  classifier  structure  and learning  algorithm  that make effective  use of unlabelled data to im(cid:173) prove performance.  The learning  is  based  on  maximization  of the  total  data  likelihood,  i.e.  over  both  the  labelled  and  unlabelled  data  sub(cid:173) sets.  Two distinct EM learning  algorithms  are proposed,  differing  in  the  EM  formalism  applied  for  unlabelled  data.  The classifier,  based  on  a  joint probability model for features  and labels,  is  a  "mixture of experts"  structure that is  equivalent to the  radial  basis function  (RBF) classifier,  but unlike RBFs, is  amenable  to likelihood-based training.  The scope of  application  for  the  new  method  is  greatly  extended  by  the  observation  that test data, or any new data to classify, is in fact  additional,  unlabelled  data - thus,  a combined learning/classification operation - much akin  to  what  is  done  in  image  segmentation  - can  be  invoked  whenever  there  is  new  data to classify.  Experiments  with  data sets from  the  UC  Irvine  database  demonstrate  that  the  new  learning  algorithms  and  structure  achieve substantial performance  gains  over alternative  approaches.
Statistical Mechanics of the Mixture of Experts
Authors: Jong-Hoon Oh Kukjin Kang
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We study generalization capability of the mixture of experts  learn(cid:173) ing  from  examples  generated  by  another  network  with  the  same  architecture.  When the number of examples is smaller than  a crit(cid:173) ical  value,  the  network  shows  a  symmetric  phase  where  the  role  of the experts  is  not specialized.  Upon  crossing  the critical  point,  the  system  undergoes  a  continuous  phase  transition  to  a  symme(cid:173) try  breaking  phase  where  the gating network  partitions  the  input  space effectively  and each expert is assigned  to an appropriate sub(cid:173) space.  We  also find  that the mixture of experts  with multiple level  of hierarchy shows  multiple phase transitions.
Reinforcement Learning for Mixed Open-loop and Closed-loop Control
Authors: Andrew G. Barto, Shlomo Zilberstein Eric A. Hansen
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Closed-loop  control  relies  on  sensory  feedback  that  is  usually  as(cid:173) sumed  to  be  free .  But  if sensing  incurs  a  cost,  it  may  be  cost(cid:173) effective  to  take  sequences  of actions  in  open-loop  mode.  We  de(cid:173) scribe  a  reinforcement  learning  algorithm  that  learns  to  combine  open-loop  and  closed-loop  control  when  sensing  incurs  a  cost.  Al(cid:173) though  we  assume reliable  sensors,  use  of open-loop control means  that  actions  must  sometimes  be  taken  when  the  current  state  of  the  controlled  system  is  uncertain.  This  is  a  special  case  of the  hidden-state  problem  in  reinforcement  learning,  and  to  cope,  our  algorithm relies  on short-term memory.  The main result  of the pa(cid:173) per is a rule that significantly limits exploration of possible  memory  states  by  pruning  memory states  for  which  the  estimated  value  of  information is  greater than its cost.  We  prove that  this  rule  allows  convergence  to an optimal  policy.
Computing with Infinite Networks
Authors: Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: For  neural  networks  with  a  wide  class  of weight-priors,  it  can  be  shown  that  in  the  limit  of an  infinite  number of hidden  units  the  prior over functions  tends to a  Gaussian process.  In  this paper an(cid:173) alytic forms are derived for the covariance function of the Gaussian  processes  corresponding  to networks with sigmoidal and Gaussian  hidden  units.  This  allows predictions  to  be  made efficiently  using  networks  with an infinite number of hidden units,  and shows  that,  somewhat paradoxically, it may be  easier  to compute with infinite  networks  than finite  ones.
Regression with Input-Dependent Noise: A Bayesian Treatment
Authors: Cazhaow S. Quazaz Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In  most  treatments  of the  regression  problem  it  is  assumed  that  the distribution of target data can be described by a  deterministic  function  of the inputs, together with additive Gaussian noise hav(cid:173) ing constant variance.  The use of maximum likelihood to train such  models then  corresponds to the minimization of a  sum-of-squares  error function.  In  many applications a more realistic model would  allow  the  noise  variance  itself to  depend  on  the  input  variables.  However, the use of maximum likelihood to train such models would  give highly biased  results.  In  this paper we  show  how  a  Bayesian  treatment  can  allow  for  an  input-dependent  variance  while  over(cid:173) coming the bias of maximum likelihood.
The Generalisation Cost of RAMnets
Authors: Michal Morciniec Richard Rohwer
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Given unlimited computational resources,  it is  best  to use  a  crite(cid:173) rion of minimal expected generalisation error to select  a model and  determine its  parameters.  However,  it may be  worthwhile to sac(cid:173) rifice  some  generalisation  performance  for  higher  learning  speed.  A  method  for  quantifying sub-optimality is  set  out  here,  so  that  this  choice  can  be  made  intelligently.  Furthermore,  the  method  is  applicable  to  a  broad  class  of models,  including  the  ultra-fast  memory-based methods such  as  RAMnets.  This brings the added  benefit  of providing,  for  the  first  time,  the  means  to  analyse  the  generalisation  properties of such  models in a  Bayesian framework .
Multilayer Neural Networks: One or Two Hidden Layers?
Authors: Claire Kenyon, Hélène Paugam-Moisy Graham Brightwell
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We study the number of hidden layers required by a multilayer neu(cid:173) ral network with threshold  units to compute a function  f  from n d  to {O, I}.  In dimension d  =  2,  Gibson  characterized  the functions  computable with just one hidden layer, under  the assumption that  there  is no  "multiple intersection  point"  and that f  is only defined  on a compact set.  We consider the restriction of f  to the neighbor(cid:173) hood of a  multiple intersection  point or of infinity,  and give  neces(cid:173) sary  and sufficient  conditions for  it  to  be  locally computable with  one  hidden  layer.  We  show  that  adding  these  conditions  to  Gib(cid:173) son's  assumptions  is  not  sufficient  to  ensure  global  computability  with one hidden layer,  by exhibiting a new  non-local configuration,  the  "critical cycle",  which  implies  that f  is  not  computable  with  one  hidden  layer.
Improving the Accuracy and Speed of Support Vector Machines
Authors: Bernhard Schölkopf Christopher J. C. Burges
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Support  Vector  Learning  Machines  (SVM)  are  finding  application  in  pattern  recognition,  regression  estimation , and  operator inver(cid:173) sion  for  ill-posed  problems.  Against  this  very  general  backdrop ,  any methods for  improving the  generalization  performance, or for  improving  the  speed  in  test  phase,  of SVMs  are  of increasing  in(cid:173) terest.  In this paper we  combine two such  techniques on  a  pattern  recognition problem. The method for improving generalization per(cid:173) formance  (the  "virtual support  vector"  method)  does  so  by  incor(cid:173) porating known invariances of the problem.  This method achieves  a  drop  in  the error  rate on  10,000 NIST  test  digit images of 1.4%  to  1.0%.  The method for  improving the speed  (the  "reduced  set"  method) does so by approximating the support vector decision sur(cid:173) face.  We  apply  this method to achieve  a  factor  of fifty  speedup  in  test phase over  the  virtual support vector machine.  The combined  approach  yields a  machine which  is  both  22  times faster  than  the  original machine, and which has better generalization performance,  achieving  1.1 % error.  The virtual support vector  method is  appli(cid:173) cable  to any SVM  problem  with  known  invariances.  The  reduced  set  method is  applicable to any support  vector machine.
Adaptive Access Control Applied to Ethernet Data
Authors: Timothy X. Brown
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper presents a method that decides which combinations of traffic  can be accepted on a packet data link,  so that quality of service  (QoS)  constraints can be met. The method uses samples of QoS  results at dif(cid:173) ferent load conditions to build a neural  network decision function.  Pre(cid:173) vious  similar approaches  to  the  problem  have  a  significant  bias.  This  bias is likely to  occur in any real system and results in accepting loads  that miss QoS targets by orders of magnitude. Preprocessing the data to  either remove  the  bias  or provide  a  confidence  level,  the  method  was  applied  to  sources  based  on  difficult-to-analyze  ethernet  data  traces.  With this data, the method produces an accurate access control function  that  dramatically  outperforms  analytic  alternatives.  Interestingly,  the  results depend on throwing away more than 99% of the data.
An Adaptive WTA using Floating Gate Technology
Authors: Paul E. Hasler, Bradley A. Minch, Christof Koch W. Fritz Kruger
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  have  designed,  fabricated,  and  tested  an  adaptive  Winner(cid:173) Take-All  (WTA)  circuit  based  upon  the  classic  WTA  of Lazzaro,  et  al  [IJ.  We  have  added  a  time  dimension  (adaptation)  to  this  circuit to make the input derivative an important factor in winner  selection.  To  accomplish  this,  we  have  modified  the  classic  WTA  circuit  by  adding floating  gate transistors which  slowly  null  their  inputs over time.  We  present a  simplified analysis and experimen(cid:173) tal data of this adaptive WTA fabricated in a standard CMOS 2f.tm  process. 
1  Winner-Take-All Circuits 
In a WTA network, each cell has one input and one output.  For any set of inputs, the  outputs will all be at zero except for the one which is from the cell with the maximum  input.  One way to accomplish this is by a global nonlinear inhibition coupled with a  self-excitation term [2J.  Each cell  inhibits all others while exciting itself;  thus a cell  with even a slightly greater input than the others will excite itself up to its maximal  state  and  inhibit  the  others  down  to  their  minimal  states.  The  WTA  function  is  important for  many classical neural  nets that involve  competitive  learning,  vector  quantization  and  feature  mapping.  The  classic  WTA  network  characterized  by  Lazzaro et.  al.  [IJ  is  an  elegant,  simple  circuit  that shares just one  common  line  among all  cells of the network to propagate the inhibition. 
Our motivation  to  add  adaptation  comes  from  the  idea of saliency  maps.  Picture  a  saliency  map  as  a  large  number  of cells  each  of which  encodes  an  analog  value 
An Adaptive wrA using Floating Gate Technology
Representation and Induction of Finite State Machines using Time-Delay Neural Networks
Authors: C. Lee Giles, Bill G. Horne, Garrison W. Cottrell Daniel S. Clouse
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This work investigates the representational and inductive capabili(cid:173) ties of time-delay neural networks (TDNNs)  in general, and of two  subclasses of TDNN, those  with delays only on the inputs (IDNN),  and those which  include delays on hidden  units (HDNN) . Both ar(cid:173) chitectures are capable of representing  the same class of languages,  the  definite  memory machine (DMM)  languages, but the delays on  the  hidden  units  in  the  HDNN  helps  it  outperform  the  IDNN  on  problems composed of repeated  features  over short  time windows.
Probabilistic Interpretation of Population Codes
Authors: Peter Dayan, Alexandre Pouget Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  present  a  theoretical  framework  for  population  codes  which  generalizes  naturally  to  the  important  case  where  the  population  provides  information  about  a  whole  probability  distribution  over  an  underlying  quantity  rather  than  just  a  single  value.  We  use  the framework  to analyze two existing models,  and to suggest and  evaluate a  third model for  encoding such probability distributions.
Analog VLSI Circuits for Attention-Based, Visual Tracking
Authors: Tonia G. Morris, Christof Koch, Stephen P. DeWeerth Timothy K. Horiuchi
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A one-dimensional visual tracking chip has been implemented us(cid:173) ing neuromorphic, analog VLSI techniques to model selective visual  attention in the control of saccadic and smooth pursuit eye move(cid:173) ments. The chip incorporates focal-plane processing to compute  image saliency and a winner-take-all circuit to select a feature for  tracking. The target position and direction of motion are reported  as the target moves across the array. We demonstrate its function(cid:173) ality in a closed-loop system which performs saccadic and smooth  pursuit tracking movements using a one-dimensional mechanical  eye.
Online Learning from Finite Training Sets: An Analytical Case Study
Authors: David Barber Peter Sollich
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  analyse  online  learning  from  finite  training  sets  at  non(cid:173) infinitesimal  learning  rates  TJ.  By  an  extension  of statistical  me(cid:173) chanics  methods,  we  obtain  exact  results  for  the  time-dependent  generalization  error  of  a  linear  network  with  a  large  number  of  weights  N.  We  find,  for  example,  that  for  small training  sets  of  size  p  ~ N,  larger  learning rates  can  be  used  without compromis(cid:173) ing  asymptotic  generalization  performance  or  convergence  speed.  Encouragingly,  for  optimal  settings  of  TJ  (and,  less  importantly,  weight decay ,)  at given final  learning time, the generalization per(cid:173) formance of online learning is  essentially  as good as  that of offline  learning.
Spatiotemporal Coupling and Scaling of Natural Images and Human Visual Sensitivities
Authors: Dawei W. Dong
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  study  the spatiotemporal  correlation  in  natural  time-varying  images  and  explore  the  hypothesis  that  the visual system  is  con(cid:173) cerned  with  the  optimal  coding  of visual  representation  through  spatiotemporal  decorrelation  of  the  input  signal.  Based  on  the  measured spatiotemporal power spectrum, the transform needed to  decorrelate input signal is derived analytically and then compared  with the actual processing observed in psychophysical experiments.
ARC-LH: A New Adaptive Resampling Algorithm for Improving ANN Classifiers
Authors: Kurt Hornik Friedrich Leisch
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We introduce arc-Ih,  a new algorithm for improvement of ANN clas(cid:173) sifier  performance,  which  measures  the  importance of patterns  by  aggregated network output errors.  On several  artificial benchmark  problems,  this  algorithm compares favorably  with other  resample  and combine techniques.
Estimating Equivalent Kernels for Neural Networks: A Data Perturbation Approach
Authors: A. Neil Burgess
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  describe  the  notion  of  "equivalent  kernels"  and  suggest  that  this  provides a framework  for comparing different classes of regression models,  including  neural  networks  and  both  parametric  and  non-parametric  statistical techniques.  Unfortunately,  standard techniques break down  when  faced with models, such as neural networks,  in which there is more than one  "layer" of adjustable parameters.  We propose an algorithm which overcomes  this limitation,  estimating the equivalent kernels for  neural network models  using  a  data  perturbation approach.  Experimental  results  indicate  that  the  networks do  not  use  the  maximum possible  number of degrees of freedom,  that  these  can  be  controlled  using  regularisation  techniques  and  that  the  equivalent kernels learnt by the network vary both  in "size"  and in "shape"  in different regions of the input space.
Monotonicity Hints
Authors: Yaser S. Abu-Mostafa Joseph Sill
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A hint is any piece of side information about the target function to  be  learned.  We  consider  the monotonicity hint,  which  states  that  the function  to be learned  is  monotonic in some or all of the input  variables.  The  application of mono tonicity  hints  is  demonstrated  on  two  real-world  problems- a  credit  card  application task,  and  a  problem in medical diagnosis.  A measure of the monotonicity error  of a candidate function is defined and an objective function for  the  enforcement  of monotonicity is  derived  from  Bayesian  principles.  We report experimental results which show that using monotonicity  hints leads to a statistically significant improvement in performance  on  both problems.
A Convergence Proof for the Softassign Quadratic Assignment Algorithm
Authors: Alan L. Yuille, Steven Gold, Eric Mjolsness Anand Rangarajan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  softassign  quadratic  assignment  algorithm  has  recently  emerged as an effective strategy for  a variety of optimization prob(cid:173) lems in pattern recognition and combinatorial optimization.  While  the  effectiveness  of the algorithm  was  demonstrated  in  thousands  of simulations,  there  was  no  known  proof of convergence.  Here,  we  provide a  proof of convergence for  the most general form of the  algorithm.
Removing Noise in On-Line Search using Adaptive Batch Sizes
Authors: Genevieve B. Orr
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Stochastic  (on-line)  learning  can  be  faster  than  batch  learning.  However,  at late times,  the learning rate must be annealed to re(cid:173) move  the noise  present  in  the stochastic weight  updates.  In  this  annealing phase,  the convergence rate (in  mean square)  is  at best  proportional to  l/T where T  is  the number of input presentations.  An alternative is  to increase the batch size to remove the noise.  In  this paper we explore convergence for LMS using 1)  small but fixed  batch sizes  and  2)  an adaptive batch size.  We  show  that the best  adaptive  batch schedule  is  exponential  and  has  a  rate  of  conver(cid:173) gence which is  the same as for  annealing, Le.,  at best proportional  to  l/T.
Size of Multilayer Networks for Exact Learning: Analytic Approach
Authors: Hélène Paugam-Moisy André Elisseeff
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This  article  presents  a  new  result  about  the  size  of a  multilayer  neural network computing real outputs for exact learning of a finite  set of real samples.  The architecture of the network is feedforward,  with  one  hidden  layer  and several  outputs.  Starting from  a  fixed  training set,  we  consider  the  network  as  a  function  of its  weights.  We  derive,  for  a  wide family  of transfer  functions,  a  lower  and  an  upper  bound  on  the  number  of hidden  units  for  exact  learning,  given  the size  of the  dataset  and  the  dimensions of the  input and  output spaces.
Support Vector Regression Machines
Authors: Christopher J. C. Burges, Linda Kaufman, Alex J. Smola, Vladimir Vapnik Harris Drucker
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  new  regression  technique  based  on  Vapnik's  concept  of  support  vectors  is  introduced.  We compare  support  vector  regression  (SVR)  with  a  committee regression  technique  (bagging)  based  on  regression  trees  and  ridge regression  done in  feature space.  On  the basis of these  experiments,  it  is  expected  that  SVR  will  have  advantages  in  high  dimensionality space because SVR optimization does not depend on the  dimensionality of the input space.
A Hierarchical Model of Visual Rivalry
Authors: Peter Dayan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Binocular rivalry is  the alternating percept that can result when  the two eyes see different scenes.  Recent psychophysical evidence  supports an account for one component of binocular rivalry similar  to that for other bistable percepts.  We  test the hypothesisl9, 16, 18  that  alternation  can be  generated by competition  between  top(cid:173) down cortical explanations for  the inputs,  rather than by direct  competition between the  inputs.  Recent neurophysiological ev(cid:173) idence  shows  that some binocular neurons  are  modulated  with  the changing percept; others are not, even if they are selective be(cid:173) tween the stimuli presented to the eyes.  We  extend our model to  a hierarchy to address these effects.
Dynamically Adaptable CMOS Winner-Take-All Neural Network
Authors: Masayuki Miyamoto, Hirofumi Matsui Kunihiko Iizuka
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The major  problem  that  has  prevented  practical  application  of analog  neuro-LSIs  has  been  poor  accuracy  due  to  fluctuating  analog  device  characteristics  inherent  in  each  device  as  a  result  of  manufacturing.  This paper proposes a dynamic control architecture that  allows analog  silicon  neural  networks  to  compensate  for  the  fluctuating  device  characteristics  and  adapt  to  a  change  in  input  DC  level.  We  have  applied  this  architecture to  compensate  for  input  offset  voltages of an  analog  CMOS  WTA (Winner-Take-AlI)  chip  that we  have  fabricated.  Experimental data show the effectiveness of the architecture.
Ensemble Methods for Phoneme Classification
Authors: Gary Cook Steve R. Waterhouse
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper investigates a  number of ensemble methods for  improv(cid:173) ing  the  performance of phoneme classification  for  use  in  a  speech  recognition system.  Two ensemble methods are described; boosting  and mixtures of experts,  both in isolation and in combination.  Re(cid:173) sults are presented on two speech recognition databases:  an isolated  word database and a large vocabulary continuous speech database.  These results show that principled ensemble methods such as boost(cid:173) ing  and  mixtures provide superior  performance to  more naive  en(cid:173) semble methods such  as  averaging.
Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA
Authors: Lucas C. Parra Barak A. Pearlmutter
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In the square linear blind source separation problem, one must find  a  linear  unmixing operator  which  can detangle  the result  Xi(t)  of  mixing n  unknown independent sources 8i(t)  through an unknown  n  x  n  mixing matrix A( t)  of causal linear filters:  Xi  =  E j  aij * 8 j .  We cast the problem as one of maximum likelihood density estima(cid:173) tion,  and  in  that framework  introduce an  algorithm  that searches  for  independent components using both temporal and spatial cues.  We  call the resulting algorithm  "Contextual ICA,"  after the  (Bell  and  Sejnowski  1995)  Infomax  algorithm,  which  we  show  to  be  a  special case of cICA.  Because cICA  can make use  of the temporal  structure of its input, it is  able separate in a  number of situations  where  standard methods  cannot,  including  sources  with  low  kur(cid:173) tosis,  colored  Gaussian sources,  and  sources  which  have  Gaussian  histograms. 
1  The Blind Source Separation Problem 
Consider  a  set  of n  indepent  sources  81 (t), . .. ,8n (t).  We  are  given  n  linearly dis(cid:173) torted sensor  reading  which  combine  these  sources,  Xi  =  E j  aij8j,  where  aij  is  a  filter  between source j  and  sensor i,  as  shown in figure  1a.  This can be expressed  as 
Xi(t) = 2: 2: aji(r)8j(t - r) = 2: aji * 8j
Temporal Low-Order Statistics of Natural Sounds
Authors: Christoph E. Schreiner Hagai Attias
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In order to process  incoming sounds  efficiently,  it is  advantageous  for the auditory system to be adapted to the statistical structure of  natural auditory scenes.  As a first step in investigating the relation  between  the system  and its inputs,  we  study low-order  statistical  properties in several sound ensembles  using a  filter  bank analysis.  Focusing on the amplitude and phase in different frequency bands,  we  find  simple  parametric  descriptions  for  their  distribution  and  power  spectrum  that  are valid  for  very  different  types  of sounds.  In  particular,  the  amplitude  distribution  has  an  exponential  tail  and  its  power  spectrum  exhibits  a  modified  power-law  behavior,  which is  manifested by self-similarity and long-range temporal cor(cid:173) relations.  Furthermore,  the statistics for  different  bands  within  a  given  ensemble  are  virtually  identical,  suggesting  translation  in(cid:173) variance along the cochlear  axis.  These results  show  that natural  sounds are highly redundant, and have possible implications to the  neural code used  by the auditory system.
Limitations of Self-organizing Maps for Vector Quantization and Multidimensional Scaling
Authors: Arthur Flexer
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  limitations  of  using  self-organizing  maps  (SaM)  for  either  clustering/vector  quantization  (VQ)  or  multidimensional  scaling  (MDS)  are  being  discussed  by  reviewing  recent  empirical findings  and the relevant theory.  SaM 's remaining ability of doing both VQ  and MDS  at the same time is challenged  by  a  new  combined  tech(cid:173) nique  of online  K-means  clustering  plus  Sammon  mapping of the  cluster  centroids.  SaM are shown to perform significantly worse in  terms of quantization error , in  recovering the structure of the clus(cid:173) ters  and  in  preserving  the  topology  in  a  comprehensive  empirical  study using a  series  of multivariate normal clustering  problems.
A Spike Based Learning Neuron in Analog VLSI
Authors: Misha Mahowald, Lloyd Watts Philipp Häfliger
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Many  popular  learning  rules  are  formulated  in  terms  of continu(cid:173) ous,  analog inputs  and outputs.  Biological  systems,  however,  use  action  potentials,  which  are  digital-amplitude  events  that  encode  analog  information  in  the  inter-event  interval.  Action-potential  representations are now  being used  to advantage in neuromorphic  VLSI  systems as  well.  We  report on a  simple learning rule,  based  on  the  Riccati  equation  described  by  Kohonen  [1],  modified  for  action-potential  neuronal  outputs.  We  demonstrate  this  learning  rule in an analog VLSI chip that uses volatile capacitive storage for  synaptic weights.  We  show that our time-dependent learning rule  is  sufficient  to achieve  approximate weight  normalization and  can  detect temporal correlations in spike trains. 
A Spike Based Learning Neuron in Analog VLSI