mtybilly/Instruct2DS huggingface dataset:
How to Dynamically Merge Markov Decision Processes
Authors: Satinder P. Singh, David Cohn
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We are frequently called upon to perform multiple tasks that com(cid:173) pete  for  our  attention  and  resource.  Often  we  know  the optimal  solution  to each  task in  isolation;  in  this  paper,  we  describe  how  this  knowledge  can  be exploited  to  efficiently  find  good  solutions  for doing the tasks in parallel.  We formulate this problem as that of  dynamically merging  multiple  Markov  decision  processes  (MDPs)  into a  composite MDP,  and present a  new  theoretically-sound dy(cid:173) namic programming algorithm for finding an optimal policy for the  composite MDP.  We  analyze various aspects of our algorithm and  illustrate its use on a  simple merging problem. 
Every day,  we  are faced  with the problem of doing mUltiple  tasks in parallel,  each  of which  competes  for  our  attention  and  resource.  If we  are  running  a  job  shop,  we  must  decide  which  machines  to allocate  to  which  jobs,  and  in  what  order,  so  that no jobs miss their deadlines.  If we  are a  mail delivery robot, we  must find  the  intended recipients of the mail while simultaneously avoiding fixed  obstacles  (such  as walls)  and mobile obstacles  (such as people),  and still manage to keep ourselves  sufficiently charged up.  Frequently we know how to perform each task in isolation; this paper considers how  we  can take the information we  have about the individual  tasks and combine it to  efficiently find an optimal solution for  doing the entire set of tasks in parallel.  More  importantly,  we  describe  a  theoretically-sound  algorithm  for  doing  this  merging  dynamically;  new tasks (such as a new job arrival at a job shop) can be assimilated  online into the solution being found for  the ongoing set of simultaneous tasks.
Refractoriness and Neural Precision
Authors: Michael J. Berry II, Markus Meister
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The relationship between a neuron's refractory period and the precision of  its response to identical stimuli was investigated. We constructed a model of  a spiking neuron that combines probabilistic firing with a refractory period.  For realistic refractoriness, the model closely reproduced both the average  firing rate and the response precision of a retinal ganglion cell. The model is  based on a "free" firing rate, which exists in the absence of refractoriness.  This function may be a better description of a spiking neuron's response  than the peri-stimulus time histogram.
Active Data Clustering
Authors: Thomas Hofmann, Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Active  data  clustering is a  novel technique for  clustering of proxim(cid:173) ity data which utilizes principles from sequential experiment design  in  order  to interleave data generation  and  data analysis.  The  pro(cid:173) posed  active data sampling strategy is  based on the  expected  value  of information, a concept rooting in statistical decision theory.  This  is considered to be an important step towards the analysis of large(cid:173) scale  data sets,  because  it  offers  a  way  to  overcome  the  inherent  data sparseness of proximity data.  '''Ie present applications to unsu(cid:173) pervised  texture segmentation in  computer vision and information  retrieval  in  document  databases.
On Parallel versus Serial Processing: A Computational Study of Visual Search
Authors: Eyal Cohen, Eytan Ruppin
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A novel neural network model of pre-attention processing in visual(cid:173) search  tasks  is  presented.  Using displays of line orientations taken  from  Wolfe's experiments  [1992], we  study the  hypothesis  that the  distinction  between  parallel  versus  serial  processes  arises  from  the  availability of global information in  the internal representations  of  the  visual  scene.  The  model  operates  in  two  phases.  First,  the  visual  displays  are  compressed  via  principal-component-analysis.  Second,  the compressed data is processed by a target detector mod(cid:173) ule in order to identify the existence of a  target in the display.  Our  main  finding  is  that  targets  in  displays  which  were  found  exper(cid:173) imentally  to  be  processed  in  parallel can  be  detected  by  the  sys(cid:173) tem,  while  targets  in  experimentally-serial  displays  cannot .  This  fundamental  difference  is  explained  via  variance  analysis  of  the  compressed representations,  providing a  numerical criterion distin(cid:173) guishing parallel from serial displays.  Our model yields a  mapping  of response-time slopes that is similar to Duncan and Humphreys's  "search  surface"  [1989],  providing  an  explicit  formulation  of their  intuitive  notion  of feature  similarity.  It presents  a  neural  realiza(cid:173) tion of the  processing that may underlie the classical metaphorical  explanations of visual search. 
On Parallel versus Serial Processing: A  Computational Study a/Visual Search
S-Map: A Network with a Simple Self-Organization Algorithm for Generative Topographic Mappings
Authors: Kimmo Kiviluoto, Erkki Oja
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The S-Map is a network with a simple learning algorithm that com(cid:173) bines  the  self-organization  capability  of the  Self-Organizing  Map  (SOM)  and the probabilistic interpretability of the Generative To(cid:173) pographic  Mapping  (GTM).  The  simulations  suggest  that  the  S(cid:173) Map algorithm has a  stronger tendency  to self-organize from  ran(cid:173) dom  initial  configuration  than  the  GTM.  The  S-Map  algorithm  can  be  further  simplified  to employ  pure  Hebbian  learning,  with(cid:173) out changing the qualitative behaviour of the network.
Bayesian Model of Surface Perception
Authors: William T. Freeman, Paul A. Viola
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Image  intensity  variations  can  result  from  several  different  object  surface  effects,  including  shading from  3-dimensional  relief of the  object, or paint on the surface itself.  An essential problem in vision ,  which  people  solve  naturally,  is  to  attribute  the  proper  physical  cause,  e.g.  surface  relief or  paint,  to  an  observed  image.  We  ad(cid:173) dressed  this  problem  with  an  approach  combining  psychophysical  and  Bayesian  computational methods.  We  assessed  human performance on a set of test images, and found  that people made fairly consistent judgements of surface properties.  Our  computational  model  assigned  simple  prior  probabilities  to  different  relief or  paint  explanations for  an  image,  and  solved  for  the  most  probable  interpretation  in  a  Bayesian  framework.  The  ratings of the  test  images  by  our  algorithm compared surprisingly  well  with  the  mean  ratings of our subjects.
Training Methods for Adaptive Boosting of Neural Networks
Authors: Holger Schwenk, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: "Boosting" is  a  general  method for  improving the  performance of any  learning algorithm that consistently generates classifiers  which  need to  perform only slightly better than random guessing.  A recently proposed  and very promising boosting algorithm is AdaBoost [5].  It has been ap(cid:173) plied with great success to several benchmark machine learning problems  using rather simple learning algorithms [4],  and decision trees [1,  2,  6].  In  this  paper we use AdaBoost to  improve the performances of neural  networks.  We compare training methods based on sampling the training  set and  weighting the cost function.  Our system  achieves  about  1.4%  error on  a  data base  of online handwritten  digits  from  more  than  200  writers.  Adaptive boosting of a multi-layer network achieved 1.5% error  on the UCI Letters and 8.1 % error on the UCI satellite data set.
An Analog VLSI Model of the Fly Elementary Motion Detector
Authors: Reid R. Harrison, Christof Koch
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Flies are capable of rapidly detecting and integrating visual motion in(cid:173) formation in behaviorly-relevant ways.  The first  stage of visual motion  processing in flies is  a retinotopic array of functional units known as  el(cid:173) ementary motion detectors (EMDs).  Several decades ago, Reichardt and  colleagues developed a correlation-based model of motion detection that  described the behavior of these neural circuits.  We  have implemented a  variant of this model in a 2.0-JLm  analog CMOS VLSI process.  The re(cid:173) sult is a low-power, continuous-time analog circuit with integrated pho(cid:173) toreceptors that responds to motion in real  time.  The  responses of the  circuit to drifting sinusoidal gratings qualitatively resemble the temporal  frequency response, spatial frequency response, and direction selectivity  of motion-sensitive neurons observed in insects.  In addition to  its pos(cid:173) sible engineering applications, the circuit could potentially be used as a  building block for constructing hardware models of higher-level insect  motion integration.
Experiences with Bayesian Learning in a Real World Application
Authors: Peter Sykacek, Georg Dorffner, Peter Rappelsberger, Josef Zeitlhofer
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This  paper  reports  about  an  application  of Bayes'  inferred  neu(cid:173) ral  network  classifiers  in  the  field  of automatic sleep  staging.  The  reason  for  using Bayesian  learning for  this  task  is  two-fold.  First,  Bayesian  inference  is  known  to  embody  regularization  automati(cid:173) cally.  Second,  a  side  effect  of  Bayesian  learning  leads  to  larger  variance of network outputs in regions  without training data.  This  results  in  well  known  moderation  effects,  which  can  be  used  to  detect  outliers.  In  a  5  fold  cross-validation  experiment  the  full  Bayesian  solution  found  with  R.  Neals  hybrid  Monte  Carlo  algo(cid:173) rithm,  was  not  better than  a single maximum a-posteriori  (MAP)  solution found  with  D.J.  MacKay's  evidence  approximation.  In  a  second  experiment  we  studied  the  properties  of both  solutions  in  rejecting  classification of movement artefacts. 
Experiences with Bayesian Learning in a Real World Application
Analytical Study of the Interplay between Architecture and Predictability
Authors: Avner Priel, Ido Kanter, David A. Kessler
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  study  model  feed  forward  networks  as  time  series  predictors  in  the stationary limit.  The focus  is  on  complex,  yet  non-chaotic,  behavior.  The main question we  address is whether the asymptotic  behavior is  governed  by  the architecture,  regardless  the details  of  the  weights .  We  find  hierarchies  among  classes  of  architectures  with  respect  to the attract or dimension  of the long term sequence  they are capable of generating;  larger number of hidden  units  can  generate higher dimensional attractors.  In the case of a perceptron,  we  develop  the  stationary  solution  for  general  weights,  and  show  that  the  flow  is  typically  one  dimensional.  The  relaxation  time  from  an  arbitrary  initial  condition  to  the  stationary  solution  is  found  to scale linearly with the size  of the network.  In multilayer  networks, the number of hidden units gives bounds on the number  and  dimension  of the  possible  attractors.  We  conclude that  long  term  prediction  (in  the  non-chaotic  regime)  with  such  models  is  governed by attractor dynamics related to the architecture. 
Neural networks provide an important tool as model free estimators for the solution  of problems when the real model is  unknown,  or  weakly known.  In the last decade  there has  been a growing interest in the application of such tools in the area of time  series  prediction  (see  Weigand  and Gershenfeld,  1994).  In this paper we  analyse a  typical class of architectures used in this field,  i.e.  a feed  forward network governed  by the following  dynamic rule:
A Hippocampal Model of Recognition Memory
Authors: Randall C. O'Reilly, Kenneth A. Norman, James L. McClelland
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A  rich  body  of data  exists  showing that  recollection of specific  infor(cid:173) mation makes an important contribution to  recognition memory,  which  is distinct from the contribution of familiarity, and is not adequately cap(cid:173) tured by existing unitary memory models. Furthennore, neuropsycholog(cid:173) ical evidence indicates that recollection is sub served by the hippocampus.  We  present a  model,  based  largely  on  known features  of hippocampal  anatomy and physiology, that accounts for the following key character(cid:173) istics of recollection:  1) false recollection is rare (i.e., participants rarely  claim to recollect having studied nonstudied items), and 2) increasing in(cid:173) terference leads to less recollection but apparently does not compromise  the quality of recollection (i.e., the extent to which recollected infonna(cid:173) tion veridically reflects events that occurred at study).
Wavelet Models for Video Time-Series
Authors: Sheng Ma, Chuanyi Ji
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In this work, we tackle the problem of time-series modeling of video  traffic.  Different from the existing methods which model the time(cid:173) series  in the time domain, we  model the wavelet  coefficients  in the  wavelet  domain.  The strength  of the wavelet  model includes  (1)  a  unified  approach to model both the long-range and the short-range  dependence  in the video traffic simultaneously, (2)  a  computation(cid:173) ally efficient  method on  developing the model and generating high  quality video traffic,  and (3) feasibility of performance analysis us(cid:173) ing the model.
Multi-time Models for Temporally Abstract Planning
Authors: Doina Precup, Richard S. Sutton
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Planning 
Doina Precup, Richard S. Sutton 
University of Massachusetts 
Amherst, MA 01003 
{dprecuplrich}@cs.umass.edu
Linear Concepts and Hidden Variables: An Empirical Study
Authors: Adam J. Grove, Dan Roth
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Some learning techniques for classification tasks work indirectly, by first trying  to fit a full probabilistic model to the observed data.  Whether this is a good idea  or not depends on the robustness with respect to deviations from  the postulated  model.  We  study this question experimentally in a restricted, yet non-trivial and  interesting case:  we consider a conditionally independent attribute (CIA) model  which  postulates  a  single  binary-valued hidden variable z  on  which all other  attributes (i.e., the target and the observables) depend. In this model, finding the  most likely value of anyone variable (given known values for the others) reduces  to testing a linear function of the observed values. 
We  learn  CIA  with  two  techniques:  the  standard  EM  algorithm,  and  a  new  algorithm we develop based on covariances.  We  compare these, in a controlled  fashion, against an algorithm (a version of Winnow) that attempts to find a good  linear classifier directly.  Our conclusions help delimit the fragility  of using the  CIA model for classification: once the data departs from this model, performance  quickly degrades and drops below that of the directly-learned linear classifier.
Comparison of Human and Machine Word Recognition
Authors: Markus Schenkel, Cyril Latimer, Marwan A. Jabri
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  study  which  is  concerned with  word  recognition  rates  for  heavily  degraded  documents.  We  compare  human  with  machine  read(cid:173) ing capabilities in a series of experiments, which  explores the interaction  of word/non-word recognition, word frequency  and legality of non-words  with  degradation level.  We  also study the influence  of character segmen(cid:173) tation, and compare human performance with that of our artificial neural  network model for reading.  We found that the proposed computer model  uses  word  context  as  efficiently  as  humans,  but performs  slightly  worse  on the pure character recognition task.
Characterizing Neurons in the Primary Auditory Cortex of the Awake Primate Using Reverse Correlation
Authors: R. Christopher DeCharms, Michael Merzenich
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: While  the  understanding of the  functional  role  of different  classes  of neurons in the awake primary visual cortex has been extensively  studied since the time of Hubel and Wiesel (Hubel and Wiesel, 1962),  our  understanding  of the  feature  selectivity  and  functional  role  of  neurons in  the primary  auditory  cortex  is  much farther  from  com(cid:173) plete.  Moving bars have long been recognized as an optimal stimulus  for  many visual cortical neurons,  and this finding  has recently been  confirmed  and extended in detail using reverse correlation methods  (Jones and Palmer,  1987;  Reid  and Alonso,  1995; Reid  et al.,  1991;  llingach et al.,  1997).  In this study, we recorded from neurons in the  primary auditory cortex of the awake  primate,  and used  a  novel  re(cid:173) verse correlation technique to compute receptive fields  (or preferred  stimuli), encompassing both multiple frequency components and on(cid:173) going time.  These spectrotemporal receptive fields  make clear that  neurons in the primary auditory cortex, as in the primary visual cor(cid:173) tex, typically show considerable structure in their feature processing  properties, often including multiple excitatory and inhibitory regions  in their receptive fields.  These neurons can be sensitive to stimulus  edges in frequency  composition or in time,  and sensitive to stimulus  transitions  such as  changes in  frequency.  These  neurons  also  show  strong responses and selectivity to continuous frequency  modulated  stimuli analogous to visual drifting gratings.
A Generic Approach for Identification of Event Related Brain Potentials via a Competitive Neural Network Structure
Authors: Daniel H. Lange, Hava T. Siegelmann, Hillel Pratt, Gideon F. Inbar
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present  a  novel generic  approach to the problem of Event  Related  Potential identification and classification,  based on a competitive N eu(cid:173) ral  Net  architecture.  The network  weights  converge  to the embedded  signal  patterns,  resulting  in  the  formation  of a  matched  filter  bank.  The network performance is analyzed via a simulation study, exploring  identification  robustness  under  low  SNR conditions  and compared  to  the  expected  performance  from  an information  theoretic  perspective.  The  classifier  is  applied  to  real  event-related  potential  data  recorded  during  a  classic  odd-ball  type  paradigm;  for  the  first  time,  within(cid:173) session  variable  signal  patterns  are  automatically  identified,  dismiss(cid:173) ing  the  strong  and  limiting  requirement  of  a-priori  stimulus-related  selective  grouping of the recorded  data. 
902 
D.  H.  Lange, H.  T.  Siegelmann, H.  Pratt and G.  F.  Inbar
Structure Driven Image Database Retrieval
Authors: Jeremy S. De Bonet, Paul A. Viola
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A  new  algorithm  is  presented  which  approximates  the  perceived  visual  similarity  between  images.  The images  are  initially  trans(cid:173) formed  into  a  feature  space  which  captures  visual  structure,  tex(cid:173) ture  and  color  using  a  tree  of filters.  Similarity  is  the  inverse  of  the distance in this  perceptual feature  space.  Using this  algorithm  we  have constructed an image database system which can perform  example based retrieval on large image databases.  Using carefully  constructed target sets, which limit variation to only a single visual  characteristic, retrieval rates are quantitatively compared to those  of standard methods.
Bayesian Robustification for Audio Visual Fusion
Authors: Javier R. Movellan, Paul Mineiro
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We discuss the problem of catastrophic fusion in multimodal recog(cid:173) nition  systems.  This  problem  arises  in systems  that  need  to  fuse  different  channels in  non-stationary environments.  Practice shows  that when  recognition modules  within each modality are tested in  contexts inconsistent with their assumptions, their influence on the  fused  product tends to increase,  with catastrophic results.  We  ex(cid:173) plore  a  principled  solution  to  this  problem  based  upon  Bayesian  ideas  of  competitive  models  and  inference  robustification:  each  sensory channel is  provided  with  simple  white-noise context mod(cid:173) els,  and  the  perceptual  hypothesis  and  context  are  jointly  esti(cid:173) mated.  Consequently, context deviations are interpreted as changes  in white noise contamination strength, automatically adjusting the  influence of the module.  The approach is  tested on  a fixed  lexicon  automatic audiovisual speech  recognition problem  with very good  results.
Shared Context Probabilistic Transducers
Authors: Yoshua Bengio, Samy Bengio, Jean-Franc Isabelle, Yoram Singer
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Recently,  a model for  supervised learning of probabilistic transduc(cid:173) ers  represented  by suffix trees  was  introduced.  However,  this  algo(cid:173) rithm tends  to build very  large  trees,  requiring very large amounts  of computer memory.  In this paper, we  propose  anew, more com(cid:173) pact, transducer model in which one shares the parameters of distri(cid:173) butions  associated  to  contexts  yielding similar conditional output  distributions .  We  illustrate the  advantages  of the  proposed  algo(cid:173) rithm  with  comparative  experiments  on  inducing  a  noun  phrase  recogmzer.
Learning to Schedule Straight-Line Code
Authors: J. Eliot B. Moss, Paul E. Utgoff, John Cavazos, Doina Precup, Darko Stefanovic, Carla E. Brodley, David Scheeff
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Program execution speed on modem computers is sensitive,  by  a factor  of  two or more, to the order in which instructions are presented to the proces(cid:173) sor.  To realize potential execution efficiency,  an optimizing compiler must  employ  a heuristic  algorithm  for  instruction  scheduling.  Such  algorithms  are painstakingly hand-crafted, which is expensive and time-consuming. We  show how to cast the instruction scheduling problem as  a learning task, ob(cid:173) taining the  heuristic  scheduling  algorithm  automatically.  Our focus  is  the  narrower problem of scheduling straight-line code (also called basic blocks  of instructions).  Our empirical results  show that just a few features  are ad(cid:173) equate for quite good performance at this  task for a real modem processor,  and that  any  of several  supervised  learning  methods  perform nearly  opti(cid:173) mally with respect to the features used.
Approximating Posterior Distributions in Belief Networks Using Mixtures
Authors: Christopher M. Bishop, Neil D. Lawrence, Tommi Jaakkola, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Exact inference in densely connected Bayesian networks is computation(cid:173) ally intractable, and so there is considerable interest in developing effec(cid:173) tive approximation schemes. One approach which has been adopted is to  bound the log likelihood using a mean-field approximating distribution.  While this leads to a tractable algorithm, the mean field distribution is as(cid:173) sumed to be factorial and hence unimodal.  In this paper we demonstrate  the feasibility of using a richer class of approximating distributions based  on mixtures of mean field distributions.  We derive an efficient algorithm  for updating the mixture parameters and apply it to the problem of learn(cid:173) ing  in  sigmoid  belief networks.  Our results  demonstrate  a  systematic  improvement over  simple  mean  field  theory  as  the  number of mixture  components is increased.
The Canonical Distortion Measure in Feature Space and 1-NN Classification
Authors: Jonathan Baxter, Peter L. Bartlett
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  prove  that  the  Canonical  Distortion Measure  (CDM)  [2,  3]  is  the  optimal distance measure to use for  I  nearest-neighbour (l-NN) classifi(cid:173) cation, and show that it reduces to squared Euclidean distance in feature  space  for function classes that can be expressed  as  linear combinations  of a  fixed  set  of features.  PAC-like  bounds  are  given  on the  sample(cid:173) complexity  required to  learn the CDM.  An experiment  is presented  in  which  a  neural network CDM was  learnt for a Japanese  OCR environ(cid:173) ment and then used to do  I-NN classification.
A Non-Parametric Multi-Scale Statistical Model for Natural Images
Authors: Jeremy S. De Bonet, Paul A. Viola
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: The  observed  distribution  of natural  images  is  far  from  uniform.  On  the  contrary,  real  images  have  complex  and  important  struc(cid:173) ture  that  can  be  exploited  for  image  processing,  recognition  and  analysis.  There have been many proposed approaches to the prin(cid:173) cipled  statistical modeling of images, but each has been  limited  in  either  the complexity  of the  models  or  the  complexity  of the  im(cid:173) ages.  We present a non-parametric multi-scale statistical model for  images  that can be  used for  recognition,  image  de-noising,  and  in  a  "generative mode"  to synthesize high  quality textures.
A Solution for Missing Data in Recurrent Neural Networks with an Application to Blood Glucose Prediction
Authors: Volker Tresp, Thomas Briegel
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  consider neural  network models for stochastic nonlinear dynamical  systems  where measurements  of the  variable of interest are only avail(cid:173) able at irregular intervals i.e.  most realizations are missing.  Difficulties  arise  since the solutions for  prediction and maximum  likelihood learn(cid:173) ing with missing data lead  to  complex integrals, which even  for simple  cases  cannot  be  solved  analytically.  In  this  paper  we  propose  a  spe(cid:173) cific  combination of a nonlinear recurrent  neural  predictive model  and  a  linear error model  which  leads  to  tractable prediction and  maximum  likelihood adaptation rules.  In particular,  the  recurrent  neural  network  can be trained using the real-time recurrent learning rule and  the linear  error model  can be trained by an EM  adaptation rule,  implemented us(cid:173) ing forward-backward Kalman filter equations. The model is applied to  predict the glucose/insulin metabolism of a diabetic patient where blood  glucose measurements  are  only available a few  times a day  at  irregular  intervals.  The new model shows considerable improvement with respect  to both recurrent neural networks trained with teacher forcing or in a free  running mode and various linear models.
Regularisation in Sequential Learning Algorithms
Authors: João F. G. de Freitas, Mahesan Niranjan, Andrew H. Gee
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: In this paper,  we  discuss regularisation in online/sequential learn(cid:173) ing  algorithms.  In environments  where  data  arrives  sequentially,  techniques  such  as  cross-validation  to  achieve  regularisation  or  model  selection  are  not  possible.  Further,  bootstrapping  to  de(cid:173) termine  a  confidence  level  is  not  practical.  To  surmount  these  problems, a minimum variance estimation approach that makes use  of the extended Kalman algorithm for  training multi-layer percep(cid:173) trons is employed.  The novel contribution of this paper is  to show  the  theoretical links  between  extended  Kalman filtering,  Sutton's  variable  learning  rate  algorithms  and  Mackay's  Bayesian  estima(cid:173) tion  framework.  In  doing  so,  we  propose  algorithms to overcome  the  need  for  heuristic  choices  of  the  initial  conditions  and  noise  covariance matrices in  the Kalman approach.
Learning Human-like Knowledge by Singular Value Decomposition: A Progress Report
Authors: Thomas K. Landauer, Darrell Laham, Peter W. Foltz
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Singular value decomposition  (SVD)  can  be  viewed  as  a  method  for  unsupervised training of a network that associates two  classes  of events  reciprocally by linear connections  through  a single  hidden  layer.  SVD  was used to learn and represent relations among  very large numbers  of  words  (20k-60k)  and  very  large numbers  of natural  text  passages  (lk- 70k)  in  which  they  occurred.  The  result  was  100-350  dimensional  "semantic spaces" in which any trained or newly aibl word or passage  could be represented as  a vector,  and  similarities  were measured by  the  cosine  of  the  contained  angle  between  vectors.  Good  accmacy  in  simulating  human judgments and behaviors has  been  demonstrated  by  performance  on  multiple-choice  vocabulary  and  domain  knowledge  tests, emulation of expert essay evaluations,  and  in  several other ways.  Examples are also given of how the kind of knowledge extracted by  this  method can be applied.
Toward a Single-Cell Account for Binocular Disparity Tuning: An Energy Model May Be Hiding in Your Dendrites
Authors: Bartlett W. Mel, Daniel L. Ruderman, Kevin A. Archie
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Converging  evidence  has  shown  that  human  object  recognition  depends  on  familiarity  with  the  images  of  an  object.  Further,  the  greater  the  similarity  between  objects,  the  stronger  is  the  dependence  on  object  appearance,  and  the  more  important  two(cid:173) dimensional (2D) image information becomes.  These findings,  how(cid:173) ever, do not rule out the use of 3D structural information in recog(cid:173) nition,  and  the  degree  to  which  3D  information  is  used  in  visual  memory is an important issue.  Liu, Knill, & Kersten (1995) showed  that  any model  that  is  restricted  to  rotations  in  the  image  plane  of independent  2D  templates  could not  account for  human perfor(cid:173) mance in discriminating novel object views.  We now present results  from models of generalized radial basis functions  (GRBF), 2D near(cid:173) est  neighbor  matching that  allows  2D  affine  transformations,  and  a Bayesian statistical estimator that integrates over all possible 2D  affine  transformations.  The  performance  of the  human  observers  relative  to  each  of  the  models  is  better for  the  novel  views  than  for  the familiar  template views,  suggesting that humans generalize  better to novel  views  from  template views.  The  Bayesian estima(cid:173) tor yields the optimal performance with  2D  affine  transformations  and  independent  2D  templates.  Therefore,  models  of  2D  affine  matching  operations  with  independent  2D  templates  are unlikely  to account for  human recognition performance.
Recurrent Neural Networks Can Learn to Implement Symbol-Sensitive Counting
Authors: Paul Rodriguez, Janet Wiles
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Recently researchers have derived formal complexity analysis of analog  computation in the setting of discrete-time dynamical  systems.  As  an  empirical constrast, training recurrent neural networks (RNNs) produces  self -organized systems that are realizations of analog mechanisms.  Pre(cid:173) vious work showed that a RNN can learn to process a simple context-free  language (CFL) by counting. Herein, we extend that work to show that a  RNN can learn a harder CFL, a simple palindrome, by organizing its re(cid:173) sources into a symbol-sensitive counting solution, and we provide a dy(cid:173) namical systems analysis which demonstrates how the network: can  not  only count, but also copy and store counting infonnation.
Modeling Acoustic Correlations by Factor Analysis
Authors: Lawrence K. Saul, Mazin G. Rahim
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Hidden  Markov  models (HMMs)  for  automatic speech  recognition  rely  on  high  dimensional feature  vectors  to  summarize the  short(cid:173) time properties  of speech.  Correlations between  features  can  arise  when the speech signal is non-stationary or corrupted by  noise.  We  investigate  how  to  model  these  correlations  using  factor  analysis,  a  statistical method for  dimensionality reduction .  Factor  analysis  uses  a  small number of parameters to model  the  covariance struc(cid:173) ture  of  high  dimensional  data.  These  parameters  are  estimated  by  an  Expectation-Maximization (EM)  algorithm that can  be  em(cid:173) bedded  in  the  training  procedures  for  HMMs.  We  evaluate  the  combined  use  of  mixture  densities  and  factor  analysis  in  HMMs  that  recognize  alphanumeric strings.  Holding the  total  number of  parameters fixed,  we  find  that  these  methods, properly  combined,  yield  better models than either  method on its own.
Hippocampal Model of Rat Spatial Abilities Using Temporal Difference Learning
Authors: David J. Foster, Richard G. M. Morris, Peter Dayan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We provide a model of the standard watermaze task, and of a more  challenging task involving novel platform locations, in which rats  exhibit one-trial learning after a few days of training.  The model  uses hippocampal place cells to support reinforcement learning,  and also,  in an integrated manner,  to build  and use  allocentric  coordinates.
Hierarchical Non-linear Factor Analysis and Topographic Maps
Authors: Zoubin Ghahramani, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  first  describe  a  hierarchical,  generative  model  that  can  be  viewed  as  a  non-linear  generalisation  of factor  analysis  and  can  be  implemented  in  a  neural  network.  The  model  performs  per(cid:173) ceptual  inference  in  a  probabilistically consistent  manner by  using  top-down,  bottom-up and  lateral  connections.  These  connections  can  be  learned  using  simple  rules  that  require  only  locally  avail(cid:173) able  information.  We  then  show  how  to  incorporate  lateral  con(cid:173) nections  into the  generative  model.  The  model extracts  a  sparse,  distributed,  hierarchical  representation  of  depth  from  simplified  random-dot  stereograms  and  the  localised  disparity  detectors  in  the  first  hidden  layer  form  a  topographic  map.  When  presented  with image patches from  natural scenes,  the  model develops  topo(cid:173) graphically organised local feature detectors.
Learning Path Distributions Using Nonequilibrium Diffusion Networks
Authors: Paul Mineiro, Javier R. Movellan, Ruth J. Williams
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  propose diffusion  networks, a  type of recurrent neural network  with probabilistic dynamics, as models for  learning natural signals  that are continuous in  time and space.  We  give  a  formula for  the  gradient  of the  log-likelihood  of  a  path  with  respect  to  the  drift  parameters for  a  diffusion  network.  This  gradient  can  be used  to  optimize diffusion networks in the nonequilibrium regime for a wide  variety of problems paralleling techniques which have succeeded in  engineering  fields  such  as  system  identification,  state estimation  and  signal  filtering.  An  aspect  of this  work  which  is  of  particu(cid:173) lar interest to  computational neuroscience and hardware design  is  that with a suitable choice of activation function,  e.g.,  quasi-linear  sigmoidal, the gradient formula is  local in  space and time.
Unsupervised On-line Learning of Decision Trees for Hierarchical Data Analysis
Authors: Marcus Held, Joachim M. Buhmann
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: An adaptive on-line algorithm is proposed to estimate hierarchical  data structures for non-stationary data sources. The approach  is based on the principle of minimum cross entropy to derive a  decision tree for data clustering and it employs a metalearning idea  (learning to learn) to adapt to changes in data characteristics. Its  efficiency is demonstrated by grouping non-stationary artifical data  and by hierarchical segmentation of LANDSAT images.
Recovering Perspective Pose with a Dual Step EM Algorithm
Authors: Andrew D. J. Cross, Edwin R. Hancock
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This paper describes  a  new approach to extracting 3D  perspective  structure  from  2D  point-sets.  The  novel  feature  is  to  unify  the  tasks of estimating transformation geometry and identifying point(cid:173) correspondence matches.  Unification  is  realised  by  constructing a  mixture model over the bi-partite graph representing the correspon(cid:173) dence match and by effecting optimisation using the EM algorithm.  According to our EM framework the probabilities of structural cor(cid:173) respondence gate contributions to the expected likelihood function  used to estimate maximum likelihood perspective pose parameters.  This provides a  means of rejecting structural outliers.
Learning to Order Things
Authors: William W. Cohen, Robert E. Schapire, Yoram Singer
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: There are many applications in which it is  desirable to order rather than classify  instances.  Here we consider the problem of learning how to order, given feedback  in the form of preference judgments, i.e., statements to the effect that one instance  should be ranked ahead of another.  We outline a two-stage approach in which one  first learns by conventional means a preference Junction, of the form PREF( u, v),  which  indicates  whether  it is advisable  to  rank  u  before  v.  New  instances  are  then  ordered  so  as  to  maximize  agreements  with  the  learned  preference  func(cid:173) tion.  We  show  that  the  problem  of finding  the  ordering  that  agrees  best  with  a preference function  is  NP-complete,  even under very  restrictive assumptions.  Nevertheless, we describe a simple greedy algorithm that is guaranteed to find  a  good approximation.  We then discuss an on-line learning algorithm, based on the  "Hedge" algorithm,  for  finding  a good linear combination of ranking "experts."  We use the ordering algorithm combined with the on-line learning algorithm to  find a combination of "search experts," each of which is a domain-specific query  expansion strategy for a WWW search engine,  and present experimental  results  that demonstrate the merits of our approach.
Analog VLSI Model of Intersegmental Coordination with Nearest-Neighbor Coupling
Authors: Girish N. Patel, Jeremy H. Holleman, Stephen P. DeWeerth
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We have a developed an analog VLSI system that models the coordina(cid:173) tion of neurobiological segmental oscillators. We have implemented and  tested a system that consists of a chain of eleven pattern generating cir(cid:173) cuits that are synaptically coupled to their nearest neighbors. Each pat(cid:173) tern  generating  circuit  is  implemented  with  two  silicon  Morris-Lecar  neurons that are connected in a reciprocally inhibitory network. We dis(cid:173) cuss the mechanisms of oscillations in the two-cell network and explore  system  behavior based  on  isotropic  and  anisotropic  coupling, and  fre(cid:173) quency gradients along the chain of oscillators.
Globally Optimal On-line Learning Rules
Authors: Magnus Rattray, David Saad
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: We  present a method for  determining the globally optimal on-line  learning rule for  a  soft  committee machine under a  statistical me(cid:173) chanics  framework.  This  work  complements  previous  results  on  locally  optimal  rules,  where  only  the  rate  of change  in  general(cid:173) ization error was  considered.  We  maximize the total reduction in  generalization error over the whole learning process and show how  the resulting rule can significantly outperform the locally optimal  rule.
Ensemble Learning for Multi-Layer Networks
Authors: David Barber, Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: Bayesian  treatments  of learning  in  neural  networks  are  typically  based  either  on  local  Gaussian  approximations  to  a  mode  of  the  posterior  weight  distribution,  or  on  Markov  chain  Monte  Carlo  simulations.  A  third  approach,  called  ensemble  learning,  was  in(cid:173) troduced by Hinton and van  Camp  (1993).  It aims to approximate  the  posterior  distribution  by  minimizing  the  Kullback-Leibler  di(cid:173) vergence  between the true posterior and a parametric approximat(cid:173) ing distribution.  However,  the  derivation  of a  deterministic  algo(cid:173) rithm  relied  on  the  use  of a  Gaussian  approximating distribution  with  a  diagonal  covariance  matrix  and  so  was  unable  to capture  the  posterior  correlations  between  parameters.  In this  paper,  we  show  how the ensemble learning approach can  be extended to full(cid:173) covariance Gaussian distributions while remaining computationally  tractable.  We also extend the framework to deal with hyperparam(cid:173) eters,  leading  to  a  simple  re-estimation  procedure.  Initial  results  from  a standard benchmark problem are encouraging.
Estimating Dependency Structure as a Hidden Variable
Authors: Marina Meila, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: This  paper introduces a probability model,  the mixture of trees  that can  account for sparse, dynamically changing dependence relationships.  We  present a  family  of efficient  algorithms  that use EM and  the Minimum  Spanning Tree algorithm to find the ML and MAP mixture of trees for a  variety of priors, including the Dirichlet and the MDL priors.
Ensemble and Modular Approaches for Face Detection: A Comparison
Authors: Raphaël Feraud, Olivier Bernier
Published in: Advances in Neural Information Processing Systems 10  (NIPS 1997) (Main Conference Track), 1997

Abstract: A  new  learning  model  based  on  autoassociative  neural  networks  is  developped  and  applied  to  face  detection.  To  extend  the  de(cid:173) tection  ability  in  orientation  and  to  decrease  the  number  of false  alarms,  different  combinations  of networks  are  tested:  ensemble,  conditional  ensemble  and  conditional  mixture  of  networks.  The  use  of a  conditional mixture of networks  allows  to  obtain state of  the  art results on different  benchmark face  databases. 
1  A  constrained generative model 
Our  purpose  is  to  classify  an  extracted  window  x  from  an  image  as  a  face  (x  E  V)  or  non-face  (x  EN).  The  set  of all  possible  windows  is  E  =  V uN,  with  V n N  = 0.  Since collecting a representative set of non-face examples is impossible,  face  detection  by a statistical model is  a difficult  task.  An autoassociative network,  using five  layers of neurons, is able to perform a non-linear dimensionnality reduction  [Kramer,  1991].  However,  its use  as  an  estimator, to classify  an  extracted  window  as  face  or  non-face,  raises  two  problems: 

V',  the obtained sub-manifold can contain  non-face examples (V  C  V'),  2.  owing  to  local  minima,  the  obtained  solution  can  be  close  to  the  linear 

solution:  the  principal components  analysis. 
Our approach is  to use counter-examples in  order to find  a sub-manifold as  close  as  possible  to  V  and  to  constrain  the  algorithm to  converge  to  a  non-linear  solution  [Feraud,  R.  et  al.,  1997].  Each non-face example is  constrained to be  reconstructed  as  its  projection on V.  The projection  P  of a  point  x  of the  input space  E on V,  is  defined  by: 
·email:  feraud@lannion.cnet.fr  t email:  bernier@lannion.cnet.fr 
Ensemble and Modular Approaches for Face Detection: A Comparison 
473 
•  if x  E V,  then P{ x)  =  x,  •  if x rJ.  V:  P{x)  = argminYEv{d(x, V)),  where  d is  the  Euclidian distance. 
During  the  learning  process,  the  projection  P  of x  on  V  is  approximated  by:  P(x)  ,....,  ~ 2:7=1 Vi,  where  VI, V2,  •.. , Vn ,  are  the  n  nearest  neighbours,  in  the training set  of faces,  of v,  the nearest  face  example of x. 
The goal of the learning process  is  to approximate the distance V  of an input space  element  x  to the set  of faces  V: 
•  V{x, V)  =  Ilx  - P(x)11  ,....,  it (x  - £)2,  where  M  is  the size  of input image x 
and £  the  image reconstructed  by  the neural  network, 
•  let  x  E £,  then  x  E V if and only if V{x, V)  S  T,  with  T  E IR,  where  T  is  a 
threshold  used  to adjust the sensitivity of the model.