mtybilly/Instruct2DS huggingface dataset:
An Information-Theoretic Framework for Understanding Saccadic Eye Movements
Authors: Tai Sing Lee, Stella X. Yu
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this paper,  we  propose that information maximization can  pro(cid:173) vide  a  unified  framework  for  understanding  saccadic  eye  move(cid:173) ments.  In  this framework,  the mutual information among the cor(cid:173) tical  representations  of the  retinal  image,  the  priors  constructed  from  our  long  term  visual  experience,  and  a  dynamic  short-term  internal  representation  constructed  from  recent  saccades  provides  a  map for  guiding eye  navigation .  By  directing  the  eyes  to loca(cid:173) tions  of maximum complexity in  neuronal  ensemble  responses  at  each  step,  the  automatic saccadic  eye  movement system  greedily  collects information about the external world, while modifying the  neural  representations  in  the  process.  This  framework  attempts  to  connect  several  psychological  phenomena,  such  as  pop-out  and  inhibition of return,  to long term visual experience  and short term  working  memory.  It  also  provides  an  interesting  perspective  on  contextual computation and formation of neural  representation  in  the visual system.
Noisy Neural Networks and Generalizations
Authors: Hava T. Siegelmann, Alexander Roitershtein, Asa Ben-Hur
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this paper we  define  a  probabilistic computational model which  generalizes many noisy neural network models, including the recent  work  of Maass and Sontag [5].  We identify weak  ergodicjty  as  the  mechanism responsible for  restriction  of the computational power  of  probabilistic  models  to  definite  languages,  independent  of the  characteristics  of the noise:  whether  it is  discrete  or analog,  or if  it  depends  on  the  input  or  not,  and  independent  of whether  the  variables  are  discrete  or continuous.  We  give examples of weakly  ergodic  models including  noisy  computational systems  with noise  depending on the current state and inputs, aggregate models,  and  computational systems which update in continuous time.
The Nonnegative Boltzmann Machine
Authors: Oliver B. Downs, David J. C. MacKay, Daniel D. Lee
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The nonnegative Boltzmann machine (NNBM) is a recurrent neural net(cid:173) work model that can describe multimodal nonnegative data.  Application  of maximum likelihood estimation to this model gives a learning rule that  is analogous to the binary Boltzmann machine. We examine the utility of  the mean field  approximation for the  NNBM,  and describe  how Monte  Carlo sampling techniques can be  used to  learn its parameters.  Reflec(cid:173) tive  slice  sampling  is  particularly well-suited  for  this  distribution,  and  can efficiently be implemented to sample the  distribution.  We  illustrate  learning of the NNBM on a transiationally invariant distribution, as well  as on a generative model for images of human faces.
Boosting Algorithms as Gradient Descent
Authors: Llew Mason, Jonathan Baxter, Peter L. Bartlett, Marcus R. Frean
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  provide an abstract characterization of boosting algorithms as  gradient  decsent  on  cost-functionals  in  an  inner-product  function  space.  We  prove convergence  of these functional-gradient-descent  algorithms under quite  weak conditions.  Following previous theo(cid:173) retical  results  bounding the generalization  performance of convex  combinations of classifiers in terms of general cost functions of the  margin,  we  present  a  new  algorithm  (DOOM  II)  for  performing a  gradient descent  optimization of such cost functions.  Experiments  on  several  data sets  from  the  UC  Irvine  repository  demonstrate  that DOOM II generally outperforms AdaBoost, especially in high  noise situations, and that the overfitting behaviour of AdaBoost is  predicted by our cost functions.
Bayesian Reconstruction of 3D Human Motion from Single-Camera Video
Authors: Nicholas R. Howe, Michael E. Leventon, William T. Freeman
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The three-dimensional motion of humans is  underdetermined when  the  observation is  limited to a single camera, due to the inherent 3D ambi(cid:173) guity of 2D video.  We present a system that reconstructs the 3D motion  of human subjects from single-camera video, relying on prior knowledge  about  human motion,  learned  from  training  data,  to  resolve  those am(cid:173) biguities.  After initialization in  2D, the tracking and 3D reconstruction  is  automatic;  we show results  for several  video sequences.  The results  show the power of treating 3D body tracking as an inference problem.
Mixture Density Estimation
Authors: Jonathan Q. Li, Andrew R. Barron
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Gaussian mixtures (or so-called radial basis function networks)  for  density estimation provide a natural counterpart to sigmoidal neu(cid:173) ral networks for function fitting and approximation.  In both cases,  it is  possible to give  simple expressions for  the iterative improve(cid:173) ment of performance as components of the network are introduced  one at a time.  In particular, for mixture density estimation we show  that a k-component mixture estimated by maximum likelihood  (or  by an iterative likelihood improvement that we introduce) achieves  log-likelihood  within order  1/k of the log-likelihood  achievable by  any convex combination.  Consequences for  approximation and es(cid:173) timation  using  Kullback-Leibler  risk  are  also  given.  A  Minimum  Description Length principle selects the optimal number of compo(cid:173) nents k that minimizes the risk bound.
Information Capacity and Robustness of Stochastic Neuron Models
Authors: Elad Schneidman, Idan Segev, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  reliability  and  accuracy  of  spike  trains  have  been  shown  to  depend  on  the  nature  of  the  stimulus  that  the  neuron  encodes.  Adding  ion  channel  stochasticity  to  neuronal  models  results  in  a  macroscopic behavior that replicates the input-dependent reliabili(cid:173) ty and precision of real neurons.  We  calculate the amount of infor(cid:173) mation that an ion channel based stochastic Hodgkin-Huxley (HH)  neuron model can encode about a wide set of stimuli.  We show that  both the information  rate and the information  per  spike  of the s(cid:173) tochastic model are similar to the values  reported experimentally.  Moreover,  the  amount  of information  that  the  neuron  encodes  is  correlated with the amplitude of fluctuations in the input, and less  so with the average firing rate of the neuron.  We also show that for  the HH  ion  channel density,  the information capacity is  robust  to  changes  in  the density  of ion  channels  in  the  membrane,  whereas  changing  the  ratio  between  the  Na+  and  K+  ion  channels  has  a  considerable effect on the information that the neuron can encode.  Finally,  we  suggest  that neurons  may maximize  their information  capacity by appropriately balancing the density of the different ion  channels that underlie neuronal excitability.
Bayesian Transduction
Authors: Thore Graepel, Ralf Herbrich, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Transduction  is  an  inference  principle  that  takes  a  training sam(cid:173) ple  and aims at estimating the values of a function  at given points  contained in the so-called working sample as  opposed  to the whole  of input space  for  induction.  Transduction  provides  a  confidence  measure  on  single  predictions  rather  than  classifiers  - a  feature  particularly important for  risk-sensitive  applications.  The possibly  infinite number of functions  is  reduced  to a finite  number of equiv(cid:173) alence classes on the working sample.  A rigorous Bayesian analysis  reveals  that for  standard classification loss  we  cannot benefit  from  considering  more  than  one  test  point  at  a  time.  The  probability  of  the  label  of a  given  test  point  is  determined  as  the  posterior  measure of the corresponding subset  of hypothesis  space.  We  con(cid:173) sider the PAC setting of binary classification by linear discriminant  functions  (perceptrons)  in kernel space such that the probability of  labels  is  determined  by  the  volume  ratio  in  version  space.  We  suggest  to sample this  region  by  an  ergodic  billiard.  Experimen(cid:173) tal  results  on  real  world data indicate that Bayesian Transduction  compares  favourably  to  the  well-known  Support  Vector  Machine,  in  particular  if the  posterior  probability  of labellings  is  used  as  a  confidence  measure to exclude test  points of low  confidence.
Constructing Heterogeneous Committees Using Input Feature Grouping: Application to Economic Forecasting
Authors: Yuansong Liao, John E. Moody
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  committee  approach  has  been  proposed  for  reducing  model  uncertainty  and  improving  generalization  performance.  The  ad(cid:173) vantage of committees depends  on  (1)  the performance of individ(cid:173) ual members  and  (2)  the correlational structure of errors between  members.  This paper presents an input grouping technique for  de(cid:173) signing a  heterogeneous  committee.  With  this  technique,  all input  variables are first grouped based on their mutual information.  Sta(cid:173) tistically  similar  variables  are  assigned  to  the  same  group.  Each  member's  input  set  is  then  formed  by  input  variables  extracted  from different groups.  Our designed  committees have less error cor(cid:173) relation between its members, since each member observes different  input variable combinations.  The individual member's feature sets  contain less redundant information, because highly correlated vari(cid:173) ables will not be combined together.  The member feature sets con(cid:173) tain almost complete information, since each set contains a feature  from  each  information group.  An  empirical study for  a  noisy  and  nonstationary  economic  forecasting  problem  shows  that  commit(cid:173) tees constructed by our proposed technique outperform committees  formed using several existing techniques.
An Analysis of Turbo Decoding with Gaussian Densities
Authors: Paat Rusmevichientong, Benjamin Van Roy
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  provide  an  analysis  of  the  turbo  decoding  algorithm  (TDA)  in  a  setting involving  Gaussian  densities.  In  this  context,  we  are  able  to show  that  the  algorithm  converges  and  that  - somewhat  surprisingly - though the density generated by the TDA may differ  significantly from  the desired posterior density,  the means of these  two densities  coincide. 
1
Neural System Model of Human Sound Localization
Authors: Craig T. Jin, Simon Carlile
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: This paper examines the role of biological constraints in the human audi(cid:173) tory localization process.  A psychophysical and neural system modeling  approach  was  undertaken  in which  performance  comparisons between  competing  models  and  a  human  subject  explore  the  relevant  biologi(cid:173) cally plausible  "realism  constraints".  The  directional  acoustical  cues,  upon which sound localization is  based,  were  derived  from  the  human  subject's  head-related transfer functions  (HRTFs).  Sound stimuli  were  generated by convolving bandpass noise with the HRTFs and were pre(cid:173) sented to both the subject and the model.  The input stimuli to the model  was processed using the Auditory Image Model of cochlear processing.  The  cochlear  data  was  then  analyzed  by  a  time-delay  neural  network  which integrated temporal and spectral information to determine the spa(cid:173) tial  location  of the  sound  source.  The  combined  cochlear  model  and  neural network provided a system model of the sound localization pro(cid:173) cess.  Human-like  localization performance  was  qualitatively  achieved  for broadband and bandpass stimuli when the model architecture incor(cid:173) porated frequency division (or tonotopicity), and was trained using vari(cid:173) able bandwidth and center-frequency sounds.
A Neuromorphic VLSI System for Modeling the Neural Control of Axial Locomotion
Authors: Girish N. Patel, Edgar A. Brown, Stephen P. DeWeerth
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We have developed and tested an analog/digital VLSI system that mod(cid:173) els the coordination of biological segmental oscillators underlying axial  locomotion in animals such as leeches and lampreys. In its current form  the system consists of a chain of twelve pattern generating circuits that  are capable of arbitrary contralateral inhibitory synaptic coupling. Each  pattern generating circuit is implemented with two independent silicon  Morris-Lecar neurons with a total of 32 programmable (floating-gate  based) inhibitory synapses, and an asynchronous address-event inter(cid:173) connection element that provides synaptic connectivity and implements  axonal delay. We describe and analyze the data from a set of experi(cid:173) ments exploring the system behavior in terms of synaptic coupling.
Graded Grammaticality in Prediction Fractal Machines
Authors: Shan Parfitt, Peter Tiño, Georg Dorffner
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce  a  novel  method  of constructing  language  models,  which  avoids some of the problems associated  with recurrent  neu(cid:173) ral networks.  The method of creating a Prediction Fractal Machine  (PFM)  [1]  is  briefly described  and some experiments are presented  which  demonstrate the suitability of PFMs for  language modeling.  PFMs  distinguish  reliably  between  minimal  pairs,  and  their  be(cid:173) havior is  consistent  with the hypothesis  [4]  that wellformedness  is  'graded' not absolute.  A discussion of their potential to offer fresh  insights into language acquisition and processing follows.
Learning Sparse Codes with a Mixture-of-Gaussians Prior
Authors: Bruno A. Olshausen, K. Jarrod Millman
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  describe  a  method  for  learning  an  overcomplete  set  of  basis  functions  for  the  purpose of modeling  sparse structure in images.  The  sparsity  of the  basis  function  coefficients  is  modeled  with  a  mixture-of-Gaussians  distribution.  One  Gaussian  captures  non(cid:173) active  coefficients  with  a  small-variance  distribution  centered  at  zero,  while one or more other Gaussians capture active coefficients  with a large-variance distribution.  We show that when the prior is  in such  a form,  there exist efficient  methods for  learning the basis  functions  as  well  as  the parameters of the prior.  The performance  of the  algorithm  is  demonstrated  on  a  number  of test  cases  and  also  on  natural  images.  The  basis  functions  learned  on  natural  images  are similar to those  obtained with other methods,  but the  sparse form of the coefficient distribution is much better described.  Also, since the parameters of the prior are adapted to the data, no  assumption about sparse  structure in the  images  need  be  made  a  priori,  rather it is  learned from  the data.
Semiparametric Approach to Multichannel Blind Deconvolution of Nonminimum Phase Systems
Authors: Liqing Zhang, Shun-ichi Amari, Andrzej Cichocki
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this  paper we discuss  the  semi parametric statistical  model  for  blind  deconvolution.  First we  introduce a Lie Group to  the  manifold of non(cid:173) causal  FIR  filters.  Then  blind deconvolution problem is  formulated  in  the  framework of a  semiparametric  model,  and  a  family  of estimating  functions  is  derived for  blind  deconvolution.  A  natural gradient  learn(cid:173) ing algorithm is developed for training noncausal filters.  Stability of the  natural gradient algorithm is also analyzed in this framework.
Application of Blind Separation of Sources to Optical Recording of Brain Activity
Authors: Holger Schoner, Martin Stetter, Ingo Schießl, John E. W. Mayhew, Jennifer S. Lund, Niall McLoughlin, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In the analysis of data recorded by optical imaging from intrinsic signals  (measurement of changes of light reflectance from cortical tissue) the re(cid:173) moval  of noise and  artifacts  such  as  blood  vessel  patterns  is  a  serious  problem. Often bandpass filtering is used, but the underlying assumption  that a spatial frequency  exists,  which separates  the mapping component  from  other components  (especially  the  global  signal),  is  questionable.  Here we propose alternative ways of processing optical imaging data, us(cid:173) ing blind source separation techniques based on the spatial decorre1ation  of the data.  We  first  perform  benchmarks  on  artificial  data in  order  to  select  the  way  of processing, which is  most robust with respect  to  sen(cid:173) sor noise.  We then apply it to recordings of optical imaging experiments  from  macaque primary visual cortex. We show that our BSS technique is  able  to  extract ocular dominance and orientation preference maps  from  single  condition  stacks,  for  data,  where  standard  post-processing pro(cid:173) cedures  fail.  Artifacts,  especially  blood  vessel  patterns,  can  often  be  completely removed  from  the maps.  In  summary,  our method for blind  source separation using extended spatial decorrelation is a superior tech(cid:173) nique for the analysis of optical recording data.
Spectral Cues in Human Sound Localization
Authors: Craig T. Jin, Anna Corderoy, Simon Carlile, André van Schaik
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  differential  contribution  of the  monaural  and  interaural  spectral  cues to human sound localization was examined using a combined psy(cid:173) chophysical  and  analytical  approach.  The  cues  to  a  sound's  location  were  correlated  on an individual basis  with  the  human  localization re(cid:173) sponses to a variety of spectrally manipulated sounds.  The spectral cues  derive from the acoustical filtering of an individual's auditory periphery  which  is  characterized by the measured head-related transfer functions  (HRTFs).  Auditory localization performance was determined in  virtual  auditory space (VAS).  Psychoacoustical experiments were conducted in  which the amplitude spectra of the sound stimulus was  varied indepen(cid:173) dentlyat each ear while preserving the normal timing cues, an impossibil(cid:173) ity in the free-field environment. Virtual auditory noise stimuli were gen(cid:173) erated over earphones for a specified target direction such that there was  a "false" flat  spectrum at the left eardrum.  Using the subject's HRTFs,  the sound spectrum at the right eardrum was then adjusted so that either  the  true right  monaural  spectral  cue  or the  true  interaural  spectral  cue  was preserved.  All  subjects showed systematic mislocalizations in both  the true right and true interaural spectral conditions which was absent in  their control localization performance. The analysis of the different cues  along with the subjects' localization responses suggests there are signif(cid:173) icant differences in the use of the monaural and interaural spectral cues  and that the auditory system's reliance on the spectral  cues varies with  the sound condition.
Robust Full Bayesian Methods for Neural Networks
Authors: Christophe Andrieu, João F. G. de Freitas, Arnaud Doucet
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this paper, we propose a full Bayesian model for neural networks.  This model treats the model dimension (number of neurons), model  parameters, regularisation parameters and noise parameters as ran(cid:173) dom  variables  that  need  to be estimated.  We  then  propose  a  re(cid:173) versible jump Markov chain Monte Carlo (MCMC)  method to per(cid:173) form  the necessary computations.  We  find  that the results are not  only  better than the previously  reported ones,  but  also  appear to  be  robust  with  respect  to  the  prior  specification.  Moreover,  we  present a  geometric convergence theorem for  the algorithm.
A Neurodynamical Approach to Visual Attention
Authors: Gustavo Deco, Josef Zihl
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The psychophysical evidence for "selective attention" originates mainly  from  visual search experiments. In this work, we formulate a hierarchi(cid:173) cal  system of interconnected modules consisting in  populations of neu(cid:173) rons  for  modeling  the  underlying  mechanisms  involved  in  selective  visual  attention.  We  demonstrate  that  our  neural  system  for  visual  search  works  across the  visual  field  in  parallel  but due  to the  different  intrinsic dynamics can show the two experimentally observed modes of  visual  attention,  namely:  the  serial  and  the  parallel  search  mode.  In  other words, neither explicit model of a focus of attention nor saliencies  maps are  used.  The  focus of attention  appears as an  emergent property  of the dynamic behavior of the system. The neural population dynamics  are  handled in  the  framework  of the  mean-field  approximation.  Conse(cid:173) quently, the whole process can be expressed as a system of coupled dif(cid:173) ferential equations.
Dynamics of Supervised Learning with Restricted Training Sets and Noisy Teachers
Authors: Anthony C. C. Coolen, C. W. H. Mace
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We generalize a recent formalism to describe the dynamics of supervised  learning in layered neural networks, in the regime where data recycling  is  inevitable, to the case of noisy teachers.  Our theory generates reliable  predictions for the  evolution in  time of training- and generalization er(cid:173) rors, and extends the class of mathematically solvable learning processes  in large neural networks to those situations where overfitting can occur.
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Authors: John R. Hershey, Javier R. Movellan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Psychophysical and physiological evidence shows that sound local(cid:173) ization of acoustic signals is strongly influenced by  their synchrony  with visual signals.  This effect,  known as ventriloquism, is at work  when  sound  coming  from  the  side  of a  TV  set  feels  as  if it  were  coming  from  the  mouth  of the  actors.  The  ventriloquism  effect  suggests that there is  important information about sound location  encoded in  the synchrony between the audio and video signals.  In  spite  of  this  evidence,  audiovisual  synchrony  is  rarely  used  as  a  source of information  in  computer  vision  tasks.  In  this  paper  we  explore the use  of audio  visual  synchrony to locate sound sources.  We developed a system that searches for regions of the visual land(cid:173) scape that correlate highly with the acoustic signals and tags them  as  likely  to contain an acoustic  source.  We  discuss  our experience  implementing the system,  present results on a  speaker localization  task and discuss potential applications of the approach.
Predictive Sequence Learning in Recurrent Neocortical Circuits
Authors: Rajesh P. N. Rao, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Neocortical circuits are dominated by massive excitatory feedback:  more  than eighty percent of the synapses made by excitatory cortical neurons  are onto other excitatory cortical neurons.  Why is there such massive re(cid:173) current excitation in the neocortex and what is its role in cortical compu(cid:173) tation? Recent neurophysiological experiments have shown that the plas(cid:173) ticity of recurrent neocortical synapses is governed by a temporally asym(cid:173) metric Hebbian learning rule.  We  describe how such a rule may  allow  the cortex to modify recurrent synapses for prediction of input sequences.  The goal is to predict the next cortical input from the recent past based on  previous experience of similar input sequences. We show that a temporal  difference learning rule for prediction used in conjunction with dendritic  back-propagating action potentials reproduces the temporally asymmet(cid:173) ric Hebbian plasticity observed physiologically. Biophysical simulations  demonstrate that a network of cortical neurons can learn to predict mov(cid:173) ing stimuli and develop direction selective responses as a consequence of  learning. The space-time response properties of model neurons are shown  to be similar to those of direction selective cells in alert monkey VI.
Differentiating Functions of the Jacobian with Respect to the Weights
Authors: Gary William Flake, Barak A. Pearlmutter
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: For many problems, the correct behavior of a model depends not only on  its input-output mapping but also on properties of its Jacobian matrix, the  matrix of partial derivatives of the model's outputs with respect to its in(cid:173) puts.  We introduce the J-prop algorithm, an efficient general method for  computing the exact partial derivatives of a variety of simple functions of  the Jacobian of a model with respect to its free parameters. The algorithm  applies to any parametrized feedforward model,  including nonlinear re(cid:173) gression, multilayer perceptrons, and radial basis function networks.
Effects of Spatial and Temporal Contiguity on the Acquisition of Spatial Information
Authors: Thea B. Ghiselli-Crippa, Paul W. Munro
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Spatial  information comes in two forms:  direct spatial information (for  example, retinal position) and indirect temporal contiguity information,  since objects encountered sequentially are in general spatially close. The  acquisition  of spatial  information  by  a  neural  network  is  investigated  here. Given a spatial layout of several objects, networks are trained on a  prediction task.  Networks using temporal sequences with no direct spa(cid:173) tial information are found to  develop internal representations that show  distances correlated with distances in the external layout.  The influence  of spatial information is analyzed by providing direct spatial information  to the system during training that is  either consistent with the layout or  inconsistent with  it.  This  approach  allows  examination of the relative  contributions of spatial and temporal contiguity.
Agglomerative Information Bottleneck
Authors: Noam Slonim, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  introduce a novel distributional clustering algorithm that max(cid:173) imizes  the  mutual  information  per  cluster  between  data and  giv(cid:173) en  categories.  This  algorithm  can  be considered  as  a  bottom  up  hard  version  of  the  recently  introduced  "Information  Bottleneck  Method".  The algorithm is  compared with the top-down soft  ver(cid:173) sion  of the  information  bottleneck  method  and  a  relationship  be(cid:173) tween the hard and soft results is established.  We  demonstrate the  algorithm on the 20 Newsgroups data set.  For a subset of two news(cid:173) groups  we  achieve compression  by  3  orders  of magnitudes loosing  only  10%  of the original mutual information.
Spike-based Learning Rules and Stabilization of Persistent Neural Activity
Authors: Xiaohui Xie, H. Sebastian Seung
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  analyze  the  conditions  under  which  synaptic  learning  rules  based  on  action  potential timing can  be approximated by  learning rules  based  on  firing  rates.  In  particular, we  consider a form  of plasticity in  which  synapses depress when a presynaptic spike is followed by a postsynaptic  spike, and potentiate with the opposite temporal ordering.  Such differen(cid:173) tial anti-Hebbian plasticity can be approximated under certain conditions  by a learning rule that depends on the time derivative of the postsynaptic  firing rate.  Such a learning rule acts to stabilize persistent neural activity  patterns in recurrent neural networks.
Nonlinear Discriminant Analysis Using Kernel Functions
Authors: Volker Roth, Volker Steinhage
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Fishers linear discriminant analysis  (LDA)  is  a  classical multivari(cid:173) ate technique both for  dimension reduction and classification.  The  data vectors are transformed into a low dimensional subspace such  that  the  class  centroids  are  spread  out  as  much  as  possible.  In  this subspace LDA  works  as  a  simple  prototype classifier  with lin(cid:173) ear decision  boundaries.  However,  in many applications the linear  boundaries  do  not  adequately separate the  classes.  We  present  a  nonlinear generalization of discriminant analysis that uses the ker(cid:173) nel trick of representing dot products by kernel functions.  The pre(cid:173) sented algorithm allows a simple formulation of the EM-algorithm  in terms of kernel functions which leads to a unique concept for  un(cid:173) supervised  mixture analysis,  supervised discriminant  analysis  and  semi-supervised discriminant analysis with partially unlabelled ob(cid:173) servations in feature spaces.
Uniqueness of the SVM Solution
Authors: Christopher J. C. Burges, David J. Crisp
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  give  necessary  and  sufficient  conditions  for  uniqueness  of the  support vector solution for the problems of pattern recognition and  regression estimation, for a general class of cost functions.  We show  that if the solution is not unique, all support vectors are necessarily  at  bound,  and  we  give  some  simple examples of non-unique solu(cid:173) tions.  We  note that uniqueness of the primal (dual)  solution does  not necessarily imply uniqueness of the dual (primal) solution.  We  show  how to compute the threshold b when the solution is  unique,  but when  all support vectors are at bound, in which case the usual  method for  determining b does  not work.
The Parallel Problems Server: an Interactive Tool for Large Scale Machine Learning
Authors: Charles Lee Isbell Jr., Parry Husbands
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Imagine that you wish to classify data consisting of tens of thousands of ex(cid:173) amples residing in a twenty thousand dimensional space.  How can one ap(cid:173) ply standard machine learning algorithms? We describe the Parallel Prob(cid:173) lems Server (PPServer) and MATLAB*P.  In tandem they  allow users  of  networked computers to work transparently on large data sets from within  Matlab.  This  work is motivated by the desire to  bring the many  benefits  of scientific computing algorithms and computational power to  machine  learning researchers.  We  demonstrate the usefulness  of the system on  a number of tasks.  For  example,  we perform independent components analysis on very large text  corpora consisting  of tens  of thousands of documents,  making  minimal  changes  to  the original Bell  and  Sejnowski Matlab source  (Bell  and  Se(cid:173) jnowski,  1995).  Applying ML techniques to data previously beyond their  reach leads to interesting analyses of both data and algorithms.
Search for Information Bearing Components in Speech
Authors: Howard Hua Yang, Hynek Hermansky
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  this  paper,  we  use  mutual information to characterize  the  dis(cid:173) tributions of phonetic  and speaker/channel  information in  a  time(cid:173) frequency  space.  The  mutual information  (MI)  between  the  pho(cid:173) netic label and one feature,  and the joint mutual information (JMI)  between the phonetic label and two or three features are estimated .  The Miller's bias formulas for  entropy and mutual information es(cid:173) timates  are  extended  to include  higher  order  terms.  The  MI  and  the  JMI for  speaker/channel  recognition  are  also  estimated.  The  results  are complementary to those for phonetic classification.  Our  results  show  how  the  phonetic  information is  locally  spread  and  how  the  speaker/channel  information  is  globally  spread  in  time  and frequency.
An Oscillatory Correlation Frame work for Computational Auditory Scene Analysis
Authors: Guy J. Brown, DeLiang L. Wang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  neural  model  is  described  which  uses  oscillatory  correlation  to  segregate speech from  interfering sound sources. The core of the model  is  a two-layer neural  oscillator network.  A sound  stream  is  represented  by  a  synchronized  population  of oscillators,  and  different  streams  are  represented  by  desynchronized  oscillator  populations.  The  model  has  been evaluated using a corpus of speech mixed with interfering sounds,  and produces an  improvement in signal-to-noise ratio for every  mixture.
Manifold Stochastic Dynamics for Bayesian Learning
Authors: Mark Zlochin, Yoram Baram
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We propose a new Markov Chain Monte Carlo algorithm which is a gen(cid:173) eralization of the stochastic dynamics method.  The algorithm performs  exploration of the state space using its intrinsic geometric structure, facil(cid:173) itating efficient sampling of complex distributions. Applied to Bayesian  learning in neural networks, our algorithm was found to perform at least  as well as the best state-of-the-art method while consuming considerably  less time.
Bifurcation Analysis of a Silicon Neuron
Authors: Girish N. Patel, Gennady S. Cymbalyuk, Ronald L. Calabrese, Stephen P. DeWeerth
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  have developed a VLSI  silicon  neuron  and  a corresponding  mathe(cid:173) matical  model  that  is  a two  state-variable system. We  describe the  cir(cid:173) cuit implementation and compare the  behaviors  observed in  the  silicon  neuron and the mathematical model. We also perform bifurcation analy(cid:173) sis of the mathematical model by  varying the externally applied current  and show that the behaviors exhibited by the silicon neuron under corre(cid:173) sponding  conditions  are  in  good  agreement  to  those  predicted  by  the  bifurcation analysis.
Speech Modelling Using Subspace and EM Techniques
Authors: Gavin Smith, João F. G. de Freitas, Tony Robinson, Mahesan Niranjan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The speech waveform can be modelled as  a piecewise-stationary linear  stochastic state space system, and its parameters can be estimated using  an  expectation-maximisation (EM)  algorithm.  One problem is  the  ini(cid:173) tialisation of the EM algorithm. Standard initialisation schemes can lead  to poor formant trajectories.  But these  trajectories however are impor(cid:173) tant for  vowel  intelligibility.  The aim of this paper is to investigate the  suitability of subspace identification methods to initialise EM.  The  paper  compares  the  subspace  state  space  system  identification  (4SID) method with the EM algorithm.  The 4SID and EM methods are  similar in that they both estimate a state sequence (but using Kalman fil(cid:173) ters and Kalman smoothers respectively),  and then estimate parameters  (but using least-squares and maximum likelihood respectively). The sim(cid:173) ilarity of 4SID and EM motivates the use of 4SID to initialise EM. Also,  4SID is non-iterative and requires no initialisation, whereas EM is itera(cid:173) tive and requires initialisation.  However 4SID is sub-optimal compared  to EM in a probabilistic sense. During experiments on real speech, 4SID  methods compare favourably with conventional initialisation techniques.  They produce smoother formant trajectories, have greater frequency res(cid:173) olution, and produce higher likelihoods. 
1  Work done while in Cambridge Engineering Dept., UK. 
Speech Modelling Using Subspace and EM Techniques 
797
Optimal Kernel Shapes for Local Linear Regression
Authors: Dirk Ormoneit, Trevor Hastie
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Local linear regression performs very well in many low-dimensional  forecasting problems.  In high-dimensional spaces, its performance  typically  decays  due  to the  well-known  "curse-of-dimensionality".  A possible way to approach this problem is  by varying the  "shape"  of the weighting kernel.  In this work we suggest a new, data-driven  method to estimating the optimal  kernel  shape.  Experiments  us(cid:173) ing an artificially generated data set and data from  the UC  Irvine  repository show the benefits of kernel shaping.
Robust Neural Network Regression for Offline and Online Learning
Authors: Thomas Briegel, Volker Tresp
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  replace  the  commonly  used  Gaussian  noise  model  in  nonlinear  regression  by  a  more  flexible  noise  model  based  on  the  Student-t(cid:173) distribution.  The degrees of freedom of the t-distribution can be  chosen  such that as  special cases either the Gaussian distribution or the Cauchy  distribution are  realized.  The  latter is  commonly used in  robust regres(cid:173) sion.  Since the t-distribution can be  interpreted as being an infinite mix(cid:173) ture of Gaussians,  parameters and  hyperparameters such  as  the  degrees  of freedom of the t-distribution can be learned from the data based on an  EM-learning algorithm.  We  show that modeling using the  t-distribution  leads  to  improved  predictors  on  real  world  data  sets.  In  particular,  if  outliers are present,  the  t-distribution  is  superior to  the  Gaussian  noise  model.  In  effect, by  adapting  the  degrees of freedom,  the  system  can  "learn" to distinguish  between outliers  and  non-outliers.  Especially for  online learning tasks, one is  interested in  avoiding inappropriate weight  changes  due  to  measurement  outliers  to  maintain  stable  online  learn(cid:173) ing  capability.  We  show experimentally that using the  t-distribution  as  a noise model leads to stable online learning algorithms and outperforms  state-of-the art online learning methods like  the  extended Kalman filter  algorithm.
Neural Network Based Model Predictive Control
Authors: Stephen Piche, James D. Keeler, Greg Martin, Gene Boe, Doug Johnson, Mark Gerules
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Model  Predictive  Control  (MPC),  a  control  algorithm which  uses  an  optimizer  to solve  for  the optimal  control  moves  over  a  future  time horizon based upon a model of the process, has become a stan(cid:173) dard control technique  in  the process industries over  the  past  two  decades.  In  most  industrial  applications,  a  linear  dynamic  model  developed using empirical data is  used even though the process it(cid:173) self is often nonlinear.  Linear models have been used because of the  difficulty  in  developing  a  generic  nonlinear  model  from  empirical  data and  the  computational expense  often involved  in  using  non(cid:173) linear models.  In  this  paper,  we  present  a  generic neural  network  based technique for developing nonlinear dynamic models from em(cid:173) pirical  data and show that these  models  can be efficiently  used  in  a  model predictive control framework.  This nonlinear MPC based  approach has been successfully implemented in a  number of indus(cid:173) trial  applications  in  the  refining,  petrochemical,  paper  and  food  industries.  Performance of the controller on  a  nonlinear industrial  process, a  polyethylene reactor, is  presented.
Coastal Navigation with Mobile Robots
Authors: Nicholas Roy, Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  problem  that  we  address  in  this paper  is  how  a mobile robot can  plan  in  order  to  arrive  at  its  goal  with  minimum  uncertainty.  Traditional  motion  planning algo(cid:173) rithms often assume that a mobile robot can track its position reliably, however, in real  world situations, reliable localization may not always be feasible.  Partially Observable  Markov Decision Processes (POMDPs) provide one way to maximize the certainty of  reaching  the goal  state,  but at the cost of computational  intractability for large state  spaces.  The  method  we propose  explicitly  models  the  uncertainty of the  robot's position as  a  state  variable,  and  generates  trajectories  through  the  augmented  pose-uncertainty  space.  By  minimizing  the  positional  uncertainty  at  the  goal,  the  robot reduces  the  likelihood  it  becomes  lost.  We  demonstrate  experimentally  that  coastal  navigation  reduces the uncertainty at the goal, especially with degraded localization.
Boosting with Multi-Way Branching in Decision Trees
Authors: Yishay Mansour, David A. McAllester
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: It  is  known  that  decision  tree  learning  can  be  viewed  as  a  form  of boosting.  However,  existing boosting theorems for  decision tree  learning allow only binary-branching trees and the generalization to  multi-branching trees  is  not immediate.  Practical decision tree al(cid:173) gorithms, such  as CART and C4.5, implement a  trade-off between  the  number  of branches  and  the  improvement  in  tree  quality  as  measured  by an index function.  Here  we  give a  boosting justifica(cid:173) tion for a particular quantitative trade-off curve.  Our main theorem  states,  in  essence,  that if we  require  an  improvement proportional  to the  log  of the  number of branches  then  top-down  greedy  con(cid:173) struction of decision trees  remains an effective boosting algorithm.
Neural Representation of Multi-Dimensional Stimuli
Authors: Christian W. Eurich, Stefan D. Wilke, Helmut Schwegler
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Spatial  information comes in two forms:  direct spatial information (for  example, retinal position) and indirect temporal contiguity information,  since objects encountered sequentially are in general spatially close. The  acquisition  of spatial  information  by  a  neural  network  is  investigated  here. Given a spatial layout of several objects, networks are trained on a  prediction task.  Networks using temporal sequences with no direct spa(cid:173) tial information are found to  develop internal representations that show  distances correlated with distances in the external layout.  The influence  of spatial information is analyzed by providing direct spatial information  to the system during training that is  either consistent with the layout or  inconsistent with  it.  This  approach  allows  examination of the relative  contributions of spatial and temporal contiguity.
Model Selection for Support Vector Machines
Authors: Olivier Chapelle, Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: New functionals for parameter (model) selection of Support Vector Ma(cid:173) chines are introduced based on the concepts of the span of support vec(cid:173) tors and rescaling of the feature space.  It is shown that using these func(cid:173) tionals, one can both predict the best choice of parameters of the model  and the relative quality of performance for any value of parameter.
Memory Capacity of Linear vs. Nonlinear Models of Dendritic Integration
Authors: Panayiota Poirazi, Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Previous biophysical modeling work showed that nonlinear interac(cid:173) tions among nearby synapses located on active dendritic trees can  provide a large boost in the memory capacity of a cell  (Mel,  1992a,  1992b).  The aim of our present work is  to quantify  this  boost by  estimating  the  capacity  of  (1)  a  neuron  model  with  passive  den(cid:173) dritic  integration  where  inputs  are  combined  linearly  across  the  entire cell  followed  by a  single  global  threshold,  and  (2)  an active  dendrite  model  in  which  a  threshold  is  applied  separately  to  the  output of each branch, and the branch subtotals are combined lin(cid:173) early.  We focus  here on the limiting case of binary-valued synaptic  weights,  and derive expressions which  measure model  capacity by  estimating the number of distinct input-output functions  available  to both neuron types.  We  show that  (1)  the application of a fixed  nonlinearity to each dendritic compartment substantially increases  the model's flexibility,  (2) for a neuron of realistic size, the capacity  of the nonlinear cell can exceed that of the same-sized linear cell by  more than an order of magnitude, and (3)  the largest capacity boost  occurs for cells with a relatively large number of dendritic subunits  of relatively  small  size.  We  validated  the analysis  by  empirically  measuring memory  capacity with  randomized two-class  classifica(cid:173) tion problems, where a stochastic delta rule was used to train both  linear and nonlinear models.  We  found  that large capacity boosts  predicted for  the nonlinear  dendritic  model  were  readily  achieved  in practice. 
-http://lnc.usc.edu 
158 
P.  Poirazi and B.  W.  Mel