mtybilly/Instruct2DS huggingface dataset:
Ensemble Methods for Phoneme Classification
Authors: Steve R. Waterhouse, Gary Cook
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper investigates a  number of ensemble methods for  improv(cid:173) ing  the  performance of phoneme classification  for  use  in  a  speech  recognition system.  Two ensemble methods are described; boosting  and mixtures of experts,  both in isolation and in combination.  Re(cid:173) sults are presented on two speech recognition databases:  an isolated  word database and a large vocabulary continuous speech database.  These results show that principled ensemble methods such as boost(cid:173) ing  and  mixtures provide superior  performance to  more naive  en(cid:173) semble methods such  as  averaging.
Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA
Authors: Barak A. Pearlmutter, Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In the square linear blind source separation problem, one must find  a  linear  unmixing operator  which  can detangle  the result  Xi(t)  of  mixing n  unknown independent sources 8i(t)  through an unknown  n  x  n  mixing matrix A( t)  of causal linear filters:  Xi  =  E j  aij * 8 j .  We cast the problem as one of maximum likelihood density estima(cid:173) tion,  and  in  that framework  introduce an  algorithm  that searches  for  independent components using both temporal and spatial cues.  We  call the resulting algorithm  "Contextual ICA,"  after the  (Bell  and  Sejnowski  1995)  Infomax  algorithm,  which  we  show  to  be  a  special case of cICA.  Because cICA  can make use  of the temporal  structure of its input, it is  able separate in a  number of situations  where  standard methods  cannot,  including  sources  with  low  kur(cid:173) tosis,  colored  Gaussian sources,  and  sources  which  have  Gaussian  histograms. 
1  The Blind Source Separation Problem 
Consider  a  set  of n  indepent  sources  81 (t), . .. ,8n (t).  We  are  given  n  linearly dis(cid:173) torted sensor  reading  which  combine  these  sources,  Xi  =  E j  aij8j,  where  aij  is  a  filter  between source j  and  sensor i,  as  shown in figure  1a.  This can be expressed  as 
Xi(t) = 2: 2: aji(r)8j(t - r) = 2: aji * 8j
Temporal Low-Order Statistics of Natural Sounds
Authors: Hagai Attias, Christoph E. Schreiner
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: In order to process  incoming sounds  efficiently,  it is  advantageous  for the auditory system to be adapted to the statistical structure of  natural auditory scenes.  As a first step in investigating the relation  between  the system  and its inputs,  we  study low-order  statistical  properties in several sound ensembles  using a  filter  bank analysis.  Focusing on the amplitude and phase in different frequency bands,  we  find  simple  parametric  descriptions  for  their  distribution  and  power  spectrum  that  are valid  for  very  different  types  of sounds.  In  particular,  the  amplitude  distribution  has  an  exponential  tail  and  its  power  spectrum  exhibits  a  modified  power-law  behavior,  which is  manifested by self-similarity and long-range temporal cor(cid:173) relations.  Furthermore,  the statistics for  different  bands  within  a  given  ensemble  are  virtually  identical,  suggesting  translation  in(cid:173) variance along the cochlear  axis.  These results  show  that natural  sounds are highly redundant, and have possible implications to the  neural code used  by the auditory system.
A Spike Based Learning Neuron in Analog VLSI
Authors: Philipp Häfliger, Misha Mahowald, Lloyd Watts
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Many  popular  learning  rules  are  formulated  in  terms  of continu(cid:173) ous,  analog inputs  and outputs.  Biological  systems,  however,  use  action  potentials,  which  are  digital-amplitude  events  that  encode  analog  information  in  the  inter-event  interval.  Action-potential  representations are now  being used  to advantage in neuromorphic  VLSI  systems as  well.  We  report on a  simple learning rule,  based  on  the  Riccati  equation  described  by  Kohonen  [1],  modified  for  action-potential  neuronal  outputs.  We  demonstrate  this  learning  rule in an analog VLSI chip that uses volatile capacitive storage for  synaptic weights.  We  show that our time-dependent learning rule  is  sufficient  to achieve  approximate weight  normalization and  can  detect temporal correlations in spike trains. 
A Spike Based Learning Neuron in Analog VLSI
One-unit Learning Rules for Independent Component Analysis
Authors: Aapo Hyvärinen, Erkki Oja
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Neural one-unit learning rules for the problem of Independent Com(cid:173) ponent Analysis (ICA)  and blind source separation are introduced.  In  these  new  algorithms,  every  ICA  neuron  develops  into  a  sepa(cid:173) rator that finds  one of the independent  components.  The learning  rules  use  very  simple  constrained  Hebbianjanti-Hebbian  learning  in  which  decorrelating feedback  may  be  added.  To  speed  up  the  convergence of these stochastic gradient descent rules, a novel com(cid:173) putationally efficient  fixed-point  algorithm is  introduced.
Analysis of Temporal-Diffference Learning with Function Approximation
Authors: John N. Tsitsiklis, Benjamin Van Roy
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  present  new results  about  the temporal-difference learning al(cid:173) gorithm,  as  applied  to  approximating  the  cost-to-go  function  of  a  Markov  chain  using  linear  function  approximators.  The  algo(cid:173) rithm we  analyze performs on-line updating of a  parameter vector  during a  single endless  trajectory of an aperiodic irreducible finite  state Markov chain.  Results include convergence (with probability  1),  a  characterization of the limit  of convergence,  and a  bound on  the resulting approximation error.  In addition to establishing new  and  stronger  results  than  those  previously  available,  our  analysis  is  based  on  a  new  line  of  reasoning  that  provides  new  intuition  about the dynamics of temporal-difference learning.  Furthermore,  we  discuss  the  implications  of two  counter-examples with  regards  to the Significance  of on-line updating and linearly  parameterized  function  approximators.
Fast Network Pruning and Feature Extraction by using the Unit-OBS Algorithm
Authors: Achim Stahlberger, Martin Riedmiller
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  algorithm described  in  this  article  is  based  on  the  OBS  algo(cid:173) rithm by  Hassibi,  Stork  and  Wolff  ([1]  and  [2]).  The  main  disad(cid:173) vantage of OBS  is  its high complexity.  OBS needs  to calculate the  inverse  Hessian  to delete only one  weight  (thus  needing  much time  to  prune  a  big  net) .  A  better  algorithm should  use  this matrix to  remove more than only one  weight , because  calculating the inverse  Hessian  takes  the  most time in  the  OBS  algorithm.  The  algorithm,  called  Unit- OBS,  described  in  this  article  is  a  method to overcome this disadvantage.  This algorithm only  needs  to calculate the inverse Hessian once to remove one whole unit thus  drastically  reducing  the  time to prune big nets.  A  further  advantage  of Unit- OBS  is  that  it  can  be  used  to  do  a  feature  extraction  on  the  input  data.  This  can  be  helpful  on  the  understanding of unknown  problems.
Learning with Noise and Regularizers in Multilayer Neural Networks
Authors: David Saad, Sara A. Solla
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Sara A.  Solla 
AT &T Research  Labs 
Holmdel, NJ  07733,  USA  solla@research .at t .com 
We  study  the  effect  of  noise  and  regularization  in  an  on-line  gradient-descent  learning scenario for  a  general  two-layer student  network  with  an  arbitrary  number  of hidden  units.  Training ex(cid:173) amples  are  randomly  drawn  input  vectors  labeled  by  a  two-layer  teacher  network  with an arbitrary number of hidden  units; the ex(cid:173) amples are corrupted  by Gaussian noise  affecting either the output  or  the  model itself.  We  examine the  effect  of both  types  of noise  and  that  of  weight-decay  regularization  on  the  dynamical evolu(cid:173) tion of the order parameters and the generalization error in various  phases  of the learning process.
The Learning Dynamcis of a Universal Approximator
Authors: Ansgar H. L. West, David Saad, Ian T. Nabney
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The learning properties of a  universal  approximator, a normalized  committee machine with  adjustable  biases,  are studied for  on-line  back-propagation learning.  Within  a  statistical  mechanics  frame(cid:173) work,  numerical  studies  show  that this  model  has  features  which  do not exist in  previously studied two-layer network models  with(cid:173) out adjustable biases, e.g., attractive suboptimal symmetric phases  even for  realizable cases and noiseless data.
Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo
Authors: David Barber, Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The full  Bayesian  method  for  applying  neural  networks  to  a  pre(cid:173) diction  problem is  to set  up  the prior/hyperprior structure for  the  net  and then  perform the necessary  integrals.  However,  these  inte(cid:173) grals are not tractable analytically, and Markov Chain Monte Carlo  (MCMC)  methods  are  slow,  especially  if the  parameter  space  is  high-dimensional.  Using  Gaussian  processes  we  can  approximate  the weight space integral analytically, so  that only a small number  of hyperparameters  need  be  integrated  over  by  MCMC  methods.  We  have applied  this  idea to classification  problems, obtaining ex(cid:173) cellent  results on  the  real-world  problems investigated so far .
Learning Exact Patterns of Quasi-synchronization among Spiking Neurons from Data on Multi-unit Recordings
Authors: Laura Martignon, Kathryn B. Laskey, Gustavo Deco, Eilon Vaadia
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper develops arguments for a family of temporal  log-linear models  to  represent spatio-temporal correlations  among  the  spiking  events  in  a  group of neurons.  The models can represent not just pairwise  correlations  but also correlations of higher order.  Methods  are discussed  for  inferring  the existence or absence of correlations and estimating their strength.  A  frequentist  and  a  Bayesian  approach  to  correlation  detection  are  compared.  The frequentist method is based on G 2 statistic with estimates  obtained via the  Max-Ent  principle.  In  the  Bayesian  approach  a  Markov  Chain  Monte  Carlo  Model  Composition  (MC3)  algorithm  is  applied  to  search  over  connectivity  structures  and  Laplace's  method  is  used  to  approximate their posterior probability.  Performance  of the  methods  was  tested on synthetic data.  The  methods  were  applied  to  experimental  data  obtained  by  the  fourth  author by  means  of measurements  carried  out  on  behaving Rhesus monkeys at the Hadassah Medical School  of the  Hebrew  University.  As  conjectured,  neural  connectivity  structures  need  not  be  neither hierarchical nor decomposable. 
Learning Quasi-synchronization Patterns among Spiking Neurons 
77
Interpolating Earth-science Data using RBF Networks and Mixtures of Experts
Authors: Ernest Wan, Don Bone
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We present a mixture of experts (ME) approach to interpolate sparse,  spatially correlated earth-science data. Kriging is an interpolation  method which uses a global covariation model estimated from the data  to take account of the spatial dependence in the data. Based on the  close relationship between kriging and the radial basis function (RBF)  network (Wan & Bone, 1996), we use a mixture of generalized RBF  networks to partition the input space into statistically correlated  regions and learn the local covariation model of the data in each  region. Applying the ME approach to simulated and real-world data,  we show that it is able to achieve good partitioning of the input space,  learn the local covariation models and improve generalization.
Statistically Efficient Estimations Using Cortical Lateral Connections
Authors: Alexandre Pouget, Kechen Zhang
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Coarse  codes  are  widely  used  throughout  the  brain to encode  sen(cid:173) sory  and  motor  variables.  Methods  designed  to  interpret  these  codes,  such  as population vector analysis, are either inefficient, i.e.,  the variance of the estimate is much larger than the smallest possi(cid:173) ble  variance,  or biologically implausible, like  maximum likelihood.  Moreover,  these  methods  attempt  to  compute  a  scalar  or  vector  estimate  of the  encoded  variable.  Neurons  are  faced  with  a  simi(cid:173) lar  estimation problem .  They  must  read  out  the  responses  of the  presynaptic  neurons,  but,  by  contrast,  they  typically  encode  the  variable  with  a  further  population  code  rather  than  as  a  scalar.  We  show  how  a  non-linear  recurrent  network  can  be  used  to  per(cid:173) form these estimation in an optimal way while keeping the estimate  in  a  coarse  code  format.  This  work  suggests  that  lateral  connec(cid:173) tions  in  the  cortex  may  be  involved  in  cleaning  up  uncorrelated  noise  among neurons  representing  similar variables.
Recursive Algorithms for Approximating Probabilities in Graphical Models
Authors: Tommi Jaakkola, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  develop  a  recursive  node-elimination formalism for  efficiently  approximating large probabilistic networks.  No  constraints are set  on  the  network  topologies.  Yet  the  formalism can  be straightfor(cid:173) wardly  integrated  with exact  methods whenever  they  are/become  applicable.  The approximations we  use  are controlled:  they  main(cid:173) tain consistently upper and lower bounds on the desired quantities  at  all  times.  We  show  that  Boltzmann  machines,  sigmoid  belief  networks,  or any  combination  (i.e.,  chain  graphs)  can  be  handled  within  the same framework.  The accuracy  of the methods is  veri(cid:173) fied  experimentally.
Clustering via Concave Minimization
Authors: Paul S. Bradley, Olvi L. Mangasarian, W. Nick Street
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The problem of assigning m  points in the n-dimensional real space  Rn  to k  clusters is  formulated  as  that of determining k  centers in  Rn  such  that  the  sum  of distances  of  each  point  to  the  nearest  center is  minimized.  If a polyhedral distance is  used,  the problem  can be formulated as that of minimizing a piecewise-linear concave  function  on  a  polyhedral  set  which  is  shown  to  be  equivalent  to  a  bilinear  program:  minimizing  a  bilinear  function  on  a  polyhe(cid:173) dral  set.  A  fast  finite  k-Median  Algorithm  consisting  of solving  few  linear  programs in  closed  form  leads  to a  stationary point  of  the bilinear program.  Computational testing on  a number of real(cid:173) world  databases  was  carried  out.  On  the  Wisconsin  Diagnostic  Breast  Cancer  (WDBC)  database,  k-Median  training set  correct(cid:173) ness was comparable to that of the k-Mean Algorithm,  however its  testing set correctness was  better.  Additionally,  on the Wisconsin  Prognostic  Breast  Cancer  (WPBC)  database,  distinct  and  clini(cid:173) cally  important  survival  curves  were  extracted  by  the  k-Median  Algorithm,  whereas  the  k-Mean  Algorithm  failed  to  obtain such  distinct survival curves for  the same database.
Self-Organizing and Adaptive Algorithms for Generalized Eigen-Decomposition
Authors: Chanchal Chatterjee, Vwani P. Roychowdhury
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The paper is developed in  two parts where we discuss a new approach  to self-organization in a single-layer linear feed-forward network.  First,  two novel algorithms for self-organization are derived from  a two-layer  linear hetero-associative network performing a one-of-m classification,  and trained with the constrained least-mean-squared classification error  criterion.  Second, two adaptive algorithms are derived from  these self(cid:173) organizing  procedures  the  principal  generalized  eigenvectors  of  two  correlation  matrices  from  two  sequences  of  random  vectors. These novel  adaptive  algorithms can  be  implemented  in  a  single-layer  linear  feed-forward  network.  We  give  a  rigorous  convergence  analysis  of the  adaptive  algorithms  by  using  stochastic  approximation theory. As an example, we consider a problem of online  signal detection in digital mobile communications.
Edges are the 'Independent Components' of Natural Scenes.
Authors: Anthony J. Bell, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Field  (1994)  has  suggested  that  neurons  with  line  and  edge  selectivities  found  in  primary  visual  cortex  of  cats  and  monkeys  form  a  sparse,  dis(cid:173) tributed representation of natural scenes,  and Barlow  (1989)  has  reasoned  that such responses should emerge from an unsupervised learning algorithm  that attempts to find  a  factorial  code  of independent  visual  features.  We  show  here  that non-linear  'infomax',  when  applied  to  an ensemble  of nat(cid:173) ural  scenes,  produces  sets of visual  filters  that are localised  and oriented.  Some  of these  filters  are  Gabor-like  and  resemble  those  produced  by  the  sparseness-maximisation network of Olshausen & Field (1996).  In addition,  the outputs  of these filters  are as  independent  as  possible,  since  the info(cid:173) max network is  able to perform Independent Components Analysis  (ICA).  We  compare the resulting ICA filters  and their associated basis  functions,  with other decorrelating filters produced by Principal Components Analysis  (PCA)  and zero-phase whitening filters  (ZCA).  The ICA filters  have more  sparsely distributed (kurtotic) outputs on natural scenes.  They also resem(cid:173) ble the receptive fields  of simple  cells  in  visual  cortex,  which  suggests that  these neurons form an information-theoretic co-ordinate system for images. 
1
Rapid Visual Processing using Spike Asynchrony
Authors: Simon J. Thorpe, Jacques Gautrais
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We have investigated the possibility that  rapid  processing  in  the  visual  system  could  be  achieved  by  using  the  order  of  firing  in  different  neurones as  a  code,  rather than  more  conventional  firing  rate  schemes.  Using  SPIKENET,  a  neural  net  simulator  based  on  integrate-and-fire  neurones and  in  which  neurones  in  the  input  layer function  as  analog(cid:173) to-delay  converters,  we  have  modeled  the  initial  stages  of  visual  processing. Initial  results  are extremely  promising.  Even  with  activity  in  retinal  output  cells  limited  to  one  spike  per  neuron  per  image  (effectively ruling out any form of rate coding), sophisticated processing  based on asynchronous activation was nonetheless possible.
Visual Cortex Circuitry and Orientation Tuning
Authors: Trevor Mundel, Alexander Dimitrov, Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  simple  mathematical model  for  the large-scale circuitry of  pri(cid:173) mary  visual  cortex  is  introduced.  It  is  shown  that  a  basic  cor(cid:173) tical  architecture  of  recurrent  local  excitation  and  lateral  inhi(cid:173) bition  can  account  quantitatively  for  such  properties  as  orien(cid:173) tation  tuning.  The  model  can  also  account  for  such  local  ef(cid:173) fects  as  cross-orientation suppression.  It is  also  shown  that  non(cid:173) local state-dependent coupling between similar orientation patches,  when  added  to  the  model,  can  satisfactorily  reproduce  such  ef(cid:173) fects as non-local iso--orientation suppression, and non-local cross(cid:173) orientation enhancement.  Following this an account is given of per(cid:173) ceptual  phenomena involving  object segmentation,  such  as  "pop(cid:173) out", and the direct  and indirect tilt illusions.
Orientation Contrast Sensitivity from Long-range Interactions in Visual Cortex
Authors: Klaus Pawelzik, Udo Ernst, Fred Wolf, Theo Geisel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Recently Sill ito and coworkers (Nature 378, pp.  492,1995) demon(cid:173) strated that stimulation beyond the classical receptive field  (cRF)  can not only modulate, but radically change a neuron's response to  oriented stimuli.  They revealed  that patch-suppressed cells  when  stimulated  with  contrasting orientations  inside  and  outside  their  cRF  can strongly  respond  to  stimuli  oriented  orthogonal to their  nominal  preferred  orientation.  Here  we  analyze  the  emergence  of  such  complex  response patterns in  a  simple  model  of primary  vi(cid:173) sual  cortex.  We  show that the observed sensitivity for  orientation  contrast  can  be  explained  by  a  delicate  interplay  between  local  isotropic interactions and patchy long-range connectivity between  distant iso-orientation domains.  In particular we demonstrate that  the observed properties might arise without specific connections be(cid:173) tween sites with  cross-oriented cRFs.
Regular and Irregular Gallager-zype Error-Correcting Codes
Authors: Yoshiyuki Kabashima, Tatsuto Murayama, David Saad, Renato Vicente
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  performance  of  regular  and  irregular  Gallager-type  error(cid:173) correcting code  is  investigated  via  methods  of statistical  physics.  The transmitted codeword comprises products of the original mes(cid:173) sage  bits  selected  by  two  randomly-constructed  sparse  matrices;  the  number  of  non-zero  row/column  elements  in  these  matrices  constitutes  a  family  of codes.  We  show  that  Shannon's  channel  capacity may  be saturated in equilibrium for  many  of the regular  codes while slightly lower performance is obtained for others which  may  be  of higher  practical  relevance.  Decoding  aspects  are  con(cid:173) sidered  by employing the TAP  approach  which  is  identical  to the  commonly used  belief-propagation-based decoding.  We  show that  irregular codes may saturate Shannon's capacity but with improved  dynamical properties.
An MEG Study of Response Latency and Variability in the Human Visual System During a Visual-Motor Integration Task
Authors: Akaysha C. Tang, Barak A. Pearlmutter, Tim A. Hely, Michael Zibulevsky, Michael P. Weisend
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Human  reaction  times  during  sensory-motor  tasks  vary  consider(cid:173) ably.  To  begin to understand how  this variability arises, we  exam(cid:173) ined neuronal populational response time variability at early versus  late  visual  processing  stages.  The  conventional  view  is  that  pre(cid:173) cise temporal information is gradually lost as information is passed  through a  layered network of mean-rate  "units."  We  tested in hu(cid:173) mans  whether  neuronal  populations  at different  processing  stages  behave like mean-rate "units".  A blind source separation algorithm  was  applied to MEG signals from  sensory-motor integration tasks.  Response  time  latency  and  variability  for  multiple  visual  sources  were estimated by  detecting single-trial stimulus-locked events for  each  source.  In  two  subjects  tested  on  four  visual  reaction  time  tasks,  we  reliably identified sources  belonging to early and late vi(cid:173) sual processing stages.  The standard deviation of response latency  was smaller for  early rather than late processing stages.  This sup(cid:173) ports the  hypothesis that human populational response  time vari(cid:173) ability increases from  early to late visual  processing stages.
The Entropy Regularization Information Criterion
Authors: Alex J. Smola, John Shawe-Taylor, Bernhard Schölkopf, Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Effective methods of capacity control via uniform convergence bounds  for function expansions have been largely limited to Support Vector ma(cid:173) chines,  where  good  bounds  are  obtainable  by  the  entropy  number ap(cid:173) proach. We extend these methods to systems with expansions in terms of  arbitrary (parametrized) basis functions and a wide range of regulariza(cid:173) tion methods covering the whole range of general linear additive models.  This is  achieved by  a data dependent analysis of the eigenvalues of the  corresponding design matrix.
Invariant Feature Extraction and Classification in Kernel Spaces
Authors: Sebastian Mika, Gunnar Rätsch, Jason Weston, Bernhard Schölkopf, Alex J. Smola, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In hyperspectral imagery one pixel  typically  consists of a  mixture  of the  reflectance  spectra of several  materials,  where  the  mixture  coefficients  correspond  to  the  abundances of the  constituting  ma(cid:173) terials.  We  assume linear combinations of reflectance spectra with  some additive normal sensor noise and derive a  probabilistic MAP  framework  for  analyzing  hyperspectral  data.  As  the  material re(cid:173) flectance characteristics are not know  a priori, we face  the problem  of  unsupervised  linear  unmixing.  The  incorporation  of  different  prior  information  (e.g.  positivity  and  normalization  of the  abun(cid:173) dances)  naturally  leads  to  a  family  of  interesting  algorithms,  for  example  in  the  noise-free  case  yielding  an  algorithm  that  can  be  understood as constrained independent component analysis (ICA).  Simulations underline the usefulness of our theory.
Population Decoding Based on an Unfaithful Model
Authors: Si Wu, Hiroyuki Nakahara, Noboru Murata, Shun-ichi Amari
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We study a population decoding paradigm in which the maximum likeli(cid:173) hood inference is based on an  unfaithful decoding model (UMLI). This  is usually the case for neural population decoding because the encoding  process  of the  brain  is  not exactly  known,  or because a  simplified de(cid:173) coding model  is  preferred for  saving computational cost.  We  consider  an  unfaithful  decoding model  which  neglects the  pair-wise  correlation  between neuronal activities, and prove that UMLI is asymptotically effi(cid:173) cient when the neuronal correlation is uniform or of limited-range.  The  performance of UMLI is compared with that of the maximum likelihood  inference based on  a faithful  model  and  that of the  center of mass de(cid:173) coding  method.  It turns  out  that  UMLI  has  advantages  of decreasing  the computational complexity remarkablely and maintaining a high-level  decoding  accuracy  at  the  same  time.  The  effect of correlation  on  the  decoding accuracy is also discussed.
Broadband Direction-Of-Arrival Estimation Based on Second Order Statistics
Authors: Justinian P. Rosca, Joseph Ó Ruanaidh, Alexander Jourjine, Scott Rickard
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: N  wideband  sources  recorded  using  N  closely  spaced  receivers  can  feasibly  be separated  based  only  on  second  order statistics when  using  a physical model  of the mixing process.  In  this case  we  show that  the  parameter estimation problem can  be essentially reduced to considering  directions of arrival and attenuations of each signal.  The paper presents  two demixing methods operating in  the time and frequency  domain and  experimentally shows that it is always possible to demix signals arriving at  different angles.  Moreover,  one can use spatial cues to solve the channel  selection  problem  and  a post-processing Wiener filter  to  ameliorate the  artifacts caused by demixing.
Emergence of Topography and Complex Cell Properties from Natural Images using Extensions of ICA
Authors: Aapo Hyvärinen, Patrik O. Hoyer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Independent  component  analysis of natural images  leads to emer(cid:173) gence  of  simple  cell  properties,  Le.  linear  filters  that  resemble  wavelets  or  Gabor  functions.  In  this  paper,  we  extend  ICA  to  explain further properties of VI cells.  First, we decompose natural  images  into  independent  subspaces  instead of scalar  components.  This  model  leads  to  emergence  of phase  and  shift  invariant  fea(cid:173) tures,  similar  to  those  in  VI  complex  cells.  Second,  we  define  a  topography between the linear components obtained by ICA.  The  topographic distance between  two components is  defined  by  their  higher-order correlations, so that two components are close to each  other  in  the  topography  if  they  are  strongly  dependent  on  each  other.  This  leads  to simultaneous emergence  of both  topography  and invariances similar to complex cell  properties.
Reinforcement Learning Using Approximate Belief States
Authors: Andres C. Rodriguez, Ronald Parr, Daphne Koller
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The problem of developing good policies for partially observable Markov  decision  problems (POMDPs) remains one of the  most challenging ar(cid:173) eas of research  in  stochastic planning.  One line of research  in  this  area  involves  the use  of reinforcement learning  with  belief states,  probabil(cid:173) ity  distributions  over  the  underlying  model  states.  This  is  a  promis(cid:173) ing method for  small problems,  but its  application is  limited  by  the  in(cid:173) tractability of computing or representing a full belief state for large prob(cid:173) lems.  Recent  work  shows  that,  in  many  settings,  we  can  maintain  an  approximate belief state, which is fairly close to the true belief state.  In  particular,  great success has  been shown  with  approximate belief states  that marginalize out correlations between  state  variables.  In  this  paper,  we investigate two methods of full belief state reinforcement learning and  one novel method for reinforcement learning using factored approximate  belief states. We compare the performance of these algorithms on several  well-known problem from the literature. Our results demonstrate the im(cid:173) portance of approximate belief state representations for large problems.
Building Predictive Models from Fractal Representations of Symbolic Sequences
Authors: Peter Tiño, Georg Dorffner
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We propose a novel approach for building finite memory predictive mod(cid:173) els similar in spirit to variable memory length Markov models (VLMMs).  The models are constructed by first transforming the n-block structure of  the training sequence into a spatial structure of points in a unit hypercube,  such that the  longer is the common suffix shared by any  two n-blocks,  the closer lie their point representations. Such a transformation embodies  a Markov assumption - n-blocks with long common suffixes  are likely  to produce similar continuations.  Finding a set of prediction contexts is  formulated as a resource allocation problem solved by vector quantizing  the spatial n-block representation. We compare our model with both the  classical  and variable memory length Markov models on three data sets  with different memory  and stochastic components.  Our models have  a  superior performance, yet, their construction is fully automatic, which is  shown to be problematic in the case of VLMMs.
Support Vector Method for Multivariate Density Estimation
Authors: Vladimir Vapnik, Sayan Mukherjee
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  new  method  for  multivariate  density  estimation  is  developed  based  on  the  Support  Vector  Method  (SVM)  solution  of inverse  ill-posed problems.  The solution has  the form  of a  mixture of den(cid:173) sities.  This  method  with  Gaussian  kernels  compared favorably  to  both  Parzen's  method  and  the  Gaussian  Mixture  Model  method.  For synthetic data we achieve more accurate estimates for densities  of 2,  6,  12, and 40  dimensions.
Variational Inference for Bayesian Mixtures of Factor Analysers
Authors: Zoubin Ghahramani, Matthew J. Beal
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present an algorithm that infers the model structure of a  mix(cid:173) ture of factor  analysers using  an efficient  and  deterministic  varia(cid:173) tional  approximation to  full  Bayesian  integration  over  model  pa(cid:173) rameters.  This  procedure  can  automatically  determine  the  opti(cid:173) mal  number  of components  and  the  local  dimensionality  of each  component  (Le.  the  number  of  factors  in  each  factor  analyser) .  Alternatively  it  can  be  used  to  infer  posterior  distributions  over  number of components and dimensionalities.  Since  all  parameters  are integrated out the method is  not prone to overfitting.  Using a  stochastic  procedure for  adding  components  it is  possible  to  per(cid:173) form  the variational optimisation incrementally and to avoid  local  maxima.  Results show that the method works very well in practice  and  correctly  infers  the  number  and  dimensionality  of  nontrivial  synthetic examples.  By  importance  sampling  from  the  variational  approximation  we  show  how  to  obtain  unbiased  estimates  of the  true  evidence,  the  exact predictive density,  and the KL divergence between the varia(cid:173) tional posterior and the true posterior, not only in this  model  but  for  variational approximations in general.
Topographic Transformation as a Discrete Latent Variable
Authors: Nebojsa Jojic, Brendan J. Frey
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Invariance to topographic transformations such as translation and  shearing in  an image has  been successfully incorporated into feed(cid:173) forward  mechanisms,  e.g.,  "convolutional neural  networks",  "tan(cid:173) gent propagation".  We describe a way to add transformation invari(cid:173) ance to a generative density model by approximating the nonlinear  transformation manifold  by  a  discrete  set of transformations.  An  EM  algorithm for  the original model  can  be extended to the  new  model  by  computing expectations over the set of transformations.  We show how to add a discrete transformation variable to Gaussian  mixture modeling,  factor  analysis and mixtures of factor  analysis.  We  give results on filtering microscopy images, face  and facial  pose  clustering, and handwritten digit  modeling and recognition.
Channel Noise in Excitable Neural Membranes
Authors: Amit Manwani, Peter N. Steinmetz, Christof Koch
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Stochastic  fluctuations  of voltage-gated  ion  channels  generate  current  and  voltage  noise  in  neuronal  membranes.  This  noise  may  be  a  criti(cid:173) cal  determinant of the efficacy of information processing within  neural  systems.  Using Monte-Carlo simulations, we carry out a systematic in(cid:173) vestigation of the relationship  between channel  kinetics  and  the result(cid:173) ing membrane  voltage  noise  using  a  stochastic  Markov  version  of the  Mainen-Sejnowski  model  of dendritic  excitability  in  cortical  neurons.  Our simulations show that kinetic  parameters which  lead to an increase  in  membrane excitability (increasing channel densities, decreasing tem(cid:173) perature)  also  lead to  an  increase in  the magnitude of the sub-threshold  voltage noise. Noise also increases as the membrane is depolarized from  rest towards threshold.  This  suggests that channel fluctuations  may  in(cid:173) terfere with a neuron's ability to function as an integrator of its synaptic  inputs and may limit the reliability  and precision of neural  information  processing.
Efficient Approaches to Gaussian Process Classification
Authors: Lehel Csató, Ernest Fokoué, Manfred Opper, Bernhard Schottky, Ole Winther
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present  three  simple  approximations  for  the  calculation  of  the  posterior  mean  in  Gaussian  Process  classification.  The  first  two  methods  are  related  to mean field  ideas  known  in  Statistical  Physics.  The third approach is based on Bayesian online approach  which was  motivated by recent results in the Statistical Mechanics  of Neural Networks.  We present simulation results showing:  1. that  the mean field  Bayesian evidence may be used for  hyperparameter  tuning and 2.  that the online approach may achieve a low  training  error fast.
v-Arc: Ensemble Learning in the Presence of Outliers
Authors: Gunnar Rätsch, Bernhard Schölkopf, Alex J. Smola, Klaus-Robert Müller, Takashi Onoda, Sebastian Mika
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: AdaBoost and other ensemble methods have successfully  been ap(cid:173) plied  to a  number  of classification  tasks,  seemingly  defying  prob(cid:173) lems of overfitting.  AdaBoost performs gradient descent in an error  function  with  respect  to the margin,  asymptotically concentrating  on  the  patterns which  are  hardest to learn.  For  very  noisy  prob(cid:173) lems,  however,  this  can  be  disadvantageous.  Indeed,  theoretical  analysis has shown that the margin distribution,  as opposed to just  the minimal margin, plays a crucial role in understanding this phe(cid:173) nomenon.  Loosely  speaking,  some  outliers  should  be  tolerated  if  this  has  the  benefit  of substantially  increasing  the  margin  on  the  remaining points.  We  propose a  new  boosting algorithm which  al(cid:173) lows for  the possibility of a  pre-specified fraction of points to lie  in  the margin area Or even on the wrong side of the decision boundary.
A Recurrent Model of the Interaction Between Prefrontal and Inferotemporal Cortex in Delay Tasks
Authors: Alfonso Renart, Néstor Parga, Edmund T. Rolls
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A very simple model of two reciprocally connected attractor neural net(cid:173) works is studied analytically in situations similar to those encountered  in delay match-to-sample tasks with intervening stimuli and in tasks of  memory guided attention. The model qualitatively reproduces many of  the experimental data on these types of tasks and provides a framework  for the understanding of the experimental observations in the context of  the attractor neural network scenario.
Information Factorization in Connectionist Models of Perception
Authors: Javier R. Movellan, James L. McClelland
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  examine  a  psychophysical  law  that  describes  the  influence  of  stimulus and context  on  perception.  According to this law  choice  probability  ratios  factorize  into  components  independently  con(cid:173) trolled by stimulus and context.  It has been argued that this  pat(cid:173) tern of results is  incompatible with feedback models of perception.  In this  paper we  examine this  claim  using neural  network models  defined via stochastic differential equations.  We show that the law  is  related to a  condition  named channel separability and has little  to do with the existence of feedback connections.  In essence, chan(cid:173) nels are separable if they converge into the response units without  direct lateral connections to other channels and if their sensors are  not  directly  contaminated  by  external  inputs  to  the  other  chan(cid:173) nels.  Implications of the analysis for  cognitive and computational  neurosicence are discussed.
Hierarchical Image Probability (H1P) Models
Authors: Clay Spence, Lucas C. Parra
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We formulate a model for probability distributions on image spaces.  We  show that any distribution of images can be factored exactly into condi(cid:173) tional  distributions of feature  vectors at  one resolution  (pyramid level)  conditioned on the  image information at  lower resolutions.  We  would  like to factor this over positions in the pyramid levels to make it tractable,  but such factoring may miss long-range dependencies. To fix this, we in(cid:173) troduce hidden class  labels  at  each  pixel  in  the pyramid.  The result  is  a  hierarchical mixture of conditional probabilities,  similar  to  a  hidden  Markov model on a tree.  The model parameters can be found with max(cid:173) imum likelihood estimation using the EM algorithm.  We  have obtained  encouraging preliminary results on the problems of detecting various ob(cid:173) jects in SAR images and target recognition in optical aerial images.
Reinforcement Learning for Spoken Dialogue Systems
Authors: Satinder P. Singh, Michael J. Kearns, Diane J. Litman, Marilyn A. Walker
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Recently, a number of authors have proposed treating dialogue systems as Markov  decision processes (MDPs). However, the practical application ofMDP algorithms  to dialogue systems faces a number of severe technical challenges. We have built a  general software tool (RLDS, for Reinforcement Learning for Dialogue Systems)  based on the MDP framework, and have applied it to dialogue corpora gathered  from two dialogue systems built at AT&T Labs. Our experiments demonstrate that  RLDS holds promise as a tool for "browsing" and understanding correlations in  complex, temporally dependent dialogue corpora.
Distributed Synchrony of Spiking Neurons in a Hebbian Cell Assembly
Authors: David Horn, Nir Levy, Isaac Meilijson, Eytan Ruppin
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  investigate the behavior of a  Hebbian  cell  assembly of spiking  neurons formed via a temporal synaptic learning curve.  This learn(cid:173) ing  function  is  based  on  recent  experimental findings .  It  includes  potentiation for  short time delays  between  pre- and  post-synaptic  neuronal spiking,  and depression for  spiking events occuring in the  reverse order.  The coupling  between the dynamics of the synaptic  learning and of the neuronal activation leads to interesting results.  We  find  that  the  cell  assembly  can  fire  asynchronously,  but  may  also  function  in  complete  synchrony,  or  in  distributed  synchrony.  The latter implies spontaneous division of the Hebbian cell  assem(cid:173) bly into groups of cells  that fire  in  a cyclic manner.  We  invetigate  the behavior of distributed synchrony both by  simulations and by  analytic calculations of the resulting synaptic distributions.
Image Representations for Facial Expression Coding
Authors: Marian Stewart Bartlett, Gianluca Donato, Javier R. Movellan, Joseph C. Hager, Paul Ekman, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  Facial  Action  Coding  System  (FACS)  (9)  is  an  objective  method  for  quantifying  facial  movement  in  terms  of  component  actions.  This  system  is  widely  used  in  behavioral  investigations  of emotion,  cognitive  processes,  and  social  interaction.  The  cod(cid:173) ing is  presently  performed by  highly trained human experts.  This  paper  explores  and  compares  techniques  for  automatically  recog(cid:173) nizing facial actions in sequences of images.  These methods include  unsupervised  learning techniques  for  finding  basis  images  such  as  principal component analysis, independent component analysis and  local  feature  analysis,  and supervised learning  techniques  such  as  Fisher's  linear  discriminants.  These  data-driven  bases  are  com(cid:173) pared to Gabor wavelets, in which the basis images are predefined.  Best  performances  were  obtained  using  the  Gabor  wavelet  repre(cid:173) sentation and the  independent  component  representation,  both of  which achieved 96%  accuracy for  classifying 12 facial  actions.  The  ICA  representation employs  2 orders of magnitude fewer  basis im(cid:173) ages than the  Gabor representation and takes 90%  less  CPU  time  to compute for new images.  The results provide converging support  for using local basis images, high spatial frequencies, and statistical  independence for  classifying facial  actions.
Algorithms for Independent Components Analysis and Higher Order Statistics
Authors: Daniel D. Lee, Uri Rokni, Haim Sompolinsky
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  latent  variable  generative  model  with  finite  noise  is  used  to  de(cid:173) scribe  several  different  algorithms  for  Independent Components  Anal(cid:173) ysis  (lCA).  In  particular,  the  Fixed  Point  ICA  algorithm  is  shown  to  be equivalent to  the Expectation-Maximization algorithm for maximum  likelihood  under certain  constraints,  allowing  the conditions for  global  convergence to  be  elucidated.  The algorithms can  also  be explained by  their generic  behavior  near  a  singular point where the  size  of the  opti(cid:173) mal generative bases vanishes.  An expansion of the likelihood about this  singular point indicates the role of higher order correlations in  determin(cid:173) ing the features discovered by  ICA. The application and convergence of  these algorithms are demonstrated on a simple illustrative example.