mtybilly/Instruct2DS huggingface dataset:
One-unit Learning Rules for Independent Component Analysis
Authors: Erkki Oja Aapo Hyvärinen
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Neural one-unit learning rules for the problem of Independent Com(cid:173) ponent Analysis (ICA)  and blind source separation are introduced.  In  these  new  algorithms,  every  ICA  neuron  develops  into  a  sepa(cid:173) rator that finds  one of the independent  components.  The learning  rules  use  very  simple  constrained  Hebbianjanti-Hebbian  learning  in  which  decorrelating feedback  may  be  added.  To  speed  up  the  convergence of these stochastic gradient descent rules, a novel com(cid:173) putationally efficient  fixed-point  algorithm is  introduced.
Analysis of Temporal-Diffference Learning with Function Approximation
Authors: Benjamin Van Roy John N. Tsitsiklis
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  present  new results  about  the temporal-difference learning al(cid:173) gorithm,  as  applied  to  approximating  the  cost-to-go  function  of  a  Markov  chain  using  linear  function  approximators.  The  algo(cid:173) rithm we  analyze performs on-line updating of a  parameter vector  during a  single endless  trajectory of an aperiodic irreducible finite  state Markov chain.  Results include convergence (with probability  1),  a  characterization of the limit  of convergence,  and a  bound on  the resulting approximation error.  In addition to establishing new  and  stronger  results  than  those  previously  available,  our  analysis  is  based  on  a  new  line  of  reasoning  that  provides  new  intuition  about the dynamics of temporal-difference learning.  Furthermore,  we  discuss  the  implications  of two  counter-examples with  regards  to the Significance  of on-line updating and linearly  parameterized  function  approximators.
Fast Network Pruning and Feature Extraction by using the Unit-OBS Algorithm
Authors: Martin Riedmiller Achim Stahlberger
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The  algorithm described  in  this  article  is  based  on  the  OBS  algo(cid:173) rithm by  Hassibi,  Stork  and  Wolff  ([1]  and  [2]).  The  main  disad(cid:173) vantage of OBS  is  its high complexity.  OBS needs  to calculate the  inverse  Hessian  to delete only one  weight  (thus  needing  much time  to  prune  a  big  net) .  A  better  algorithm should  use  this matrix to  remove more than only one  weight , because  calculating the inverse  Hessian  takes  the  most time in  the  OBS  algorithm.  The  algorithm,  called  Unit- OBS,  described  in  this  article  is  a  method to overcome this disadvantage.  This algorithm only  needs  to calculate the inverse Hessian once to remove one whole unit thus  drastically  reducing  the  time to prune big nets.  A  further  advantage  of Unit- OBS  is  that  it  can  be  used  to  do  a  feature  extraction  on  the  input  data.  This  can  be  helpful  on  the  understanding of unknown  problems.
Learning with Noise and Regularizers in Multilayer Neural Networks
Authors: Sara A. Solla David Saad
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Sara A.  Solla 
AT &T Research  Labs 
Holmdel, NJ  07733,  USA  solla@research .at t .com 
We  study  the  effect  of  noise  and  regularization  in  an  on-line  gradient-descent  learning scenario for  a  general  two-layer student  network  with  an  arbitrary  number  of hidden  units.  Training ex(cid:173) amples  are  randomly  drawn  input  vectors  labeled  by  a  two-layer  teacher  network  with an arbitrary number of hidden  units; the ex(cid:173) amples are corrupted  by Gaussian noise  affecting either the output  or  the  model itself.  We  examine the  effect  of both  types  of noise  and  that  of  weight-decay  regularization  on  the  dynamical evolu(cid:173) tion of the order parameters and the generalization error in various  phases  of the learning process.
The Learning Dynamcis of a Universal Approximator
Authors: David Saad, Ian T. Nabney Ansgar H. L. West
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The learning properties of a  universal  approximator, a normalized  committee machine with  adjustable  biases,  are studied for  on-line  back-propagation learning.  Within  a  statistical  mechanics  frame(cid:173) work,  numerical  studies  show  that this  model  has  features  which  do not exist in  previously studied two-layer network models  with(cid:173) out adjustable biases, e.g., attractive suboptimal symmetric phases  even for  realizable cases and noiseless data.
Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo
Authors: Christopher K. I. Williams David Barber
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The full  Bayesian  method  for  applying  neural  networks  to  a  pre(cid:173) diction  problem is  to set  up  the prior/hyperprior structure for  the  net  and then  perform the necessary  integrals.  However,  these  inte(cid:173) grals are not tractable analytically, and Markov Chain Monte Carlo  (MCMC)  methods  are  slow,  especially  if the  parameter  space  is  high-dimensional.  Using  Gaussian  processes  we  can  approximate  the weight space integral analytically, so  that only a small number  of hyperparameters  need  be  integrated  over  by  MCMC  methods.  We  have applied  this  idea to classification  problems, obtaining ex(cid:173) cellent  results on  the  real-world  problems investigated so far .
Learning Exact Patterns of Quasi-synchronization among Spiking Neurons from Data on Multi-unit Recordings
Authors: Kathryn B. Laskey, Gustavo Deco, Eilon Vaadia Laura Martignon
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper develops arguments for a family of temporal  log-linear models  to  represent spatio-temporal correlations  among  the  spiking  events  in  a  group of neurons.  The models can represent not just pairwise  correlations  but also correlations of higher order.  Methods  are discussed  for  inferring  the existence or absence of correlations and estimating their strength.  A  frequentist  and  a  Bayesian  approach  to  correlation  detection  are  compared.  The frequentist method is based on G 2 statistic with estimates  obtained via the  Max-Ent  principle.  In  the  Bayesian  approach  a  Markov  Chain  Monte  Carlo  Model  Composition  (MC3)  algorithm  is  applied  to  search  over  connectivity  structures  and  Laplace's  method  is  used  to  approximate their posterior probability.  Performance  of the  methods  was  tested on synthetic data.  The  methods  were  applied  to  experimental  data  obtained  by  the  fourth  author by  means  of measurements  carried  out  on  behaving Rhesus monkeys at the Hadassah Medical School  of the  Hebrew  University.  As  conjectured,  neural  connectivity  structures  need  not  be  neither hierarchical nor decomposable. 
Learning Quasi-synchronization Patterns among Spiking Neurons 
77
Efficient Nonlinear Control with Actor-Tutor Architecture
Authors: Kenji Doya
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  new  reinforcement  learning architecture for  nonlinear  control is  proposed.  A  direct  feedback  controller,  or  the actor,  is  trained  by  a  value-gradient  based  controller,  or  the  tutor.  This  architecture  enables both efficient use of the value function and simple computa(cid:173) tion for  real-time implementation.  Good performance was  verified  in  multi-dimensional  nonlinear  control  tasks  using  Gaussian  soft(cid:173) max networks.
Interpolating Earth-science Data using RBF Networks and Mixtures of Experts
Authors: Don Bone Ernest Wan
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We present a mixture of experts (ME) approach to interpolate sparse,  spatially correlated earth-science data. Kriging is an interpolation  method which uses a global covariation model estimated from the data  to take account of the spatial dependence in the data. Based on the  close relationship between kriging and the radial basis function (RBF)  network (Wan & Bone, 1996), we use a mixture of generalized RBF  networks to partition the input space into statistically correlated  regions and learn the local covariation model of the data in each  region. Applying the ME approach to simulated and real-world data,  we show that it is able to achieve good partitioning of the input space,  learn the local covariation models and improve generalization.
Statistically Efficient Estimations Using Cortical Lateral Connections
Authors: Kechen Zhang Alexandre Pouget
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Coarse  codes  are  widely  used  throughout  the  brain to encode  sen(cid:173) sory  and  motor  variables.  Methods  designed  to  interpret  these  codes,  such  as population vector analysis, are either inefficient, i.e.,  the variance of the estimate is much larger than the smallest possi(cid:173) ble  variance,  or biologically implausible, like  maximum likelihood.  Moreover,  these  methods  attempt  to  compute  a  scalar  or  vector  estimate  of the  encoded  variable.  Neurons  are  faced  with  a  simi(cid:173) lar  estimation problem .  They  must  read  out  the  responses  of the  presynaptic  neurons,  but,  by  contrast,  they  typically  encode  the  variable  with  a  further  population  code  rather  than  as  a  scalar.  We  show  how  a  non-linear  recurrent  network  can  be  used  to  per(cid:173) form these estimation in an optimal way while keeping the estimate  in  a  coarse  code  format.  This  work  suggests  that  lateral  connec(cid:173) tions  in  the  cortex  may  be  involved  in  cleaning  up  uncorrelated  noise  among neurons  representing  similar variables.
Recursive Algorithms for Approximating Probabilities in Graphical Models
Authors: Michael I. Jordan Tommi Jaakkola
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We  develop  a  recursive  node-elimination formalism for  efficiently  approximating large probabilistic networks.  No  constraints are set  on  the  network  topologies.  Yet  the  formalism can  be straightfor(cid:173) wardly  integrated  with exact  methods whenever  they  are/become  applicable.  The approximations we  use  are controlled:  they  main(cid:173) tain consistently upper and lower bounds on the desired quantities  at  all  times.  We  show  that  Boltzmann  machines,  sigmoid  belief  networks,  or any  combination  (i.e.,  chain  graphs)  can  be  handled  within  the same framework.  The accuracy  of the methods is  veri(cid:173) fied  experimentally.
Clustering via Concave Minimization
Authors: Olvi L. Mangasarian, W. Nick Street Paul S. Bradley
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The problem of assigning m  points in the n-dimensional real space  Rn  to k  clusters is  formulated  as  that of determining k  centers in  Rn  such  that  the  sum  of distances  of  each  point  to  the  nearest  center is  minimized.  If a polyhedral distance is  used,  the problem  can be formulated as that of minimizing a piecewise-linear concave  function  on  a  polyhedral  set  which  is  shown  to  be  equivalent  to  a  bilinear  program:  minimizing  a  bilinear  function  on  a  polyhe(cid:173) dral  set.  A  fast  finite  k-Median  Algorithm  consisting  of solving  few  linear  programs in  closed  form  leads  to a  stationary point  of  the bilinear program.  Computational testing on  a number of real(cid:173) world  databases  was  carried  out.  On  the  Wisconsin  Diagnostic  Breast  Cancer  (WDBC)  database,  k-Median  training set  correct(cid:173) ness was comparable to that of the k-Mean Algorithm,  however its  testing set correctness was  better.  Additionally,  on the Wisconsin  Prognostic  Breast  Cancer  (WPBC)  database,  distinct  and  clini(cid:173) cally  important  survival  curves  were  extracted  by  the  k-Median  Algorithm,  whereas  the  k-Mean  Algorithm  failed  to  obtain such  distinct survival curves for  the same database.
Self-Organizing and Adaptive Algorithms for Generalized Eigen-Decomposition
Authors: Vwani P. Roychowdhury Chanchal Chatterjee
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: The paper is developed in  two parts where we discuss a new approach  to self-organization in a single-layer linear feed-forward network.  First,  two novel algorithms for self-organization are derived from  a two-layer  linear hetero-associative network performing a one-of-m classification,  and trained with the constrained least-mean-squared classification error  criterion.  Second, two adaptive algorithms are derived from  these self(cid:173) organizing  procedures  the  principal  generalized  eigenvectors  of  two  correlation  matrices  from  two  sequences  of  random  vectors. These novel  adaptive  algorithms can  be  implemented  in  a  single-layer  linear  feed-forward  network.  We  give  a  rigorous  convergence  analysis  of the  adaptive  algorithms  by  using  stochastic  approximation theory. As an example, we consider a problem of online  signal detection in digital mobile communications.
Multi-Grid Methods for Reinforcement Learning in Controlled Diffusion Processes
Authors: Stephan Pareigis
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Reinforcement learning methods for  discrete and semi-Markov de(cid:173) cision  problems  such  as  Real-Time  Dynamic  Programming  can  be  generalized  for  Controlled  Diffusion  Processes.  The  optimal  control  problem  reduces  to  a  boundary  value  problem  for  a  fully  nonlinear  second-order  elliptic  differential  equation  of Hamilton(cid:173) Jacobi-Bellman  (HJB-)  type.  Numerical  analysis  provides  multi(cid:173) grid methods for this kind of equation.  In the case of Learning Con(cid:173) trol, however, the systems of equations on the various grid-levels are  obtained  using  observed  information  (transitions  and  local  cost).  To  ensure  consistency,  special  attention  needs  to  be  directed  to(cid:173) ward  the  type  of time  and  space  discretization  during  the  obser(cid:173) vation.  An  algorithm for  multi-grid  observation is  proposed.  The  multi-grid algorithm is demonstrated on a simple queuing problem.
Noisy Spiking Neurons with Temporal Coding have more Computational Power than Sigmoidal Neurons
Authors: Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We exhibit a novel  way of simulating sigmoidal  neural  nets by  net(cid:173) works  of  noisy  spiking  neurons  in  temporal  coding.  Furthermore  it  is  shown  that  networks  of noisy  spiking  neurons  with  temporal  coding  have  a  strictly  larger  computational  power  than  sigmoidal  neural  nets with  the same number of units. 
1 
Introduction and Definitions 
We  consider  a  formal  model  SNN  for  a  §piking neuron  network  that  is  basically  a  reformulation  of  the  spike  response  model  (and  of  the  leaky  integrate  and  fire  model)  without  using  6-functions  (see  [Maass,  1996a]  or  [Maass,  1996b]  for  further  backgrou nd). 
An  SNN consists of a finite set V  of spiking neurons, a set  E  ~ V  x V  of synapses, a  weight wu,v  2:  0 and  a  response function cu,v  :  R+  --+  R  for  each synapse {u, v}  E  E  (where R+  := {x E R: x  2:  O})  , and  a  threshold function  8 v  :  R+  --+  R+  for  each  neuron  v  E  V  . 
If Fu  ~ R+ is  the set of firing  times of a neuron u  , then  the  potential at the trigger  zone of neuron  v  at time t is  given  by
Edges are the 'Independent Components' of Natural Scenes.
Authors: Terrence J. Sejnowski Anthony J. Bell
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Field  (1994)  has  suggested  that  neurons  with  line  and  edge  selectivities  found  in  primary  visual  cortex  of  cats  and  monkeys  form  a  sparse,  dis(cid:173) tributed representation of natural scenes,  and Barlow  (1989)  has  reasoned  that such responses should emerge from an unsupervised learning algorithm  that attempts to find  a  factorial  code  of independent  visual  features.  We  show  here  that non-linear  'infomax',  when  applied  to  an ensemble  of nat(cid:173) ural  scenes,  produces  sets of visual  filters  that are localised  and oriented.  Some  of these  filters  are  Gabor-like  and  resemble  those  produced  by  the  sparseness-maximisation network of Olshausen & Field (1996).  In addition,  the outputs  of these filters  are as  independent  as  possible,  since  the info(cid:173) max network is  able to perform Independent Components Analysis  (ICA).  We  compare the resulting ICA filters  and their associated basis  functions,  with other decorrelating filters produced by Principal Components Analysis  (PCA)  and zero-phase whitening filters  (ZCA).  The ICA filters  have more  sparsely distributed (kurtotic) outputs on natural scenes.  They also resem(cid:173) ble the receptive fields  of simple  cells  in  visual  cortex,  which  suggests that  these neurons form an information-theoretic co-ordinate system for images. 
1
For Valid Generalization the Size of the Weights is More Important than the Size of the Network
Authors: Peter L. Bartlett
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: This paper shows that if a large neural network is used for a pattern  classification  problem, and the learning algorithm finds  a  network  with  small  weights  that  has  small  squared  error  on  the  training  patterns,  then  the generalization performance depends  on the size  of the  weights  rather  than  the  number  of weights.  More  specifi(cid:173) cally,  consider  an i-layer feed-forward  network of sigmoid units, in  which  the  sum  of the  magnitudes  of the  weights  associated  with  each  unit is  bounded  by  A.  The  misclassification probability con(cid:173) verges  to an error estimate (that is closely  related to squared error  on  the  training  set)  at  rate  O((cA)l(l+1)/2J(log n)jm)  ignoring  log  factors,  where  m  is  the  number of training  patterns,  n  is  the  input  dimension,  and  c is  a  constant.  This may explain  the  gen(cid:173) eralization performance of neural  networks,  particularly when  the  number of training examples is considerably smaller than the num(cid:173) ber  of weights.  It also  supports  heuristics  (such  as  weight  decay  and early stopping)  that attempt to keep  the weights small during  training.
Neural Network Modeling of Speech and Music Signals
Authors: Alex Röbel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Time  series  prediction  is  one of the  major  applications  of neural  net(cid:173) works.  After a short introduction into the basic theoretical foundations  we argue that the iterated prediction of a dynamical system may  be in(cid:173) terpreted as  a model of the system dynamics.  By  means of RBF  neural  networks  we describe a modeling approach and  extend it to be able  to  model instationary systems.  As a practical test for the capabilities of the  method we investigate the modeling of musical and  speech signals  and  demonstrate that the  model  may  be  used  for  synthesis  of musical  and  speech signals.
Rapid Visual Processing using Spike Asynchrony
Authors: Jacques Gautrais Simon J. Thorpe
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: We have investigated the possibility that  rapid  processing  in  the  visual  system  could  be  achieved  by  using  the  order  of  firing  in  different  neurones as  a  code,  rather than  more  conventional  firing  rate  schemes.  Using  SPIKENET,  a  neural  net  simulator  based  on  integrate-and-fire  neurones and  in  which  neurones  in  the  input  layer function  as  analog(cid:173) to-delay  converters,  we  have  modeled  the  initial  stages  of  visual  processing. Initial  results  are extremely  promising.  Even  with  activity  in  retinal  output  cells  limited  to  one  spike  per  neuron  per  image  (effectively ruling out any form of rate coding), sophisticated processing  based on asynchronous activation was nonetheless possible.
Visual Cortex Circuitry and Orientation Tuning
Authors: Alexander Dimitrov, Jack D. Cowan Trevor Mundel
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: A  simple  mathematical model  for  the large-scale circuitry of  pri(cid:173) mary  visual  cortex  is  introduced.  It  is  shown  that  a  basic  cor(cid:173) tical  architecture  of  recurrent  local  excitation  and  lateral  inhi(cid:173) bition  can  account  quantitatively  for  such  properties  as  orien(cid:173) tation  tuning.  The  model  can  also  account  for  such  local  ef(cid:173) fects  as  cross-orientation suppression.  It is  also  shown  that  non(cid:173) local state-dependent coupling between similar orientation patches,  when  added  to  the  model,  can  satisfactorily  reproduce  such  ef(cid:173) fects as non-local iso--orientation suppression, and non-local cross(cid:173) orientation enhancement.  Following this an account is given of per(cid:173) ceptual  phenomena involving  object segmentation,  such  as  "pop(cid:173) out", and the direct  and indirect tilt illusions.
Orientation Contrast Sensitivity from Long-range Interactions in Visual Cortex
Authors: Udo Ernst, Fred Wolf, Theo Geisel Klaus Pawelzik
Published in: Advances in Neural Information Processing Systems 9  (NIPS 1996) (Main Conference Track), 1996

Abstract: Recently Sill ito and coworkers (Nature 378, pp.  492,1995) demon(cid:173) strated that stimulation beyond the classical receptive field  (cRF)  can not only modulate, but radically change a neuron's response to  oriented stimuli.  They revealed  that patch-suppressed cells  when  stimulated  with  contrasting orientations  inside  and  outside  their  cRF  can strongly  respond  to  stimuli  oriented  orthogonal to their  nominal  preferred  orientation.  Here  we  analyze  the  emergence  of  such  complex  response patterns in  a  simple  model  of primary  vi(cid:173) sual  cortex.  We  show that the observed sensitivity for  orientation  contrast  can  be  explained  by  a  delicate  interplay  between  local  isotropic interactions and patchy long-range connectivity between  distant iso-orientation domains.  In particular we demonstrate that  the observed properties might arise without specific connections be(cid:173) tween sites with  cross-oriented cRFs.
Regular and Irregular Gallager-zype Error-Correcting Codes
Authors: Tatsuto Murayama, David Saad, Renato Vicente Yoshiyuki Kabashima
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The  performance  of  regular  and  irregular  Gallager-type  error(cid:173) correcting code  is  investigated  via  methods  of statistical  physics.  The transmitted codeword comprises products of the original mes(cid:173) sage  bits  selected  by  two  randomly-constructed  sparse  matrices;  the  number  of  non-zero  row/column  elements  in  these  matrices  constitutes  a  family  of codes.  We  show  that  Shannon's  channel  capacity may  be saturated in equilibrium for  many  of the regular  codes while slightly lower performance is obtained for others which  may  be  of higher  practical  relevance.  Decoding  aspects  are  con(cid:173) sidered  by employing the TAP  approach  which  is  identical  to the  commonly used  belief-propagation-based decoding.  We  show that  irregular codes may saturate Shannon's capacity but with improved  dynamical properties.
An MEG Study of Response Latency and Variability in the Human Visual System During a Visual-Motor Integration Task
Authors: Barak A. Pearlmutter, Tim A. Hely, Michael Zibulevsky, Michael P. Weisend Akaysha C. Tang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Human  reaction  times  during  sensory-motor  tasks  vary  consider(cid:173) ably.  To  begin to understand how  this variability arises, we  exam(cid:173) ined neuronal populational response time variability at early versus  late  visual  processing  stages.  The  conventional  view  is  that  pre(cid:173) cise temporal information is gradually lost as information is passed  through a  layered network of mean-rate  "units."  We  tested in hu(cid:173) mans  whether  neuronal  populations  at different  processing  stages  behave like mean-rate "units".  A blind source separation algorithm  was  applied to MEG signals from  sensory-motor integration tasks.  Response  time  latency  and  variability  for  multiple  visual  sources  were estimated by  detecting single-trial stimulus-locked events for  each  source.  In  two  subjects  tested  on  four  visual  reaction  time  tasks,  we  reliably identified sources  belonging to early and late vi(cid:173) sual processing stages.  The standard deviation of response latency  was smaller for  early rather than late processing stages.  This sup(cid:173) ports the  hypothesis that human populational response  time vari(cid:173) ability increases from  early to late visual  processing stages.
The Entropy Regularization Information Criterion
Authors: John Shawe-Taylor, Bernhard Schölkopf, Robert C. Williamson Alex J. Smola
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Effective methods of capacity control via uniform convergence bounds  for function expansions have been largely limited to Support Vector ma(cid:173) chines,  where  good  bounds  are  obtainable  by  the  entropy  number ap(cid:173) proach. We extend these methods to systems with expansions in terms of  arbitrary (parametrized) basis functions and a wide range of regulariza(cid:173) tion methods covering the whole range of general linear additive models.  This is  achieved by  a data dependent analysis of the eigenvalues of the  corresponding design matrix.
Invariant Feature Extraction and Classification in Kernel Spaces
Authors: Gunnar Rätsch, Jason Weston, Bernhard Schölkopf, Alex J. Smola, Klaus-Robert Müller Sebastian Mika
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In hyperspectral imagery one pixel  typically  consists of a  mixture  of the  reflectance  spectra of several  materials,  where  the  mixture  coefficients  correspond  to  the  abundances of the  constituting  ma(cid:173) terials.  We  assume linear combinations of reflectance spectra with  some additive normal sensor noise and derive a  probabilistic MAP  framework  for  analyzing  hyperspectral  data.  As  the  material re(cid:173) flectance characteristics are not know  a priori, we face  the problem  of  unsupervised  linear  unmixing.  The  incorporation  of  different  prior  information  (e.g.  positivity  and  normalization  of the  abun(cid:173) dances)  naturally  leads  to  a  family  of  interesting  algorithms,  for  example  in  the  noise-free  case  yielding  an  algorithm  that  can  be  understood as constrained independent component analysis (ICA).  Simulations underline the usefulness of our theory.
Population Decoding Based on an Unfaithful Model
Authors: Hiroyuki Nakahara, Noboru Murata, Shun-ichi Amari Si Wu
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We study a population decoding paradigm in which the maximum likeli(cid:173) hood inference is based on an  unfaithful decoding model (UMLI). This  is usually the case for neural population decoding because the encoding  process  of the  brain  is  not exactly  known,  or because a  simplified de(cid:173) coding model  is  preferred for  saving computational cost.  We  consider  an  unfaithful  decoding model  which  neglects the  pair-wise  correlation  between neuronal activities, and prove that UMLI is asymptotically effi(cid:173) cient when the neuronal correlation is uniform or of limited-range.  The  performance of UMLI is compared with that of the maximum likelihood  inference based on  a faithful  model  and  that of the  center of mass de(cid:173) coding  method.  It turns  out  that  UMLI  has  advantages  of decreasing  the computational complexity remarkablely and maintaining a high-level  decoding  accuracy  at  the  same  time.  The  effect of correlation  on  the  decoding accuracy is also discussed.
Broadband Direction-Of-Arrival Estimation Based on Second Order Statistics
Authors: Joseph Ó Ruanaidh, Alexander Jourjine, Scott Rickard Justinian P. Rosca
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: N  wideband  sources  recorded  using  N  closely  spaced  receivers  can  feasibly  be separated  based  only  on  second  order statistics when  using  a physical model  of the mixing process.  In  this case  we  show that  the  parameter estimation problem can  be essentially reduced to considering  directions of arrival and attenuations of each signal.  The paper presents  two demixing methods operating in  the time and frequency  domain and  experimentally shows that it is always possible to demix signals arriving at  different angles.  Moreover,  one can use spatial cues to solve the channel  selection  problem  and  a post-processing Wiener filter  to  ameliorate the  artifacts caused by demixing.
Emergence of Topography and Complex Cell Properties from Natural Images using Extensions of ICA
Authors: Patrik O. Hoyer Aapo Hyvärinen
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Independent  component  analysis of natural images  leads to emer(cid:173) gence  of  simple  cell  properties,  Le.  linear  filters  that  resemble  wavelets  or  Gabor  functions.  In  this  paper,  we  extend  ICA  to  explain further properties of VI cells.  First, we decompose natural  images  into  independent  subspaces  instead of scalar  components.  This  model  leads  to  emergence  of phase  and  shift  invariant  fea(cid:173) tures,  similar  to  those  in  VI  complex  cells.  Second,  we  define  a  topography between the linear components obtained by ICA.  The  topographic distance between  two components is  defined  by  their  higher-order correlations, so that two components are close to each  other  in  the  topography  if  they  are  strongly  dependent  on  each  other.  This  leads  to simultaneous emergence  of both  topography  and invariances similar to complex cell  properties.
Reinforcement Learning Using Approximate Belief States
Authors: Ronald Parr, Daphne Koller Andres C. Rodriguez
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The problem of developing good policies for partially observable Markov  decision  problems (POMDPs) remains one of the  most challenging ar(cid:173) eas of research  in  stochastic planning.  One line of research  in  this  area  involves  the use  of reinforcement learning  with  belief states,  probabil(cid:173) ity  distributions  over  the  underlying  model  states.  This  is  a  promis(cid:173) ing method for  small problems,  but its  application is  limited  by  the  in(cid:173) tractability of computing or representing a full belief state for large prob(cid:173) lems.  Recent  work  shows  that,  in  many  settings,  we  can  maintain  an  approximate belief state, which is fairly close to the true belief state.  In  particular,  great success has  been shown  with  approximate belief states  that marginalize out correlations between  state  variables.  In  this  paper,  we investigate two methods of full belief state reinforcement learning and  one novel method for reinforcement learning using factored approximate  belief states. We compare the performance of these algorithms on several  well-known problem from the literature. Our results demonstrate the im(cid:173) portance of approximate belief state representations for large problems.
Building Predictive Models from Fractal Representations of Symbolic Sequences
Authors: Georg Dorffner Peter Tiño
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We propose a novel approach for building finite memory predictive mod(cid:173) els similar in spirit to variable memory length Markov models (VLMMs).  The models are constructed by first transforming the n-block structure of  the training sequence into a spatial structure of points in a unit hypercube,  such that the  longer is the common suffix shared by any  two n-blocks,  the closer lie their point representations. Such a transformation embodies  a Markov assumption - n-blocks with long common suffixes  are likely  to produce similar continuations.  Finding a set of prediction contexts is  formulated as a resource allocation problem solved by vector quantizing  the spatial n-block representation. We compare our model with both the  classical  and variable memory length Markov models on three data sets  with different memory  and stochastic components.  Our models have  a  superior performance, yet, their construction is fully automatic, which is  shown to be problematic in the case of VLMMs.
Neural Computation with Winner-Take-All as the Only Nonlinear Operation
Authors: Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Everybody "knows" that neural networks need more than a single layer  of nonlinear units to compute interesting functions.  We show that this is  false if one employs winner-take-all as nonlinear unit: 
•  Any boolean function can be computed by a single k-winner-take(cid:173)
all unit applied to weighted sums of the input variables. 
•  Any continuous  function  can be approximated arbitrarily well  by  a  single soft winner-take-all unit applied to weighted sums of the  input variables. 
•  Only positive weights are needed in these (linear) weighted sums.  This may be of interest from  the point of view of neurophysiology,  since only 15% of the synapses in the cortex are inhibitory.  In addi(cid:173) tion it is  widely believed that there are special microcircuits in the  cortex that compute winner-take-all. 
•  Our results  support the  view  that winner-take-all  is  a  very  useful 
basic computational unit in Neural VLS!:  o 
it  is  wellknown  that  winner-take-all  of n  input  variables  can  be  computed  very  efficiently  with  2n  transistors  (and  a  to(cid:173) tal  wire  length  and  area  that  is  linear  in  n)  in  analog  VLSI  [Lazzaro et at.,  1989] 
o  we show that winner-take-all is  not just useful  for special pur(cid:173) pose computations, but may serve as the only nonlinear unit for  neural circuits with universal computational power 
o  we show that any multi-layer perceptron needs quadratically in  n many gates to compute winner-take-all for n  input variables,  hence  winner-take-all  provides a  substantially  more  powerful  computational  unit  than  a  perceptron (at  about the  same  cost  of implementation in analog VLSI). 
Complete  proofs  and  further  details  to  these  results  can  be  found  in  [Maass, 2000]. 
294
Support Vector Method for Multivariate Density Estimation
Authors: Sayan Mukherjee Vladimir Vapnik
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A  new  method  for  multivariate  density  estimation  is  developed  based  on  the  Support  Vector  Method  (SVM)  solution  of inverse  ill-posed problems.  The solution has  the form  of a  mixture of den(cid:173) sities.  This  method  with  Gaussian  kernels  compared favorably  to  both  Parzen's  method  and  the  Gaussian  Mixture  Model  method.  For synthetic data we achieve more accurate estimates for densities  of 2,  6,  12, and 40  dimensions.
Leveraged Vector Machines
Authors: Yoram Singer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We describe an iterative algorithm for building vector machines used in  classification tasks.  The algorithm builds  on  ideas from  support vector  machines, boosting, and generalized additive models. The algorithm can  be  used  with  various continuously differential functions  that bound the  discrete (0-1) classification loss and is very simple to implement. We test  the proposed algorithm with two different loss functions on synthetic and  natural data. We also describe a norm-penalized version of the algorithm  for the exponential loss function used in AdaBoost.  The performance of  the algorithm on  natural data is  comparable to  support vector machines  while typically its running time is shorter than of SVM.
Learning Factored Representations for Partially Observable Markov Decision Processes
Authors: Brian Sallans
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The problem of reinforcement learning in a non-Markov environment is  explored using a dynamic Bayesian network, where conditional indepen(cid:173) dence assumptions between random variables are compactly represented  by network parameters.  The parameters are learned on-line, and approx(cid:173) imations are used to perform inference and to compute the optimal value  function.  The  relative  effects of inference  and  value  function  approxi(cid:173) mations on the quality of the final  policy are investigated, by learning to  solve a moderately difficult driving task. The two value function approx(cid:173) imations,  linear and quadratic, were found to  perform similarly, but the  quadratic model was more sensitive to initialization.  Both performed be(cid:173) low the level of human performance on the task.  The dynamic Bayesian  network performed comparably to  a model  using a localist hidden state  representation, while requiring exponentially fewer parameters.
Variational Inference for Bayesian Mixtures of Factor Analysers
Authors: Matthew J. Beal Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present an algorithm that infers the model structure of a  mix(cid:173) ture of factor  analysers using  an efficient  and  deterministic  varia(cid:173) tional  approximation to  full  Bayesian  integration  over  model  pa(cid:173) rameters.  This  procedure  can  automatically  determine  the  opti(cid:173) mal  number  of components  and  the  local  dimensionality  of each  component  (Le.  the  number  of  factors  in  each  factor  analyser) .  Alternatively  it  can  be  used  to  infer  posterior  distributions  over  number of components and dimensionalities.  Since  all  parameters  are integrated out the method is  not prone to overfitting.  Using a  stochastic  procedure for  adding  components  it is  possible  to  per(cid:173) form  the variational optimisation incrementally and to avoid  local  maxima.  Results show that the method works very well in practice  and  correctly  infers  the  number  and  dimensionality  of  nontrivial  synthetic examples.  By  importance  sampling  from  the  variational  approximation  we  show  how  to  obtain  unbiased  estimates  of the  true  evidence,  the  exact predictive density,  and the KL divergence between the varia(cid:173) tional posterior and the true posterior, not only in this  model  but  for  variational approximations in general.
Topographic Transformation as a Discrete Latent Variable
Authors: Brendan J. Frey Nebojsa Jojic
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Invariance to topographic transformations such as translation and  shearing in  an image has  been successfully incorporated into feed(cid:173) forward  mechanisms,  e.g.,  "convolutional neural  networks",  "tan(cid:173) gent propagation".  We describe a way to add transformation invari(cid:173) ance to a generative density model by approximating the nonlinear  transformation manifold  by  a  discrete  set of transformations.  An  EM  algorithm for  the original model  can  be extended to the  new  model  by  computing expectations over the set of transformations.  We show how to add a discrete transformation variable to Gaussian  mixture modeling,  factor  analysis and mixtures of factor  analysis.  We  give results on filtering microscopy images, face  and facial  pose  clustering, and handwritten digit  modeling and recognition.
Channel Noise in Excitable Neural Membranes
Authors: Peter N. Steinmetz, Christof Koch Amit Manwani
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Stochastic  fluctuations  of voltage-gated  ion  channels  generate  current  and  voltage  noise  in  neuronal  membranes.  This  noise  may  be  a  criti(cid:173) cal  determinant of the efficacy of information processing within  neural  systems.  Using Monte-Carlo simulations, we carry out a systematic in(cid:173) vestigation of the relationship  between channel  kinetics  and  the result(cid:173) ing membrane  voltage  noise  using  a  stochastic  Markov  version  of the  Mainen-Sejnowski  model  of dendritic  excitability  in  cortical  neurons.  Our simulations show that kinetic  parameters which  lead to an increase  in  membrane excitability (increasing channel densities, decreasing tem(cid:173) perature)  also  lead to  an  increase in  the magnitude of the sub-threshold  voltage noise. Noise also increases as the membrane is depolarized from  rest towards threshold.  This  suggests that channel fluctuations  may  in(cid:173) terfere with a neuron's ability to function as an integrator of its synaptic  inputs and may limit the reliability  and precision of neural  information  processing.
Efficient Approaches to Gaussian Process Classification
Authors: Ernest Fokoué, Manfred Opper, Bernhard Schottky, Ole Winther Lehel Csató
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present  three  simple  approximations  for  the  calculation  of  the  posterior  mean  in  Gaussian  Process  classification.  The  first  two  methods  are  related  to mean field  ideas  known  in  Statistical  Physics.  The third approach is based on Bayesian online approach  which was  motivated by recent results in the Statistical Mechanics  of Neural Networks.  We present simulation results showing:  1. that  the mean field  Bayesian evidence may be used for  hyperparameter  tuning and 2.  that the online approach may achieve a low  training  error fast.
Optimal Sizes of Dendritic and Axonal Arbors
Authors: Dmitri B. Chklovskii
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: I consider a topographic projection between two neuronal layers with dif(cid:173) ferent densities of neurons.  Given  the  number of output  neurons con(cid:173) nected to each input neuron (divergence or fan-out) and the number of  input neurons synapsing on each output neuron (convergence or fan-in) I  determine the widths of axonal and dendritic arbors which minimize the  total  volume ofaxons and dendrites.  My analytical results can be sum(cid:173) marized qualitatively in the following rule:  neurons of the sparser layer  should have arbors wider than those of the denser layer.  This agrees with  the anatomical data from retinal and cerebellar neurons whose morphol(cid:173) ogy and connectivity are known.  The rule may be used to infer connec(cid:173) tivity of neurons from their morphology.
v-Arc: Ensemble Learning in the Presence of Outliers
Authors: Bernhard Schölkopf, Alex J. Smola, Klaus-Robert Müller, Takashi Onoda, Sebastian Mika Gunnar Rätsch
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: AdaBoost and other ensemble methods have successfully  been ap(cid:173) plied  to a  number  of classification  tasks,  seemingly  defying  prob(cid:173) lems of overfitting.  AdaBoost performs gradient descent in an error  function  with  respect  to the margin,  asymptotically concentrating  on  the  patterns which  are  hardest to learn.  For  very  noisy  prob(cid:173) lems,  however,  this  can  be  disadvantageous.  Indeed,  theoretical  analysis has shown that the margin distribution,  as opposed to just  the minimal margin, plays a crucial role in understanding this phe(cid:173) nomenon.  Loosely  speaking,  some  outliers  should  be  tolerated  if  this  has  the  benefit  of substantially  increasing  the  margin  on  the  remaining points.  We  propose a  new  boosting algorithm which  al(cid:173) lows for  the possibility of a  pre-specified fraction of points to lie  in  the margin area Or even on the wrong side of the decision boundary.
Monte Carlo POMDPs
Authors: Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present  a Monte Carlo algorithm for  learning to  act  in  partially observable  Markov decision processes (POMDPs) with real-valued state and action spaces.  Our approach uses importance sampling for representing beliefs, and Monte Carlo  approximation for belief propagation.  A reinforcement learning algorithm, value  iteration, is employed to learn value functions over belief states. Finally, a sample(cid:173) based  version  of nearest  neighbor  is  used  to  generalize  across  states.  Initial  empirical results suggest that our approach works well in practical applications.
A Recurrent Model of the Interaction Between Prefrontal and Inferotemporal Cortex in Delay Tasks
Authors: Néstor Parga, Edmund T. Rolls Alfonso Renart
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A very simple model of two reciprocally connected attractor neural net(cid:173) works is studied analytically in situations similar to those encountered  in delay match-to-sample tasks with intervening stimuli and in tasks of  memory guided attention. The model qualitatively reproduces many of  the experimental data on these types of tasks and provides a framework  for the understanding of the experimental observations in the context of  the attractor neural network scenario.