mtybilly/Instruct2DS huggingface dataset:
Unsupervised and Supervised Clustering: The Mutual Information between Parameters and Observations
Authors: Didier Herschkowitz, Jean-Pierre Nadal
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Recent  works  in  parameter  estimation  and  neural  coding  have  demonstrated that optimal performance are related to the  mutual  information between parameters and data.  We  consider the mutual  information in  the case where  the dependency in the parameter (a  vector  8)  of  the  conditional  p.d.f.  of each  observation  (a  vector  0, is  through  the  scalar  product  8.~ only.  We  derive  bounds  and  asymptotic behaviour for  the mutual information and compare with  results  obtained on  the same model with the" replica technique" .
Stationarity and Stability of Autoregressive Neural Network Processes
Authors: Friedrich Leisch, Adrian Trapletti, Kurt Hornik
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  analyze  the  asymptotic behavior of autoregressive  neural  net(cid:173) work  (AR-NN)  processes  using techniques from Markov chains and  non-linear time series  analysis.  It is  shown  that standard AR-NNs  without shortcut connections are  asymptotically stationary.  If lin(cid:173) ear  shortcut  connections  are  allowed,  only  the  shortcut  weights  determine whether the overall system is stationary, hence standard  conditions for  linear AR processes  can be  used.
Gradient Descent for General Reinforcement Learning
Authors: Leemon C. Baird III, Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: A  simple  learning  rule  is  derived,  the  VAPS  algorithm,  which  can  be  instantiated  to  generate  a  wide  range  of  new  reinforcement(cid:173) learning  algorithms.  These  algorithms  solve  a  number  of  open  problems, define several new approaches to reinforcement learning,  and  unify  different  approaches  to  reinforcement  learning  under  a  single  theory.  These  algorithms  all  have  guaranteed  convergence,  and  include  modifications  of several  existing  algorithms  that  were  known  to  fail  to  converge  on  simple  MOPs.  These  include  Q(cid:173) In  addition  to  these  learning,  SARSA,  and  advantage  learning.  it  also  generates  pure  policy-search  value-based  algorithms  reinforcement-learning  algorithms,  which  learn  optimal  policies  without  learning  a  value  function.  search  and  value-based  algorithms  to  be  combined,  thus  unifying  two  very  different  approaches  to  reinforcement  learning  into  a  single  Value  and  Policy  Search  (V APS)  algorithm.  And  these  algorithms converge for  POMDPs without requiring a  proper belief  state .  Simulations  results  are  given,  and  several  areas  for  future  research are discussed. 
In  addition,  it  allows  policy(cid:173)
1  CONVERGENCE  OF  GREEDY  EXPLORATION 
Many  reinforcement-learning  algorithms  are  known  that  use  a  parameterized  function  approximator  to  represent  a  value  function,  and  adjust  the  weights  include  Q-learning,  SARSA,  and  incrementally  during  advantage  learning.  There  are  simple  MOPs  where  the  original  form  of  these  algorithms  fails  to  converge,  as  summarized  in  Table  1.  For the  cases  with..J,  the  algorithms  are  guaranteed  to  converge  under  reasonable  assumptions  such  as
Vertex Identification in High Energy Physics Experiments
Authors: Gideon Dror, Halina Abramowicz, David Horn
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In High Energy Physics experiments one has to sort through a high  flux  of events, at a  rate of tens of MHz,  and select  the few  that are  of interest.  One  of the  key  factors  in  making  this  decision  is  the  location of the vertex  where  the  interaction , that led  to the event ,  took  place.  Here  we  present  a  novel  solution  to  the  problem  of  finding  the  location  of the  vertex,  based  on  two  feedforward  neu(cid:173) ral  networks with fixed  architectures, whose parameters are chosen  so  as  to  obtain  a  high  accuracy.  The  system  is  tested  on  simu(cid:173) lated  data sets , and is  shown  to  perform better  than  conventional  algorithms.
Computation of Smooth Optical Flow in a Feedback Connected Analog Network
Authors: Alan Stocker, Rodney J. Douglas
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  1986, Tanner and Mead [1] implemented an interesting constraint sat(cid:173) isfaction  circuit  for  global  motion  sensing  in  a VLSI.  We  report  here  a  new  and  improved a VLSI implementation that provides smooth optical  flow as well as global motion in a two dimensional visual field.  The com(cid:173) putation of optical flow  is  an ill-posed problem, which expresses itself as  the aperture problem.  However, the optical flow  can be estimated by the  use of regularization methods, in  which additional constraints are intro(cid:173) duced in  terms of a global energy functional that must be minimized . We  show how the algorithmic constraints of Hom and Schunck [2]  on com(cid:173) puting smooth optical flow can be mapped onto the physical constraints  of an equivalent electronic network.
Applications of Multi-Resolution Neural Networks to Mammography
Authors: Clay Spence, Paul Sajda
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  have  previously  presented  a  coarse-to-fine  hierarchical  pyra(cid:173) mid/neural  network  (HPNN)  architecture  which  combines  multi(cid:173) scale  image  processing  techniques  with  neural  networks.  In  this  paper  we  present  applications  of  this  general  architecture  to  two  problems  in  mammographic  Computer-Aided  Diagnosis  (CAD).  The  first  application  is  the  detection  of  microcalcifications.  The  <:oarse-to-fine  HPNN  was  designed  to learn  large-scale  context  in(cid:173) formation  for  detecting  small objects  like  microcalcifications.  Re(cid:173) ceiver  operating  characteristic  (ROC)  analysis  suggests  that  the  hierarchical  architecture  improves  detection  performance  of a  well  established  CAD  system by  roughly  50 %.  The second  application  is  to detect mammographic masses directly.  Since masses are large,  extended objects,  the coarse-to-fine HPNN architecture is  not suit(cid:173) able for  this  problem.  Instead we  construct  a fine-to-coarse  HPNN  architecture which  is  designed  to  learn  small-scale detail  structure  associated  with  the extended  objects.  Our initial  results  applying  the  fine-to-coarse  HPNN  to  mass  detection  are  encouraging,  with  detection  performance  improvements  of about  36 %.  We  conclude  that the ability of the HPNN architecture to integrate information  across  scales,  both  coarse-to-fine  and  fine-to-coarse,  makes  it  well  suited  for  detecting  objects  which  may  have  contextual  clues  or  detail structure occurring at scales  other than  the  natural scale of  the object.
Robot Docking Using Mixtures of Gaussians
Authors: Matthew M. Williamson, Roderick Murray-Smith, Volker Hansen
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper applies  the  Mixture  of Gaussians  probabilistic model,  com(cid:173) bined  with  Expectation Maximization  optimization  to  the  task  of sum(cid:173) marizing three dimensional range data for a mobile robot.  This provides  a flexible way of dealing with uncertainties in sensor information, and al(cid:173) lows the introduction of prior knowledge into low-level perception mod(cid:173) ules.  Problems with the basic approach were solved in  several ways:  the  mixture of Gaussians was reparameterized to reflect the types of objects  expected  in  the  scene,  and  priors  on  model  parameters  were  included  in  the  optimization  process.  Both approaches force  the  optimization  to  find  'interesting'  objects,  given the  sensor and object characteristics.  A  higher level  classifier  was  used  to  interpret the  results  provided by  the  model, and to reject spurious solutions.
Semi-Supervised Support Vector Machines
Authors: Kristin P. Bennett, Ayhan Demiriz
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  introduce  a  semi-supervised  support  vector  machine  (S3yM)  method.  Given  a  training  set  of labeled  data  and  a  working  set  of unlabeled  data,  S3YM  constructs  a  support vector  machine us(cid:173) ing  both  the  training  and  working  sets.  We  use  S3 YM  to  solve  the  transduction  problem  using  overall  risk  minimization  (ORM)  posed  by  Yapnik.  The  transduction  problem  is  to  estimate  the  value of a  classification function at the given points in the working  set.  This  contrasts  with  the  standard inductive  learning  problem  of estimating the  classification  function  at  all  possible values  and  then  using  the  fixed  function  to  deduce  the classes  of the working  set  data.  We  propose  a  general  S3YM  model  that minimizes both  the  misclassification  error  and  the  function  capacity  based  on  all  the available data.  We show  how the S3YM  model  for  I-norm lin(cid:173) ear  support  vector  machines  can  be  converted  to  a  mixed-integer  program  and then solved exactly using  integer  programming.  Re(cid:173) sults  of S3YM  and  the  standard  I-norm  support  vector  machine  approach  are  compared  on  ten  data sets.  Our  computational  re(cid:173) sults  support  the statistical  learning  theory  results  showing  that  incorporating  working  data  improves  generalization  when  insuffi(cid:173) cient  training information is  available.  In every case,  S3YM  either  improved or showed no significant difference in generalization com(cid:173) pared to the traditional approach. 
Semi-Supervised Support  Vector Machines
A Phase Space Approach to Minimax Entropy Learning and the Minutemax Approximations
Authors: James M. Coughlan, Alan L. Yuille
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: There  has  been  much  recent  work  on  measuring  image  statistics  and  on  learning  probability  distributions  on  images.  We  observe  that  the  mapping  from  images  to  statistics  is  many-to-one  and  show  it  can  be  quantified  by  a  phase  space  factor.  This  phase  space approach throws light on the Minimax Entropy technique for  learning Gibbs distributions on images with potentials derived from  image statistics and elucidates the ambiguities that are inherent to  determining the potentials.  In  addition, it shows that if the phase  factor  can  be  approximated  by  an  analytic  distribution  then  this  approximation  yields  a  swift  "Minutemax"  algorithm  that  vastly  reduces  the  computation  time  for  Minimax entropy  learning.  An  illustration  of this  concept,  using  a  Gaussian  to  approximate  the  phase  factor,  gives  a  good  approximation  to  the  results  of  Zhu  and  Mumford  (1997)  in  just  seconds  of  CPU  time.  The  phase  space  approach  also  gives  insight  into  the  multi-scale  potentials  found  by  Zhu and Mumford  (1997)  and suggests that the forms  of  the potentials are influenced greatly by phase space considerations.  Finally,  we  prove  that probability distributions  learned  in  feature  space  alone  are  equivalent  to  Minimax  Entropy  learning  with  a  multinomial approximation of the phase factor.
Almost Linear VC Dimension Bounds for Piecewise Polynomial Networks
Authors: Peter L. Bartlett, Vitaly Maiorov, Ron Meir
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  compute  upper  and  lower  bounds  on  the  VC  dimension  of  feedforward  networks  of  units  with  piecewise  polynomial  activa(cid:173) tion functions.  We  show that if the number of layers is  fixed,  then  the  VC  dimension  grows  as  W log W,  where  W  is  the  number of  parameters in the network.  This result stands in opposition to the  case  where  the  number of  layers  is  unbounded,  in  which  case  the  VC  dimension grows as W 2 •
Bayesian Modeling of Facial Similarity
Authors: Baback Moghaddam, Tony Jebara, Alex Pentland
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  previous work  [6, 9, 10],  we  advanced a  new technique for  direct  visual  matching  of  images  for  the  purposes  of  face  recognition  and  image  retrieval ,  using  a  probabilistic  measure  of  similarity  based  primarily  on  a  Bayesian  (MAP)  analysis  of image  differ(cid:173) ences,  leading  to  a  "dual"  basis  similar  to  eigenfaces  [13].  The  performance  advantage  of  this  probabilistic  matching  technique  over  standard  Euclidean  nearest-neighbor  eigenface  matching was  recently demonstrated using results from DARPA's 1996  "FERET"  face  recognition  competition, in  which  this  probabilistic matching  algorithm  was  found  to  be  the  top  performer.  We  have  further  developed  a  simple  method  of replacing  the  costly  com put ion  of  nonlinear  (online)  Bayesian  similarity  measures  by  the  relatively  inexpensive  computation  of  linear  (offline)  subspace  projections  and simple (online)  Euclidean norms, thus resulting in a significant  computational speed-up for  implementation with very  large image  databases  as typically encountered  in  real-world  applications.
The Bias-Variance Tradeoff and the Randomized GACV
Authors: Grace Wahba, Xiwu Lin, Fangyu Gao, Dong Xiang, Ronald Klein, Barbara Klein
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  propose a new in-sample cross validation based method (randomized  GACV) for choosing smoothing or bandwidth parameters that govern the  bias-variance or fit-complexity tradeoff in  'soft' classification.  Soft clas(cid:173) sification refers to  a  learning procedure which  estimates the probability  that an example with a given attribute vector is  in  class  1 vs  class O.  The  target  for  optimizing  the  the  tradeoff is  the  Kullback-Liebler distance  between  the  estimated  probability  distribution  and  the  'true'  probabil(cid:173) ity  distribution,  representing  knowledge of an  infinite  population.  The  method uses a randomized estimate of the trace of a Hessian and mimics  cross validation at the cost of a single relearning with perturbed outcome  data.
The Role of Lateral Cortical Competition in Ocular Dominance Development
Authors: Christian Piepenbrock, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Lateral competition within a layer of neurons sharpens and localizes the  response to an input stimulus. Here, we investigate a model for the ac(cid:173) tivity dependent development of ocular dominance maps which allows  to vary the degree of lateral competition. For weak competition, it re(cid:173) sembles a correlation-based learning model and for strong competition,  it becomes a self-organizing map. Thus, in the regime of weak compe(cid:173) tition the receptive fields are shaped by the second order statistics of the  input patterns, whereas in the regime of strong competition, the higher  moments and "features" of the individual patterns become important.  When correlated localized stimuli from two eyes drive the cortical de(cid:173) velopment we find (i) that a topographic map and binocular, localized  receptive fields emerge when the degree of competition exceeds a critical  value and (ii) that receptive fields exhibit eye dominance beyond a sec(cid:173) ond critical value. For anti-correlated activity between the eyes, the sec(cid:173) ond order statistics drive the system to develop ocular dominance even  for weak competition, but no topography emerges. Topography is estab(cid:173) lished only beyond a critical degree of competition.
General Bounds on Bayes Errors for Regression with Gaussian Processes
Authors: Manfred Opper, Francesco Vivarelli
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Based on a  simple convexity lemma, we  develop bounds for  differ(cid:173) ent types of Bayesian prediction errors for regression with Gaussian  processes.  The basic bounds are formulated for a fixed training set.  Simpler expressions are obtained for sampling from an input distri(cid:173) bution  which  equals  the  weight  function  of the covariance kernel,  yielding  asymptotically  tight  results.  The  results  are  compared  with numerical experiments.
A Precise Characterization of the Class of Languages Recognized by Neural Nets under Gaussian and Other Common Noise Distributions
Authors: Wolfgang Maass, Eduardo D. Sontag
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  consider recurrent analog  neural  nets  where each gate is  subject to  Gaussian noise, or any other common noise distribution whose probabil(cid:173) ity density function is nonzero on a large set.  We  show that many regular  languages cannot be  recognized by  networks of this  type,  for  example  the  language  {w  E  {O, I} * I w begins with O},  and  we  give  a  precise  characterization of those languages which can be recognized. This result  implies severe constraints on possibilities for constructing recurrent ana(cid:173) log neural  nets that are robust against realistic types of analog noise.  On  the other hand we present a method for constructing feed forward analog  neural nets that are robust with regard to analog noise of this type.
Outcomes of the Equivalence of Adaptive Ridge with Least Absolute Shrinkage
Authors: Yves Grandvalet, Stéphane Canu
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Adaptive Ridge is a special form of Ridge regression, balancing the  quadratic penalization on each parameter of the model. It was shown to  be equivalent to Lasso (least absolute shrinkage and selection operator),  in the sense that both procedures produce the same estimate. Lasso can  thus be viewed as a particular quadratic penalizer.  From this observation, we derive a fixed point algorithm to compute the  Lasso solution. The analogy provides also a new hyper-parameter for tun(cid:173) ing effectively the model complexity. We finally present a series ofpossi(cid:173) ble extensions oflasso performing sparse regression in kernel smoothing,  additive modeling and neural net training.
Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm
Authors: Tony Jebara, Alex Pentland
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present  the  CEM  (Conditional  Expectation  Maximi::ation)  al(cid:173) gorithm  as  an  extension  of  the  EM  (Expectation  M aximi::ation)  algorithm to conditional density estimation under  missing data.  A  bounding and maximization process is given to specifically optimize  conditional likelihood instead  of the  usual  joint likelihood.  We  ap(cid:173) ply  the  method  to  conditioned  mixture models  and  use  bounding  techniques  to  derive  the  model's  update  rules .  Monotonic  conver(cid:173) gence,  computational efficiency  and  regression  results  superior  to  EM  are  demonstrated.
Source Separation as a By-Product of Regularization
Authors: Sepp Hochreiter, Jürgen Schmidhuber
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper  reveals  a  previously  ignored  connection  between  two  important fields:  regularization and independent component anal(cid:173) ysis  (ICA).  We  show  that  at  least  one  representative  of a  broad  class  of algorithms  (regularizers  that  reduce  network complexity)  extracts independent  features  as  a  by-product.  This  algorithm  is  Flat  Minimum  Search  (FMS),  a  recent general method for  finding  low-complexity networks with high generalization capability.  FMS  works  by minimizing  both training error and required weight  pre(cid:173) cision.  According  to  our  theoretical  analysis  the  hidden  layer  of  an  FMS-trained  autoassociator attempts  at  coding  each  input  by  a  sparse  code  with  as  few  simple  features  as  possible.  In experi(cid:173) ments  the  method  extracts optimal  codes  for  difficult  versions  of  the  "noisy bars"  benchmark problem by separating the underlying  sources,  whereas ICA and PCA fail.  Real world images are coded  with fewer  bits per pixel than by ICA or PCA.
Analog VLSI Cellular Implementation of the Boundary Contour System
Authors: Gert Cauwenberghs, James Waskiewicz
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present an analog VLSI cellular architecture implementing a simpli(cid:173) . fied  version of the Boundary Contour System (BCS) for real-time image  processing.  Inspired by  neuromorphic models  across  several  layers  of  visual  cortex,  the  design  integrates  in  each pixel  the  functions  of sim(cid:173) ple cells,  complex cells,  hyper-complex cells,  and  bipole cells,  in  three  orientations interconnected on a  hexagonal grid.  Analog current-mode  CMOS circuits are used throughout to perform edge detection, local inhi(cid:173) bition, directionally selective long-range diffusive kernels, and renormal(cid:173) izing global gain control. Experimental results from a fabricated 12 x  10  pixel prototype in  1.2 J-tm  CMOS technology demonstrate the robustness  of the  architecture in  selecting image contours in  a cluttered and noisy  background.
Unsupervised Classification with Non-Gaussian Mixture Models Using ICA
Authors: Te-Won Lee, Michael S. Lewicki, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present  an  unsupervised  classification  algorithm  based  on  an  ICA  mixture  model.  The  ICA  mixture  model  assumes  that  the  observed  data can  be  categorized  into  several  mutually  exclusive  data classes  in  which  the components  in each  class  are generated  by  a  linear  mixture  of independent  sources.  The  algorithm  finds  the independent sources, the mixing matrix for  each class and also  computes  the  class  membership  probability  for  each  data  point.  This  approach  extends  the  Gaussian  mixture  model  so  that  the  classes  can  have  non-Gaussian  structure.  We  demonstrate  that  this method can learn efficient codes to represent images of natural  scenes and text.  The learned classes of basis functions yield a better  approximation of the underlying distributions of the data, and thus  can provide greater coding efficiency.  We  believe that this method  is  well  suited to  modeling structure in  high-dimensional  data and  has many potential applications. 
1
Barycentric Interpolators for Continuous Space and Time Reinforcement Learning
Authors: Rémi Munos, Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  order  to find  the  optimal control of continuous state-space  and  time  reinforcement  learning  (RL)  problems,  we  approximate  the  value function  (VF)  with a  particular class  of functions  called  the  barycentric  interpolators.  We  establish sufficient  conditions  under  which  a  RL algorithm converges  to the optimal VF,  even  when  we  use  approximate models of the  state  dynamics  and  the  reinforce(cid:173) ment functions .
Exploiting Generative Models in Discriminative Classifiers
Authors: Tommi Jaakkola, David Haussler
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Generative probability models such as hidden  ~larkov models pro(cid:173) vide  a  principled  way of treating  missing  information  and  dealing  with variable  length sequences.  On the other hand , discriminative  methods  such  as  support  vector  machines  enable  us  to  construct  flexible  decision  boundaries  and  often  result  in  classification  per(cid:173) formance superior to that of the model based approaches.  An ideal  classifier should  combine these two  complementary approaches.  In  this  paper,  we  develop  a  natural  way  of  achieving  this  combina(cid:173) tion  by deriving kernel functions for  use in discriminative methods  such  as support vector  machines from  generative probability mod(cid:173) els.  We  provide a  theoretical justification for  this  combination  as  well as demonstrate a substantial improvement in  the classification  performance in  the context of D~A and  protein sequence analysis.
Learning a Continuous Hidden Variable Model for Binary Data
Authors: Daniel D. Lee, Haim Sompolinsky
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: A directed generative model for binary data using a  small number  of hidden  continuous  units  is  investigated.  A  clipping  nonlinear(cid:173) ity distinguishes the model from conventional principal components  analysis.  The relationships between the correlations of the underly(cid:173) ing continuous Gaussian variables  and the binary output variables  are utilized  to learn  the  appropriate weights  of the network.  The  advantages of this  approach are illustrated on a  translationally in(cid:173) variant binary distribution and on handwritten digit  images.
Perceiving without Learning: From Spirals to Inside/Outside Relations
Authors: Ke Chen, DeLiang L. Wang
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: As  a benchmark task,  the  spiral  problem  is  well  known  in  neural  net(cid:173) works.  Unlike  previous  work  that  emphasizes  learning,  we  approach  the  problem from  a generic perspective that does  not involve learning.  We  point out that the spiral problem is intrinsically connected to the in(cid:173) side/outside problem.  A  generic  solution  to  both problems is proposed  based on  oscillatory correlation using a time delay network.  Our simu(cid:173) lation results are qualitatively  consistent with  human performance,  and  we  interpret human limitations in  terms  of synchrony and  time  delays,  both biologically plausible.  As a special case, our network without time  delays can always distinguish these figures regardless of shape, position,  size, and orientation.
Optimizing Classifers for Imbalanced Training Sets
Authors: Grigoris I. Karakoulas, John Shawe-Taylor
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Following  recent  results  [9,  8]  showing  the  importance of the  fat(cid:173) shattering  dimension  in  explaining  the  beneficial  effect  of a  large  margin  on  generalization  performance,  the  current  paper  investi(cid:173) gates  the  implications of these  results  for  the  case  of imbalanced  datasets  and  develops  two  approaches  to  setting  the  threshold.  The  approaches  are  incorporated  into ThetaBoost,  a  boosting  al(cid:173) gorithm for  dealing with  unequal  loss  functions.  The  performance  of ThetaBoost  and the  two approaches  are  tested  experimentally. 
Keywords:  Computational Learning Theory,  Generalization, fat-shattering,  large  margin, pac estimates,  unequal  loss,  imbalanced datasets
Blind Separation of Filtered Sources Using State-Space Approach
Authors: Liqing Zhang, Andrzej Cichocki
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  this  paper  we  present  a  novel  approach  to  multichannel  blind  separation/generalized deconvolution,  assuming that  both mixing  and demixing models are described by stable linear state-space sys(cid:173) tems.  We  decompose  the  blind  separation  problem  into  two  pro(cid:173) cess:  separation and state estimation.  Based on  the minimization  of Kullback-Leibler  Divergence,  we  develop  a  novel  learning algo(cid:173) rithm to train the matrices in the output equation.  To estimate the  state  of the  demixing  model,  we  introduce  a  new  concept,  called  hidden  innovation,  to  numerically  implement  the  Kalman  filter.  Computer simulations are given  to show  the validity  and  high  ef(cid:173) fectiveness of the state-space approach. 
1
Recurrent Cortical Amplification Produces Complex Cell Responses
Authors: Frances S. Chance, Sacha B. Nelson, L. F. Abbott
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Cortical amplification has been proposed as a mechanism for enhancing  the selectivity of neurons in the primary visual  cortex.  Less appreciated  is the fact that the same form of amplification can also be used to de-tune  or  broaden selectivity.  Using  a  network  model  with  recurrent  cortical  circuitry,  we  propose  that the  spatial  phase  invariance  of complex  cell  responses  arises  through  recurrent  amplification  of feedforward  input.  Neurons in  the network respond  like simple cells at low  gain and com(cid:173) plex ceUs at high  gain.  Similar recurrent  mechanisms  may  playa role  in generating invariant representations of feedforward input elsewhere in  the visual processing pathway.
Viewing Classifier Systems as Model Free Learning in POMDPs
Authors: Akira Hayashi, Nobuo Suematsu
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Classifier systems are now  viewed disappointing because of their prob(cid:173) lems such as the rule  strength vs rule  set performance problem and the  credit assignment problem.  In order to solve the problems, we have de(cid:173) veloped a hybrid classifier  system:  GLS  (Generalization Learning Sys(cid:173) tem).  In designing GLS, we view CSs as model free learning in POMDPs  and  take a  hybrid approach to finding  the  best generalization,  given the  total  number of rules.  GLS  uses  the policy improvement procedure by  Jaakkola  et al.  for  an locally  optimal  stochastic policy  when  a  set of  rule conditions is  given.  GLS  uses GA to search for  the best set of rule  conditions.
Reinforcement Learning Based on On-Line EM Algorithm
Authors: Masa-aki Sato, Shin Ishii
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  this  article,  we  propose  a  new  reinforcement  learning  (RL)  method  based  on  an  actor-critic  architecture.  The  actor  and  the  critic  are  approximated  by  Normalized  Gaussian  Networks  (NGnet),  which  are  networks  of local  linear regression  units.  The  NGnet is trained by the on-line EM algorithm proposed in our pre(cid:173) vious  paper.  We  apply our  RL  method  to the task of swinging-up  and stabilizing a  single pendulum and the task of balancing a  dou(cid:173) ble  pendulum near the upright  position.  The experimental results  show  that our RL  method can  be applied  to optimal control prob(cid:173) lems  having  continuous  state/action  spaces  and  that  the  method  achieves  good  control  with  a  small  number of trial-and-errors.
Learning Multi-Class Dynamics
Authors: Andrew Blake, Ben North, Michael Isard
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Standard  techniques  (eg.  Yule-Walker)  are  available  for  learning  Auto-Regressive process models of simple,  directly observable,  dy(cid:173) namical  processes.  When  sensor  noise  means  that  dynamics  are  observed  only  approximately,  learning can  still  been  achieved  via  Expectation-Maximisation  (EM)  together  with  Kalman  Filtering.  However,  this  does  not  handle  more  complex  dynamics,  involving  multiple  classes  of motion.  For  that  problem,  we  show  here  how  EM  can  be combined  with  the  CONDENSATION  algorithm,  which  is  based on propagation of random sample-sets.  Experiments have  been  performed  with  visually  observed juggling,  and plausible dy(cid:173) namical models are found  to emerge from  the learning process.
Markov Processes on Curves for Automatic Speech Recognition
Authors: Lawrence K. Saul, Mazin G. Rahim
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  investigate  a  probabilistic  framework  for  automatic  speech  recognition  based  on  the  intrinsic  geometric  properties  of curves.  In  particular,  we  analyze  the  setting  in  which  two  variables-one  continuous  (~), one  discrete  (s )-evolve jointly in  time.  We  sup(cid:173) pose  that the vector ~ traces out a smooth multidimensional curve  and  that  the  variable  s  evolves  stochastically  as  a  function  of the  arc  length  traversed  along  this  curve.  Since  arc  length  does  not  depend  on  the  rate  at  which  a  curve  is  traversed,  this  gives  rise  to  a  family  of  Markov  processes  whose  predictions,  Pr[sl~]'  are  invariant  to  nonlinear  warpings  of time.  We  describe  the  use  of  such  models,  known  as  Markov  processes  on  curves  (MPCs),  for  automatic speech  recognition,  where  ~ are  acoustic feature  trajec(cid:173) tories and s are phonetic transcriptions.  On two tasks-recognizing  New  Jersey  town names and connected  alpha-digits- we  find  that  MPCs yield lower word error rates than comparably trained hidden  Markov  models.
Discovering Hidden Features with Gaussian Processes Regression
Authors: Francesco Vivarelli, Christopher K. I. Williams
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  study  the  dynamics  of supervised  learning  in  layered  neural  net(cid:173) works,  in  the regime where the size p of the training set is  proportional  to the number N  of inputs.  Here the local fields  are no longer described  by  Gaussian  distributions.  We  use  dynamical  replica theory  to  predict  the  evolution  of macroscopic  observables,  including  the  relevant  error  measures, incorporating the old formalism  in the limit piN --t  00.
Exploratory Data Analysis Using Radial Basis Function Latent Variable Models
Authors: Alan D. Marrs, Andrew R. Webb
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Two  developments of nonlinear latent variable models  based  on  radial  basis functions are discussed:  in the first, the use of priors or constraints  on allowable models is considered as a means of preserving data structure  in  low-dimensional representations for  visualisation  purposes.  Also,  a  resampling approach  is  introduced which  makes  more  effective use  of  the latent samples in evaluating the likelihood.
A High Performance k-NN Classifier Using a Binary Correlation Matrix Memory
Authors: Ping Zhou, Jim Austin, John Kennedy
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper  presents  a  novel  and  fast  k-NN  classifier  that  is  based  on  a  binary  CMM  (Correlation  Matrix  Memory)  neural  network.  A  robust  encoding  method  is  developed  to  meet  CMM  input  requirements .  A  hardware implementation of the CMM is  described, which gives  over 200  times  the  speed  of a  current  mid-range  workstation,  and  is  scaleable  to  very  large  problems.  When  tested  on  several  benchmarks  and  compared  with a simple k-NN method, the CMM classifier gave  less than  I %  lower  accuracy  and  over  4  and  12  times  speed-up  in  software  and  hardware  respectively.
Inference in Multilayer Networks via Large Deviation Bounds
Authors: Michael J. Kearns, Lawrence K. Saul
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  study  probabilistic  inference  in  large,  layered  Bayesian  net(cid:173) works  represented  as  directed  acyclic  graphs.  We  show  that  the  intractability of exact inference in such  networks does  not preclude  their effective  use.  We  give algorithms for  approximate probabilis(cid:173) tic inference  that exploit averaging phenomena occurring at nodes  with  large  numbers  of parents.  We  show  that  these  algorithms  compute  rigorous  lower  and  upper  bounds  on  marginal probabili(cid:173) ties  of interest,  prove  that these  bounds  become exact in  the limit  of large networks,  and provide rates  of convergence.
Basis Selection for Wavelet Regression
Authors: Kevin R. Wheeler, Atam P. Dhawan
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: A  wavelet  basis  selection  procedure  is  presented  for  wavelet  re(cid:173) gression.  Both  the  basis  and  threshold  are  selected  using  cross(cid:173) validation.  The  method  includes  the  capability  of  incorporating  prior knowledge on the smoothness (or shape of the basis functions)  into  the  basis  selection  procedure.  The  results  of the  method  are  demonstrated  using  widely  published  sampled  functions.  The  re(cid:173) sults of the method are contrasted with other basis function  based  methods.
Where Does the Population Vector of Motor Cortical Cells Point during Reaching Movements?
Authors: Pierre Baraduc, Emmanuel Guigon, Yves Burnod
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Visually-guided  arm  reaching  movements  are  produced  by  distributed  neural networks within parietal and frontal regions of the cerebral cortex.  Experimental  data  indicate  that (I) single  neurons  in  these  regions  are  broadly tuned to parameters of movement; (2) appropriate commands are  elaborated by  populations of neurons; (3) the coordinated action of neu(cid:173) rons can be  visualized  using a neuronal population vector (NPV).  How(cid:173) ever,  the  NPV  provides only  a rough  estimate  of movement parameters  (direction, velocity) and may even fail  to reflect the parameters of move(cid:173) ment when arm posture is changed.  We designed a model of the cortical  motor command to  investigate the relation between the desired direction  of the  movement, the  actual direction of movement and  the direction of  the NPV in  motor cortex. The model is a two-layer self-organizing neural  network  which  combines  broadly-tuned  (muscular)  proprioceptive  and  (cartesian) visual information to calculate (angular) motor commands for  the  initial  part  of the  movement of a  two-link  arm.  The  network  was  trained  by  motor babbling  in  5  positions.  Simulations  showed  that  (1)  the  network produced appropriate movement direction over a  large part  of the  workspace;  (2) small  deviations of the  actual  trajectory  from  the  desired  trajectory existed  at the extremities of the  workspace;  (3)  these  deviations were accompanied by  large deviations of the NPV from both  trajectories. These results suggest the NPV does not give a faithful image  of cortical processing during arm reaching movements. 
• to  whom correspondence should be addressed 
84 
P.  Baraduc,  E. Guigon and Y.  Burnod
Inferring neural population dynamics from multiple partial recordings of the same neural circuit
Authors: Srini Turaga, Lars Buesing, Adam M Packer, Henry Dalgleish, Noah Pettit, Michael Hausser, Jakob H Macke
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Simultaneous recordings of the activity of large neural populations are extremely valuable as they can be used to infer the dynamics and interactions of neurons in a local circuit, shedding light on the computations performed. It is now possible to measure the activity of hundreds of neurons  using 2-photon calcium imaging. However, many computations are thought to involve circuits consisting of thousands of neurons, such as cortical barrels in rodent somatosensory cortex. Here we contribute a statistical method for stitching" together sequentially imaged sets of neurons into one model  by phrasing the problem as fitting a latent dynamical system with missing observations. This method allows us to substantially expand the population-sizes for which population dynamics can be characterized---beyond the number of simultaneously imaged neurons. In particular, we demonstrate using recordings in mouse somatosensory cortex that this method makes it possible to predict noise correlations between non-simultaneously recorded neuron pairs."
Approximate Gaussian process inference for the drift function in stochastic differential equations
Authors: Andreas Ruttor, Philipp Batz, Manfred Opper
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: We introduce a nonparametric approach for estimating drift functions in systems of stochastic differential equations from incomplete observations of the state vector. Using a Gaussian process prior over the drift as a function of the state vector, we develop an approximate EM algorithm to deal with the unobserved, latent dynamics between observations. The posterior over states is approximated by a piecewise linearized process and the MAP estimation of the drift is facilitated by a sparse Gaussian process regression.
Third-Order Edge Statistics: Contour Continuation, Curvature, and Cortical Connections
Authors: Matthew Lawlor, Steven W Zucker
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Association field models have been used to explain human contour grouping performance and to explain the mean frequency of long-range horizontal connections across cortical columns in V1. However, association fields essentially depend on pairwise statistics of edges in natural scenes. We develop a spectral test of the sufficiency of pairwise statistics and show that there is significant higher-order structure.  An analysis using a probabilistic spectral embedding reveals curvature-dependent components to the association field, and reveals a challenge for biological learning algorithms.
Transportability from Multiple Environments with Limited Experiments
Authors: Elias Bareinboim, Sanghack Lee, Vasant Honavar, Judea Pearl
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: This paper considers the problem of transferring experimental findings learned from multiple heterogeneous domains to a target environment, in which only limited experiments can be performed. We reduce questions of transportability from multiple domains and with limited scope to symbolic derivations in the do-calculus, thus extending the treatment of transportability from full experiments introduced in Pearl and Bareinboim (2011). We further provide different graphical and algorithmic conditions for computing the transport formula for this setting, that is, a way of fusing the observational and experimental information scattered throughout different domains to synthesize a consistent estimate of the desired effects.
On model selection consistency of penalized M-estimators: a geometric theory
Authors: Jason Lee, Yuekai Sun, Jonathan E Taylor
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Penalized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Often, the penalties are \emph{geometrically decomposable}, \ie\ can be expressed as a sum of (convex) support functions. We generalize the notion of irrepresentable to geometrically decomposable penalties and develop a general framework for establishing consistency and model selection consistency of M-estimators with such penalties. We then use this framework to derive results for some special cases of interest in bioinformatics and statistical learning.
Robust Bloom Filters for Large MultiLabel Classification Tasks
Authors: Moustapha M Cisse, Nicolas Usunier, Thierry Artières, Patrick Gallinari
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: This  paper presents an approach to multilabel classification (MLC) with a large number of labels. Our approach is a reduction to binary classification in which label sets are represented by low dimensional binary vectors. This representation follows the principle of Bloom filters, a space-efficient data structure originally designed for approximate membership testing. We show that a naive application of Bloom filters in MLC is not robust to individual binary classifiers' errors. We then present an approach that exploits a specific feature of real-world datasets when the number of labels is large: many labels (almost) never appear together.  Our approch is provably robust, has sublinear training and inference complexity with respect to the number of labels, and compares favorably to state-of-the-art algorithms on two large scale multilabel datasets.
On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation
Authors: Harikrishna Narasimhan, Shivani Agarwal
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: We investigate the relationship between three fundamental problems in machine learning: binary classification, bipartite ranking, and binary class probability estimation (CPE). It is known that a good binary CPE model can be used to obtain a good binary classification model (by thresholding at 0.5), and also to obtain a good bipartite ranking model (by using the CPE model directly as a ranking model); it is also known that a binary classification model does not necessarily yield a CPE model. However, not much is known about other directions. Formally, these relationships involve regret transfer bounds. In this paper, we introduce the notion of weak regret transfer bounds, where the mapping needed to transform a model from one problem to another depends on the underlying probability distribution (and in practice, must be estimated from data). We then show that, in this weaker sense, a good bipartite ranking model can be used to construct a good classification model (by thresholding at a suitable point), and more surprisingly, also to construct a good binary CPE model (by calibrating the scores of the ranking model).