mtybilly/Instruct2DS huggingface dataset:
The Role of Lateral Cortical Competition in Ocular Dominance Development
Authors: Klaus Obermayer Christian Piepenbrock
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Lateral competition within a layer of neurons sharpens and localizes the  response to an input stimulus. Here, we investigate a model for the ac(cid:173) tivity dependent development of ocular dominance maps which allows  to vary the degree of lateral competition. For weak competition, it re(cid:173) sembles a correlation-based learning model and for strong competition,  it becomes a self-organizing map. Thus, in the regime of weak compe(cid:173) tition the receptive fields are shaped by the second order statistics of the  input patterns, whereas in the regime of strong competition, the higher  moments and "features" of the individual patterns become important.  When correlated localized stimuli from two eyes drive the cortical de(cid:173) velopment we find (i) that a topographic map and binocular, localized  receptive fields emerge when the degree of competition exceeds a critical  value and (ii) that receptive fields exhibit eye dominance beyond a sec(cid:173) ond critical value. For anti-correlated activity between the eyes, the sec(cid:173) ond order statistics drive the system to develop ocular dominance even  for weak competition, but no topography emerges. Topography is estab(cid:173) lished only beyond a critical degree of competition.
General Bounds on Bayes Errors for Regression with Gaussian Processes
Authors: Francesco Vivarelli Manfred Opper
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Based on a  simple convexity lemma, we  develop bounds for  differ(cid:173) ent types of Bayesian prediction errors for regression with Gaussian  processes.  The basic bounds are formulated for a fixed training set.  Simpler expressions are obtained for sampling from an input distri(cid:173) bution  which  equals  the  weight  function  of the covariance kernel,  yielding  asymptotically  tight  results.  The  results  are  compared  with numerical experiments.
Bayesian PCA
Authors: Christopher M. Bishop
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: The technique of principal component analysis (PCA) has recently been  expressed  as  the  maximum  likelihood  solution  for  a  generative  latent  variable  model.  In  this  paper  we  use  this  probabilistic  reformulation  as  the basis  for a  Bayesian treatment of PCA.  Our key  result  is  that  ef(cid:173) fective  dimensionality  of the  latent  space  (equivalent to  the  number of  retained principal components) can be determined automatically as  part  of the  Bayesian  inference  procedure.  An  important application  of this  framework  is  to  mixtures  of probabilistic  PCA  models,  in  which  each  component can determine its own effective complexity.
A Precise Characterization of the Class of Languages Recognized by Neural Nets under Gaussian and Other Common Noise Distributions
Authors: Eduardo D. Sontag Wolfgang Maass
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  consider recurrent analog  neural  nets  where each gate is  subject to  Gaussian noise, or any other common noise distribution whose probabil(cid:173) ity density function is nonzero on a large set.  We  show that many regular  languages cannot be  recognized by  networks of this  type,  for  example  the  language  {w  E  {O, I} * I w begins with O},  and  we  give  a  precise  characterization of those languages which can be recognized. This result  implies severe constraints on possibilities for constructing recurrent ana(cid:173) log neural  nets that are robust against realistic types of analog noise.  On  the other hand we present a method for constructing feed forward analog  neural nets that are robust with regard to analog noise of this type.
General-Purpose Localization of Textured Image Regions
Authors: Ruth Rosenholtz
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  suggest a working  definition  of texture:  Texture  is  stuff that is  more  compactly represented  by  its  statistics  than  by  specifying  the  configuration  of  its  parts.  This  definition  suggests  that  to  fmd  texture  we  look  for  outliers  to  the  local  statistics,  and  label  as  texture  the  regions  with  no  outliers.  We  present a  method,  based  upon  this  idea, for  labeling points  in  natural scenes  as  belonging to  texture  regions,  while  simultaneously  allowing  us  to  label  low(cid:173) level,  bottom-up  cues  for  visual  attention.  This  method  is  based  upon  recent  psychophysics  results  on  processing  of texture  and  popout. 
1  WHAT  IS  TEXTURE,  AND  WHY  DO  WE  WANT  TO  FIND  IT? 
In  a  number  of problems  in  computer  VlSlon  and  image  processing,  one  must  distinguish  between  image  regions  that  correspond  to  objects  and  those  which  correspond to  texture,  and  perform different processing depending  upon  the  type  of  region.  Current  computer  vision  algorithms  assume  one  magically  knows  this  region  labeling.  But  what  is  texture?  We  have  the  notion  that texture  involves  a  pattern  that  is  somehow  homogeneous,  or  in  which  signal  changes  are  "too  complex"  to  describe,  so  that  aggregate  properties  must  be  used  instead  (Saund,  1998).  There is  by no means a firm  division between texture and  objects; rather, the  characterization often depends upon  the scale of interest (Saund, 1998). 
• Email: rruth@parc.xerox.com 
818
Making Templates Rotationally Invariant. An Application to Rotated Digit Recognition
Authors: Shumeet Baluja
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper  describes  a  simple  and  efficient  method  to  make  template-based  object classification invariant to in-plane rotations. The task is divided into two  parts:  orientation discrimination and classification. The key idea is  to perform  the  orientation  discrimination  before  the  classification.  This  can  be  accom(cid:173) plished by hypothesizing, in  turn, that the input image belongs to each class of  interest. The image can then be rotated to  maximize its  similarity to the train(cid:173) ing images in each class (these contain the prototype object in an upright orien(cid:173) tation). This process yields a set of images, at least one of which will have the  object  in  an  upright position.  The resulting  images  can  then be classified by  models  which  have been  trained  with  only  upright examples.  This  approach  has  been  successfully  applied  to  two  real-world  vision-based  tasks:  rotated  handwritten digit recognition and rotated face detection in cluttered scenes.
Outcomes of the Equivalence of Adaptive Ridge with Least Absolute Shrinkage
Authors: Stéphane Canu Yves Grandvalet
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Adaptive Ridge is a special form of Ridge regression, balancing the  quadratic penalization on each parameter of the model. It was shown to  be equivalent to Lasso (least absolute shrinkage and selection operator),  in the sense that both procedures produce the same estimate. Lasso can  thus be viewed as a particular quadratic penalizer.  From this observation, we derive a fixed point algorithm to compute the  Lasso solution. The analogy provides also a new hyper-parameter for tun(cid:173) ing effectively the model complexity. We finally present a series ofpossi(cid:173) ble extensions oflasso performing sparse regression in kernel smoothing,  additive modeling and neural net training.
Bayesian Modeling of Human Concept Learning
Authors: Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: I consider the problem of learning concepts from small numbers of pos(cid:173) itive examples,  a feat  which humans perform routinely but which com(cid:173) puters  are  rarely  capable  of.  Bridging machine  learning  and  cognitive  science perspectives, I present both theoretical analysis and an empirical  study with human subjects for the simple task oflearning concepts corre(cid:173) sponding to axis-aligned rectangles in a multidimensional feature space.  Existing learning models, when applied to this task, cannot explain how  subjects generalize from only a few  examples of the concept.  I propose  a principled Bayesian model based on the assumption that the examples  are  a random sample from  the concept to be  learned.  The  model  gives  precise fits to human behavior on this simple task and provides qualitati ve  insights into more complex, realistic cases of concept learning.
Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm
Authors: Alex Pentland Tony Jebara
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present  the  CEM  (Conditional  Expectation  Maximi::ation)  al(cid:173) gorithm  as  an  extension  of  the  EM  (Expectation  M aximi::ation)  algorithm to conditional density estimation under  missing data.  A  bounding and maximization process is given to specifically optimize  conditional likelihood instead  of the  usual  joint likelihood.  We  ap(cid:173) ply  the  method  to  conditioned  mixture models  and  use  bounding  techniques  to  derive  the  model's  update  rules .  Monotonic  conver(cid:173) gence,  computational efficiency  and  regression  results  superior  to  EM  are  demonstrated.
Source Separation as a By-Product of Regularization
Authors: Jürgen Schmidhuber Sepp Hochreiter
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper  reveals  a  previously  ignored  connection  between  two  important fields:  regularization and independent component anal(cid:173) ysis  (ICA).  We  show  that  at  least  one  representative  of a  broad  class  of algorithms  (regularizers  that  reduce  network complexity)  extracts independent  features  as  a  by-product.  This  algorithm  is  Flat  Minimum  Search  (FMS),  a  recent general method for  finding  low-complexity networks with high generalization capability.  FMS  works  by minimizing  both training error and required weight  pre(cid:173) cision.  According  to  our  theoretical  analysis  the  hidden  layer  of  an  FMS-trained  autoassociator attempts  at  coding  each  input  by  a  sparse  code  with  as  few  simple  features  as  possible.  In experi(cid:173) ments  the  method  extracts optimal  codes  for  difficult  versions  of  the  "noisy bars"  benchmark problem by separating the underlying  sources,  whereas ICA and PCA fail.  Real world images are coded  with fewer  bits per pixel than by ICA or PCA.
Analog VLSI Cellular Implementation of the Boundary Contour System
Authors: James Waskiewicz Gert Cauwenberghs
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present an analog VLSI cellular architecture implementing a simpli(cid:173) . fied  version of the Boundary Contour System (BCS) for real-time image  processing.  Inspired by  neuromorphic models  across  several  layers  of  visual  cortex,  the  design  integrates  in  each pixel  the  functions  of sim(cid:173) ple cells,  complex cells,  hyper-complex cells,  and  bipole cells,  in  three  orientations interconnected on a  hexagonal grid.  Analog current-mode  CMOS circuits are used throughout to perform edge detection, local inhi(cid:173) bition, directionally selective long-range diffusive kernels, and renormal(cid:173) izing global gain control. Experimental results from a fabricated 12 x  10  pixel prototype in  1.2 J-tm  CMOS technology demonstrate the robustness  of the  architecture in  selecting image contours in  a cluttered and noisy  background.
Unsupervised Classification with Non-Gaussian Mixture Models Using ICA
Authors: Michael S. Lewicki, Terrence J. Sejnowski Te-Won Lee
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  present  an  unsupervised  classification  algorithm  based  on  an  ICA  mixture  model.  The  ICA  mixture  model  assumes  that  the  observed  data can  be  categorized  into  several  mutually  exclusive  data classes  in  which  the components  in each  class  are generated  by  a  linear  mixture  of independent  sources.  The  algorithm  finds  the independent sources, the mixing matrix for  each class and also  computes  the  class  membership  probability  for  each  data  point.  This  approach  extends  the  Gaussian  mixture  model  so  that  the  classes  can  have  non-Gaussian  structure.  We  demonstrate  that  this method can learn efficient codes to represent images of natural  scenes and text.  The learned classes of basis functions yield a better  approximation of the underlying distributions of the data, and thus  can provide greater coding efficiency.  We  believe that this method  is  well  suited to  modeling structure in  high-dimensional  data and  has many potential applications. 
1
Barycentric Interpolators for Continuous Space and Time Reinforcement Learning
Authors: Andrew W. Moore Rémi Munos
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  order  to find  the  optimal control of continuous state-space  and  time  reinforcement  learning  (RL)  problems,  we  approximate  the  value function  (VF)  with a  particular class  of functions  called  the  barycentric  interpolators.  We  establish sufficient  conditions  under  which  a  RL algorithm converges  to the optimal VF,  even  when  we  use  approximate models of the  state  dynamics  and  the  reinforce(cid:173) ment functions .
Exploiting Generative Models in Discriminative Classifiers
Authors: David Haussler Tommi Jaakkola
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Generative probability models such as hidden  ~larkov models pro(cid:173) vide  a  principled  way of treating  missing  information  and  dealing  with variable  length sequences.  On the other hand , discriminative  methods  such  as  support  vector  machines  enable  us  to  construct  flexible  decision  boundaries  and  often  result  in  classification  per(cid:173) formance superior to that of the model based approaches.  An ideal  classifier should  combine these two  complementary approaches.  In  this  paper,  we  develop  a  natural  way  of  achieving  this  combina(cid:173) tion  by deriving kernel functions for  use in discriminative methods  such  as support vector  machines from  generative probability mod(cid:173) els.  We  provide a  theoretical justification for  this  combination  as  well as demonstrate a substantial improvement in  the classification  performance in  the context of D~A and  protein sequence analysis.
Learning a Continuous Hidden Variable Model for Binary Data
Authors: Haim Sompolinsky Daniel D. Lee
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: A directed generative model for binary data using a  small number  of hidden  continuous  units  is  investigated.  A  clipping  nonlinear(cid:173) ity distinguishes the model from conventional principal components  analysis.  The relationships between the correlations of the underly(cid:173) ing continuous Gaussian variables  and the binary output variables  are utilized  to learn  the  appropriate weights  of the network.  The  advantages of this  approach are illustrated on a  translationally in(cid:173) variant binary distribution and on handwritten digit  images.
Perceiving without Learning: From Spirals to Inside/Outside Relations
Authors: DeLiang L. Wang Ke Chen
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: As  a benchmark task,  the  spiral  problem  is  well  known  in  neural  net(cid:173) works.  Unlike  previous  work  that  emphasizes  learning,  we  approach  the  problem from  a generic perspective that does  not involve learning.  We  point out that the spiral problem is intrinsically connected to the in(cid:173) side/outside problem.  A  generic  solution  to  both problems is proposed  based on  oscillatory correlation using a time delay network.  Our simu(cid:173) lation results are qualitatively  consistent with  human performance,  and  we  interpret human limitations in  terms  of synchrony and  time  delays,  both biologically plausible.  As a special case, our network without time  delays can always distinguish these figures regardless of shape, position,  size, and orientation.
Optimizing Classifers for Imbalanced Training Sets
Authors: John Shawe-Taylor Grigoris I. Karakoulas
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Following  recent  results  [9,  8]  showing  the  importance of the  fat(cid:173) shattering  dimension  in  explaining  the  beneficial  effect  of a  large  margin  on  generalization  performance,  the  current  paper  investi(cid:173) gates  the  implications of these  results  for  the  case  of imbalanced  datasets  and  develops  two  approaches  to  setting  the  threshold.  The  approaches  are  incorporated  into ThetaBoost,  a  boosting  al(cid:173) gorithm for  dealing with  unequal  loss  functions.  The  performance  of ThetaBoost  and the  two approaches  are  tested  experimentally. 
Keywords:  Computational Learning Theory,  Generalization, fat-shattering,  large  margin, pac estimates,  unequal  loss,  imbalanced datasets
Blind Separation of Filtered Sources Using State-Space Approach
Authors: Andrzej Cichocki Liqing Zhang
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  this  paper  we  present  a  novel  approach  to  multichannel  blind  separation/generalized deconvolution,  assuming that  both mixing  and demixing models are described by stable linear state-space sys(cid:173) tems.  We  decompose  the  blind  separation  problem  into  two  pro(cid:173) cess:  separation and state estimation.  Based on  the minimization  of Kullback-Leibler  Divergence,  we  develop  a  novel  learning algo(cid:173) rithm to train the matrices in the output equation.  To estimate the  state  of the  demixing  model,  we  introduce  a  new  concept,  called  hidden  innovation,  to  numerically  implement  the  Kalman  filter.  Computer simulations are given  to show  the validity  and  high  ef(cid:173) fectiveness of the state-space approach. 
1
Recurrent Cortical Amplification Produces Complex Cell Responses
Authors: Sacha B. Nelson, L. F. Abbott Frances S. Chance
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Cortical amplification has been proposed as a mechanism for enhancing  the selectivity of neurons in the primary visual  cortex.  Less appreciated  is the fact that the same form of amplification can also be used to de-tune  or  broaden selectivity.  Using  a  network  model  with  recurrent  cortical  circuitry,  we  propose  that the  spatial  phase  invariance  of complex  cell  responses  arises  through  recurrent  amplification  of feedforward  input.  Neurons in  the network respond  like simple cells at low  gain and com(cid:173) plex ceUs at high  gain.  Similar recurrent  mechanisms  may  playa role  in generating invariant representations of feedforward input elsewhere in  the visual processing pathway.
Viewing Classifier Systems as Model Free Learning in POMDPs
Authors: Nobuo Suematsu Akira Hayashi
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Classifier systems are now  viewed disappointing because of their prob(cid:173) lems such as the rule  strength vs rule  set performance problem and the  credit assignment problem.  In order to solve the problems, we have de(cid:173) veloped a hybrid classifier  system:  GLS  (Generalization Learning Sys(cid:173) tem).  In designing GLS, we view CSs as model free learning in POMDPs  and  take a  hybrid approach to finding  the  best generalization,  given the  total  number of rules.  GLS  uses  the policy improvement procedure by  Jaakkola  et al.  for  an locally  optimal  stochastic policy  when  a  set of  rule conditions is  given.  GLS  uses GA to search for  the best set of rule  conditions.
Reinforcement Learning Based on On-Line EM Algorithm
Authors: Shin Ishii Masa-aki Sato
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: In  this  article,  we  propose  a  new  reinforcement  learning  (RL)  method  based  on  an  actor-critic  architecture.  The  actor  and  the  critic  are  approximated  by  Normalized  Gaussian  Networks  (NGnet),  which  are  networks  of local  linear regression  units.  The  NGnet is trained by the on-line EM algorithm proposed in our pre(cid:173) vious  paper.  We  apply our  RL  method  to the task of swinging-up  and stabilizing a  single pendulum and the task of balancing a  dou(cid:173) ble  pendulum near the upright  position.  The experimental results  show  that our RL  method can  be applied  to optimal control prob(cid:173) lems  having  continuous  state/action  spaces  and  that  the  method  achieves  good  control  with  a  small  number of trial-and-errors.
Learning Multi-Class Dynamics
Authors: Ben North, Michael Isard Andrew Blake
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Standard  techniques  (eg.  Yule-Walker)  are  available  for  learning  Auto-Regressive process models of simple,  directly observable,  dy(cid:173) namical  processes.  When  sensor  noise  means  that  dynamics  are  observed  only  approximately,  learning can  still  been  achieved  via  Expectation-Maximisation  (EM)  together  with  Kalman  Filtering.  However,  this  does  not  handle  more  complex  dynamics,  involving  multiple  classes  of motion.  For  that  problem,  we  show  here  how  EM  can  be combined  with  the  CONDENSATION  algorithm,  which  is  based on propagation of random sample-sets.  Experiments have  been  performed  with  visually  observed juggling,  and plausible dy(cid:173) namical models are found  to emerge from  the learning process.
Markov Processes on Curves for Automatic Speech Recognition
Authors: Mazin G. Rahim Lawrence K. Saul
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  investigate  a  probabilistic  framework  for  automatic  speech  recognition  based  on  the  intrinsic  geometric  properties  of curves.  In  particular,  we  analyze  the  setting  in  which  two  variables-one  continuous  (~), one  discrete  (s )-evolve jointly in  time.  We  sup(cid:173) pose  that the vector ~ traces out a smooth multidimensional curve  and  that  the  variable  s  evolves  stochastically  as  a  function  of the  arc  length  traversed  along  this  curve.  Since  arc  length  does  not  depend  on  the  rate  at  which  a  curve  is  traversed,  this  gives  rise  to  a  family  of  Markov  processes  whose  predictions,  Pr[sl~]'  are  invariant  to  nonlinear  warpings  of time.  We  describe  the  use  of  such  models,  known  as  Markov  processes  on  curves  (MPCs),  for  automatic speech  recognition,  where  ~ are  acoustic feature  trajec(cid:173) tories and s are phonetic transcriptions.  On two tasks-recognizing  New  Jersey  town names and connected  alpha-digits- we  find  that  MPCs yield lower word error rates than comparably trained hidden  Markov  models.
Discovering Hidden Features with Gaussian Processes Regression
Authors: Christopher K. I. Williams Francesco Vivarelli
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  study  the  dynamics  of supervised  learning  in  layered  neural  net(cid:173) works,  in  the regime where the size p of the training set is  proportional  to the number N  of inputs.  Here the local fields  are no longer described  by  Gaussian  distributions.  We  use  dynamical  replica theory  to  predict  the  evolution  of macroscopic  observables,  including  the  relevant  error  measures, incorporating the old formalism  in the limit piN --t  00.
Utilizing lime: Asynchronous Binding
Authors: Bradley C. Love
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Historically,  connectionist  systems  have  not excelled  at  represent(cid:173) ing and manipulating complex structures.  How  can a  system com(cid:173) posed  of simple  neuron-like  computing  elements  encode  complex  relations?  Recently,  researchers have begun to appreciate that rep(cid:173) resentations can extend in  both time and space.  Many researchers  have proposed that the synchronous firing of units can encode com(cid:173) plex  representations.  I  identify  the  limitations  of  this  approach  and present an  asynchronous model of binding that effectively rep(cid:173) resents  complex  structures.  The  asynchronous model  extends  the  synchronous approach.  I argue that our cognitive architecture uti(cid:173) lizes  a similar mechanism.
Very Fast EM-Based Mixture Model Clustering Using Multiresolution Kd-Trees
Authors: Andrew W. Moore
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Clust ering  is  impor ta nt  in  m any  fields  including  m anufac tlll'ing ,  biol og~',  fin ance , a nd astronomy.  l\Iixturp  models arp a  popula r  ap(cid:173) proach  due  to  their  st.atist.ical  found a t.ions,  and  EM  is  a  very  pop(cid:173) ular  l1wthocl  for  fillding  mixture  models.  EM,  however,  requires  lllany accesses  of the dat a , a nd  thus h as  been  dismissed  as  imprac(cid:173) t ical  (e.g.  [9])  for  d ata mining of enormous dataset.s.  We  present  a  nt' \·  algorit.hm,  baspd  on  thp  l1lultiresolution  ~.'Cl- trees of [5] ,  which  dramatically reelucps  the cost  of EtlI-baspd  clusteriug , wit.h  savings  rising  linearl:;  wit.h  the number  of datapoints.  Although  prespnt.pd  lwre for  maximum likplihoocl estimation of Gaussian mixt.ure mod(cid:173) f'ls ,  it.  is  also  applicable  to  non-(~aussian models  (provided  class  densit.ies  are  monotonic in  Mahalanobis dist.ance), mixed  categori(cid:173) cal/ nUllwric  clusters.  anel  Bayesian  nwthocls  such  as  Antoclass  [1]. 
1  Learning Mixture Models  In  a  Gaussian mixture lllod f'l  (e.g.  [3]) , we  aSSUI1W  t.hat  d ata points {Xl  .. . XR}  ha\'p  bef'n  gelw r
. 
where  8  den otps  all  the  parameters of the  mixture:  the class  probabilities Vi  (wlwre  Vi  =  P(Cj  18)) , the class  centers  fl j  and  the  class covariances  ~j' 
Tlw  job  of a  mixture  m odel  learn er  is  to  find  a  good  estimat e  of  t.he  modeL  and  Expectation  MaximizRtion  (EM) ,  also  known  a::l  "Fuzzy  ~'-me a n::l",  i::l  a  popular 
544
Tight Bounds for the VC-Dimension of Piecewise Polynomial Networks
Authors: Akito Sakurai
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: O(ws(s log d+log(dqh/ s))) and O(ws((h/ s) log q) +log(dqh/ s)) are  upper bounds for  the VC-dimension of a  set of neural networks of  units  with  piecewise  polynomial  activation  functions,  where  s  is  the  depth  of  the  network,  h  is  the  number  of  hidden  units,  w  is  the  number  of  adjustable  parameters,  q  is  the  maximum  of  the  number of polynomial segments of the activation function, and d is  the  maximum degree  of  the polynomials;  also  n(wslog(dqh/s))  is  a  lower  bound  for  the VC-dimension  of such  a  network set,  which  are tight for  the cases  s =  8(h)  and  s is  constant.  For the special  case q =  1,  the VC-dimension is  8(ws log d).
The Effect of Eligibility Traces on Finding Optimal Memoryless Policies in Partially Observable Markov Decision Processes
Authors: John Loch
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Agents  acting  in  the  real  world  are  confronted  with  the  problem  of  making  good  decisions  with  limited  knowledge  of the  environment.  Partially  observable  Markov  decision  processes  (POMDPs)  model  decision problems in which an  agent tries to maximize its reward in  the  face  of limited sensor feedback.  Recent work has shown empirically that  a  reinforcement  learning  (RL)  algorithm  called  Sarsa(A)  can  efficiently  find  optimal  memoryless  policies,  which  map  current  observations  to  actions,  for  POMDP  problems  (Loch  and  Singh  1998).  The  Sarsa(A)  algorithm uses a form  of short-term memory  called an eligibility trace,  which  distributes  temporally  delayed  rewards  to  observation-action  pairs  which  lead  up  to  the  reward.  This  paper  explores  the  effect  of  eligibility traces on the ability of the Sarsa(A) algorithm to find  optimal  memoryless  policies.  A  variant  of  Sarsa(A)  called  k-step  truncated  Sarsa(A)  is  applied to four test  problems taken  from  the  recent  work of  Littman,  Littman,  Cassandra  and  Kaelbling,  Parr  and  Russell,  and  Chrisman.  The  empirical  results  show  that  eligibility  traces  can  be  significantly  truncated  without  affecting  the  ability  of Sarsa(A)  to  find  optimal memoryless policies for POMDPs.
Exploratory Data Analysis Using Radial Basis Function Latent Variable Models
Authors: Andrew R. Webb Alan D. Marrs
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Two  developments of nonlinear latent variable models  based  on  radial  basis functions are discussed:  in the first, the use of priors or constraints  on allowable models is considered as a means of preserving data structure  in  low-dimensional representations for  visualisation  purposes.  Also,  a  resampling approach  is  introduced which  makes  more  effective use  of  the latent samples in evaluating the likelihood.
A V1 Model of Pop Out and Asymmetty in Visual Search
Authors: Zhaoping Li
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Visual  search is  the task of finding  a  target in  an image against  a  background of distractors.  Unique features  of targets enable them  to pop out against the background, while targets defined by lacks of  features or conjunctions of features are more difficult to spot.  It is  known that the ease of target detection can change when the roles  of figure  and  ground  are  switched.  The  mechanisms  underlying  the  ease  of  pop  out  and  asymmetry  in  visual  search  have  been  elusive.  This paper shows that a model of segmentation in VI based  on  intracortical  interactions  can  explain  many  of the  qualitative  aspects of visual search.
A High Performance k-NN Classifier Using a Binary Correlation Matrix Memory
Authors: Jim Austin, John Kennedy Ping Zhou
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: This  paper  presents  a  novel  and  fast  k-NN  classifier  that  is  based  on  a  binary  CMM  (Correlation  Matrix  Memory)  neural  network.  A  robust  encoding  method  is  developed  to  meet  CMM  input  requirements .  A  hardware implementation of the CMM is  described, which gives  over 200  times  the  speed  of a  current  mid-range  workstation,  and  is  scaleable  to  very  large  problems.  When  tested  on  several  benchmarks  and  compared  with a simple k-NN method, the CMM classifier gave  less than  I %  lower  accuracy  and  over  4  and  12  times  speed-up  in  software  and  hardware  respectively.
Exploring Unknown Environments with Real-Time Search or Reinforcement Learning
Authors: Sven Koenig
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Learning Real-Time A*  (LRTA*) is a popular control method that interleaves plan(cid:173) ning  and  plan  execution  and  has  been  shown  to  solve  search  problems  in  known  environments efficiently. In this paper, we apply LRTA * to the problem of getting to  a given goal location in an initially unknown environment.  Uninformed LRTA * with  maximal  lookahead always moves  on a  shortest path to the closest unvisited  state,  that is, to the closest potential goal state.  This was believed to be a good exploration  heuristic, but we show that it does not minimize the worst-case plan-execution time  compared to other uninformed exploration methods.  This result is also of interest to  reinforcement-learning researchers  since many  reinforcement learning methods use  asynchronous dynamic  programming,  interleave  planning and  plan  execution,  and  exhibit optimism in the face of uncertainty, just like LRTA *.
Inference in Multilayer Networks via Large Deviation Bounds
Authors: Lawrence K. Saul Michael J. Kearns
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We  study  probabilistic  inference  in  large,  layered  Bayesian  net(cid:173) works  represented  as  directed  acyclic  graphs.  We  show  that  the  intractability of exact inference in such  networks does  not preclude  their effective  use.  We  give algorithms for  approximate probabilis(cid:173) tic inference  that exploit averaging phenomena occurring at nodes  with  large  numbers  of parents.  We  show  that  these  algorithms  compute  rigorous  lower  and  upper  bounds  on  marginal probabili(cid:173) ties  of interest,  prove  that these  bounds  become exact in  the limit  of large networks,  and provide rates  of convergence.
Basis Selection for Wavelet Regression
Authors: Atam P. Dhawan Kevin R. Wheeler
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: A  wavelet  basis  selection  procedure  is  presented  for  wavelet  re(cid:173) gression.  Both  the  basis  and  threshold  are  selected  using  cross(cid:173) validation.  The  method  includes  the  capability  of  incorporating  prior knowledge on the smoothness (or shape of the basis functions)  into  the  basis  selection  procedure.  The  results  of the  method  are  demonstrated  using  widely  published  sampled  functions.  The  re(cid:173) sults of the method are contrasted with other basis function  based  methods.
Where Does the Population Vector of Motor Cortical Cells Point during Reaching Movements?
Authors: Emmanuel Guigon, Yves Burnod Pierre Baraduc
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: Visually-guided  arm  reaching  movements  are  produced  by  distributed  neural networks within parietal and frontal regions of the cerebral cortex.  Experimental  data  indicate  that (I) single  neurons  in  these  regions  are  broadly tuned to parameters of movement; (2) appropriate commands are  elaborated by  populations of neurons; (3) the coordinated action of neu(cid:173) rons can be  visualized  using a neuronal population vector (NPV).  How(cid:173) ever,  the  NPV  provides only  a rough  estimate  of movement parameters  (direction, velocity) and may even fail  to reflect the parameters of move(cid:173) ment when arm posture is changed.  We designed a model of the cortical  motor command to  investigate the relation between the desired direction  of the  movement, the  actual direction of movement and  the direction of  the NPV in  motor cortex. The model is a two-layer self-organizing neural  network  which  combines  broadly-tuned  (muscular)  proprioceptive  and  (cartesian) visual information to calculate (angular) motor commands for  the  initial  part  of the  movement of a  two-link  arm.  The  network  was  trained  by  motor babbling  in  5  positions.  Simulations  showed  that  (1)  the  network produced appropriate movement direction over a  large part  of the  workspace;  (2) small  deviations of the  actual  trajectory  from  the  desired  trajectory existed  at the extremities of the  workspace;  (3)  these  deviations were accompanied by  large deviations of the NPV from both  trajectories. These results suggest the NPV does not give a faithful image  of cortical processing during arm reaching movements. 
• to  whom correspondence should be addressed 
84 
P.  Baraduc,  E. Guigon and Y.  Burnod
Inferring neural population dynamics from multiple partial recordings of the same neural circuit
Authors: Lars Buesing, Adam M Packer, Henry Dalgleish, Noah Pettit, Michael Hausser, Jakob H Macke Srini Turaga
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Simultaneous recordings of the activity of large neural populations are extremely valuable as they can be used to infer the dynamics and interactions of neurons in a local circuit, shedding light on the computations performed. It is now possible to measure the activity of hundreds of neurons  using 2-photon calcium imaging. However, many computations are thought to involve circuits consisting of thousands of neurons, such as cortical barrels in rodent somatosensory cortex. Here we contribute a statistical method for stitching" together sequentially imaged sets of neurons into one model  by phrasing the problem as fitting a latent dynamical system with missing observations. This method allows us to substantially expand the population-sizes for which population dynamics can be characterized---beyond the number of simultaneously imaged neurons. In particular, we demonstrate using recordings in mouse somatosensory cortex that this method makes it possible to predict noise correlations between non-simultaneously recorded neuron pairs."
Approximate Gaussian process inference for the drift function in stochastic differential equations
Authors: Philipp Batz, Manfred Opper Andreas Ruttor
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: We introduce a nonparametric approach for estimating drift functions in systems of stochastic differential equations from incomplete observations of the state vector. Using a Gaussian process prior over the drift as a function of the state vector, we develop an approximate EM algorithm to deal with the unobserved, latent dynamics between observations. The posterior over states is approximated by a piecewise linearized process and the MAP estimation of the drift is facilitated by a sparse Gaussian process regression.
Third-Order Edge Statistics: Contour Continuation, Curvature, and Cortical Connections
Authors: Steven W Zucker Matthew Lawlor
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Association field models have been used to explain human contour grouping performance and to explain the mean frequency of long-range horizontal connections across cortical columns in V1. However, association fields essentially depend on pairwise statistics of edges in natural scenes. We develop a spectral test of the sufficiency of pairwise statistics and show that there is significant higher-order structure.  An analysis using a probabilistic spectral embedding reveals curvature-dependent components to the association field, and reveals a challenge for biological learning algorithms.
Transportability from Multiple Environments with Limited Experiments
Authors: Sanghack Lee, Vasant Honavar, Judea Pearl Elias Bareinboim
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: This paper considers the problem of transferring experimental findings learned from multiple heterogeneous domains to a target environment, in which only limited experiments can be performed. We reduce questions of transportability from multiple domains and with limited scope to symbolic derivations in the do-calculus, thus extending the treatment of transportability from full experiments introduced in Pearl and Bareinboim (2011). We further provide different graphical and algorithmic conditions for computing the transport formula for this setting, that is, a way of fusing the observational and experimental information scattered throughout different domains to synthesize a consistent estimate of the desired effects.
On model selection consistency of penalized M-estimators: a geometric theory
Authors: Yuekai Sun, Jonathan E Taylor Jason Lee
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Penalized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Often, the penalties are \emph{geometrically decomposable}, \ie\ can be expressed as a sum of (convex) support functions. We generalize the notion of irrepresentable to geometrically decomposable penalties and develop a general framework for establishing consistency and model selection consistency of M-estimators with such penalties. We then use this framework to derive results for some special cases of interest in bioinformatics and statistical learning.
Robust Bloom Filters for Large MultiLabel Classification Tasks
Authors: Nicolas Usunier, Thierry Artières, Patrick Gallinari Moustapha M Cisse
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: This  paper presents an approach to multilabel classification (MLC) with a large number of labels. Our approach is a reduction to binary classification in which label sets are represented by low dimensional binary vectors. This representation follows the principle of Bloom filters, a space-efficient data structure originally designed for approximate membership testing. We show that a naive application of Bloom filters in MLC is not robust to individual binary classifiers' errors. We then present an approach that exploits a specific feature of real-world datasets when the number of labels is large: many labels (almost) never appear together.  Our approch is provably robust, has sublinear training and inference complexity with respect to the number of labels, and compares favorably to state-of-the-art algorithms on two large scale multilabel datasets.
On the Relationship Between Binary Classification, Bipartite Ranking, and Binary Class Probability Estimation
Authors: Shivani Agarwal Harikrishna Narasimhan
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: We investigate the relationship between three fundamental problems in machine learning: binary classification, bipartite ranking, and binary class probability estimation (CPE). It is known that a good binary CPE model can be used to obtain a good binary classification model (by thresholding at 0.5), and also to obtain a good bipartite ranking model (by using the CPE model directly as a ranking model); it is also known that a binary classification model does not necessarily yield a CPE model. However, not much is known about other directions. Formally, these relationships involve regret transfer bounds. In this paper, we introduce the notion of weak regret transfer bounds, where the mapping needed to transform a model from one problem to another depends on the underlying probability distribution (and in practice, must be estimated from data). We then show that, in this weaker sense, a good bipartite ranking model can be used to construct a good classification model (by thresholding at a suitable point), and more surprisingly, also to construct a good binary CPE model (by calibrating the scores of the ranking model).
Sequential Transfer in Multi-armed Bandit with Finite Set of Models
Authors: Alessandro Lazaric, Emma Brunskill Mohammad Gheshlaghi azar
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Learning from prior tasks and transferring that experience to improve future performance is critical for building lifelong learning agents. Although results in supervised and reinforcement learning show that transfer may significantly improve the learning performance, most of the literature on transfer is focused on batch learning tasks. In this paper we study the problem of sequential transfer in online learning, notably in the multi-arm bandit framework, where the objective is to minimize the cumulative regret over a sequence of tasks by incrementally transferring knowledge from prior tasks.  We introduce a novel bandit algorithm based on a method-of-moments approach for the estimation of the possible tasks and derive regret bounds for it.
A Graphical Transformation for Belief Propagation: Maximum Weight Matchings and Odd-Sized Cycles
Authors: Andrew E Gelfand, Misha Chertkov Jinwoo Shin
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: Max-product ‘belief propagation’ (BP) is a popular distributed heuristic for finding the Maximum A Posteriori (MAP) assignment in a joint probability distribution represented by a Graphical Model (GM). It was recently shown that BP converges to the correct MAP assignment for a class of loopy GMs with the following common feature: the Linear Programming (LP) relaxation to the MAP problem is tight (has no integrality gap). Unfortunately, tightness of the LP relaxation does not, in general, guarantee convergence and correctness of the BP algorithm. The failure of BP in such cases motivates reverse engineering a solution – namely, given a tight LP, can we design a ‘good’ BP algorithm.  In this paper, we design a BP algorithm for the Maximum Weight Matching (MWM) problem over general graphs. We prove that the algorithm converges to the correct optimum if the respective LP relaxation, which may include inequalities associated with non-intersecting odd-sized cycles, is tight. The most significant part of our approach is the introduction of a novel graph transformation designed to force convergence of BP. Our theoretical result suggests an efficient BP-based heuristic for the MWM problem, which consists of making sequential, “cutting plane”, modifications to the underlying GM. Our experiments show that this heuristic performs as well as traditional cutting-plane algorithms using LP solvers on MWM problems.
A Kernel Test for Three-Variable Interactions
Authors: Arthur Gretton, Wicher Bergsma Dino Sejdinovic
Published in: Advances in Neural Information Processing Systems 26  (NIPS 2013) (Main Conference Track), 2013

Abstract: We introduce kernel nonparametric tests for Lancaster three-variable interaction and for total independence, using embeddings of signed measures into a reproducing kernel Hilbert space. The resulting test statistics are straightforward to compute, and are used in powerful three-variable interaction tests, which are consistent against all alternatives for a large family of reproducing kernels. We show the Lancaster test to be sensitive to cases where two independent causes individually have weak influence on a third dependent variable, but their combined effect has a strong influence. This makes the Lancaster test especially suited to finding structure in directed graphical models, where it outperforms competing nonparametric tests in detecting such V-structures.