mtybilly/Instruct2DS huggingface dataset:
A SNoW-Based Face Detector
Authors: Ming-Hsuan Yang, Dan Roth, Narendra Ahuja
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A novel learning approach for human face detection using a network  of linear units is  presented.  The SNoW  learning architecture is  a  sparse network  of linear functions  over a  pre-defined or incremen(cid:173) tally learned feature  space  and  is  specifically tailored for  learning  in the presence of a  very large number of features.  A wide range of  face  images in different poses,  with different expressions and under  different  lighting conditions  are  used  as  a  training  set  to  capture  the  variations of human faces.  Experimental results on commonly  used benchmark data sets of a wide range of face images show that  the  SNoW-based  approach  outperforms  methods  that  use  neural  networks,  Bayesian  methods,  support  vector  machines  and  oth(cid:173) ers.  Furthermore, learning and  evaluation using  the  SNoW-based  method are significantly more efficient  than with other methods.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
Authors: Richard S. Sutton, David A. McAllester, Satinder P. Singh, Yishay Mansour
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Function  approximation  is  essential  to  reinforcement  learning,  but  the standard approach of approximating a  value function and deter(cid:173) mining  a  policy  from  it  has so  far  proven theoretically  intractable.  In this paper we explore an alternative approach in which the policy  is explicitly represented by its own function approximator,  indepen(cid:173) dent of the value function,  and is  updated according to the gradient  of expected reward with respect to the policy parameters.  Williams's  REINFORCE method and actor-critic methods are examples of this  approach.  Our  main  new  result  is  to  show  that  the  gradient  can  be  written  in  a  form  suitable  for  estimation  from  experience  aided  by  an  approximate  action-value  or  advantage  function.  Using  this  result,  we  prove for  the first  time that a  version  of policy  iteration  with arbitrary differentiable function approximation is convergent to  a  locally optimal policy. 
Large applications of reinforcement learning (RL) require the use of generalizing func(cid:173) tion approximators such neural networks,  decision-trees,  or instance-based methods.  The dominant approach for the last decade has been the value-function approach,  in  which  all  function  approximation  effort  goes  into  estimating  a  value  function,  with  the action-selection policy represented implicitly as the  "greedy"  policy with respect  to the estimated values  (e.g.,  as the policy that selects in each state the action with  highest estimated value).  The value-function approach has worked well in many appli(cid:173) cations,  but has several limitations.  First, it is oriented toward finding  deterministic  policies, whereas the optimal policy is often stochastic, selecting different actions with  specific probabilities (e.g.,  see Singh,  Jaakkola,  and Jordan,  1994).  Second,  an arbi(cid:173) trarily small change in the estimated value of an action can cause it to be, or not be,  selected.  Such discontinuous changes have been identified as a  key obstacle to estab(cid:173) lishing  convergence  assurances for  algorithms  following  the value-function approach  (Bertsekas  and Tsitsiklis,  1996).  For example,  Q-Iearning,  Sarsa,  and dynamic pro(cid:173) gramming methods have all  been shown unable to converge to any policy for  simple  MDPs  and simple  function  approximators  (Gordon,  1995,  1996;  Baird,  1995;  Tsit(cid:173) siklis and van Roy,  1996;  Bertsekas  and Tsitsiklis,  1996).  This can occur even if the  best approximation is found at each step before changing the policy,  and whether the  notion of "best"  is in the mean-squared-error sense or the slightly different  senses of  residual-gradient,  temporal-difference,  and dynamic-programming methods. 
In this  paper we  explore an  alternative  approach to  function  approximation  in  RL.
Large Margin DAGs for Multiclass Classification
Authors: John C. Platt, Nello Cristianini, John Shawe-Taylor
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present a  new  learning  architecture:  the  Decision  Directed  Acyclic  Graph  (DDAG),  which  is  used  to  combine  many  two-class  classifiers  into  a  multiclass  classifier.  For  an  N -class  problem,  the  DDAG  con(cid:173) tains N(N - 1)/2 classifiers, one for each pair of classes.  We present a  VC analysis of the case when the node classifiers are hyperplanes; the re(cid:173) sulting bound on the test error depends on N  and on the margin achieved  at  the  nodes,  but not  on  the  dimension  of the  space.  This motivates an  algorithm, DAGSVM, which operates in  a kernel-induced feature  space  and  uses  two-class maximal  margin  hyperplanes at each  decision-node  of the  DDAG.  The DAGSVM  is  substantially faster  to  train  and  evalu(cid:173) ate  than  either the  standard  algorithm  or Max Wins,  while maintaining  comparable accuracy to both of these algorithms.
Approximate Planning in Large POMDPs via Reusable Trajectories
Authors: Michael J. Kearns, Yishay Mansour, Andrew Y. Ng
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We consider the problem of reliably choosing a near-best strategy from  a restricted class  of strategies TI  in  a partially observable Markov deci(cid:173) sion process (POMDP). We assume we are given the ability to simulate  the POMDP,  and study  what might be called the sample complexity - that  is,  the  amount of data one must  generate in  the  POMDP  in  order  to choose a good strategy.  We  prove upper bounds on  the sample com(cid:173) plexity  showing  that,  even  for  infinitely  large  and arbitrarily  complex  POMDPs,  the  amount  of data needed  can  be finite,  and  depends  only  linearly  on  the complexity of the  restricted strategy class  TI,  and  expo(cid:173) nentially  on  the horizon time.  This latter dependence can be eased in  a  variety  of ways,  including the  application  of gradient and  local  search  algorithms.  Our measure of complexity generalizes the classical super(cid:173) vised  learning notion  of VC  dimension to the settings  of reinforcement  learning and planning.
Maximum Entropy Discrimination
Authors: Tommi Jaakkola, Marina Meila, Tony Jebara
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We present a general framework for discriminative estimation based  on the maximum entropy principle and its extensions.  All  calcula(cid:173) tions involve distributions over structures and/or parameters rather  than  specific  settings  and  reduce  to  relative  entropy  projections.  This holds  even  when  the data is  not  separable within  the chosen  parametric class,  in  the context of anomaly  detection  rather than  classification, or when the labels in the training set are uncertain or  incomplete.  Support  vector  machines  are  naturally subsumed  un(cid:173) der  this  class  and we  provide several extensions.  We  are also  able  to estimate exactly and efficiently discriminative distributions over  tree structures of class-conditional  models  within  this framework.  Preliminary experimental results  are  indicative of the potential in  these techniques.
The Relaxed Online Maximum Margin Algorithm
Authors: Yi Li, Philip M. Long
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  describe  a  new  incremental  algorithm  for  training  linear  thresh(cid:173) old  functions:  the  Relaxed  Online  Maximum  Margin  Algorithm,  or  ROMMA. ROMMA can be viewed as  an approximation to the algorithm  that repeatedly chooses the hyperplane that classifies previously seen ex(cid:173) amples  correctly  with  the  maximum  margin.  It  is  known  that  such  a  maximum-margin hypothesis can be computed by minimizing the length  of the weight vector subject to a number of linear constraints.  ROMMA  works  by  maintaining a relatively simple relaxation of these constraints  that can be efficiently updated.  We prove a mistake bound for ROMMA  that is the same as that proved for the perceptron algorithm.  Our analysis  implies that the more computationally intensive maximum-margin algo(cid:173) rithm also satisfies this mistake bound; this is the first worst-case perfor(cid:173) mance guarantee for this algorithm.  We describe some experiments us(cid:173) ing ROMMA and  a variant that updates its hypothesis more aggressively  as  batch  algorithms to recognize handwritten digits.  The computational  complexity  and  simplicity of these algorithms is  similar to  that of per(cid:173) ceptron algorithm , but their generalization is much better.  We describe a  sense  in  which the performance of ROMMA converges  to that of SVM  in the limit if bias isn't considered.
Bayesian Modelling of fMRI lime Series
Authors: Pedro A. d. F. R. Højen-Sørensen, Lars Kai Hansen, Carl Edward Rasmussen
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We present a Hidden Markov Model (HMM) for inferring the hidden  psychological state (or neural activity) during single trial tMRI activa(cid:173) tion experiments with blocked task paradigms. Inference is based on  Bayesian methodology, using a combination of analytical and a variety  of Markov Chain Monte Carlo (MCMC) sampling techniques. The ad(cid:173) vantage of this method is that detection of short time learning effects be(cid:173) tween repeated trials is possible since inference is based only on single  trial experiments.
Policy Search via Density Estimation
Authors: Andrew Y. Ng, Ronald Parr, Daphne Koller
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  propose  a  new  approach  to  the  problem  of searching  a  space  of  stochastic controllers for a Markov decision process (MDP) or a partially  observable Markov decision process (POMDP). Following several other  authors,  our approach  is  based  on  searching  in  parameterized  families  of policies (for example, via gradient descent) to optimize solution qual(cid:173) ity.  However,  rather than  trying  to  estimate  the  values  and  derivatives  of a policy  directly,  we do  so  indirectly  using  estimates  for  the  proba(cid:173) bility  densities  that  the  policy  induces  on  states  at  the  different points  in  time.  This enables our algorithms to  exploit the many techniques for  efficient  and  robust approximate density  propagation  in  stochastic  sys(cid:173) tems.  We show how our techniques can  be applied both to deterministic  propagation schemes (where the MDP's dynamics are given explicitly in  compact form,)  and  to  stochastic  propagation schemes (where we  have  access only to a generative model, or simulator, of the MDP). We present  empirical results for both of these variants on complex problems.
Learning to Parse Images
Authors: Geoffrey E. Hinton, Zoubin Ghahramani, Yee Whye Teh
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  describe a class of probabilistic models  that we  call credibility  networks.  Using  parse trees as  internal representations of images,  credibility  networks  are able  to  perform  segmentation  and  recog(cid:173) nition simultaneously,  removing the need for  ad hoc  segmentation  heuristics.  Promising results  in  the  problem of segmenting  hand(cid:173) written digits  were obtained. 
1
Robust Recognition of Noisy and Superimposed Patterns via Selective Attention
Authors: Soo-Young Lee, Michael Mozer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In many classification tasks, recognition accuracy is low because input  patterns  are  corrupted  by  noise  or  are  spatially  or  temporally  overlapping. We propose an approach to overcoming these  limitations  based  on  a  model  of human  selective  attention.  The  model,  an  early  selection filter guided by top-down attentional control, entertains each  candidate  output  class  in  sequence  and  adjusts  attentional  gain  coefficients  in  order to produce a strong response  for that class.  The  chosen class is then the one that  obtains the strongest response with the  least  modulation  of  attention.  We  present  simulation  results  on  classification of corrupted and superimposed handwritten digit patterns,  showing a significant improvement in recognition rates.  The algorithm  has  also  been  applied  in  the  domain  of  speech  recognition,  with  comparable results.
Bayesian Network Induction via Local Neighborhoods
Authors: Dimitris Margaritis, Sebastian Thrun
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In  recent  years,  Bayesian  networks  have  become  highly  successful  tool  for  di(cid:173) agnosis,  analysis,  and  decision  making  in  real-world domains.  We  present  an  efficient algorithm for  learning Bayes  networks from  data.  Our approach  con(cid:173) structs Bayesian networks by first identifying each node's Markov blankets, then  connecting nodes  in  a maximally consistent way.  In  contrast to the majority of  work, which typically uses hill-climbing approaches that may produce dense and  causally incorrect nets, our approach yields much more compact causal networks  by heeding independencies in the data.  Compact causal networks facilitate fast in(cid:173) ference and are also easier to understand. We prove that under mild assumptions,  our approach requires time polynomial in the size of the data and the number of  nodes.  A randomized variant,  also presented here,  yields comparable results at  much higher speeds.
Spiking Boltzmann Machines
Authors: Geoffrey E. Hinton, Andrew D. Brown
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We first show how to represent sharp posterior probability distribu(cid:173) tions using real valued coefficients on broadly-tuned basis functions.  Then we  show how  the precise times of spikes can be used to con(cid:173) vey the real-valued  coefficients on  the basis functions  quickly and  accurately.  Finally we  describe a  simple  simulation in which spik(cid:173) ing neurons learn to model an image sequence by fitting a dynamic  generative model. 
1  Population codes and  energy landscapes 
A perceived object is represented in the brain by the activities of many neurons, but  there is no general consensus on how the activities of individual neurons combine to  represent the multiple properties of an object.  We start by focussing on the case of  a single object that has multiple instantiation parameters such as position, velocity,  size and orientation.  We assume that each neuron has an ideal stimulus in the space  of instantiation parameters and that its activation rate or probability of activation  falls off monotonically in all directions as the actual stimulus departs from this ideal.  The semantic problem is  to define  exactly what instantiation parameters are being  represented when  the activities of many such neurons are specified. 
Hinton,  Rumelhart and  McClelland  (1986)  consider  binary neurons with  receptive  fields  that  are  convex  in  instantiation  space.  They  assume  that  when  an  object  is  present  it  activates  all  of the neurons  in whose  receptive  fields  its  instantiation  parameters  lie.  Consequently,  if it  is  known  that  only  one  object  is  present,  the  parameter  values  of the  object  must  lie  within  the  feasible  region  formed  by  the  intersection of the receptive fields  of the active neurons.  This will  be called a  con(cid:173) junctive  distributed  representation.  Assuming  that  each  receptive  field  occupies  only  a  small  fraction  of the  whole  space,  an  interesting  property  of this  type  of  "coarse coding"  is that the bigger the receptive fields,  the more accurate the repre(cid:173) sentation.  However,  large receptive fields  lead to a  loss  of resolution when  several  objects are present simultaneously. 
When the sensory input  is  noisy,  it  is  impossible  to infer  the exact  parameters of  objects so it  makes sense  for  a  perceptual system to represent  the probability dis(cid:173) tribution  across  parameters  rather  than  just  a  single  best  estimate  or  a  feasible  region.  The full  probability distribution is  essential for  correctly  combining  infor-
Spiking Boltzmann Machines
Actor-Critic Algorithms
Authors: Vijay R. Konda, John N. Tsitsiklis
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  propose  and  analyze  a  class  of  actor-critic  algorithms  for  simulation-based  optimization  of  a  Markov  decision  process  over  a  parameterized  family  of randomized  stationary  policies.  These  are two-time-scale  algorithms in  which  the critic uses TD learning  with  a  linear approximation architecture and the actor is  updated  in  an  approximate  gradient  direction  based  on  information  pro(cid:173) vided by the critic.  We  show that the features for  the critic should  span a subspace prescribed by the choice of parameterization of the  actor.  We  conclude by discussing convergence properties and some  open problems.
Training Data Selection for Optimal Generalization in Trigonometric Polynomial Networks
Authors: Masashi Sugiyama, Hidemitsu Ogawa
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this  paper,  we  consider the problem of active learning  in trigonomet(cid:173) ric  polynomial  networks  and  give  a  necessary  and sufficient  condition of  sample  points to  provide the  optimal generalization capability.  By ana(cid:173) lyzing the condition from  the functional analytic point of view,  we  clarify  the  mechanism  of  achieving  the  optimal  generalization  capability.  We  also  show  that  a  set  of  training  examples  satisfying  the  condition  does  not only provide the optimal generalization  but also  reduces the compu(cid:173) tational  complexity and memory required  for  the  calculation of learning  results.  Finally, examples  of sample  points  satisfying  the  condition  are  given and computer simulations are performed to demonstrate the effec(cid:173) tiveness of the proposed  active learning method.
Image Recognition in Context: Application to Microscopic Urinalysis
Authors: Xubo B. Song, Joseph Sill, Yaser S. Abu-Mostafa, Harvey Kasdan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  propose a  new  and efficient technique for  incorporating contextual  information into object classification. Most of the current techniques face  the problem of exponential computation cost.  In this paper, we propose a  new general framework that incorporates partial context at a linear cost.  This technique  is  applied  to  microscopic  urinalysis  image  recognition,  resulting in a significant improvement of recognition rate over the context  free approach. This gain would have been impossible using conventional  context incorporation techniques. 
1  BACKGROUND: RECOGNITION IN CONTEXT 
There are a number of pattern recognition problem domains where the classification of an  object should be based on more than simply the appearance of the object itself.  In remote  sensing image classification, where each pixel is part of ground cover, a pixel is more like(cid:173) ly  to  be a glacier if it is  in  a mountainous area, than  if surrounded by  pixels of residential  areas.  In text analysis, one can expect to find certain letters occurring regularly in  particu(cid:173) lar arrangement with  other letters(qu, ee,est, tion,  etc.).  The information conveyed by  the  accompanying entities is referred to as contextual information.  Human experts apply con(cid:173) textual information in their decision making [2][ 6].  It makes sense to design techniques and  algorithms to make computers aggregate and utilize a more complete set of information in  their decision making the way human experts do.  In  pattern recognition systems, however, 
*Author for correspondence 
964 
X  B.  Song, J  Sill,  Y.  Abu-Mostafa and H.  Kasdan 
the primary (and often only) source of information used to identify an object is the set of  measurements, or features, associated with the object itself.  Augmenting this information  by incorporating context into the classification process can yield significant benefits. 
i  =  1, ... N.  With  each  object  we  associate  a  Consider  a  set  of  N  objects  Ti ,  class  label  Ci  that  is  a  member  of  a  label  set  n  =  {1 , ... , D} .  Each  object  Ti  is  characterized  by  a  set  of  measurements  Xi  E  R P,  which  we  call  a  feature  vec(cid:173) tor.  Many  techniques  [1][2][4J[6}  incorporate  context  by  conditioning  the  posterior  probability  of objects'  identities  on  the joint features  of all  accompanying objects.  i.e .•  P(Cl, C2,··· , cNlxl , . . . , XN). and then maximizing it with respectto Cl, C2, . .. , CN . It can  be  shown  thatp(cl,c2, . . . ,cNlxl, . . . ,xN)  ex  p(cllxl) ... p(CNlxN)  (~ci""'(N\ given  certain reasonable assumptions.
Statistical Dynamics of Batch Learning
Authors: Song Li, K. Y. Michael Wong
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: An important issue in neural computing concerns the description of  learning dynamics  with  macroscopic  dynamical  variables.  Recen(cid:173) t  progress on  on-line learning only  addresses the often unrealistic  case of an infinite  training set.  We  introduce a  new  framework  to  model batch learning of restricted sets of examples, widely applica(cid:173) ble to  any learning cost function,  and fully  taking into account the  temporal correlations introduced by the recycling of the examples.  For  illustration  we  analyze  the  effects  of weight  decay  and  early  stopping during the learning of teacher-generated examples.
Scale Mixtures of Gaussians and the Statistics of Natural Images
Authors: Martin J. Wainwright, Eero P. Simoncelli
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The statistics of photographic images, when represented using  multiscale (wavelet) bases, exhibit two striking types of non(cid:173) Gaussian behavior. First, the marginal densities of the coefficients  have extended heavy tails. Second, the joint densities exhibit vari(cid:173) ance dependencies not captured by second-order models. We ex(cid:173) amine properties of the class of Gaussian scale mixtures, and show  that these densities can accurately characterize both the marginal  and joint distributions of natural image wavelet coefficients. This  class of model suggests a Markov structure, in which wavelet coeffi(cid:173) cients are linked by hidden scaling variables corresponding to local  image structure. We derive an estimator for these hidden variables,  and show that a nonlinear "normalization" procedure can be used  to Gaussianize the coefficients. 
Recent years have witnessed a surge of interest in modeling the statistics of natural  images. Such models are important for applications in image processing and com(cid:173) puter vision, where many techniques rely (either implicitly or explicitly) on a prior  density. A number of empirical studies have demonstrated that the power spectra  of natural images follow a 1/ f'Y law in radial frequency, where the exponent "f is  typically close to two [e.g., 1]. Such second-order characterization is inadequate,  however, because images usually exhibit highly non-Gaussian behavior. For in(cid:173) stance, the marginals of wavelet coefficients typically have much heavier tails than  a Gaussian [2]. Furthermore, despite being approximately decorrelated (as sug(cid:173) gested by theoretical analysis of 1/ f processes [3]), orthonormal wavelet coefficients  exhibit striking forms of statistical dependency [4, 5]. In particular, the standard  deviation of a wavelet coefficient typically scales with the absolute values of its  neighbors [5]. 
A number of researchers have modeled the marginal distributions of wavelet coef(cid:173) ficients with generalized Laplacians, py(y) ex exp( -Iy/ AlP) [e.g. 6, 7, 8]. Special  cases include the Gaussian (p = 2) and the Laplacian (p = 1), but appropriate ex- Research supported by NSERC 1969 fellowship 160833 to MJW, and NSF CAREER grant  MIP-9796040 to EPS. 
856 
M J Wainwright and E. P. Simoncelli
Potential Boosters?
Authors: Nigel Duffy, David P. Helmbold
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Recent  interpretations  of the  Adaboost  algorithm  view  it  as  per(cid:173) forming a gradient descent on a  potential function.  Simply chang(cid:173) ing the potential function  allows  one to create new  algorithms re(cid:173) lated  to  AdaBoost.  However,  these  new  algorithms are generally  not  known  to have  the formal  boosting property.  This  paper ex(cid:173) amines  the question  of which  potential  functions  lead  to  new  al(cid:173) gorithms that are boosters.  The two main results are general sets  of  conditions  on  the  potential;  one  set  implies  that  the  resulting  algorithm is  a  booster,  while the other implies  that the algorithm  is  not.  These conditions are applied to previously studied potential  functions , such as those used by LogitBoost and  Doom II.
Resonance in a Stochastic Neuron Model with Delayed Interaction
Authors: Toru Ohira, Yuzuru Sato, Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We study here a simple stochastic single neuron model with delayed  self-feedback capable of generating spike  trains.  Simulations show  that its spike trains exhibit resonant behavior between "noise"  and  "delay".  In  order to gain  insight  into this  resonance,  we  simplify  the model and study a  stochastic binary element whose  transition  probability  depends  on  its  state  at  a  fixed  interval  in  the  past.  With this simplified model we  can analytically compute interspike  interval histograms, and show how the resonance between noise and  delay  arises.  The resonance  is  also  observed  when  such  elements  are coupled through delayed interaction. 
1
Wiring Optimization in the Brain
Authors: Dmitri B. Chklovskii, Charles F. Stevens
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: The complexity of cortical circuits may be characterized by the number  of synapses per neuron.  We study the dependence of complexity on the  fraction of the cortical volume that is made up of "wire" (that is, ofaxons  and dendrites),  and find  that complexity is  maximized when wire takes  up about 60% of the cortical volume.  This prediction is  in  good agree(cid:173) ment with experimental observations.  A consequence of our arguments  is that any rearrangement of neurons that takes more wire would sacrifice  computational power. 
Wiring a brain presents formidable problems because of the extremely large number of con(cid:173) nections:  a microliter of cortex contains approximately 105  neurons, 109  synapses, and 4  km ofaxons, with 60% of the cortical  volume being taken up with "wire", half of this  by  axons and the other half by dendrites. [ 1] Each cortical neighborhood must have exactly the  right balance of components; if too many cell bodies were present in a particular mm cube,  for example, insufficient space would remain for the axons, dendrites and synapses.  Here  we ask "What fraction of the cortical volume should be wires (axons + dendrites)?" We ar(cid:173) gue that physiological properties ofaxons and dendrites dictate an optimal wire fraction of  0.6, just what is actually observed. 
To calculate the optimal wire fraction, we start with a real cortical region containing a fixed  number of neurons, a mm cube, for example, and imagine perturbing it by adding or sub(cid:173) tracting synapses and the axons and dendrites needed to support them.  The rules for per(cid:173) turbing the cortical cube require that the existing circuit connections and function remain  intact (except for what may have been removed in  the perturbation), that no holes are cre(cid:173) ated, and that all added (or subtracted) synapses are typical of those present; as wire volume  is added, the volume of the cube of course increases.  The ratio of the number of synapses  per neuron in  the perturbed cortex to  that in  the real  cortex is  denoted by 8,  a  parameter  we call the relative complexity.  We  require that the volume of non-wire components (cell  bodies, blood vessels,  glia, etc) is  unchanged by  our perturbation and use 4>  to denote the  volume fraction of the perturbed cortical region that is made up of wires (axons + dendrites;  4> can vary between zero and one), with the fraction for the real brain being 4>0.  The relation  between relative complexity 8 and wire volume fraction 4>  is given by the equation (derived  in Methods)
Learning from User Feedback in Image Retrieval Systems
Authors: Nuno Vasconcelos, Andrew Lippman
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  formulate  the  problem  of retrieving  images  from  visual  databases  as  a problem of Bayesian inference.  This leads to  natural and effective  solutions for two of the most challenging issues in the design of a retrieval  system:  providing  support  for  region-based  queries  without  requiring  prior  image  segmentation,  and  accounting  for  user-feedback  during  a  retrieval  session.  We  present  a  new  learning  algorithm  that  relies  on  belief propagation to account for both positive and negative examples of  the user's interests.
Online Independent Component Analysis with Local Learning Rate Adaptation
Authors: Nicol N. Schraudolph, Xavier Giannakopoulos
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Stochastic meta-descent (SMD) is a new technique for online adap(cid:173) tation  of local  learning  rates  in  arbitrary twice-differentiable  sys(cid:173) tems.  Like matrix momentum it uses full  second-order information  while  retaining  O(n)  computational  complexity  by  exploiting  the  efficient  computation  of Hessian-vector  products.  Here  we  apply  SMD  to independent  component  analysis,  and  employ  the  result(cid:173) ing  algorithm  for  the  blind  separation  of time-varying  mixtures.  By matching individual learning rates to the rate of change in each  source signal's  mixture coefficients,  our technique is  capable of si(cid:173) multaneously tracking sources that move at very different,  a priori  unknown speeds.
Model Selection in Clustering by Uniform Convergence Bounds
Authors: Joachim M. Buhmann, Marcus Held
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Unsupervised  learning  algorithms  are  designed  to  extract  struc(cid:173) ture from  data samples.  Reliable  and robust  inference  requires  a  guarantee that extracted structures are typical for the data source,  Le.,  similar  structures  have  to be  inferred  from  a  second  sample  set of the same data source.  The overfitting phenomenon in max(cid:173) imum  entropy  based  annealing  algorithms  is  exemplarily  studied  for  a  class of histogram clustering  models.  Bernstein's  inequality  for  large deviations is used to determine the maximally achievable  approximation  quality  parameterized  by  a  minimal  temperature.  Monte Carlo simulations support the proposed model selection cri(cid:173) terion by finite  temperature annealing.
Unmixing Hyperspectral Data
Authors: Lucas C. Parra, Clay Spence, Paul Sajda, Andreas Ziehe, Klaus-Robert Müller
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In hyperspectral imagery one pixel  typically  consists of a  mixture  of the  reflectance  spectra of several  materials,  where  the  mixture  coefficients  correspond  to  the  abundances of the  constituting  ma(cid:173) terials.  We  assume linear combinations of reflectance spectra with  some additive normal sensor noise and derive a  probabilistic MAP  framework  for  analyzing  hyperspectral  data.  As  the  material re(cid:173) flectance characteristics are not know  a priori, we face  the problem  of  unsupervised  linear  unmixing.  The  incorporation  of  different  prior  information  (e.g.  positivity  and  normalization  of the  abun(cid:173) dances)  naturally  leads  to  a  family  of  interesting  algorithms,  for  example  in  the  noise-free  case  yielding  an  algorithm  that  can  be  understood as constrained independent component analysis (ICA).  Simulations underline the usefulness of our theory.
Inference for the Generalization Error
Authors: Claude Nadeau, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In order to to compare learning algorithms, experimental results reported  in  the  machine  learning  litterature often  use  statistical  tests  of signifi(cid:173) cance.  Unfortunately,  most of these  tests  do  not take  into account the  variability  due  to  the  choice  of training  set.  We  perform  a  theoretical  investigation of the variance of the cross-validation estimate of the gen(cid:173) eralization error that takes into account the  variability due to  the choice  of training  sets.  This  allows  us  to  propose  two  new  ways  to  estimate  this variance. We show, via simulations, that these new statistics perform  well relative to the statistics considered by Dietterich (Dietterich, 1998).
LTD Facilitates Learning in a Noisy Environment
Authors: Paul W. Munro, Gerardina Hernández
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Long-term potentiation (LTP) has long been held as a biological  substrate for associative learning. Recently, evidence has emerged  that long-term depression (LTD) results when the presynaptic cell  fires after the postsynaptic cell. The computational utility of LTD  is explored here. Synaptic modification kernels for both LTP and  LTD have been proposed by other laboratories based studies of one  postsynaptic unit. Here, the interaction between time-dependent  LTP and LTD is studied in small networks.
An Oculo-Motor System with Multi-Chip Neuromorphic Analog VLSI Control
Authors: Oliver Landolt, Steve Gyger
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A system emulating the functionality of a moving eye-hence the name  oculo-motor system-has been built and successfully tested.  It is  made  of an optical device for shifting the field of view of an image sensor by up  to 45 ° in  any direction, four neuromorphic analog VLSI circuits imple(cid:173) menting an oculo-motor control loop, and some off-the-shelf electronics.  The custom integrated circuits communicate with each other primarily by  non-arbitrated address-event buses.  The system  implements the behav(cid:173) iors of saliency-based saccadic exploration, and smooth pursuit of light  spots.  The duration of saccades ranges from  45 ms  to  100 ms,  which  is  comparable to human eye performance. Smooth pursuit operates on light  sources moving at up to 50 0 /s in the visual field.
Understanding Stepwise Generalization of Support Vector Machines: a Toy Model
Authors: Sebastian Risau-Gusman, Mirta B. Gordon
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: In this  article  we  study the effects  of introducing structure in the  input distribution of the data to be learnt by a  simple perceptron.  We  determine the learning curves within the framework of Statis(cid:173) tical  Mechanics.  Stepwise  generalization  occurs  as  a  function  of  the number of examples when the distribution of patterns is highly  anisotropic.  Although  extremely simple,  the model  seems  to cap(cid:173) ture  the  relevant  features  of  a  class  of  Support  Vector  Machines  which was  recently shown to present this behavior.
A Geometric Interpretation of v-SVM Classifiers
Authors: David J. Crisp, Christopher J. C. Burges
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We show that the recently proposed variant of the Support Vector  machine  (SVM)  algorithm,  known  as  v-SVM,  can  be  interpreted  as a maximal separation between subsets of the convex hulls of the  data,  which  we  call  soft  convex  hulls.  The  soft  convex  hulls  are  controlled by  choice of the parameter v.  If the intersection of the  convex hulls is empty, the hyperplane is positioned halfway between  them such that the distance between convex hulls, measured along  the normal, is  maximized; and if it is not, the hyperplane's normal  is  similarly determined  by  the  soft  convex  hulls,  but  its  position  (perpendicular  distance  from  the  origin)  is  adjusted  to  minimize  the  error  sum.  The  proposed  geometric  interpretation of v-SVM  also leads to necessary and sufficient conditions for the existence of  a choice of v  for  which  the v-SVM solution is nontrivial.
Robust Learning of Chaotic Attractors
Authors: Rembrandt Bakker, Jaap C. Schouten, Marc-Olivier Coppens, Floris Takens, C. Lee Giles, Cor M. van den Bleek
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A fundamental problem with the modeling of chaotic time series data is that  minimizing short-term prediction errors does not guarantee a match  between the reconstructed attractors of model and experiments. We  introduce a modeling paradigm that simultaneously learns to short-tenn  predict and to locate the outlines of the attractor by a new way of nonlinear  principal component analysis. Closed-loop predictions are constrained to  stay within these outlines, to prevent divergence from the attractor. Learning  is exceptionally fast: parameter estimation for the 1000 sample laser data  from the 1991 Santa Fe time series competition took less than a minute on  a 166 MHz Pentium PC.
Recurrent Cortical Competition: Strengthen or Weaken?
Authors: Péter Adorján, Lars Schwabe, Christian Piepenbrock, Klaus Obermayer
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We investigate the short term .dynamics of the recurrent competition and  neural  activity in the primary visual cortex in terms of information pro(cid:173) cessing and in the context of orientation selectivity.  We propose that af(cid:173) ter stimulus onset, the strength of the recurrent excitation decreases due  to  fast  synaptic depression.  As  a consequence,  the network shifts from  an  initially highly  nonlinear to a  more  linear operating  regime.  Sharp  orientation tuning is established in the first highly competitive phase.  In  the second and less competitive phase, precise signaling of multiple ori(cid:173) entations and long range modulation, e.g.,  by intra- and inter-areal con(cid:173) nections becomes possible (surround effects).  Thus the network first ex(cid:173) tracts  the  salient  features  from  the  stimulus,  and  then  starts  to  process  the details.  We  show  that  this  signal  processing  strategy  is  optimal  if  the neurons have limited bandwidth and their objective is to transmit the  maximum amount of information in any time interval beginning with the  stimulus onset.
Approximate Inference A lgorithms for Two-Layer Bayesian Networks
Authors: Andrew Y. Ng, Michael I. Jordan
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We  present  a  class  of approximate  inference  algorithms  for  graphical  models  of the  QMR-DT type.  We  give  convergence rates  for  these  al(cid:173) gorithms and  for  the  Jaakkola and  Jordan  (1999) algorithm,  and  verify  these  theoretical predictions empirically.  We  also present empirical re(cid:173) sults on the difficult QMR-DT network problem, obtaining performance  of the  new  algorithms roughly  comparable  to  the  Jaakkola and  Jordan  algorithm.
Perceptual Organization Based on Temporal Dynamics
Authors: Xiuwen Liu, DeLiang L. Wang
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: A figure-ground  segregation  network is  proposed  based on a  novel  boundary  pair  representation.  Nodes  in  the  network  are  bound(cid:173) ary  segments  obtained  through  local  grouping.  Each  node  is  ex(cid:173) citatorily  coupled  with  the  neighboring  nodes  that  belong  to  the  same region, and inhibitorily coupled with the corresponding paired  node.  Gestalt grouping rules are incorporated by  modulating con(cid:173) nections.  The  status  of  a  node  represents  its  probability  being  figural  and  is  updated  according  to  a  differential  equation.  The  system solves the figure-ground segregation problem through tem(cid:173) poral  evolution.  Different  perceptual  phenomena,  such  as  modal  and  amodal completion,  virtual contours,  grouping and  shape de(cid:173) composition are then explained through local diffusion.  The system  eliminates combinatorial optimization and accounts for  many psy(cid:173) chophysical results with a  fixed  set of parameters.
Learning Informative Statistics: A Nonparametnic Approach
Authors: John W. Fisher III, Alexander T. Ihler, Paul A. Viola
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We discuss an information theoretic approach for categorizing and mod(cid:173) eling dynamic processes. The approach can learn a compact and informa(cid:173) tive statistic which summarizes past states to predict future observations.  Furthermore, the  uncertainty of the prediction is characterized nonpara(cid:173) metrically by a joint density over the learned statistic and present obser(cid:173) vation.  We  discuss the  application of the technique to both noise driven  dynamical systems and random processes sampled from a density which  is conditioned on the past. In the first case we show results in which both  the  dynamics of random walk and the  statistics of the  driving noise  are  captured.  In the second case we  present results in  which a summarizing  statistic  is  learned  on  noisy  random telegraph  waves  with  differing de(cid:173) pendencies on past states.  In both cases the algorithm yields a principled  approach for discriminating processes with differing dynamics and/or de(cid:173) pendencies.  The method is  grounded in  ideas  from  information theory  and nonparametric statistics.
Support Vector Method for Novelty Detection
Authors: Bernhard Schölkopf, Robert C. Williamson, Alex J. Smola, John Shawe-Taylor, John C. Platt
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Suppose you are given some dataset drawn from an underlying probabil(cid:173) ity distribution P  and you want to estimate a "simple" subset S  of input  space such that the probability that a test point drawn from P lies outside  of S equals some a priori specified l/ between 0 and 1.  We  propose a method to  approach this  problem by trying to estimate a  function f  which is positive on S  and negative on the complement.  The  functional form of f  is given by a kernel expansion in terms of a poten(cid:173) tially small subset of the training data; it is regularized by controlling the  length of the weight vector in  an associated feature space.  We provide a  theoretical analysis of the statistical performance of our algorithm.  The algorithm is  a natural extension of the  support vector algorithm to  the case of unlabelled data.
Generalized Model Selection for Unsupervised Learning in High Dimensions
Authors: Shivakumar Vaithyanathan, Byron Dom
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: We describe a Bayesian approach to  model  selection in unsupervised  learning  that  determines  both  the  feature  set  and  the  number  of  clusters. We then evaluate this scheme (based on marginal likelihood)  and  one  based  on  cross-validated  likelihood.  For  the  Bayesian  scheme  we  derive  a  closed-form  solution  of the  marginal  likelihood  by  assuming  appropriate  forms  of the  likelihood  function  and  prior.  Extensive  experiments  compare these  approaches  and  all  results  are  verified  by comparison against ground truth.  In  these experiments the  Bayesian scheme using our objective function gave better results than  cross-validation.
From Coexpression to Coregulation: An Approach to Inferring Transcriptional Regulation among Gene Classes from Large-Scale Expression Data
Authors: Eric Mjolsness, Tobias Mann, Rebecca Castaño, Barbara J. Wold
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: small-scale  gene 
regulation  networks 
We  provide  preliminary  evidence  that  eXlstmg  algorithms  for  inferring  from  gene  expression  data  can  be  adapted  to  large-scale  gene  expression  data  coming  from  hybridization  microarrays.  The essential  steps are  (1)  clustering  many  genes  by  their  expression  time-course  data  into  a  minimal  set  of  clusters  of  co-expressed  genes,  (2)  theoretically  modeling  the  various  conditions  under  which  the  time-courses  are  measured  using  a  continious-time  analog  recurrent  neural  network  for  the  cluster  mean  time-courses,  (3)  fitting  such  a  regulatory  model  to  the  cluster  mean  time  courses  by  simulated  annealing  with  weight  decay,  and  (4)  analysing  several  such  fits  for  commonalities  the  connection  matrices.  This  procedure  can  be  used  to  assess  the  adequacy  of existing  and  future  gene  expression  time-course  data  sets  for  determ ining  transcriptional  regulatory  relationships  such  as  coregulation . 
the  circuit  parameter  sets
Data Visualization and Feature Selection: New Algorithms for Nongaussian Data
Authors: Howard Hua Yang, John Moody
Published in: Advances in Neural Information Processing Systems 12  (NIPS 1999) (Main Conference Track), 1999

Abstract: Data  visualization  and  feature  selection  methods  are  proposed  based on the )oint mutual information and ICA.  The visualization  methods can find  many good 2-D  projections for  high dimensional  data interpretation,  which  cannot be easily found by  the other ex(cid:173) isting methods.  The new  variable selection  method is found  to be  better in eliminating redundancy in the inputs than other methods  based  on  simple mutual information.  The efficacy  of the  methods  is illustrated on a radar signal analysis problem to find  2-D viewing  coordinates for  data visualization and to select  inputs for  a  neural  network  classifier.  Keywords:  feature  selection,  joint mutual information,  ICA,  vi(cid:173) sualization, classification.