mtybilly/Instruct2DS huggingface dataset:
Learning the Architecture of Sum-Product Networks Using Clustering on Variables
Authors: Dan Ventura Aaron Dennis
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: The sum-product network (SPN) is a recently-proposed deep model consisting of a network of sum and product nodes, and has been shown to be competitive with state-of-the-art deep models on certain difficult tasks such as image completion. Designing an SPN network architecture that is suitable for the task at hand is an open question. We propose an algorithm for learning the SPN architecture from data. The idea is to cluster variables (as opposed to data instances) in order to identify variable subsets that strongly interact with one another. Nodes in the SPN network are then allocated towards explaining these interactions. Experimental evidence shows that learning the SPN architecture significantly improves its performance compared to using a previously-proposed static architecture.
Towards a learning-theoretic analysis of spike-timing dependent plasticity
Authors: Michel Besserve David Balduzzi
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to theoretical analysis. We show that the selectron encodes reward estimates into spikes and that an error bound on spikes is controlled by a spiking margin and the sum of synaptic weights. Moreover, the efficacy of spikes (their usefulness to other reward maximizing selectrons) also depends on total synaptic strength. Finally, based on our analysis, we propose a regularized version of STDP, and show the regularization improves the robustness of neuronal learning when faced with multiple stimuli.
Angular Quantization-based Binary Codes for Fast Similarity Search
Authors: Sanjiv Kumar, Vishal Verma, Svetlana Lazebnik Yunchao Gong
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: This paper focuses on the problem of learning binary embeddings for efficient retrieval of high-dimensional non-negative data. Such data typically arises in a large number of vision and text applications where counts or frequencies are used as features.  Also, cosine distance is commonly used as a measure of dissimilarity between such vectors. In this work, we introduce a novel spherical quantization scheme to generate binary embedding of such data and analyze its properties. The number of quantization landmarks in this scheme grows exponentially with data dimensionality resulting in low-distortion quantization.  We propose a very efficient method for computing the binary embedding using such large number of landmarks. Further, a linear transformation is learned to minimize the quantization error by adapting the method to the input data resulting in improved embedding.  Experiments on image and text retrieval applications show superior performance of the proposed method over other existing state-of-the-art methods.
Matrix reconstruction with the local max norm
Authors: Nathan Srebro, Ruslan Salakhutdinov Rina Foygel
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: We introduce a new family of matrix norms, the ''local max'' norms, generalizing existing methods such as the max norm, the trace norm (nuclear norm), and the weighted or smoothed weighted trace norms, which have been extensively used in the literature as regularizers for matrix reconstruction problems. We show that this new family can be used to interpolate between the (weighted or unweighted) trace norm and the more conservative max norm. We test this interpolation on simulated data and on the large-scale Netflix and MovieLens ratings data, and find improved accuracy relative to the existing matrix norms. We also provide theoretical results showing learning guarantees for some of the new norms.
Near-optimal Differentially Private Principal Components
Authors: Anand Sarwate, Kaushik Sinha Kamalika Chaudhuri
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Principal components analysis (PCA) is a standard tool for identifying good low-dimensional approximations to data sets in high dimension.  Many current data sets of interest contain private or sensitive information about individuals.  Algorithms which operate on such data should be sensitive to the privacy risks in publishing their outputs.  Differential privacy is a framework for developing tradeoffs between privacy and the utility of these outputs.  In this paper we investigate the theory and empirical performance of differentially private approximations to PCA and propose a new method which explicitly optimizes the utility of the output.  We demonstrate that on real data, there this a large performance gap between the existing methods and our method.  We show that the sample complexity for the two procedures differs in the scaling with the data dimension, and that our method is nearly optimal in terms of this scaling.
Mandatory Leaf Node Prediction in Hierarchical Multilabel Classification
Authors: James T. Kwok Wei Bi
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: In hierarchical classification, the prediction paths may be required to always end at leaf nodes. This is called mandatory leaf node prediction (MLNP) and is particularly useful when the leaf nodes have much stronger semantic meaning than the internal nodes. However, while there have been a lot of MLNP methods in hierarchical multiclass classification, performing MLNP in hierarchical multilabel classification is much more difficult. In this paper, we propose a novel MLNP algorithm that (i) considers the global hierarchy structure; and (ii) can  be used on hierarchies of both trees and DAGs. We show that one can efficiently maximize the joint posterior probability of all the node labels by a simple greedy algorithm. Moreover, this can be further extended to the minimization of the expected symmetric loss. Experiments are performed on a number of real-world data sets with tree- and DAG-structured label hierarchies. The  proposed method consistently outperforms other hierarchical and flat multilabel classification methods.
Synchronization can Control Regularization in Neural Systems via Correlated Noise Processes
Authors: Jean-jeacques Slotine Jake Bouvrie
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: To learn reliable rules that can generalize to novel situations, the brain must be capable of imposing some form of regularization. Here we suggest, through theoretical and computational arguments, that the combination of noise with synchronization provides a plausible mechanism for regularization in the nervous system. The functional role of regularization is considered in a general context in which coupled computational systems receive inputs corrupted by correlated noise. Noise on the inputs is shown to impose regularization, and when synchronization upstream induces time-varying correlations across noise variables, the degree of regularization can be calibrated over time. The resulting qualitative behavior matches experimental data from visual cortex.
Online Regret Bounds for Undiscounted Continuous Reinforcement Learning
Authors: Daniil Ryabko Ronald Ortner
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: We derive sublinear regret bounds for undiscounted reinforcement learning in continuous state space. The proposed algorithm combines state aggregation with the use of upper confidence bounds for implementing optimism in the face of uncertainty.  Beside the existence of an optimal policy which satisfies the Poisson equation, the only assumptions made are Hoelder continuity of rewards and transition probabilities.
Perceptron Learning of SAT
Authors: Matthew Blaschko Alex Flint
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Boolean satisfiability (SAT) as a canonical NP-complete decision problem is one of the most important problems in computer science.  In practice, real-world SAT sentences are drawn from a distribution that may result in efficient algorithms for their solution. Such SAT instances are likely to have shared characteristics and substructures. This work approaches the exploration of a family of SAT solvers as a learning problem.  In particular, we relate polynomial time solvability of a SAT subset to a notion of margin between sentences mapped by a feature function into a Hilbert space.  Provided this mapping is based on polynomial time computable statistics of a sentence, we show that the existance of a margin between these data points implies the existance of a polynomial time solver for that SAT subset based on the Davis-Putnam-Logemann-Loveland algorithm.  Furthermore, we show that a simple perceptron-style learning rule will find an optimal SAT solver with a bounded number of training updates.  We derive a linear time computable set of features and show analytically that margins exist for important polynomial special cases of SAT.  Empirical results show an order of magnitude improvement over a state-of-the-art SAT solver on a hardware verification task.
On Lifting the Gibbs Sampling Algorithm
Authors: Vibhav Gogate Deepak Venugopal
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Statistical relational learning models combine the power of first-order logic, the de facto tool for handling relational structure, with that of probabilistic graphical models, the de facto tool for handling uncertainty. Lifted probabilistic inference algorithms for them have been the subject of much recent research. The main idea in these algorithms is to improve the speed, accuracy and scalability of existing graphical models' inference algorithms by exploiting symmetry in the first-order representation. In this paper, we consider blocked Gibbs sampling, an advanced variation of the classic Gibbs sampling algorithm and lift it to the first-order level. We propose to achieve this by partitioning the first-order atoms in the relational model into a set of disjoint clusters such that exact lifted inference is polynomial in each cluster given an assignment to all other atoms not in the cluster. We propose an approach for constructing such clusters and determining their complexity and show how it can be used to trade accuracy with computational complexity in a principled manner. Our experimental evaluation shows that lifted Gibbs sampling is superior to the propositional algorithm in terms of accuracy and convergence.
Q-MKL: Matrix-induced Regularization in Multi-Kernel Learning with Applications to Neuroimaging
Authors: Vikas Singh, Jiming Peng, Sterling Johnson Chris Hinrichs
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Multiple Kernel Learning (MKL) generalizes SVMs to the setting where one simultaneously trains a linear classifier and chooses an optimal combination of given base kernels. Model complexity is typically controlled using various norm regularizations on the vector of base kernel mixing coefficients. Existing methods, however, neither regularize nor exploit potentially useful information pertaining to how kernels in the input set 'interact'; that is, higher order kernel-pair relationships that can be easily obtained via unsupervised (similarity, geodesics), supervised (correlation in errors), or domain knowledge driven mechanisms (which features were used to construct the kernel?). We show that by substituting the norm penalty with an arbitrary quadratic function Q \succeq 0, one can impose a desired covariance structure on mixing coefficient selection, and use this as an inductive bias when learning the concept. This formulation significantly generalizes the widely used 1- and 2-norm MKL objectives. We explore the model’s utility via experiments on a challenging Neuroimaging problem, where the goal is to predict a subject’s conversion to Alzheimer’s Disease (AD) by exploiting aggregate information from several distinct imaging modalities. Here, our new model outperforms the state of the art (p-values << 10−3 ). We briefly discuss ramifications in terms of learning bounds (Rademacher complexity).
Label Ranking with Partial Abstention based on Thresholded Probabilistic Models
Authors: Eyke Hüllermeier, Willem Waegeman, Volkmar Welker Weiwei Cheng
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Several machine learning methods allow for abstaining from uncertain predictions. While being common for settings like conventional classification, abstention has been studied much less in learning to rank. We address abstention for the label ranking setting, allowing the learner to declare certain pairs of labels as being incomparable and, thus, to predict partial instead of total orders. In our method, such predictions are produced via thresholding the probabilities of pairwise preferences between labels, as induced by a predicted probability distribution on the set of all rankings. We formally analyze this approach for the Mallows and the Plackett-Luce model, showing that it produces proper partial orders as predictions and characterizing the expressiveness of the induced class of partial orders. These theoretical results are complemented by experiments demonstrating the practical usefulness of the approach.
Weighted Likelihood Policy Search with Model Selection
Authors: Kohei Hayashi, Takashi Washio, Yoshinobu Kawahara Tsuyoshi Ueno
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Reinforcement learning (RL) methods based on direct policy search (DPS) have been actively discussed to achieve an efficient approach to complicated Markov decision processes (MDPs). Although they have brought much progress in practical applications of RL, there still remains an unsolved problem in DPS related to model  selection for the policy. In this paper, we propose a novel DPS method, {\it  weighted likelihood policy search (WLPS)}, where a policy is efficiently learned through the weighted likelihood estimation. WLPS naturally connects DPS to the statistical inference problem and thus various sophisticated techniques in statistics can be applied to DPS problems directly. Hence, by following the idea of the {\it information criterion}, we develop a new measurement for model comparison in DPS based on the weighted log-likelihood.
Parameterising Feature Sensitive Cell Formation in Linsker Networks in the Auditory System
Authors: David L. Bisset Lance C. Walton
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper examines and extends the work of Linsker (1986) on  self organising feature detectors. Linsker concentrates on the vi(cid:173) sual processing system, but infers that the weak assumptions made  will allow the model to be used in the processing of other sensory  information. This claim is examined here, with special attention  paid to the auditory system, where there is much lower connec(cid:173) tivity and therefore more statistical variability. On-line training is  utilised, to obtain an idea of training times. These are then com(cid:173) pared to the time available to pre-natal mammals for the formation  of feature sensitive cells.
Learning Spatio-Temporal Planning from a Dynamic Programming Teacher: Feed-Forward Neurocontrol for Moving Obstacle Avoidance
Authors: Rolf Eckmiller Gerald Fahner
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Within a simple test-bed,  application of feed-forward  neurocontrol  for short-term planning of robot trajectories in a dynamic environ(cid:173) ment  is  studied.  The  action  network  is  embedded  in  a  sensory(cid:173) motoric system architecture that contains  a separate world model.  It is  continuously  fed  with  short-term  predicted  spatio-temporal  obstacle  trajectories,  and  receives  robot  state  feedback.  The  ac(cid:173) tion  net  allows  for  external  switching  between  alternative  plan(cid:173) ning  tasks.  It  generates  goal-directed  motor  actions  - subject  to  the  robot's  kinematic  and  dynamic  constraints  - such  that  colli(cid:173) sions  with  moving obstacles  are  avoided.  Using  supervised  learn(cid:173) ing,  we  distribute  examples  of the  optimal planner  mapping over  a  structure-level  adapted  parsimonious higher  order  network.  The  training  database  is  generated  by  a  Dynamic  Programming algo(cid:173) rithm.  Extensive  simulations reveal,  that  the  local  planner  map(cid:173) ping is  highly nonlinear, but can be effectively  and sparsely repre(cid:173) sented  by  the  chosen  powerful  net model.  Excellent generalization  occurs for unseen obstacle configurations.  We  also discuss the limi(cid:173) tations of feed-forward  neurocontrol for growing planning horizons. 
*Tel.:  (228)-550-364 
FAX:  (228)-550-425 
e-mail:  gerald@nero.uni-bonn.de
Hidden Markov Models in Molecular Biology: New Algorithms and Applications
Authors: Yves Chauvin, Tim Hunkapiller, Marcella A. McClure Pierre Baldi
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Hidden  Markov Models  (HMMs)  can  be applied  to several impor(cid:173) tant problems in molecular biology.  We introduce a new convergent  learning algorithm for HMMs that, unlike the classical Baum-Welch  algorithm is smooth and can be applied  on-line or in  batch mode,  with or  without the usual Viterbi most  likely  path approximation.  Left-right HMMs with insertion and deletion states are then trained  to represent several protein families including immunoglobulins and  kinases.  In  all cases,  the models derived capture all  the important  statistical  properties  of the families  and  can  be  used  efficiently  in  a  number of important tasks such as multiple alignment, motif de(cid:173) tection,  and  classification. 
*and Division  of Biology,  California  Institute of Technology.  t and Department of Psychology,  Stanford University.
Statistical and Dynamical Interpretation of ISIH Data from Periodically Stimulated Sensory Neurons
Authors: Frank Moss, André Longtin John K. Douglass
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We interpret the time interval data obtained from periodically stimulated  sensory neurons in terms of two simple dynamical systems driven by noise  with an embedded weak periodic function called the signal: 1) a bistable  system defined by two potential wells separated by a barrier, and 2) a Fit(cid:173) zHugh-Nagumo system. The implementation is by analog simulation: elec(cid:173) tronic circuits which mimic the dynamics. For a given signal frequency, our  simulators have only two adjustable parameters, the signal and noise intensi(cid:173) ties. We show that experimental data obtained from the periodically stimu(cid:173) lated mechanoreceptor in the crayfish tail fan can be accurately approximated  by these simulations. Finally, we discuss stochastic resonance in the two  models.
Spiral Waves in Integrate-and-Fire Neural Networks
Authors: Po Hsiang Chu, Jack D. Cowan John G. Milton
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The formation of propagating spiral waves is studied in a  randomly  connected neural network composed of integrate-and-fire neurons  with  recovery  period and  excitatory  connections  using  computer  simulations.  Network  activity is  initiated by  periodic stimulation  at a single point.  The results suggest that spiral waves can arise in  such a  network via a  sub-critical Hopf bifurcation.
Adaptive Stimulus Representations: A Computational Theory of Hippocampal-Region Function
Authors: Catherine E. Myers Mark A. Gluck
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present a theory of cortico-hippocampal interaction in discrimination learning. The  hippocampal region is presumed to form new stimulus representations which facilitate  learning by enhancing the discriminability of predictive stimuli and compressing  stimulus-stimulus redundancies. The cortical and cerebellar regions, which are the sites  of long-term memory. may acquire these new representations but are not assumed to be  capable of forming new representations themselves.  Instantiated as a connectionist  model. this theory accounts for a wide range of trial-level classical conditioning  phenomena in normal (intact) and hippocampal-Iesioned animals. It also makes several  novel predictions which remain to be investigated empirically. The theory implies that  the hippocampal region is involved in even the simplest learning tasks; although  hippocampal-Iesioned animals may be able to use other strategies to learn these tasks. the  theory predicts that they will show consistently different patterns of transfer and  generalization when the task demands change.
Computing with Almost Optimal Size Neural Networks
Authors: Vwani Roychowdhury, Thomas Kailath Kai-Yeung Siu
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Artificial neural networks are comprised of an interconnected collection  of certain nonlinear devices; examples of commonly used devices include  linear threshold elements, sigmoidal elements and radial-basis elements.  We employ results from harmonic analysis and the theory of rational ap(cid:173) proximation to obtain almost tight lower bounds on the size (i.e. number  of elements) of neural networks. The class of neural networks to which  our techniques can be applied is quite general; it includes any feedforward  network in which each element can be piecewise approximated by a low  degree rational function. For example, we prove that any depth-( d + 1)  network of sigmoidal units or linear threshold elements computing the par(cid:173) ity function of n variables must have O(dnl/d-£) size, for any fixed i > O.  In addition, we prove that this lower bound is almost tight by showing  that the parity function can be computed with O(dnl/d) sigmoidal units  or linear threshold elements in a depth-(d + 1) network. These almost  tight bounds are the first known complexity results on the size of neural  networks with depth more than two. Our lower bound techniques yield  a unified approach to the complexity analysis of various models of neural  networks with feedforward structures. Moreover, our results indicate that  in the context of computing highly oscillating symmetric Boolean func-
Q-Learning with Hidden-Unit Restarting
Authors: Charles W. Anderson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Platt's  resource-allocation  network  (RAN)  (Platt,  1991a,  1991b)  is  modified for  a  reinforcement-learning paradigm and  to  "restart"  existing hidden  units  rather than adding  new  units.  After restart(cid:173) ing,  units  continue  to  learn  via  back-propagation.  The  resulting  restart  algorithm is  tested  in  a  Q-Iearning  network  that  learns  to  solve an inverted pendulum problem.  Solutions are found faster on  average  with  the restart  algorithm than  without it.
Synaptic Weight Noise During MLP Learning Enhances Fault-Tolerance, Generalization and Learning Trajectory
Authors: Peter J. Edwards Alan F. Murray
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  analyse  the  effects  of analog  noise  on  the  synaptic  arithmetic  during MultiLayer Perceptron training, by expanding the cost func(cid:173) tion to include noise-mediated penalty terms.  Predictions are made  in the light of these  calculations which suggest  that fault tolerance,  generalisation  ability  and  learning  trajectory  should  be  improved  by  such  noise-injection.  Extensive  simulation experiments  on  two  distinct  classification  problems  substantiate  the  claims.  The  re(cid:173) sults  appear to  be  perfectly  general for  all training schemes  where  weights are adjusted incrementally, and have wide-ranging implica(cid:173) tions for  all applications, particularly those involving  "inaccurate"  analog neural  VLSI.
Rational Parametrizations of Neural Networks
Authors: Robert C. Williamson Uwe Helmke
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A connection is drawn between rational functions, the realization  theory of dynamical systems, and feedforward neural networks.  This allows us to parametrize single hidden layer scalar neural  networks with (almost) arbitrary analytic activation functions in  terms of strictly proper rational functions. Hence, we can solve the  uniqueness of parametrization problem for such networks.
Directional-Unit Boltzmann Machines
Authors: Christopher K. I. Williams, Michael Mozer Richard S. Zemel
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  present  a  general  formulation  for  a  network  of stochastic  di(cid:173) rectional units.  This formulation is  an extension of the Boltzmann  machine in which  the units are  not  binary, but take on values in a  cyclic  range,  between  0  and 271'  radians.  The state of each  unit  in  a  Directional-Unit  Boltzmann  Machine  (DUBM)  is  described  by  a  complex variable, where  the phase component specifies  a direction;  the  weights  are  also  complex variables.  We  associate  a  quadratic  energy  function,  and  corresponding  probability,  with  each  DUBM  configuration.  The  conditional  distribution  of a  unit's  stochastic  state is  a  circular  version of the Gaussian probability distribution,  known  as  the  von  Mises  distribution.  In  a  mean-field  approxima(cid:173) tion  to  a  stochastic  DUBM,  the  phase  component of a  unit's state  represents  its mean direction,  and the magnitude component spec(cid:173) ifies  the  degree  of certainty  associated  with  this  direction.  This  combination of a  value  and  a  certainty  provides  additional  repre(cid:173) sentational power  in a  unit.  We  describe  a  learning algorithm and  simulations that demonstrate  a  mean-field  DUBM'S  ability to learn  interesting  mappings. 
Many  kinds  of information  can  naturally  be  represented  in  terms  of angular,  or  directional, variables.  A circular range forms a suitable representation for  explicitly  directional  information,  such  as  wind  direction,  as  well  as  for  information  where  the  underlying  range  is  periodic,  such  as  days  of the  week  or  months of the  year.  In  computer  vision,  tangent  fields  and optic flow  fields  are  represented  as fields  of  oriented line segments, each of which can be described by a magnitude and direction.  Directions  can  also  be  used  to represent  a set  of symbolic labels,  e.g.,  object  label  A  at  0,  and  object  label  B  at 71'/2  radians.  We  discuss  below  some  advantages  of  representing  symbolic labels with  directional units.
Stimulus Encoding by Multidimensional Receptive Fields in Single Cells and Cell Populations in V1 of Awake Monkey
Authors: Ad Aertsen, Eilon Vaadia, Shaul Hochstein Edward Stern
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Ad  Aertsen 
Institut fur  Neuroinfonnatik  Ruhr-Universitat-Bochum
On the Use of Projection Pursuit Constraints for Training Neural Networks
Authors: Nathan Intrator
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: \Ve  present  a  novel  classifica t.ioll  and  regression  met.hod  that  com(cid:173) bines  exploratory  projection  pursuit.  (unsupervised  traiuing)  with  pro(cid:173) jection  pursuit.  regression  (supervised  t.raining),  t.o  yield  a.  nev,,'  family  of  cost./complexity  penalLy  terms .  Some  improved  generalization  properties  are  demonstrat.ed  on  real  \vorld  problems.
Assessing and Improving Neural Network Predictions by the Bootstrap Algorithm
Authors: Gerhard Paass
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The bootstrap algorithm is  a  computational intensive procedure to  derive  nonparametric  confidence  intervals  of statistical estimators  in  situations  where  an  analytic  solution  is  intractable.  It is  ap(cid:173) plied to neural networks to estimate the predictive distribution for  unseen  inputs.  The  consistency  of different  bootstrap  procedures  and  their convergence  speed is  discussed.  A small scale  simulation  experiment  shows  the  applicability  of the  bootstrap  to  practical  problems and its potential use.
Self-Organizing Rules for Robust Principal Component Analysis
Authors: Alan L. Yuille Lei Xu
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In  the  presence  of outliers,  the  existing  self-organizing  rules  for  Principal  Component  Analysis  (PCA)  perform  poorly.  Using  sta(cid:173) tistical physics techniques including the Gibbs distribution, binary  decision  fields  and  effective  energies,  we  propose  self-organizing  PCA  rules  which  are  capable  of  resisting  outliers  while  fulfilling  various PCA-related tasks such as obtaining the first  principal com(cid:173) ponent vector,  the first  k  principal component vectors,  and directly  finding  the  subspace  spanned  by  the  first  k  vector  principal  com(cid:173) ponent  vectors  without solving for  each  vector  individually.  Com(cid:173) parative  experiments  have  shown  that  the  proposed  robust  rules  improve  the  performances  of the  existing  PCA  algorithms signifi(cid:173) cantly when outliers are  present.
Intersecting regions: The Key to combinatorial structure in hidden unit space
Authors: Mark Ollila Janet Wiles
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Hidden units in multi-layer networks form a representation space in which each  region can be identified with a class of equivalent outputs (Elman, 1989) or a  logical state in a finite state machine (Cleeremans, Servan-Schreiber &  McClelland, 1989; Giles, Sun, Chen, Lee, & Chen, 1990). We extend the  analysis of the spatial structure of hidden unit space to a combinatorial task,  based on binding features together in a visual scene. The logical structure  requires a combinatorial number of states to represent all valid scenes. On  analysing our networks, we find that the high dimensionality of hidden unit  space is exploited by using the intersection of neighboring regions to represent  conjunctions of features. These results show how combinatorial structure can  be based on the spatial nature of networks, and not just on their emulation of  logical structure. 
1 TECHNIQUES FOR ANALYSING THE SPATIAL AND  LOGICAL STRUCTURE OF HIDDEN UNIT SPACE 
In multi-layer networks, regions of hidden unit space can be identified with classes of  equivalent outputs. For example, Elman (1989) showed that the hidden unit patterns for  words in simple grammatical sentences cluster into regions, with similar patterns  representing similar grammatical entities. For example, different tokens of the same word  are clustered tightly, indicating that they are represented within a small region. These  regions can be grouped into larger regions, reflecting a hierarchical structure. The largest
A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks
Authors: R. Meir, B. Yuhas, A. Jayakumar, D. Lippe J. Alspector
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Typical methods for gradient descent in neural network learning involve  calculation of derivatives based on a detailed knowledge of the network  model. This requires extensive, time consuming calculations for each pat(cid:173) tern presentation and high precision that makes it difficult to implement  in VLSI. We present here a perturbation technique that measures, not  calculates, the gradient. Since the technique uses the actual network as  a measuring device, errors in modeling neuron activation and synaptic  weights do not cause errors in gradient descent. The method is parallel  in nature and easy to implement in VLSI. We describe the theory of such  an algorithm, an analysis of its domain of applicability, some simulations  using it and an outline of a hardware implementation.
A Model of Feedback to the Lateral Geniculate Nucleus
Authors: Carlos D. Brody
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Simplified  models  of the  lateral geniculate  nucles  (LGN)  and stri(cid:173) ate cortex  illustrate the  possibility  that feedback  to the  LG N may  be  used  for  robust,  low-level  pattern  analysis.  The  information  fed  back  to the  LG N is  rebroadcast  to cortex  using the LG N 's full  fan-out,  so  the  cortex-LGN-cortex pathway  mediates extensive  cortico-cortical communication while  keeping the number of neces(cid:173) sary  connections small.
History-Dependent Attractor Neural Networks
Authors: Eytan Ruppin Isaac Meilijson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  present  a  methodological  framework  enabling  a  detailed  de(cid:173) scription  of the  performance of Hopfield-like  attractor  neural  net(cid:173) works  (ANN)  in  the  first  two  iterations.  Using  the  Bayesian  ap(cid:173) proach,  we find  that performance is improved when a history-based  term is included in the neuron's dynamics.  A further enhancement  of the  network's  performance  is  achieved  by  judiciously  choosing  the  censored neurons  (those  which  become active in  a  given  itera(cid:173) tion)  on  the  basis  of the  magnitude of their  post-synaptic  poten(cid:173) tials.  The contribution of biologically plausible,  censored,  history(cid:173) dependent dynamics is especially marked in conditions of low firing  activity  and  sparse  connectivity,  two  important  characteristics  of  the  mammalian  cortex.  In  such  networks,  the  performance  at(cid:173) tained  is  higher  than  the  performance  of two  'independent'  iter(cid:173) ations,  which  represents  an  upper  bound  on  the  performance  of  history-independent  networks.
Single-Iteration Threshold Hamming Networks
Authors: Eytan Ruppin, Moshe Sipper Isaac Meilijson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  analyze in  detail  the  performance of a  Hamming network  clas(cid:173) sifying  inputs  that  are  distorted  versions  of one  of its  m  stored  memory patterns.  The activation function  of the memory neurons  in the original Hamming network  is  replaced  by a simple threshold  function.  The resulting Threshold  Hamming Network  (THN)  cor(cid:173) rectly  classifies  the  input  pattern,  with  probability approaching 1,  using only O(mln m)  connections,  in  a single iteration.  The THN  drastically reduces the time and space complexity of Hamming Net(cid:173) work  classifiers.
Reinforcement Learning Applied to Linear Quadratic Regulation
Authors: Steven J. Bradtke
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Recent  research  on  reinforcement  learning  has  focused  on  algo(cid:173) rithms  based  on  the  principles  of  Dynamic  Programming  (DP).  One  of  the  most  promising  areas  of  application  for  these  algo(cid:173) rithms  is  the  control  of dynamical  systems,  and  some  impressive  results  have  been  achieved.  However,  there  are  significant  gaps  between  practice  and  theory.  In  particular,  there  are  no  con ver(cid:173) gence  proofs for  problems with continuous state and action  spaces,  or  for  systems  involving  non-linear  function  approximators  (such  as  multilayer  perceptrons).  This  paper  presents  research  applying  DP-based  reinforcement  learning  theory  to Linear  Quadratic  Reg(cid:173) ulation  (LQR),  an  important  class  of control  problems  involving  continuous  state  and  action  spaces  and  requiring  a  simple  type of  non-linear function  approximator.  We describe  an algorithm based  on  Q-Iearning  that is  proven  to converge  to the optimal  controller  for  a  large  class  of  LQR  problems.  We  also  describe  a  slightly  different  algorithm  that  is  only  locally  convergent  to  the  optimal  Q-function,  demonstrating  one  of the  possible  pitfalls  of using  a  non-linear  function  approximator  with  DP-based  learning.
The Computation of Stereo Disparity for Transparent and for Opaque Surfaces
Authors: Daniel Kersten, Ting-Chuen Pong Suthep Madarasmi
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  classical  computational  model  for  stereo  vision  incorporates  a  uniqueness  inhibition  constraint  to enforce  a  one-to-one  feature  match, thereby sacrificing the ability to handle transparency.  Crit(cid:173) ics  of the  model  disregard  the  uniqueness  constraint  and  argue  that the smoothness constraint  can provide the excitation support  required  for  transparency  computation.  However,  this  modifica(cid:173) tion  fails  in  neighborhoods  with  sparse  features.  We  propose  a  Bayesian  approach  to  stereo  vision  with  priors  favoring  cohesive  over transparent surfaces.  The disparity and its segmentation into a  multi-layer "depth planes"  representation  are simultaneously com(cid:173) puted.  The smoothness constraint propagates support  within each  layer, providing mutual excitation for non-neighboring transparent  or  partially occluded  regions.  Test  results  for  various  random-dot  and other stereograms are presented.
Forecasting Demand for Electric Power
Authors: Terrence Fine Jen-Lun Yuan
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  are  developing  a  forecaster  for  daily  extremes  of demand  for  electric  power encountered  in  the service  area of a  large  midwest(cid:173) ern  utility  and  using  this  application  as  a  testbed  for  approaches  to input dimension reduction and decomposition of network train(cid:173) ing.  Projection  pursuit  regression  representations  and  the  ability  of algorithms like SIR to quickly  find  reasonable weighting vectors  enable us  to confront the vexing architecture selection problem  by  reducing  high-dimensional  gradient  searchs  to  fitting  single-input  single-output  (SISO)  subnets.  We  introduce  dimension  reduction  algorithms,  to select  features  or  relevant  subsets of a  set  of many  variables,  based  on  minimizing  an  index  of level-set  dispersions  (closely  related  to  a  projection  index  and  to  SIR),  and  combine  them  with  backfitting  to  implement  a  neural  network  version  of  projection  pursuit.  The  performance  achieved  by  our  approach,  when trained on 1989,  1990 data and tested on  1991  data, is  com(cid:173) parable  to  that  achieved  in  our  earlier  study  of  backpropagation  trained networks.
Visual Motion Computation in Analog VLSI Using Pulses
Authors: Wyeth Bair, Christof Koch Rahul Sarpeshkar
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  real  time  computation  of  motion  from  real  images  using a  single chip with integrated sensors is a  hard prob(cid:173) lem.  We  present two  analog VLSI schemes that use  pulse  domain neuromorphic circuits to compute motion.  Pulses  of variable width, rather than graded potentials, represent  a  natural  medium  for  evaluating temporal relationships.  Both algorithms measure speed by  timing a  moving edge  in  the image.  Our first  model is  inspired  by  Reichardt's  algorithm in the fiy  and yields  a  non-monotonic response  vs.  velocity  curve.  We  present  data from  a  chip  that  implements  this  model.  Our  second  algorithm  yields  a  monotonic  response  vs.  velocity  curve  and  is  currently  being translated into silicon. 
1
Learning Control Under Extreme Uncertainty
Authors: Vijaykumar Gullapalli
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A  peg-in-hole  insertion  task  is  used  as  an  example  to  illustrate  the  utility of direct associative reinforcement learning methods for  learning  control  under  real-world  conditions  of  uncertainty  and  noise.  Task  complexity  due  to  the  use  of  an  unchamfered  hole  and a clearance of less than 0.2mm is  compounded  by  the presence  of positional uncertainty of magnitude exceeding 10  to 50 times the  clearance.  Despite  this  extreme  degree  of uncertainty,  our  results  indicate  that  direct  reinforcement learning  can  be  used  to  learn  a  robust  reactive  control strategy  that  results  in  skillful  peg-in-hole  insertions.
Efficient Pattern Recognition Using a New Transformation Distance
Authors: Yann LeCun, John S. Denker Patrice Simard
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Memory-based  classification  algorithms such  as  radial  basis  func(cid:173) tions or K-nearest  neighbors typically rely on simple distances (Eu(cid:173) clidean,  dot  product ... ),  which  are  not  particularly  meaningful  on  pattern vectors.  More complex, better suited distance measures are  often  expensive  and  rather  ad-hoc  (elastic  matching,  deformable  templates).  We  propose  a  new  distance  measure which  (a)  can  be  made  locally  invariant to  any  set  of transformations  of the  input  and  (b)  can  be  computed  efficiently.  We  tested  the  method  on  large handwritten character  databases  provided by the  Post  Office  and the NIST.  Using invariances with  respect  to translation, rota(cid:173) tion,  scaling,  shearing and line  thickness,  the  method  consistently  outperformed  all other systems  tested  on  the same databases.
Planar Hidden Markov Modeling: From Speech to Optical Character Recognition
Authors: Roberto Pieraccini Esther Levin
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We propose in  this paper a  statistical  model  (planar hidden  Markov model  - PHMM)  describing  statistical  properties  of images.  The  model generalizes  the single-dimensional HMM,  used for speech processing, to  the  planar case.  For this model to be useful an efficient segmentation algorithm, similar to the  Viterbi  algorithm  for  HMM,  must exist  We  present conditions  in  terms  of  the  PHMM  parameters  that  are  sufficient  to  guarantee  that  the  planar  segmentation  problem  can  be  solved  in  polynomial  time,  and  describe  an  algorithm for that.  This algorithm aligns optimally the image with the model,  and  therefore  is  insensitive  to  elastic  distortions  of  images.  Using  this  algorithm a joint optima1  segmentation and recognition of the  image  can  be  performed, thus overcoming the  weakness of traditional OCR systems where  segmentation  is  performed  independently  before  the  recognition  leading  to  unrecoverable recognition errors. 
Tbe  PHMM  approach  was  evaluated  using  a  set  of  isolated  band-written  digits.  An  overall  digit  recognition  accuracy  of  95%  was  acbieved.  An  analysis of the results showed  that even in  the  simple case of recognition  of  isolated  characters,  the  elimination  of  elastic  distortions  enhances  the  performance Significantly. We expect that the advantage of this approach  will  be  even  more  such  as  connected  writing  recognition/spotting,  for  whicb  there  is  no  known  high  accuracy  method  of  recognition.
Global Regularization of Inverse Kinematics for Redundant Manipulators
Authors: Kenneth Kreutz-Delgado David DeMers
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  inverse  kinematics  problem  for  redundant  manipulators  is  ill-posed  and  nonlinear.  There are two fundamentally different issues which result in the need  for  some  form  of regularization;  the  existence  of multiple  solution branches  (global ill-posedness) and the existence of excess degrees of freedom (local ill(cid:173) posedness).  For certain  classes  of manipulators,  learning  methods  applied  to  input-output data generated from  the forward function can  be used  to globally  regularize the problem by partitioning the domain of the forward mapping into  a  finite  set  of regions  over which  the  inverse  problem  is  well-posed.  Local  regularization  can  be accomplished  by  an  appropriate parameterization  of the  redundancy consistently over each region.  As a result, the ill-posed problem can  be transformed into a finite set of well-posed problems. Each can then be solved  separately to construct approximate direct inverse functions.
Object-Based Analog VLSI Vision Circuits
Authors: Binnal Mathur, Shih-Chii Liu, John G. Harris, Jin Luo, Massimo Sivilotti Christof Koch
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  describe  two successfully  working,  analog  VLSI  vision  circuits  that move beyond pixel-based early vision algorithms.  One circuit,  implementing the dynamic wires model, provides for dedicated lines  of communication  among  groups  of pixels  that  share  a  common  property.  The chip  uses  the  dynamic wires  model to  compute the  arclength of visual contours.  Another circuit labels all points inside  a  given  contour  with one  voltage  and  all other  with  another  volt(cid:173) age.  Its behavior is  very  robust,  since small breaks in  contours are  automatically sealed,  providing  for  Figure-Ground segregation  in  a  noisy  environment.  Both  chips  are implemented using  networks  of resistors  and switches  and  represent  a  step  towards object  level  processing  since  a  single  voltage value encodes  the property  of an  ensemble of pixels. 
1  CONTOUR-LENGTH CHIP 
Contour  length  computation  is  useful  for  further  processing  such  as  structural  saliency  (Shaashua and  Ullman,  1988),  which  is  thought  to be  an important stage  before  object  recognition.  This computation is  impossible on  an  analog chip  if we