mtybilly/Instruct2DS huggingface dataset:
Density-Difference Estimation
Authors: Masashi Sugiyama, Takafumi Kanamori, Taiji Suzuki, Marthinus D. Plessis, Song Liu, Ichiro Takeuchi
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: We address the problem of estimating the difference between two probability densities. A naive approach  is a two-step procedure of first estimating two densities separately and then computing their difference. However, such a two-step procedure does not necessarily work well because the first step is performed without regard to the second step and thus a small estimation error incurred in the first stage can cause a big error in the second stage. In this paper, we propose a single-shot procedure  for directly estimating the density difference without separately estimating two densities. We derive a non-parametric finite-sample error bound for the proposed single-shot density-difference estimator and show that it achieves the optimal convergence rate. We then show how the proposed density-difference estimator can be utilized in L2-distance approximation. Finally, we experimentally demonstrate the usefulness of the proposed method in robust distribution comparison such as class-prior estimation and change-point detection.
Learning the Architecture of Sum-Product Networks Using Clustering on Variables
Authors: Aaron Dennis, Dan Ventura
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: The sum-product network (SPN) is a recently-proposed deep model consisting of a network of sum and product nodes, and has been shown to be competitive with state-of-the-art deep models on certain difficult tasks such as image completion. Designing an SPN network architecture that is suitable for the task at hand is an open question. We propose an algorithm for learning the SPN architecture from data. The idea is to cluster variables (as opposed to data instances) in order to identify variable subsets that strongly interact with one another. Nodes in the SPN network are then allocated towards explaining these interactions. Experimental evidence shows that learning the SPN architecture significantly improves its performance compared to using a previously-proposed static architecture.
Towards a learning-theoretic analysis of spike-timing dependent plasticity
Authors: David Balduzzi, Michel Besserve
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to theoretical analysis. We show that the selectron encodes reward estimates into spikes and that an error bound on spikes is controlled by a spiking margin and the sum of synaptic weights. Moreover, the efficacy of spikes (their usefulness to other reward maximizing selectrons) also depends on total synaptic strength. Finally, based on our analysis, we propose a regularized version of STDP, and show the regularization improves the robustness of neuronal learning when faced with multiple stimuli.
Angular Quantization-based Binary Codes for Fast Similarity Search
Authors: Yunchao Gong, Sanjiv Kumar, Vishal Verma, Svetlana Lazebnik
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: This paper focuses on the problem of learning binary embeddings for efficient retrieval of high-dimensional non-negative data. Such data typically arises in a large number of vision and text applications where counts or frequencies are used as features.  Also, cosine distance is commonly used as a measure of dissimilarity between such vectors. In this work, we introduce a novel spherical quantization scheme to generate binary embedding of such data and analyze its properties. The number of quantization landmarks in this scheme grows exponentially with data dimensionality resulting in low-distortion quantization.  We propose a very efficient method for computing the binary embedding using such large number of landmarks. Further, a linear transformation is learned to minimize the quantization error by adapting the method to the input data resulting in improved embedding.  Experiments on image and text retrieval applications show superior performance of the proposed method over other existing state-of-the-art methods.
Matrix reconstruction with the local max norm
Authors: Rina Foygel, Nathan Srebro, Ruslan Salakhutdinov
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: We introduce a new family of matrix norms, the ''local max'' norms, generalizing existing methods such as the max norm, the trace norm (nuclear norm), and the weighted or smoothed weighted trace norms, which have been extensively used in the literature as regularizers for matrix reconstruction problems. We show that this new family can be used to interpolate between the (weighted or unweighted) trace norm and the more conservative max norm. We test this interpolation on simulated data and on the large-scale Netflix and MovieLens ratings data, and find improved accuracy relative to the existing matrix norms. We also provide theoretical results showing learning guarantees for some of the new norms.
Near-optimal Differentially Private Principal Components
Authors: Kamalika Chaudhuri, Anand Sarwate, Kaushik Sinha
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Principal components analysis (PCA) is a standard tool for identifying good low-dimensional approximations to data sets in high dimension.  Many current data sets of interest contain private or sensitive information about individuals.  Algorithms which operate on such data should be sensitive to the privacy risks in publishing their outputs.  Differential privacy is a framework for developing tradeoffs between privacy and the utility of these outputs.  In this paper we investigate the theory and empirical performance of differentially private approximations to PCA and propose a new method which explicitly optimizes the utility of the output.  We demonstrate that on real data, there this a large performance gap between the existing methods and our method.  We show that the sample complexity for the two procedures differs in the scaling with the data dimension, and that our method is nearly optimal in terms of this scaling.
Mandatory Leaf Node Prediction in Hierarchical Multilabel Classification
Authors: Wei Bi, James T. Kwok
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: In hierarchical classification, the prediction paths may be required to always end at leaf nodes. This is called mandatory leaf node prediction (MLNP) and is particularly useful when the leaf nodes have much stronger semantic meaning than the internal nodes. However, while there have been a lot of MLNP methods in hierarchical multiclass classification, performing MLNP in hierarchical multilabel classification is much more difficult. In this paper, we propose a novel MLNP algorithm that (i) considers the global hierarchy structure; and (ii) can  be used on hierarchies of both trees and DAGs. We show that one can efficiently maximize the joint posterior probability of all the node labels by a simple greedy algorithm. Moreover, this can be further extended to the minimization of the expected symmetric loss. Experiments are performed on a number of real-world data sets with tree- and DAG-structured label hierarchies. The  proposed method consistently outperforms other hierarchical and flat multilabel classification methods.
Synchronization can Control Regularization in Neural Systems via Correlated Noise Processes
Authors: Jake Bouvrie, Jean-jeacques Slotine
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: To learn reliable rules that can generalize to novel situations, the brain must be capable of imposing some form of regularization. Here we suggest, through theoretical and computational arguments, that the combination of noise with synchronization provides a plausible mechanism for regularization in the nervous system. The functional role of regularization is considered in a general context in which coupled computational systems receive inputs corrupted by correlated noise. Noise on the inputs is shown to impose regularization, and when synchronization upstream induces time-varying correlations across noise variables, the degree of regularization can be calibrated over time. The resulting qualitative behavior matches experimental data from visual cortex.
Online Regret Bounds for Undiscounted Continuous Reinforcement Learning
Authors: Ronald Ortner, Daniil Ryabko
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: We derive sublinear regret bounds for undiscounted reinforcement learning in continuous state space. The proposed algorithm combines state aggregation with the use of upper confidence bounds for implementing optimism in the face of uncertainty.  Beside the existence of an optimal policy which satisfies the Poisson equation, the only assumptions made are Hoelder continuity of rewards and transition probabilities.
Perceptron Learning of SAT
Authors: Alex Flint, Matthew Blaschko
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Boolean satisfiability (SAT) as a canonical NP-complete decision problem is one of the most important problems in computer science.  In practice, real-world SAT sentences are drawn from a distribution that may result in efficient algorithms for their solution. Such SAT instances are likely to have shared characteristics and substructures. This work approaches the exploration of a family of SAT solvers as a learning problem.  In particular, we relate polynomial time solvability of a SAT subset to a notion of margin between sentences mapped by a feature function into a Hilbert space.  Provided this mapping is based on polynomial time computable statistics of a sentence, we show that the existance of a margin between these data points implies the existance of a polynomial time solver for that SAT subset based on the Davis-Putnam-Logemann-Loveland algorithm.  Furthermore, we show that a simple perceptron-style learning rule will find an optimal SAT solver with a bounded number of training updates.  We derive a linear time computable set of features and show analytically that margins exist for important polynomial special cases of SAT.  Empirical results show an order of magnitude improvement over a state-of-the-art SAT solver on a hardware verification task.
On Lifting the Gibbs Sampling Algorithm
Authors: Deepak Venugopal, Vibhav Gogate
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Statistical relational learning models combine the power of first-order logic, the de facto tool for handling relational structure, with that of probabilistic graphical models, the de facto tool for handling uncertainty. Lifted probabilistic inference algorithms for them have been the subject of much recent research. The main idea in these algorithms is to improve the speed, accuracy and scalability of existing graphical models' inference algorithms by exploiting symmetry in the first-order representation. In this paper, we consider blocked Gibbs sampling, an advanced variation of the classic Gibbs sampling algorithm and lift it to the first-order level. We propose to achieve this by partitioning the first-order atoms in the relational model into a set of disjoint clusters such that exact lifted inference is polynomial in each cluster given an assignment to all other atoms not in the cluster. We propose an approach for constructing such clusters and determining their complexity and show how it can be used to trade accuracy with computational complexity in a principled manner. Our experimental evaluation shows that lifted Gibbs sampling is superior to the propositional algorithm in terms of accuracy and convergence.
Q-MKL: Matrix-induced Regularization in Multi-Kernel Learning with Applications to Neuroimaging
Authors: Chris Hinrichs, Vikas Singh, Jiming Peng, Sterling Johnson
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Multiple Kernel Learning (MKL) generalizes SVMs to the setting where one simultaneously trains a linear classifier and chooses an optimal combination of given base kernels. Model complexity is typically controlled using various norm regularizations on the vector of base kernel mixing coefficients. Existing methods, however, neither regularize nor exploit potentially useful information pertaining to how kernels in the input set 'interact'; that is, higher order kernel-pair relationships that can be easily obtained via unsupervised (similarity, geodesics), supervised (correlation in errors), or domain knowledge driven mechanisms (which features were used to construct the kernel?). We show that by substituting the norm penalty with an arbitrary quadratic function Q \succeq 0, one can impose a desired covariance structure on mixing coefficient selection, and use this as an inductive bias when learning the concept. This formulation significantly generalizes the widely used 1- and 2-norm MKL objectives. We explore the model’s utility via experiments on a challenging Neuroimaging problem, where the goal is to predict a subject’s conversion to Alzheimer’s Disease (AD) by exploiting aggregate information from several distinct imaging modalities. Here, our new model outperforms the state of the art (p-values << 10−3 ). We briefly discuss ramifications in terms of learning bounds (Rademacher complexity).
Label Ranking with Partial Abstention based on Thresholded Probabilistic Models
Authors: Weiwei Cheng, Eyke Hüllermeier, Willem Waegeman, Volkmar Welker
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Several machine learning methods allow for abstaining from uncertain predictions. While being common for settings like conventional classification, abstention has been studied much less in learning to rank. We address abstention for the label ranking setting, allowing the learner to declare certain pairs of labels as being incomparable and, thus, to predict partial instead of total orders. In our method, such predictions are produced via thresholding the probabilities of pairwise preferences between labels, as induced by a predicted probability distribution on the set of all rankings. We formally analyze this approach for the Mallows and the Plackett-Luce model, showing that it produces proper partial orders as predictions and characterizing the expressiveness of the induced class of partial orders. These theoretical results are complemented by experiments demonstrating the practical usefulness of the approach.
Weighted Likelihood Policy Search with Model Selection
Authors: Tsuyoshi Ueno, Kohei Hayashi, Takashi Washio, Yoshinobu Kawahara
Published in: Advances in Neural Information Processing Systems 25  (NIPS 2012) (Main Conference Track), 2012

Abstract: Reinforcement learning (RL) methods based on direct policy search (DPS) have been actively discussed to achieve an efficient approach to complicated Markov decision processes (MDPs). Although they have brought much progress in practical applications of RL, there still remains an unsolved problem in DPS related to model  selection for the policy. In this paper, we propose a novel DPS method, {\it  weighted likelihood policy search (WLPS)}, where a policy is efficiently learned through the weighted likelihood estimation. WLPS naturally connects DPS to the statistical inference problem and thus various sophisticated techniques in statistics can be applied to DPS problems directly. Hence, by following the idea of the {\it information criterion}, we develop a new measurement for model comparison in DPS based on the weighted log-likelihood.
Parameterising Feature Sensitive Cell Formation in Linsker Networks in the Auditory System
Authors: Lance C. Walton, David L. Bisset
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper examines and extends the work of Linsker (1986) on  self organising feature detectors. Linsker concentrates on the vi(cid:173) sual processing system, but infers that the weak assumptions made  will allow the model to be used in the processing of other sensory  information. This claim is examined here, with special attention  paid to the auditory system, where there is much lower connec(cid:173) tivity and therefore more statistical variability. On-line training is  utilised, to obtain an idea of training times. These are then com(cid:173) pared to the time available to pre-natal mammals for the formation  of feature sensitive cells.
Learning Spatio-Temporal Planning from a Dynamic Programming Teacher: Feed-Forward Neurocontrol for Moving Obstacle Avoidance
Authors: Gerald Fahner, Rolf Eckmiller
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Within a simple test-bed,  application of feed-forward  neurocontrol  for short-term planning of robot trajectories in a dynamic environ(cid:173) ment  is  studied.  The  action  network  is  embedded  in  a  sensory(cid:173) motoric system architecture that contains  a separate world model.  It is  continuously  fed  with  short-term  predicted  spatio-temporal  obstacle  trajectories,  and  receives  robot  state  feedback.  The  ac(cid:173) tion  net  allows  for  external  switching  between  alternative  plan(cid:173) ning  tasks.  It  generates  goal-directed  motor  actions  - subject  to  the  robot's  kinematic  and  dynamic  constraints  - such  that  colli(cid:173) sions  with  moving obstacles  are  avoided.  Using  supervised  learn(cid:173) ing,  we  distribute  examples  of the  optimal planner  mapping over  a  structure-level  adapted  parsimonious higher  order  network.  The  training  database  is  generated  by  a  Dynamic  Programming algo(cid:173) rithm.  Extensive  simulations reveal,  that  the  local  planner  map(cid:173) ping is  highly nonlinear, but can be effectively  and sparsely repre(cid:173) sented  by  the  chosen  powerful  net model.  Excellent generalization  occurs for unseen obstacle configurations.  We  also discuss the limi(cid:173) tations of feed-forward  neurocontrol for growing planning horizons. 
*Tel.:  (228)-550-364 
FAX:  (228)-550-425 
e-mail:  gerald@nero.uni-bonn.de
Hidden Markov Models in Molecular Biology: New Algorithms and Applications
Authors: Pierre Baldi, Yves Chauvin, Tim Hunkapiller, Marcella A. McClure
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Hidden  Markov Models  (HMMs)  can  be applied  to several impor(cid:173) tant problems in molecular biology.  We introduce a new convergent  learning algorithm for HMMs that, unlike the classical Baum-Welch  algorithm is smooth and can be applied  on-line or in  batch mode,  with or  without the usual Viterbi most  likely  path approximation.  Left-right HMMs with insertion and deletion states are then trained  to represent several protein families including immunoglobulins and  kinases.  In  all cases,  the models derived capture all  the important  statistical  properties  of the families  and  can  be  used  efficiently  in  a  number of important tasks such as multiple alignment, motif de(cid:173) tection,  and  classification. 
*and Division  of Biology,  California  Institute of Technology.  t and Department of Psychology,  Stanford University.
Statistical and Dynamical Interpretation of ISIH Data from Periodically Stimulated Sensory Neurons
Authors: John K. Douglass, Frank Moss, André Longtin
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We interpret the time interval data obtained from periodically stimulated  sensory neurons in terms of two simple dynamical systems driven by noise  with an embedded weak periodic function called the signal: 1) a bistable  system defined by two potential wells separated by a barrier, and 2) a Fit(cid:173) zHugh-Nagumo system. The implementation is by analog simulation: elec(cid:173) tronic circuits which mimic the dynamics. For a given signal frequency, our  simulators have only two adjustable parameters, the signal and noise intensi(cid:173) ties. We show that experimental data obtained from the periodically stimu(cid:173) lated mechanoreceptor in the crayfish tail fan can be accurately approximated  by these simulations. Finally, we discuss stochastic resonance in the two  models.
Spiral Waves in Integrate-and-Fire Neural Networks
Authors: John G. Milton, Po Hsiang Chu, Jack D. Cowan
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The formation of propagating spiral waves is studied in a  randomly  connected neural network composed of integrate-and-fire neurons  with  recovery  period and  excitatory  connections  using  computer  simulations.  Network  activity is  initiated by  periodic stimulation  at a single point.  The results suggest that spiral waves can arise in  such a  network via a  sub-critical Hopf bifurcation.
Adaptive Stimulus Representations: A Computational Theory of Hippocampal-Region Function
Authors: Mark A. Gluck, Catherine E. Myers
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present a theory of cortico-hippocampal interaction in discrimination learning. The  hippocampal region is presumed to form new stimulus representations which facilitate  learning by enhancing the discriminability of predictive stimuli and compressing  stimulus-stimulus redundancies. The cortical and cerebellar regions, which are the sites  of long-term memory. may acquire these new representations but are not assumed to be  capable of forming new representations themselves.  Instantiated as a connectionist  model. this theory accounts for a wide range of trial-level classical conditioning  phenomena in normal (intact) and hippocampal-Iesioned animals. It also makes several  novel predictions which remain to be investigated empirically. The theory implies that  the hippocampal region is involved in even the simplest learning tasks; although  hippocampal-Iesioned animals may be able to use other strategies to learn these tasks. the  theory predicts that they will show consistently different patterns of transfer and  generalization when the task demands change.
Computing with Almost Optimal Size Neural Networks
Authors: Kai-Yeung Siu, Vwani Roychowdhury, Thomas Kailath
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Artificial neural networks are comprised of an interconnected collection  of certain nonlinear devices; examples of commonly used devices include  linear threshold elements, sigmoidal elements and radial-basis elements.  We employ results from harmonic analysis and the theory of rational ap(cid:173) proximation to obtain almost tight lower bounds on the size (i.e. number  of elements) of neural networks. The class of neural networks to which  our techniques can be applied is quite general; it includes any feedforward  network in which each element can be piecewise approximated by a low  degree rational function. For example, we prove that any depth-( d + 1)  network of sigmoidal units or linear threshold elements computing the par(cid:173) ity function of n variables must have O(dnl/d-£) size, for any fixed i > O.  In addition, we prove that this lower bound is almost tight by showing  that the parity function can be computed with O(dnl/d) sigmoidal units  or linear threshold elements in a depth-(d + 1) network. These almost  tight bounds are the first known complexity results on the size of neural  networks with depth more than two. Our lower bound techniques yield  a unified approach to the complexity analysis of various models of neural  networks with feedforward structures. Moreover, our results indicate that  in the context of computing highly oscillating symmetric Boolean func-
Synaptic Weight Noise During MLP Learning Enhances Fault-Tolerance, Generalization and Learning Trajectory
Authors: Alan F. Murray, Peter J. Edwards
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  analyse  the  effects  of analog  noise  on  the  synaptic  arithmetic  during MultiLayer Perceptron training, by expanding the cost func(cid:173) tion to include noise-mediated penalty terms.  Predictions are made  in the light of these  calculations which suggest  that fault tolerance,  generalisation  ability  and  learning  trajectory  should  be  improved  by  such  noise-injection.  Extensive  simulation experiments  on  two  distinct  classification  problems  substantiate  the  claims.  The  re(cid:173) sults  appear to  be  perfectly  general for  all training schemes  where  weights are adjusted incrementally, and have wide-ranging implica(cid:173) tions for  all applications, particularly those involving  "inaccurate"  analog neural  VLSI.
Rational Parametrizations of Neural Networks
Authors: Uwe Helmke, Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A connection is drawn between rational functions, the realization  theory of dynamical systems, and feedforward neural networks.  This allows us to parametrize single hidden layer scalar neural  networks with (almost) arbitrary analytic activation functions in  terms of strictly proper rational functions. Hence, we can solve the  uniqueness of parametrization problem for such networks.
Directional-Unit Boltzmann Machines
Authors: Richard S. Zemel, Christopher K. I. Williams, Michael Mozer
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  present  a  general  formulation  for  a  network  of stochastic  di(cid:173) rectional units.  This formulation is  an extension of the Boltzmann  machine in which  the units are  not  binary, but take on values in a  cyclic  range,  between  0  and 271'  radians.  The state of each  unit  in  a  Directional-Unit  Boltzmann  Machine  (DUBM)  is  described  by  a  complex variable, where  the phase component specifies  a direction;  the  weights  are  also  complex variables.  We  associate  a  quadratic  energy  function,  and  corresponding  probability,  with  each  DUBM  configuration.  The  conditional  distribution  of a  unit's  stochastic  state is  a  circular  version of the Gaussian probability distribution,  known  as  the  von  Mises  distribution.  In  a  mean-field  approxima(cid:173) tion  to  a  stochastic  DUBM,  the  phase  component of a  unit's state  represents  its mean direction,  and the magnitude component spec(cid:173) ifies  the  degree  of certainty  associated  with  this  direction.  This  combination of a  value  and  a  certainty  provides  additional  repre(cid:173) sentational power  in a  unit.  We  describe  a  learning algorithm and  simulations that demonstrate  a  mean-field  DUBM'S  ability to learn  interesting  mappings. 
Many  kinds  of information  can  naturally  be  represented  in  terms  of angular,  or  directional, variables.  A circular range forms a suitable representation for  explicitly  directional  information,  such  as  wind  direction,  as  well  as  for  information  where  the  underlying  range  is  periodic,  such  as  days  of the  week  or  months of the  year.  In  computer  vision,  tangent  fields  and optic flow  fields  are  represented  as fields  of  oriented line segments, each of which can be described by a magnitude and direction.  Directions  can  also  be  used  to represent  a set  of symbolic labels,  e.g.,  object  label  A  at  0,  and  object  label  B  at 71'/2  radians.  We  discuss  below  some  advantages  of  representing  symbolic labels with  directional units.
Stimulus Encoding by Multidimensional Receptive Fields in Single Cells and Cell Populations in V1 of Awake Monkey
Authors: Edward Stern, Ad Aertsen, Eilon Vaadia, Shaul Hochstein
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Ad  Aertsen 
Institut fur  Neuroinfonnatik  Ruhr-Universitat-Bochum
Self-Organizing Rules for Robust Principal Component Analysis
Authors: Lei Xu, Alan L. Yuille
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In  the  presence  of outliers,  the  existing  self-organizing  rules  for  Principal  Component  Analysis  (PCA)  perform  poorly.  Using  sta(cid:173) tistical physics techniques including the Gibbs distribution, binary  decision  fields  and  effective  energies,  we  propose  self-organizing  PCA  rules  which  are  capable  of  resisting  outliers  while  fulfilling  various PCA-related tasks such as obtaining the first  principal com(cid:173) ponent vector,  the first  k  principal component vectors,  and directly  finding  the  subspace  spanned  by  the  first  k  vector  principal  com(cid:173) ponent  vectors  without solving for  each  vector  individually.  Com(cid:173) parative  experiments  have  shown  that  the  proposed  robust  rules  improve  the  performances  of the  existing  PCA  algorithms signifi(cid:173) cantly when outliers are  present.
Intersecting regions: The Key to combinatorial structure in hidden unit space
Authors: Janet Wiles, Mark Ollila
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Hidden units in multi-layer networks form a representation space in which each  region can be identified with a class of equivalent outputs (Elman, 1989) or a  logical state in a finite state machine (Cleeremans, Servan-Schreiber &  McClelland, 1989; Giles, Sun, Chen, Lee, & Chen, 1990). We extend the  analysis of the spatial structure of hidden unit space to a combinatorial task,  based on binding features together in a visual scene. The logical structure  requires a combinatorial number of states to represent all valid scenes. On  analysing our networks, we find that the high dimensionality of hidden unit  space is exploited by using the intersection of neighboring regions to represent  conjunctions of features. These results show how combinatorial structure can  be based on the spatial nature of networks, and not just on their emulation of  logical structure. 
1 TECHNIQUES FOR ANALYSING THE SPATIAL AND  LOGICAL STRUCTURE OF HIDDEN UNIT SPACE 
In multi-layer networks, regions of hidden unit space can be identified with classes of  equivalent outputs. For example, Elman (1989) showed that the hidden unit patterns for  words in simple grammatical sentences cluster into regions, with similar patterns  representing similar grammatical entities. For example, different tokens of the same word  are clustered tightly, indicating that they are represented within a small region. These  regions can be grouped into larger regions, reflecting a hierarchical structure. The largest
A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks
Authors: J. Alspector, R. Meir, B. Yuhas, A. Jayakumar, D. Lippe
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Typical methods for gradient descent in neural network learning involve  calculation of derivatives based on a detailed knowledge of the network  model. This requires extensive, time consuming calculations for each pat(cid:173) tern presentation and high precision that makes it difficult to implement  in VLSI. We present here a perturbation technique that measures, not  calculates, the gradient. Since the technique uses the actual network as  a measuring device, errors in modeling neuron activation and synaptic  weights do not cause errors in gradient descent. The method is parallel  in nature and easy to implement in VLSI. We describe the theory of such  an algorithm, an analysis of its domain of applicability, some simulations  using it and an outline of a hardware implementation.
History-Dependent Attractor Neural Networks
Authors: Isaac Meilijson, Eytan Ruppin
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  present  a  methodological  framework  enabling  a  detailed  de(cid:173) scription  of the  performance of Hopfield-like  attractor  neural  net(cid:173) works  (ANN)  in  the  first  two  iterations.  Using  the  Bayesian  ap(cid:173) proach,  we find  that performance is improved when a history-based  term is included in the neuron's dynamics.  A further enhancement  of the  network's  performance  is  achieved  by  judiciously  choosing  the  censored neurons  (those  which  become active in  a  given  itera(cid:173) tion)  on  the  basis  of the  magnitude of their  post-synaptic  poten(cid:173) tials.  The contribution of biologically plausible,  censored,  history(cid:173) dependent dynamics is especially marked in conditions of low firing  activity  and  sparse  connectivity,  two  important  characteristics  of  the  mammalian  cortex.  In  such  networks,  the  performance  at(cid:173) tained  is  higher  than  the  performance  of two  'independent'  iter(cid:173) ations,  which  represents  an  upper  bound  on  the  performance  of  history-independent  networks.
Single-Iteration Threshold Hamming Networks
Authors: Isaac Meilijson, Eytan Ruppin, Moshe Sipper
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  analyze in  detail  the  performance of a  Hamming network  clas(cid:173) sifying  inputs  that  are  distorted  versions  of one  of its  m  stored  memory patterns.  The activation function  of the memory neurons  in the original Hamming network  is  replaced  by a simple threshold  function.  The resulting Threshold  Hamming Network  (THN)  cor(cid:173) rectly  classifies  the  input  pattern,  with  probability approaching 1,  using only O(mln m)  connections,  in  a single iteration.  The THN  drastically reduces the time and space complexity of Hamming Net(cid:173) work  classifiers.
The Computation of Stereo Disparity for Transparent and for Opaque Surfaces
Authors: Suthep Madarasmi, Daniel Kersten, Ting-Chuen Pong
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  classical  computational  model  for  stereo  vision  incorporates  a  uniqueness  inhibition  constraint  to enforce  a  one-to-one  feature  match, thereby sacrificing the ability to handle transparency.  Crit(cid:173) ics  of the  model  disregard  the  uniqueness  constraint  and  argue  that the smoothness constraint  can provide the excitation support  required  for  transparency  computation.  However,  this  modifica(cid:173) tion  fails  in  neighborhoods  with  sparse  features.  We  propose  a  Bayesian  approach  to  stereo  vision  with  priors  favoring  cohesive  over transparent surfaces.  The disparity and its segmentation into a  multi-layer "depth planes"  representation  are simultaneously com(cid:173) puted.  The smoothness constraint propagates support  within each  layer, providing mutual excitation for non-neighboring transparent  or  partially occluded  regions.  Test  results  for  various  random-dot  and other stereograms are presented.
Forecasting Demand for Electric Power
Authors: Jen-Lun Yuan, Terrence Fine
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  are  developing  a  forecaster  for  daily  extremes  of demand  for  electric  power encountered  in  the service  area of a  large  midwest(cid:173) ern  utility  and  using  this  application  as  a  testbed  for  approaches  to input dimension reduction and decomposition of network train(cid:173) ing.  Projection  pursuit  regression  representations  and  the  ability  of algorithms like SIR to quickly  find  reasonable weighting vectors  enable us  to confront the vexing architecture selection problem  by  reducing  high-dimensional  gradient  searchs  to  fitting  single-input  single-output  (SISO)  subnets.  We  introduce  dimension  reduction  algorithms,  to select  features  or  relevant  subsets of a  set  of many  variables,  based  on  minimizing  an  index  of level-set  dispersions  (closely  related  to  a  projection  index  and  to  SIR),  and  combine  them  with  backfitting  to  implement  a  neural  network  version  of  projection  pursuit.  The  performance  achieved  by  our  approach,  when trained on 1989,  1990 data and tested on  1991  data, is  com(cid:173) parable  to  that  achieved  in  our  earlier  study  of  backpropagation  trained networks.
Visual Motion Computation in Analog VLSI Using Pulses
Authors: Rahul Sarpeshkar, Wyeth Bair, Christof Koch
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  real  time  computation  of  motion  from  real  images  using a  single chip with integrated sensors is a  hard prob(cid:173) lem.  We  present two  analog VLSI schemes that use  pulse  domain neuromorphic circuits to compute motion.  Pulses  of variable width, rather than graded potentials, represent  a  natural  medium  for  evaluating temporal relationships.  Both algorithms measure speed by  timing a  moving edge  in  the image.  Our first  model is  inspired  by  Reichardt's  algorithm in the fiy  and yields  a  non-monotonic response  vs.  velocity  curve.  We  present  data from  a  chip  that  implements  this  model.  Our  second  algorithm  yields  a  monotonic  response  vs.  velocity  curve  and  is  currently  being translated into silicon. 
1
Efficient Pattern Recognition Using a New Transformation Distance
Authors: Patrice Simard, Yann LeCun, John S. Denker
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Memory-based  classification  algorithms such  as  radial  basis  func(cid:173) tions or K-nearest  neighbors typically rely on simple distances (Eu(cid:173) clidean,  dot  product ... ),  which  are  not  particularly  meaningful  on  pattern vectors.  More complex, better suited distance measures are  often  expensive  and  rather  ad-hoc  (elastic  matching,  deformable  templates).  We  propose  a  new  distance  measure which  (a)  can  be  made  locally  invariant to  any  set  of transformations  of the  input  and  (b)  can  be  computed  efficiently.  We  tested  the  method  on  large handwritten character  databases  provided by the  Post  Office  and the NIST.  Using invariances with  respect  to translation, rota(cid:173) tion,  scaling,  shearing and line  thickness,  the  method  consistently  outperformed  all other systems  tested  on  the same databases.
Planar Hidden Markov Modeling: From Speech to Optical Character Recognition
Authors: Esther Levin, Roberto Pieraccini
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We propose in  this paper a  statistical  model  (planar hidden  Markov model  - PHMM)  describing  statistical  properties  of images.  The  model generalizes  the single-dimensional HMM,  used for speech processing, to  the  planar case.  For this model to be useful an efficient segmentation algorithm, similar to the  Viterbi  algorithm  for  HMM,  must exist  We  present conditions  in  terms  of  the  PHMM  parameters  that  are  sufficient  to  guarantee  that  the  planar  segmentation  problem  can  be  solved  in  polynomial  time,  and  describe  an  algorithm for that.  This algorithm aligns optimally the image with the model,  and  therefore  is  insensitive  to  elastic  distortions  of  images.  Using  this  algorithm a joint optima1  segmentation and recognition of the  image  can  be  performed, thus overcoming the  weakness of traditional OCR systems where  segmentation  is  performed  independently  before  the  recognition  leading  to  unrecoverable recognition errors. 
Tbe  PHMM  approach  was  evaluated  using  a  set  of  isolated  band-written  digits.  An  overall  digit  recognition  accuracy  of  95%  was  acbieved.  An  analysis of the results showed  that even in  the  simple case of recognition  of  isolated  characters,  the  elimination  of  elastic  distortions  enhances  the  performance Significantly. We expect that the advantage of this approach  will  be  even  more  such  as  connected  writing  recognition/spotting,  for  whicb  there  is  no  known  high  accuracy  method  of  recognition.
Global Regularization of Inverse Kinematics for Redundant Manipulators
Authors: David DeMers, Kenneth Kreutz-Delgado
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  inverse  kinematics  problem  for  redundant  manipulators  is  ill-posed  and  nonlinear.  There are two fundamentally different issues which result in the need  for  some  form  of regularization;  the  existence  of multiple  solution branches  (global ill-posedness) and the existence of excess degrees of freedom (local ill(cid:173) posedness).  For certain  classes  of manipulators,  learning  methods  applied  to  input-output data generated from  the forward function can  be used  to globally  regularize the problem by partitioning the domain of the forward mapping into  a  finite  set  of regions  over which  the  inverse  problem  is  well-posed.  Local  regularization  can  be accomplished  by  an  appropriate parameterization  of the  redundancy consistently over each region.  As a result, the ill-posed problem can  be transformed into a finite set of well-posed problems. Each can then be solved  separately to construct approximate direct inverse functions.
Object-Based Analog VLSI Vision Circuits
Authors: Christof Koch, Binnal Mathur, Shih-Chii Liu, John G. Harris, Jin Luo, Massimo Sivilotti
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  describe  two successfully  working,  analog  VLSI  vision  circuits  that move beyond pixel-based early vision algorithms.  One circuit,  implementing the dynamic wires model, provides for dedicated lines  of communication  among  groups  of pixels  that  share  a  common  property.  The chip  uses  the  dynamic wires  model to  compute the  arclength of visual contours.  Another circuit labels all points inside  a  given  contour  with one  voltage  and  all other  with  another  volt(cid:173) age.  Its behavior is  very  robust,  since small breaks in  contours are  automatically sealed,  providing  for  Figure-Ground segregation  in  a  noisy  environment.  Both  chips  are implemented using  networks  of resistors  and switches  and  represent  a  step  towards object  level  processing  since  a  single  voltage value encodes  the property  of an  ensemble of pixels. 
1  CONTOUR-LENGTH CHIP 
Contour  length  computation  is  useful  for  further  processing  such  as  structural  saliency  (Shaashua and  Ullman,  1988),  which  is  thought  to be  an important stage  before  object  recognition.  This computation is  impossible on  an  analog chip  if we
Second order derivatives for network pruning: Optimal Brain Surgeon
Authors: Babak Hassibi, David G. Stork
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We investigate the use of information from all second order derivatives of the error  function to perfonn network pruning (i.e., removing unimportant weights from a trained  network) in order to improve generalization, simplify networks, reduce hardware or  storage requirements, increase the speed of further training, and in some cases enable rule  extraction. Our method, Optimal Brain Surgeon (OBS), is Significantly better than  magnitude-based methods and Optimal Brain Damage [Le Cun, Denker and Sol1a, 1990],  which often remove the wrong weights. OBS permits the pruning of more weights than  other methods (for the same error on the training set), and thus yields better  generalization on test data. Crucial to OBS is a recursion relation for calculating the  inverse Hessian matrix H-I from training data and structural information of the net. OBS  permits a 90%, a 76%, and a 62% reduction in weights over backpropagation with weighL  decay on three benchmark MONK's problems [Thrun et aI., 1991]. Of OBS, Optimal  Brain Damage, and magnitude-based methods, only OBS deletes the correct weights from  a trained XOR network in every case. Finally, whereas Sejnowski and Rosenberg [1987J  used 18,000 weights in their NETtalk network, we used OBS to prune a network to just  1560 weights, yielding better generalization.
Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors
Authors: Yann LeCun, Patrice Y. Simard, Barak Pearlmutter
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We propose a very simple, and well principled way of computing  the optimal step size in gradient descent algorithms. The on-line  version is very efficient computationally, and is applicable to large  backpropagation networks trained on large data sets. The main  ingredient is a technique for estimating the principal eigenvalue(s)  and eigenvector(s) of the objective function's second derivative ma(cid:173) trix (Hessian), which does not require to even calculate the Hes(cid:173) sian. Several other applications of this technique are proposed for  speeding up learning, or for eliminating useless parameters.
Computation of Heading Direction from Optic Flow in Visual Cortex
Authors: Markus Lappe, Josef P. Rauschecker
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have designed a neural network which detects the direction of ego(cid:173) motion from optic flow in the presence of eye movements (Lappe and  Rauschecker, 1993). The performance of the network is consistent with  human psychophysical data, and its output neurons show great similarity  to "triple component" cells in area MSTd of monkey visual cortex. We  now show that by using assumptions about the kind of eye movements  that the obsenrer is likely to perform, our model can generate various  other cell types found in MSTd as well.
Statistical Modeling of Cell Assemblies Activities in Associative Cortex of Behaving Monkeys
Authors: Itay Gat, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: So far there has been no general method for relating extracellular  electrophysiological measured activity of neurons in the associative  cortex to underlying network or "cognitive" states. We propose  to model such data using a multivariate Poisson Hidden Markov  Model. We demonstrate the application of this approach for tem(cid:173) poral segmentation of the firing patterns, and for characterization  of the cortical responses to external stimuli. Using such a statisti(cid:173) cal model we can significantly discriminate two behavioral modes  of the monkey, and characterize them by the different firing pat(cid:173) terns, as well as by the level of coherency of their multi-unit firing  activity.  Our study utilized measurements carried out on behaving Rhesus  monkeys by M. Abeles, E. Vaadia, and H. Bergman, of the Hadassa  Medical School of the Hebrew University.