mtybilly/Instruct2DS huggingface dataset:
Neural Network Recognizer for Hand-Written Zip Code Digits
Authors: John S. Denker, W. R. Gardner, Hans Peter Graf, Donnie Henderson, R. E. Howard, W. Hubbard, L. D. Jackel, Henry S. Baird, Isabelle Guyon
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper describes the construction of a system that recognizes hand-printed  digits, using a combination of classical techniques and neural-net methods. The  system has been trained and tested on real-world data, derived from zip codes seen  on actual U.S. Mail. The system rejects a small percentage of the examples as  unclassifiable, and achieves a very low error rate on the remaining examples. The  system compares favorably with other state-of-the art recognizers. While some of  the methods are specific to this task, it is hoped that many of the techniques will  be applicable to a wide range of recognition tasks.
Analog Implementation of Shunting Neural Networks
Authors: Bahram Nabet, Robert B. Darling, Robert B. Pinter
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: An  extremely  compact,  all  analog  and  fully  parallel  implementa(cid:173) tion  of a  class  of shunting  recurrent  neural  networks  that  is  ap(cid:173) plicable to a  wide variety of FET-based integration  technologies is  proposed.  While the contrast enhancement, data compression, and  adaptation to mean input intensity capabilities of the network  are  well suited for  processing of sensory information or feature  extrac(cid:173) tion for a content addressable memory (CAM) system, the network  also admits a global Liapunov function  and can thus achieve stable  CAM storage  itself.  In  addition  the model  can  readily function  as  a front-end  processor to an analog adaptive resonance  circuit.
Range Image Restoration Using Mean Field Annealing
Authors: Griff L. Bilbro, Wesley E. Snyder
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  new  optimization strategy,  Mean  Field  Annealing, is  presented.  Its application to MAP restoration of noisy range images is derived  and experimentally verified.
Neural Approach for TV Image Compression Using a Hopfield Type Network
Authors: Martine Naillon, Jean-Bernard Theeten
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A self-organizing Hopfield network has been  developed in the context of Vector Ouantiza(cid:173) -tion, aiming at compression of  television  images. The metastable states of the spin  glass-like network are used as  an extra  the Minimal Overlap  storage resource using  and Mezard 1987) to  rule (Krauth  learning  the organization of the attractors.  optimize  The sel f-organi zi ng  that we have  scheme  devised  the generation of an  in  adaptive codebook for any qiven TV image.
Convergence and Pattern-Stabilization in the Boltzmann Machine
Authors: Moshe Kam, Roger Cheng
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The Boltzmann Machine has been introduced as a means to perform  global optimization for multimodal objective functions using the  principles of simulated annealing. In this paper we consider its utility  as a spurious-free content-addressable memory, and provide bounds on  its performance in this context. We show how to exploit the machine's  ability to escape local minima, in order to use it, at a constant  temperature, for unambiguous associative pattern-retrieval in noisy  environments. An association rule, which creates a sphere of influence  around each stored pattern, is used along with the Machine's dynamics  to match the machine's noisy input with one of the pre-stored patterns.  Spurious fIxed points, whose regions of attraction are not recognized by  the rule, are skipped, due to the Machine's fInite probability to escape  from any state. The results apply to the Boltzmann machine and to the  asynchronous net of binary threshold elements (Hopfield model'). They  provide the network designer with worst-case and best-case bounds for  the network's performance, and allow polynomial-time tradeoff studies  of design parameters.
Models of Ocular Dominance Column Formation: Analytical and Computational Results
Authors: Kenneth D. Miller, Joseph B. Keller, Michael P. Stryker
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: We  have  previously  developed  a  simple  mathemati(cid:173)
cal model  for  formation  of ocular  dominance  columns  in  mammalian  visual  cortex.  The  model  provides  a  com(cid:173) mon framework  in  which a  variety  of activity-dependent  biological machanisms can be studied.  Analytic and com(cid:173) putational  results  together  now  reveal  the  following:  if  inputs  specific  to  each eye  are  locally  correlated  in  their  firing,  and are not  anticorrelated within an arbor radius,  monocular  cells  will  robustly  form  and  be  organized  by  intra-cortical  interactions  into  columns.  Broader  corre(cid:173) lations  withln  each  eye,  or anti-correlations  between the  eyes, create a  more purely monocular cortex; positive cor(cid:173) relation  over  an  arbor  radius  yields  an  almost  perfectly  monocular cortex.  Most features of the model can be un(cid:173) derstood  analytically  through  decomposition  into  eigen(cid:173) functions and linear stability analysis.  This allows predic(cid:173) tion of the widths of the columns and other features from  measurable biological parameters.
Scaling and Generalization in Neural Networks: A Case Study
Authors: Subutai Ahmad, Gerald Tesauro
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The  issues  of scaling  and  generalization  have  emerged  as  key  issues  in  current studies of supervised learning from examples in neural networks.  Questions such  as  how  many  training  patterns  and  training  cycles  are  needed for  a problem of a given size  and difficulty,  how  to represent the  inllUh  and how  to choose useful training exemplars,  are of considerable  theoretical  and  practical  importance.  Several  intuitive  rules  of thumb  have been obtained from empirical studies, but as yet there are few  rig(cid:173) orous  results.  In  this  paper we  summarize  a  study Qf generalization in  the simplest possible case-perceptron networks learning linearly separa(cid:173) ble  functions.  The  task  chosen  was  the majority function  (i.e.  return  a  1  if a  majority  of the  input  units  are  on),  a  predicate  with  a  num(cid:173) ber  of useful  properties.  We  find  that  many  aspects  of.generalization  in  multilayer  networks  learning  large,  difficult  tasks  are  reproduced  in  this simple domain, in which  concrete numerical results and even some  analytic understanding can be achieved.
A Model of Neural Oscillator for a Unified Submodule
Authors: Alexandr B. Kirillov, G. N. Borisyuk, R. M. Borisyuk, Ye. I. Kovalenko, V. I. Makarenko, V. A. Chulaevsky, V. I. Kryukov
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Abstract Unavailable
Neural Analog Diffusion-Enhancement Layer and Spatio-Temporal Grouping in Early Vision
Authors: Allen M. Waxman, Michael Seibert, Robert K. Cunningham, Jian Wu
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A  new  class of neural  network  aimed  at  early  visual  processing  is  described; we call it a Neural  Analog Diffusion-Enhancement Layer or  "NADEL." The  network  consists  of two  levels  which  are  coupled  through feedfoward and shunted feedback connections. The lower level  is  a  two-dimensional  diffusion map which  accepts  visual  features  as  input, and spreads activity over larger scales as a function of time. The  upper layer is periodically fed the  activity from  the diffusion layer and  locates local maxima in it (an extreme form  of contrast enhancement)  using a network of local comparators. These local maxima are fed back  to  the  diffusion  layer  using  an  on-center/off-surround  shunting  anatomy. The maxima are also available  as output of the network.  The  network dynamics  serves  to  cluster features  on  multiple  scales  as  a  function of time, and can be used in a variety of early visual processing  tasks such  as:  extraction of comers  and high  curvature  points  along  edge contours, line end detection, gap filling in contours, generation of  fixation points, perceptual grouping on multiple scales, correspondence  and path impletion  in long-range  apparent  motion,  and building  2-D  shape representations that are invariant to  location, orientation, scale,  and small deformation on the visual field.
Optimization by Mean Field Annealing
Authors: Griff Bilbro, Reinhold Mann, Thomas K. Miller, Wesley E. Snyder, David E. van den Bout, Mark White
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Nearly optimal solutions to many combinatorial problems can be  found using stochastic simulated annealing. This paper extends  the concept of simulated annealing from its original formulation  as a Markov process to a new formulation based on mean field  theory. Mean field annealing essentially replaces the discrete de(cid:173) grees of freedom in simulated annealing with their average values  as computed by the mean field approximation. The net result is  that equilibrium at a given temperature is achieved 1-2 orders of  magnitude faster than with simulated annealing. A general frame(cid:173) work for the mean field annealing algorithm is derived, and its re(cid:173) lationship to Hopfield networks is shown. The behavior of MFA is  examined both analytically and experimentally for a generic combi(cid:173) natorial optimization problem: graph bipartitioning. This analysis  indicates the presence of critical temperatures which could be im(cid:173) portant in improving the performance of neural networks. 
STOCHASTIC VERSUS MEAN FIELD 
In combinatorial optimization problems, an objective function or Hamiltonian,  H(s), is presented which depends on a vector of interacting 3pim, S = {81," .,8N},  in some complex nonlinear way. Stochastic simulated annealing (SSA) (S. Kirk(cid:173) patrick, C. Gelatt, and M. Vecchi (1983)) finds a global minimum of H by com(cid:173) bining gradient descent with a random process. This combination allows, under  certain conditions, choices of s which actually increa3e H, thus providing SSA with  a mechanism for escaping from local minima. The frequency and severity of these  uphill moves is reduced by slowly decreasing a parameter T (often referred to as  the temperature) such that the system settles into a global optimum. 
Two conceptual operationo; are involved in simulated annealing: a thermodatic op(cid:173) eration which schedules decreases in the temperature, and a relazation operation 
92
Constraints on Adaptive Networks for Modeling Human Generalization
Authors: Mark A. Gluck, M. Pavel, Van Henkle
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: The potential of adaptive  networks  to learn categorization rules and to  model  human  performance  is  studied  by  comparing  how  natural  and  artificial systems respond to new inputs, i.e., how they generalize.  Like  humans,  networks  can  learn  a  detenninistic  categorization  task  by  a  variety  of  alternative  individual  solutions.  An  analysis  of  the  con(cid:173) straints imposed by using networks with the minimal number of hidden  units  shows  that  this  "minimal  configuration"  constraint  is  not  sufficient to explain and predict human performance;  only a few  solu(cid:173) tions  were  found  to be  shared by both  humans and  minimal  adaptive  networks.  A  further  analysis  of human  and  network  generalizations  indicates  that  initial  conditions  may  provide  important constraints  on  generalization.  A new  technique,  which  we  call  "reversed learning",  is described for finding appropriate initial conditions.
Self Organizing Neural Networks for the Identification Problem
Authors: Manoel Fernando Tenorio, Wei-Tsih Lee
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This  work  introduces  a  new  method called Self Organizing  Neural  Network  (SONN)  algorithm  and  demonstrates  its  use  in  a  system  identification  task.  The  algorithm  constructs  the  network,  chooses the neuron functions, and adjusts the weights. It is compared to  the Back-Propagation algorithm in the identification of the chaotic time  series.  The  results  shows  that  SONN  constructs  a  simpler,  more  accurate model. requiring less training data and epochs. The algorithm  can be applied and generalized to appilications as a classifier.
Learning with Temporal Derivatives in Pulse-Coded Neuronal Systems
Authors: David B. Parker, Mark Gluck, Eric S. Reifsnider
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: A number of learning models have recently been proposed which  involve calculations of temporal differences (or derivatives in  continuous-time models). These models. like most adaptive network  models. are formulated in tenns of frequency (or activation), a useful  abstraction of neuronal firing rates. To more precisely evaluate the  implications of a neuronal model. it may be preferable to develop a  model which transmits discrete pulse-coded information. We point out  that many functions and properties of neuronal processing and learning  may depend. in subtle ways. on the pulse-coded nature of the informa(cid:173) tion coding and transmission properties of neuron systems. When com(cid:173) pared to formulations in terms of activation. computing with temporal  derivatives (or differences) as proposed by Kosko (1986). Klopf  (1988). and Sutton (1988). is both more stable and easier when refor(cid:173) mulated for a more neuronally realistic pulse-coded system. In refor(cid:173) mulating these models in terms of pulse-coding. our motivation has  been to enable us to draw further parallels and connections between  real-time behavioral models of learning and biological circuit models  of the substrates underlying learning and memory.
Using Backpropagation with Temporal Windows to Learn the Dynamics of the CMU Direct-Drive Arm II
Authors: Kenneth Y. Goldberg, Barak A. Pearlmutter
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: Computing the inverse dynamics  of a robot ann is an active area of research  in the control literature.  We hope to  learn the  inverse dynamics  by training  a neural network on the  measured  response of a physical ann.  The  input  to  the  network is  a  temporal  window of measured positions;  output is  a vector  of torques.  We  train  the  network on  data measured from  the  first  two joints  of the CMU Direct-Drive Arm II  as  it moves  through a randomly-generated  sample  of "pick-and-place"  trajectories.  We  then  test  generalization  with  a  new  trajectory  and  compare  its  output  with  the  torque  measured  at  the  physical arm.  The network  is  shown  to  generalize with  a root mean  square  error/standard deviation  (RMSS)  of 0.10.  We  interpreted the weights  of the  network in tenns of the velocity and acceleration filters  used in  conventional  control  theory.
Storing Covariance by the Associative Long-Term Potentiation and Depression of Synaptic Strengths in the Hippocampus
Authors: Patric K. Stanton, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: In modeling studies or memory based on neural networks, both the selective  enhancement and depression or synaptic strengths are required ror effident storage  or inrormation (Sejnowski, 1977a,b; Kohonen, 1984; Bienenstock et aI, 1982;  Sejnowski and Tesauro, 1989). We have tested this assumption in the hippocampus,  a cortical structure or the brain that is involved in long-term memory. A brier,  high-frequency activation or excitatory synapses in the hippocampus produces an  increase in synaptic strength known as long-term potentiation, or L TP (BUss and  Lomo, 1973), that can last ror many days. LTP is known to be Hebbian since it  requires the simultaneous release or neurotransmitter from presynaptic terminals  coupled with postsynaptic depolarization (Kelso et al, 1986; Malinow and Miller,  1986; Gustatrson et al, 1987). However, a mechanism ror the persistent reduction or  synaptic strength that could balance LTP has not yet been demonstrated. We stu(cid:173) died the associative interactions between separate inputs onto the same dendritic  trees or hippocampal pyramidal cells or field CAl, and round that a low-frequency  input which, by itselr, does not persistently change synaptic strength, can either  increase (associative L TP) or decrease in strength (associative long-term depression  or LTD) depending upon whether it is positively or negatively correlated in time  with a second, high-frequency bursting input. LTP or synaptic strength is Hebbian,  and LTD is anti-Hebbian since it is elicited by pairing presynaptic firing with post(cid:173) synaptic hyperpolarization sufficient to block postsynaptic activity. Thus, associa(cid:173) tive L TP and associative L TO are capable or storing inrormation contained in the  covariance between separate, converging hippocampal inputs • 
• Present address: Dep~ents of NeW'Oscience and Neurology, Albert Einstein College  of Medicine, 1410 Pelham Parkway South, Bronx, NY 10461 USA. 
tPresent address: Computational Neurobiology Laboratory, The Salk Institute, P.O. Box  85800, San Diego, CA 92138 USA. 
Storing Covariance by Synaptic Strengths in the Hippocampus 
395
Dynamic, Non-Local Role Bindings and Inferencing in a Localist Network for Natural Language Understanding
Authors: Trent E. Lange, Michael G. Dyer
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This  paper introduces  a means to  handle the critical problem  of non(cid:173) local  role-bindings  in  localist  spreading-activation  networks.  Every  conceptual node in the network broadcasts a stable, uniquely-identifying  activation pattern, called its signature.  A dynamic role-binding is cre(cid:173) ated  when  a  role's  binding  node  has  an  activation  that  matches  the  bound concept's signature.  Most importantly, signatures are propagated  across long paths of nodes to handle the non-local role-bindings neces(cid:173) sary  for  inferencing.  Our  localist  network  model,  ROBIN  (ROle  Binding  and  Inferencing  Network),  uses  signature  activations  to  ro(cid:173) bustly represent schemata role-bindings and thus perfonn the inferenc(cid:173) ing, plan/goal analysis,  schema instantiation, word-sense disambigua(cid:173) tion, and dynamic re-interpretation portions of the natural language un(cid:173) derstanding process.
Fixed Point Analysis for Recurrent Networks
Authors: Patrice Y. Simard, Mary B. Ottaway, Dana H. Ballard
Published in: Advances in Neural Information Processing Systems 1  (NIPS 1988) (Main Conference Track), 1988

Abstract: This paper provides a systematic analysis of the recurrent backpropaga(cid:173) tion (RBP) algorithm, introducing a number of new results. The main  limitation of the RBP algorithm is that it assumes the convergence of  the network to a stable fixed point in order to backpropagate the error  signals. We show by experiment and eigenvalue analysis that this condi(cid:173) tion can be violated and that chaotic behavior can be avoided. Next we  examine the advantages of RBP over the standard backpropagation al(cid:173) gorithm. RBP is shown to build stable fixed points corresponding to the  input patterns. This makes it an appropriate tool for content address(cid:173) able memories, one-to-many function learning, and inverse problems.
From Stochastic Mixability to Fast Rates
Authors: Nishant A. Mehta, Robert C. Williamson
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell, \mathcal{F},\mathsf{P})$ ERM can have slow $(1/\sqrt{n})$ or fast $(1/n)$ rates of convergence of the excess risk as a function of the sample size $n$. There exist several results that give sufficient conditions for fast rates in terms of joint properties of $\ell$, $\mathcal{F}$, and $\mathsf{P}$, such as the margin condition and the Bernstein condition. In the non-statistical prediction with expert advice setting, there is an analogous slow and fast rate phenomenon, and it is entirely characterized in terms of the mixability of the loss $\ell$ (there being no role there for $\mathcal{F}$ or $\mathsf{P}$). The notion of stochastic mixability builds a bridge between these two models of learning, reducing to classical mixability in a special case. The present paper presents a direct proof of fast rates for ERM in terms of stochastic mixability of $(\ell,\mathcal{F}, \mathsf{P})$, and in so doing provides new insight into the fast-rates phenomenon. The proof exploits an old result of Kemperman on the solution to the general moment problem. We also show a partial converse that suggests a characterization of fast rates for ERM in terms of stochastic mixability is possible.
Active Regression by Stratification
Authors: Sivan Sabato, Remi Munos
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We propose a new active learning algorithm for parametric linear regression with random design. We provide finite sample convergence guarantees for general distributions in the misspecified model. This is the first active learner for this setting that provably can improve over passive learning. Unlike other learning settings (such as classification), in regression the passive learning rate of O(1/epsilon) cannot in general be improved upon. Nonetheless, the so-called `constant' in the rate of convergence, which is characterized by a distribution-dependent risk, can be improved in many cases. For a given distribution, achieving the optimal risk requires prior knowledge of the distribution. Following the stratification technique advocated in Monte-Carlo function integration, our active learner approaches a the optimal risk using piecewise constant approximations.
Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition
Authors: Hanie Sedghi, Anima Anandkumar, Edmond Jonckheere
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: In this paper, we consider a multi-step version of the stochastic ADMM method with efficient guarantees for high-dimensional problems. We first analyze the simple setting, where the optimization problem consists of a loss function and a single regularizer (e.g. sparse optimization), and then extend to the multi-block setting with multiple regularizers and multiple variables (e.g. matrix decomposition into sparse and low rank components). For the sparse optimization problem, our method achieves the minimax rate of $O(s\log d/T)$ for $s$-sparse problems in $d$ dimensions in $T$ steps, and is thus, unimprovable by any method up to constant factors. For the matrix decomposition problem with a general loss function, we analyze the multi-step ADMM with multiple blocks. We establish $O(1/T)$ rate and efficient scaling as the size of matrix grows. For natural noise models (e.g. independent noise), our convergence rate is minimax-optimal. Thus, we establish tight convergence guarantees for multi-block ADMM in high dimensions. Experiments show that for both sparse optimization and matrix decomposition problems, our algorithm outperforms the state-of-the-art methods.
Spatio-temporal Representations of Uncertainty in Spiking Neural Networks
Authors: Cristina Savin, Sophie Deneve
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: It has been long argued that, because of inherent ambiguity and noise, the brain needs to represent uncertainty in the form of probability distributions. The neural encoding of such distributions remains however highly controversial. Here we present a novel circuit model for representing multidimensional real-valued distributions using a spike based spatio-temporal code. Our model combines the computational advantages of the currently competing models for probabilistic codes and exhibits realistic neural responses along a variety of classic measures. Furthermore, the model highlights the challenges associated with interpreting neural activity in relation to behavioral uncertainty and points to alternative population-level approaches for the experimental validation of distributed representations.
Biclustering Using Message Passing
Authors: Luke O'Connor, Soheil Feizi
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Biclustering is the analog of clustering on a bipartite graph. Existent methods infer biclusters through local search strategies that find one cluster at a time; a common technique is to update the row memberships based on the current column memberships, and vice versa. We propose a biclustering algorithm that maximizes a global objective function using message passing. Our objective function closely approximates a general likelihood function, separating a cluster size penalty term into row- and column-count penalties. Because we use a global optimization framework, our approach excels at resolving the overlaps between biclusters, which are important features of biclusters in practice. Moreover, Expectation-Maximization can be used to learn the model parameters if they are unknown. In simulations, we find that our method outperforms two of the best existing biclustering algorithms, ISA and LAS, when the planted clusters overlap. Applied to three gene expression datasets, our method finds coregulated gene clusters that have high quality in terms of cluster size and density.
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Authors: Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.
Clustered factor analysis of multineuronal spike data
Authors: Lars Buesing, Timothy A. Machado, John P. Cunningham, Liam Paninski
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: High-dimensional, simultaneous recordings of neural spiking activity are often explored, analyzed and visualized with the help of latent variable or factor models. Such models are however ill-equipped to extract structure beyond shared, distributed aspects of firing activity across multiple cells. Here, we extend unstructured factor models by proposing a model that discovers subpopulations or groups of cells from the pool of recorded neurons. The model combines aspects of mixture of factor analyzer models for capturing clustering structure, and aspects of latent dynamical system models for capturing temporal dependencies. In the resulting model, we infer the subpopulations and the latent factors from data using variational inference and model parameters are estimated by Expectation Maximization (EM). We also address the crucial problem of initializing parameters for EM by extending a sparse subspace clustering algorithm to integer-valued spike count observations. We illustrate the merits of the proposed model by applying it to calcium-imaging data from spinal cord neurons, and we show that it uncovers meaningful clustering structure in the data.
Gaussian Process Volatility Model
Authors: Yue Wu, José Miguel Hernández Lobato, Zoubin Ghahramani
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: The prediction of time-changing variances is an important task in the modeling of financial data. Standard econometric models are often limited as they assume rigid functional relationships for the evolution of the variance. Moreover, functional parameters are usually learned by maximum likelihood, which can lead to overfitting. To address these problems we introduce GP-Vol, a novel non-parametric model for time-changing variances based on Gaussian Processes. This new model can capture highly flexible functional relationships for the variances. Furthermore, we introduce a new online algorithm for fast inference in GP-Vol. This method is much faster than current offline inference procedures and it avoids overfitting problems by following a fully Bayesian approach. Experiments with financial data show that GP-Vol performs significantly better than current standard alternatives.
Distributed Estimation, Information Loss and Exponential Families
Authors: Qiang Liu, Alexander Ihler
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE, based on the whole dataset jointly. We study the statistical properties of this framework, showing that the loss of efficiency compared to the global setting relates to how much the underlying distribution families deviate from full exponential families, drawing connection to the theory of information loss by Fisher, Rao and Efron. We show that the full-exponential-family-ness" represents the lower bound of the error rate of arbitrary combinations of local MLEs, and is achieved by a KL-divergence-based combination method but not by a more common linear combination method. We also study the empirical properties of the KL and linear combination methods, showing that the KL method significantly outperforms linear combination in practical settings with issues such as model misspecification, non-convexity, and heterogeneous data partitions."
Cone-Constrained Principal Component Analysis
Authors: Yash Deshpande, Andrea Montanari, Emile Richard
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Estimating a vector from noisy quadratic observations is a task that arises naturally in many contexts, from dimensionality reduction, to synchronization and phase retrieval problems. It is often the case that additional information is available about the unknown vector (for instance, sparsity, sign or magnitude of its entries). Many authors propose non-convex quadratic optimization problems that aim at exploiting optimally this information. However, solving these problems is typically NP-hard. We consider a simple model for noisy quadratic observation of an unknown vector $\bvz$. The unknown vector is constrained to belong to a cone $\Cone \ni \bvz$. While optimal estimation appears to be intractable for the general problems in this class, we provide evidence that it is tractable when $\Cone$ is a convex cone with an efficient projection. This is surprising, since the corresponding optimization problem is non-convex and --from a worst case perspective-- often NP hard. We characterize the resulting minimax risk in terms of the statistical dimension of the cone $\delta(\Cone)$. This quantity is already known to control the risk of estimation from gaussian observations and random linear measurements. It is rather surprising that the same quantity plays a role in the estimation risk from quadratic measurements.
Dynamic Rank Factor Model for Text Streams
Authors: Shaobo Han, Lin Du, Esther Salazar, Lawrence Carin
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We propose a semi-parametric and dynamic rank factor model for topic modeling, capable of (1) discovering topic prevalence over time, and (2) learning contemporary multi-scale dependence structures, providing topic and word correlations as a byproduct. The high-dimensional and time-evolving ordinal/rank observations (such as word counts), after an arbitrary monotone transformation, are well accommodated through an underlying dynamic sparse factor model. The framework naturally admits heavy-tailed innovations, capable of inferring abrupt temporal jumps in the importance of topics. Posterior inference is performed through straightforward Gibbs sampling, based on the forward-filtering backward-sampling algorithm. Moreover, an efficient data subsampling scheme is leveraged to speed up inference on massive datasets. The modeling framework is illustrated on two real datasets: the US State of the Union Address and the JSTOR collection from Science.
Online combinatorial optimization with stochastic decision sets and adversarial losses
Authors: Gergely Neu, Michal Valko
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Most work on sequential learning assumes a fixed set of actions that are available all the time. However, in practice, actions can consist of picking subsets of readings from sensors that may break from time to time, road segments that can be blocked or goods that are out of stock. In this paper we study learning algorithms that are able to deal with stochastic availability of such unreliable composite actions. We propose and analyze algorithms based on the Follow-The-Perturbed-Leader prediction method for several learning settings differing in the feedback provided to the learner. Our algorithms rely on a novel loss estimation technique that we call Counting Asleep Times. We deliver regret bounds for our algorithms for the previously studied full information and (semi-)bandit settings, as well as a natural middle point between the two that we call the restricted information setting. A special consequence of our results is a significant improvement of the best known performance guarantees achieved by an efficient algorithm for the sleeping bandit problem with stochastic availability. Finally, we evaluate our algorithms empirically and show their improvement over the known approaches.
Magnitude-sensitive preference formation`
Authors: Nisheeth Srivastava, Edward Vul, Paul R Schrater
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Our understanding of the neural computations that underlie the ability of animals to choose among options has advanced through a synthesis of computational modeling, brain imaging and behavioral choice experiments. Yet, there remains a gulf between theories of preference learning and accounts of the real, economic choices that humans face in daily life, choices that are usually between some amount of money and an item. In this paper, we develop a theory of magnitude-sensitive preference learning that permits an agent to rationally infer its preferences for items compared with money options of different magnitudes. We show how this theory yields classical and anomalous supply-demand curves and predicts choices for a large panel of risky lotteries. Accurate replications of such phenomena without recourse to utility functions suggest that the theory proposed is both psychologically realistic and econometrically viable.
Sparse PCA via Covariance Thresholding
Authors: Yash Deshpande, Andrea Montanari
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: In sparse principal component analysis we are given noisy observations of a low-rank matrix of dimension $n\times p$ and seek to reconstruct it under additional sparsity assumptions. In particular, we assume here that the principal components $\bv_1,\dots,\bv_r$ have at most $k_1, \cdots, k_q$ non-zero entries respectively, and study the high-dimensional regime in which $p$ is of the same order as $n$. In an influential paper, Johnstone and Lu \cite{johnstone2004sparse} introduced a simple algorithm that estimates the support of the principal vectors $\bv_1,\dots,\bv_r$ by the largest entries in the diagonal of the empirical covariance. This method can be shown to succeed with high probability if $k_q \le C_1\sqrt{n/\log p}$, and to fail with high probability if $k_q\ge C_2 \sqrt{n/\log p}$ for two constants $0 < C_1,C_2 < \infty$. Despite a considerable amount of work over the last ten years, no practical algorithm exists with provably better support recovery guarantees. Here we analyze a covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler and Vilenchik \cite{KrauthgamerSPCA}. We confirm empirical evidence presented by these authors and rigorously prove that the algorithm succeeds with high probability for $k$ of order $\sqrt{n}$. Recent conditional lower bounds \cite{berthet2013computational} suggest that it might be impossible to do significantly better. The key technical component of our analysis develops new bounds on the norm of kernel random matrices, in regimes that were not considered before.
Online Optimization for Max-Norm Regularization
Authors: Jie Shen, Huan Xu, Ping Li
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Max-norm regularizer has been extensively studied in the last decade as it promotes an effective low rank estimation of the underlying data. However, max-norm regularized problems are typically formulated and solved in a batch manner, which prevents it from processing big data due to possible memory bottleneck. In this paper, we propose an online algorithm for solving max-norm regularized problems that is scalable to large problems. Particularly, we consider the matrix decomposition problem as an example, although our analysis can also be applied in other problems such as matrix completion. The key technique in our algorithm is to reformulate the max-norm into a matrix factorization form, consisting of a basis component and a coefficients one. In this way, we can solve the optimal basis and coefficients alternatively. We prove that the basis produced by our algorithm converges to a stationary point asymptotically. Experiments demonstrate encouraging results for the effectiveness and robustness of our algorithm. See the full paper at arXiv:1406.3190.
Optimizing Energy Production Using Policy Search and Predictive State Representations
Authors: Yuri Grinberg, Doina Precup, Michel Gendreau
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We consider the challenging practical problem of optimizing the power production of a complex of hydroelectric power plants, which involves control over three continuous action variables, uncertainty in the amount of water inflows and a variety of constraints that need to be satisfied. We propose a policy-search-based approach coupled with predictive modelling to address this problem. This approach has some key advantages compared to other alternatives, such as dynamic programming: the policy representation and search algorithm can conveniently incorporate domain knowledge; the resulting policies are easy to interpret, and the algorithm is naturally parallelizable. Our algorithm obtains a policy which outperforms the solution found by dynamic programming both quantitatively and qualitatively.
Dependent nonparametric trees for dynamic hierarchical clustering
Authors: Avinava Dubey, Qirong Ho, Sinead Williamson, Eric P. Xing
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: Hierarchical clustering methods offer an intuitive and powerful way to model a wide variety of data sets. However, the assumption of a fixed hierarchy is often overly restrictive when working with data generated over a period of time: We expect both the structure of our hierarchy, and the parameters of the clusters, to evolve with time. In this paper, we present a distribution over collections of time-dependent, infinite-dimensional trees that can be used to model evolving hierarchies, and present an efficient and scalable algorithm for performing approximate inference in such a model. We demonstrate the efficacy of our model and inference algorithm on both synthetic data and real-world document corpora.
Kernel Mean Estimation via Spectral Filtering
Authors: Krikamol Muandet, Bharath Sriperumbudur, Bernhard Schölkopf
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: The problem of estimating the kernel mean in a reproducing kernel Hilbert space (RKHS) is central to kernel methods in that it is used by classical approaches (e.g., when centering a kernel PCA matrix), and it also forms the core inference step of modern kernel methods (e.g., kernel-based non-parametric tests) that rely on embedding probability distributions in RKHSs. Previous work [1] has shown that shrinkage can help in constructing “better” estimators of the kernel mean than the empirical estimator. The present paper studies the consistency and admissibility of the estimators in [1], and proposes a wider class of shrinkage estimators that improve upon the empirical estimator by considering appropriate basis functions. Using the kernel PCA basis, we show that some of these estimators can be constructed using spectral filtering algorithms which are shown to be consistent under some technical assumptions. Our theoretical analysis also reveals a fundamental connection to the kernel-based supervised learning framework. The proposed estimators are simple to implement and perform well in practice.
Beyond Disagreement-Based Agnostic Active Learning
Authors: Chicheng Zhang, Kamalika Chaudhuri
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We study agnostic active learning, where the goal is to learn a classifier in a pre-specified hypothesis class interactively with as few label queries as possible, while making no assumptions on the true function generating the labels. The main algorithms for this problem are {\em{disagreement-based active learning}}, which has a high label requirement, and {\em{margin-based active learning}}, which only applies to fairly restricted settings. A major challenge is to find an algorithm which achieves better label complexity, is consistent in an agnostic setting, and applies to general classification problems. In this paper, we provide such an algorithm. Our solution is based on two novel contributions -- a reduction from consistent active learning to confidence-rated prediction with guaranteed error, and a novel confidence-rated predictor.
Distance-Based Network Recovery under Feature Correlation
Authors: David Adametz, Volker Roth
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We present an inference method for Gaussian graphical models when only pairwise distances of n objects are observed. Formally, this is a problem of estimating an n x n covariance matrix from the Mahalanobis distances dMH(xi, xj), where object xi lives in a latent feature space. We solve the problem in fully Bayesian fashion by integrating over the Matrix-Normal likelihood and a Matrix-Gamma prior; the resulting Matrix-T posterior enables network recovery even under strongly correlated features. Hereby, we generalize TiWnet, which assumes Euclidean distances with strict feature independence. In spite of the greatly increased flexibility, our model neither loses statistical power nor entails more computational cost. We argue that the extension is highly relevant as it yields significantly better results in both synthetic and real-world experiments, which is successfully demonstrated for a network of biological pathways in cancer patients.
Inference by Learning: Speeding-up Graphical Model Optimization via a Coarse-to-Fine Cascade of Pruning Classifiers
Authors: Bruno Conejo, Nikos Komodakis, Sebastien Leprince, Jean Philippe Avouac
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We propose a general and versatile framework that significantly speeds-up graphical model optimization while maintaining an excellent solution accuracy. The proposed approach, refereed as Inference by Learning or IbyL, relies on a multi-scale pruning scheme that progressively reduces the solution space by use of a coarse-to-fine cascade of learnt classifiers. We thoroughly experiment with classic computer vision related MRF problems, where our novel framework constantly yields a significant time speed-up (with respect to the most efficient inference methods) and obtains a more accurate solution than directly optimizing the MRF. We make our code available on-line.
A Complete Variational Tracker
Authors: Ryan Turner, Steven Bottone, Bhargav Avasarala
Published in: Advances in Neural Information Processing Systems 27  (NIPS 2014) (Main Conference Track), 2014

Abstract: We introduce a novel probabilistic tracking algorithm that incorporates combinatorial data association constraints and model-based track management using variational Bayes. We use a Bethe entropy approximation to incorporate data association constraints that are often ignored in previous probabilistic tracking algorithms. Noteworthy aspects of our method include a model-based mechanism to replace heuristic logic typically used to initiate and destroy tracks, and an assignment posterior with linear computation cost in window length as opposed to the exponential scaling of previous MAP-based approaches. We demonstrate the applicability of our method on radar tracking and computer vision problems.