mtybilly/Instruct2DS huggingface dataset:
A Connectionist Learning Approach to Analyzing Linguistic Stress
Authors: David S. Touretzky Prahlad Gupta
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We use connectionist modeling to develop an analysis of stress systems in terms  of ease  of learnability.  In traditional linguistic analyses,  learnability arguments  determine default parameter settings based on the feasibilty of logicall y deducing  correct settings from an initial state.  Our approach provides an empirical alter(cid:173) native to such arguments.  Based on perceptron learning experiments using data  from  nineteen  human  languages,  we  develop a  novel  characterization of stress  patterns in terms of six parameters.  These provide both a partial description of the  stress pattern itself and a prediction of its learnability, without invoking abstract  theoretical constructs  such  as  metrical  feet.  This  work  demonstrates  that ma(cid:173) chine learning methods can provide a fresh  approach to understanding linguistic  phenomena. 
1  LINGUISTIC STRESS 
The domain of stress systems in language is considered to have a relatively good linguistic  theory,  called metrical phonologyl.  In this  theory,  the stress  patterns of many  languages  can be described concisely, and characterized in terms of a set of linguistic "parameters,"  such as  bounded vs.  unbounded metrical feet,  left vs.  right dominant feet,  etc.2  In many  languages, stress tends to be placed on certain kinds of syllables rather than on others; the  former are termed heavy syllables, and the latter light syllables. Languages that distinguish 
lFor an overview of the theory, see [Goldsmith 90, chapter 4].  2See [Dresher 90]  for one such parameter scheme.
Stationarity of Synaptic Coupling Strength Between Neurons with Nonstationary Discharge Properties
Authors: Eric D. Young Mark R. Sydorenko
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Based on a general non-stationary point process model, we computed estimates of  the synaptic coupling strength (efficacy) as a function of time after stimulus onset  between an inhibitory interneuron and its target postsynaptic cell in the feline dorsal  cochlear nucleus. The data consist of spike trains from pairs of neurons responding  to brief tone bursts recorded in vivo. Our results suggest that the synaptic efficacy is  non-stationary. Further. synaptic efficacy is shown to be inversely and  approximately linearly related to average presynaptic spike rate. A second-order  analysis suggests that the latter result is not due to non-linear interactions. Synaptic  efficacy is less strongly correlated with postsynaptic rate and the correlation is not  consistent across neural pairs.
Time-Warping Network: A Hybrid Framework for Speech Recognition
Authors: Roberto Pieraccini, Enrico Bocchieri Esther Levin
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: interest  has  been  generated  regarding  speech  Recently.  much  recognition  systems  based  on  Hidden  Markov  Models  (HMMs)  and  neural  network  (NN)  hybrids.  Such  systems  attempt  to  combine  the  best  features  of  both  models:  the  temporal  structure  of HMMs  and  the  discriminative  power  of neural  networks.  In this  work  we  define  a  time-warping  (1W)  neuron  that extends  the  operation of the  fonnal  neuron  of a  back-propagation  network by  warping  the input pattern  to  match  it  optimally  to  its  weights.  We  show  that  a  single-layer  network  of TW  neurons  is  equivalent  to  a  Gaussian  density  HMM(cid:173) the  based  discriminative  power  of  this  system  by  using  back-propagation  discriminative  training.  and/or  by  generalizing  the  structure  of  the  recognizer  to  a  multi-layered  net  The  performance  of the  proposed  network  was  evaluated  on  a  highly  confusable,  isolated  word.  multi  speaker recognition  task.  The  results  indicate  that  not  only  does  the  recognition  performance  improve.  but the  separation  between  classes  to  set  up  a  rejection  criterion  to  is  enhanced  also,  allowing  us  improve the confidence of the system. 
recognition  system.  and  we  propose
The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems
Authors: John E. Moody
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  present  an  analysis  of how  the  generalization  performance  (expected  test set error) relates to the expected training set  error for  nonlinear learn(cid:173) ing systems, such as multilayer perceptrons and radial basis functions.  The  principal  result  is  the  following  relationship  (computed  to  second  order)  between  the  expected  test  set  and tlaining set  errors: 
(1)  Here,  n  is  the  size  of the  training  sample e,  u;f f  is  the  effective  noise  variance in the response  variable( s),  ,x  is  a  regularization or  weight  decay  parameter,  and Peff(,x)  is  the  effective  number  of parameters in  the non(cid:173) linear  model.  The  expectations  ( )  of training set  and  test  set  errors  are  taken  over  possible  training  sets e and  training  and  test  sets  e'  respec(cid:173) tively.  The effective number of parameters Peff(,x)  usually differs from the  true  number  of model  parameters  P for  nonlinear  or  regularized  models;  this  theoretical  conclusion  is  supported  by  Monte  Carlo experiments.  In  addition to the surprising result  that Peff(,x)  ;/;  p,  we  propose an estimate  of (1)  called the  generalized prediction  error (GPE)  which generalizes well  established  estimates  of prediction  risk  such  as  Akaike's  F P E  and  AI C,  Mallows Cp, and  Barron's PSE to the nonlinear setting.! 
lCPE and Peff(>")  were previously  introduced  in  Moody  (1991).
Dual Inhibitory Mechanisms for Definition of Receptive Field Characteristics in a Cat Striate Cortex
Authors: A. B. Bonds
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: In  single  cells  of the  cat  striate  cortex,  lateral  inhibition  across  orienta(cid:173) tion  and/or spatial frequency  is  found  to enhance  pre-existing  biases.  A  contrast-dependent but spatially non-selective inhibitory component is also  found.  Stimulation with  ascending  and  descending  contrasts  reveals  the  latter  as  a  response  hysteresis  that  is  sensitive,  powerful  and  rapid,  sug(cid:173) gesting that it is  active in day-to-day vision.  Both forms of inhibition are  not  recurrent  but  are  rather  network  properties.  These  findings  suggest  two  fundamental  inhibitory mechanisms:  a  global mechanism that limits  dynamic range  and  creates  spatial selectivity  through  thresholding  and a  local mechanism that specifically refines  spatial filter  properties.  Analysis  of burst  patterns in spike  trains demonstrates that these  two mechanisms  have  unique  physiological origins. 
1 
INFORMATION PROCESSING IN STRIATE  CORTICAL  CELLS 
The  most  popular  current  model  of single  cells  in  the  striate  cortex  casts  them  in  terms  of spatial  and  temporal  filters.  The  input  to  visual  cortical  cells  from  lower  visual areas,  primarily the LGN, is fairly broadband (e.g., Soodak, Shapley &  Kaplan,  1987;  Maffei  & Fiorentini,  1973).  Cortical  cells  perform significant  band(cid:173) width restrictions on this information in at least three domains:  orientation, spatial  frequency  and  temporal frequency.  The  most  interesting  quality  of these  cells  is  75
Statistical Reliability of a Blowfly Movement-Sensitive Neuron
Authors: William Bialek Rob de Ruyter van Steveninck
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We develop a model-independent method for characterizing the reliability  of neural responses to brief stimuli. This approach allows us to measure  the discriminability of similar stimuli, based on the real-time response of a  single neuron. Neurophysiological data were obtained from a movement(cid:173) sensitive neuron (HI) in the visual system of the blowfly Calliphom ery(cid:173) throcephala. Furthermore, recordings were made from blowfly photore(cid:173) ceptor cells to quantify the signal to noise ratios in the peripheral visual  system. As photoreceptors form the input to the visual system, the reli(cid:173) ability of their signals ultimately determines the reliability of any visual  discrimination task. For the case of movement detection, this limit can  be computed, and compared to the HI neuron's reliability. Under favor(cid:173) able conditions, the performance of the HI neuron closely approaches the  theoretical limit, which means that under these conditions the nervous  system adds little noise in the process of computing movement from the  correlations of signals in the photoreceptor array.
Networks with Learned Unit Response Functions
Authors: Norman Yarvin John Moody
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Feedforward networks composed of units which compute a sigmoidal func(cid:173) tion  of a  weighted  sum of their  inputs  have  been  much investigated.  We  tested  the  approximation  and  estimation  capabilities  of networks  using  functions  more  complex  than  sigmoids.  Three  classes  of functions  were  tested:  polynomials,  rational  functions,  and  flexible  Fourier  series.  Un(cid:173) like  sigmoids,  these  classes  can  fit  non-monotonic functions.  They  were  compared  on  three  problems:  prediction  of  Boston  housing  prices,  the  sunspot  count,  and  robot  arm inverse  dynamics.  The  complex units  at(cid:173) tained  clearly superior  performance  on  the  robot  arm  problem,  which  is  a  highly  non-monotonic,  pure  approximation problem.  On  the  noisy  and  only  mildly  nonlinear  Boston  housing  and  sunspot  problems,  differences  among the  complex units  were  revealed;  polynomials did poorly,  whereas  rationals and flexible  Fourier series  were  comparable to sigmoids.
Adaptive Elastic Models for Hand-Printed Character Recognition
Authors: Christopher K. I. Williams, Michael D. Revow Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Hand-printed digits can be modeled as splines that are governed by about  8 control points. For each known digit. the control points have preferred  ., home" locations, and deformations of the digit are generated by moving  the control points away from their home locations. Images of digits can be  produced by placing Gaussian ink generators uniformly along the spline.  Real images can be recognized by finding the digit model most likely to  have generated the data. For each digit model we use an elastic matching  algorithm to minimize an energy function that includes both the defor(cid:173) mation energy of the digit model and the log probability that the model  would generate the inked pixels in the image. The model with the lowest  total energy wins. If a uniform noise process is included in the model of  image generation, some of the inked pixels can be rejected as noise as a  digit model is fitting a poorly segmented image. The digit models learn  by modifying the home locations of the control points.
Modeling Applications with the Focused Gamma Net
Authors: Bert de Vries, Jyh-Ming Kuo, Pedro Guedes de Oliveira José Carlos Príncipe
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The  focused  gamma  network  is  proposed  as  one  of  the  possible  implementations of the  gamma  neural  model.  The  focused  gamma  network is compared with the focused backpropagation network and  TDNN  for a time  series  prediction problem, and with  ADALINE in  a  system identification problem.
Towards Faster Stochastic Gradient Search
Authors: John Moody Christian Darken
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Stochastic gradient descent is a general algorithm which includes LMS,  on-line backpropagation, and adaptive k-means clustering as special cases.  The standard choices of the learning rate 1] (both adaptive and fixed func(cid:173) tions of time) often perform quite poorly. In contrast, our recently pro(cid:173) posed class of "search then converge" learning rate schedules (Darken and  Moody, 1990) display the theoretically optimal asymptotic convergence rate  and a superior ability to escape from poor local minima. However, the user  is responsible for setting a key parameter. We propose here a new method(cid:173) ology for creating the first completely automatic adaptive learning rates  which achieve the optimal rate of convergence.
Gradient Descent: Second Order Momentum and Saturating Error
Authors: Barak Pearlmutter
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Batch gradient descent, ~w(t) = -7JdE/dw(t) , conver~es to a minimum  of quadratic form with a time constant no better than '4Amax/ Amin where  Amin and Amax are the minimum and maximum eigenvalues of the Hessian  matrix of E with respect to w.  It was recently shown that adding a  momentum term ~w(t) = -7JdE/dw(t) + Q'~w(t - 1) improves this to  ~ VAmax/ Amin, although only in the batch case. Here we show that second(cid:173) order momentum, ~w(t) = -7JdE/dw(t) + Q'~w(t -1) + (3~w(t - 2), can  lower this no further. We then regard gradient descent with momentum  as a dynamic system and explore a non quadratic error surface, showing  that saturation of the error accounts for a variety of effects observed in  simulations and justifies some popular heuristics.
Active Exploration in Dynamic Environments
Authors: Knut Möller Sebastian B. Thrun
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: \Vhenever an agent learns to control an unknown environment, two oppos(cid:173) ing principles have to be combined, namely: exploration (long-term opti(cid:173) mization) and exploitation (short-term optimization). Many real-valued  connectionist approaches to learning control realize exploration by ran(cid:173) domness in action selection. This might be disadvantageous when costs  are assigned to "negative experiences" . The basic idea presented in this  paper is to make an agent explore unknown regions in a more directed  manner. This is achieved by a so-called competence map, which is trained  to predict the controller's accuracy, and is used for guiding exploration.  Based on this, a bistable system enables smoothly switching attention  between two behaviors - exploration and exploitation - depending on ex(cid:173) pected costs and knowledge gain.  The appropriateness of this method is demonstrated by a simple robot  navigation task.
Data Analysis using G/SPLINES
Authors: David Rogers
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: G/SPLINES is an algorithm for building functional models of data. It  uses genetic search to discover combinations of basis functions which  are then used to build a least-squares regression model. Because it  produces a population of models which evolve over time rather than a  single model, it allows analysis not possible with other regression-based  approaches.
A Contrast Sensitive Silicon Retina with Reciprocal Synapses
Authors: Andreas G. Andreou Kwabena A. Boahen
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The goal of perception is to extract invariant properties of the underly(cid:173) ing world. By computing contrast at edges, the retina reduces incident  light intensities spanning twelve decades to a twentyfold variation. In one  stroke, it solves the dynamic range problem and extracts relative reflec(cid:173) tivity, bringing us a step closer to the goal. We have built a contrast(cid:173) sensitive silicon retina that models all major synaptic interactions in the  outer-plexiform layer of the vertebrate retina using current-mode CMOS  circuits: namely, reciprocal synapses between cones and horizontal cells,  which produce the antagonistic center/surround receptive field, and cone  and horizontal cell gap junctions, which determine its size. The chip has  90 x 92 pixels on a 6.8 x 6.9mm die in 2/lm n-well technology and is fully  functional.
Markov Random Fields Can Bridge Levels of Abstraction
Authors: Peter N. Prokopowicz Paul R. Cooper
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Network  vision  systems  must  make  inferences  from  evidential  informa(cid:173) tion across levels of representational abstraction, from low level invariants,  through  intermediate scene  segments,  to  high  level  behaviorally  relevant  object  descriptions.  This paper shows  that such  networks  can  be realized  as  Markov  Random  Fields  (MRFs).  We  show  first  how  to  construct  an  MRF  functionally  equivalent  to  a  Hough  transform  parameter  network,  thus establishing a  principled  probabilistic basis for  visual networks.  Sec(cid:173) ond,  we  show  that these  MRF  parameter networks  are  more capable and  flexible  than traditional methods.  In particular,  they  have a  well-defined  probabilistic  interpretation,  intrinsically  incorporate  feedback,  and  offer  richer  representations  and decision capabilities.
Information Measure Based Skeletonisation
Authors: Lorien Y. Pratt Sowmya Ramachandran
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Automatic determination of proper neural network  topology by  trimming  over-sized  networks  is  an  important  area  of study,  which  has  previously  been  addressed  using  a  variety  of techniques.  In  this  paper,  we  present  Information  Measure  Based  Skeletonisation  (IMBS),  a  new  approach  to  this  problem where  superfluous  hidden  units  are  removed  based  on  their  information measure (1M).  This measure,  borrowed from decision  tree  in(cid:173) duction  techniques,  reflects  the  degree  to  which  the  hyperplane  formed  by  a  hidden  unit  discriminates  between  training  data  classes.  We  show  the results of applying IMBS  to three classification tasks and demonstrate  that it removes a  substantial number of hidden  units without significantly  affecting  network performance.
Interpretation of Artificial Neural Networks: Mapping Knowledge-Based Neural Networks into Rules
Authors: Jude W. Shavlik Geoffrey Towell
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We propose and empirically evaluate a method for the extraction of expert(cid:173) comprehensible rules from trained neural networks. Our method operates in  the context of a three-step process for learning that uses rule-based domain  knowledge in combination with neural networks. Empirical tests using real(cid:173) worlds problems from molecular biology show that the rules our method extracts  from trained neural networks: closely reproduce the accuracy of the network  from which they came, are superior to the rules derived by a learning system that  directly refines symbolic rules, and are expert-comprehensible.
Competitive Anti-Hebbian Learning of Invariants
Authors: Terrence J. Sejnowski Nicol N. Schraudolph
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Although the detection  of invariant structure in  a given set  of input patterns is  vital  to  many  recognition  tasks,  connectionist  learning  rules  tend  to  focus  on  directions of high variance (principal components).  The prediction paradigm is  often used to  reconcile this dichotomy; here we suggest a more direct approach  to  invariant learning based  on  an  anti-Hebbian learning rule.  An  unsupervised  tWO-layer  network implementing this  method in a competitive setting learns  to  extract coherent depth information from random-dot stereograms. 
1 
INTRODUCTION: LEARNING INVARIANT STRUCTURE 
Many connectionist learning algorithms share with principal component analysis (Jolliffe,  1986) the strategy of extracting the directions of highest variance from the input.  A single  Hebbian  neuron,  for  instance,  will come  to encode the input's first  principal component  (Oja and Karhunen, 1985); various forms of lateral interaction can be used to force a layer  of such  nodes to differentiate and span  the principal component subspace - cf.  (Sanger,  1989; Kung,  1990; Leen,  1991), and others. The same type of representation also develops  in the hidden layer of backpropagation autoassociator networks (Baldi and Hornik, 1989). 
as  the  case  may  be -
However,  the directions of highest variance  need not always be those that yield the most  information,  or - the information we  are  interested in  (Intrator,  1991).  In  fact,  it  is  sometimes  desirable to extract the invariant structure  of a stimulus  instead, learning to encode those aspects that vary the least.  The problem, then, is how to  achieve this within a connectionist framework that is so closely tied to the maximization of  variance.
3D Object Recognition Using Unsupervised Feature Extraction
Authors: Joshua I. Gold, Heinrich H. Bülthoff, Shimon Edelman Nathan Intrator
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Intrator  (1990)  proposed  a  feature  extraction  method  that  is  related  to  recent  statistical  theory  (Huber,  1985;  Friedman,  1987),  and is  based  on  a  biologically motivated model of neuronal  plasticity  (Bienenstock  et  al.,  1982).  This method has  been recently applied to feature extraction in the  context of recognizing 3D objects from single 2D views  (Intrator and Gold,  1991).  Here  we  describe  experiments designed to analyze the nature of the  extracted features,  and their  relevance  to the theory and psychophysics of  object  recognition.
Information Processing to Create Eye Movements
Authors: David A. Robinson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Because  eye  muscles  never  cocontract  and  do  not deal  with  external  loads,  one can write an equation that relates  motoneuron firing rate to  eye  position and  velocity  - a  very  uncommon  situation  in  the  CNS.  The semicircular canals  transduce head velocity in a linear manner by  using  a  high  background  discharge  rate,  imparting  linearity  to  the  premotor  circuits  that  generate  eye  movements.  This  has  allowed  deducing  some  of the  signal  processing  involved,  including  a  neural  network that integrates.  These  ideas  are  often  summarized  by block  diagrams.  Unfortunately,  they  are  of little  value  in  describing  the  behavior  of single  neurons  - a  fmding  supported  by  neural  network  models.
Propagation Filters in PDS Networks for Sequencing and Ambiguity Resolution
Authors: Michael G. Dyer Ronald A. Sumida
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  present  a  Parallel  Distributed  Semantic  (PDS)  Network  architecture  that  addresses  the  problems  of sequencing  and  ambiguity  resolution  in  natural language understanding.  A PDS Network stores phrases and their  meanings  using  multiple  PDP  networks,  structured  in  the  form  of a  se(cid:173) mantic net.  A  mechanism called  Propagation  Filters  is  employed:  (1)  to  control  communication  between  networks,  (2)  to  properly  sequence  the  components of a  phrase, and (3)  to resolve ambiguities.  Simulation results  indicate that PDS Networks and Propagation Filters can successfully  rep(cid:173) resent  high-level knowledge, can be trained relatively quickly, and provide  for  parallel inferencing at the knowledge level.
Neural Network Routing for Random Multistage Interconnection Networks
Authors: C. Lee Giles Mark W. Goudreau
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A routing scheme that uses  a neural network has been developed  that can  aid  in  establishing  point-to-point  communication  routes  through  multi(cid:173) stage interconnection  networks  (MINs).  The neural network  is  a  network  of the  type  that  was  examined  by  Hopfield  (Hopfield,  1984  and  1985).  In  this  work,  the  problem  of establishing  routes  through  random  MINs  (RMINs)  in a shared-memory, distributed computing system is  addressed.  The performance of the neural network routing scheme is  compared to two  more traditional approaches - exhaustive search  routing  and greedy  rout(cid:173) ing.  The results suggest  that a neural network  router may be  competitive  for  certain RMIN s.
Direction Selective Silicon Retina that uses Null Inhibition
Authors: Tobi Delbrück Ronald G. Benson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Biological retinas extract spatial and temporal features in an attempt to  reduce the complexity of performing visual tasks. We have built and tested  a silicon retina which encodes several useful temporal features found in ver(cid:173) tebrate retinas. The cells in our silicon retina are selective to direction,  highly sensitive to positive contrast changes around an ambient light level,  and tuned to a particular velocity. Inhibitory connections in the null di(cid:173) rection perform the direction selectivity we desire. This silicon retina is  on a 4.6 x 6.8mm die and consists of a 47 x 41 array of photoreceptors.
Software for ANN training on a Ring Array Processor
Authors: Jeff Bilmes, Nelson Morgan, James Beck Phil Kohn
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Experimental research on Artificial Neural Network (ANN) algorithms requires  either writing variations on the same program or making one monolithic program  with many parameters and options. By using an object-oriented library, the size  of these experimental programs is reduced while making them easier to read,  write and modify. An efficient and flexible realization of this idea is Connection(cid:173) ist Layered Object-oriented Network Simulator (CLONES). CLONES runs on  UNIX1 workstations and on the 100-1000 MFLOP Ring Array Processor (RAP)  that we built with ANN algorithms in mind. In this report we describe CLONES  and show how it is implemented on the RAP.
The Clusteron: Toward a Simple Abstraction for a Complex Neuron
Authors: Bartlett W. Mel
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Are  single  neocortical  neurons  as  powerful as  multi-layered  networks?  A  recent  compartmental modeling study  has shown  that voltage-dependent  membrane  nonlinearities  present  in  a  complex dendritic  tree  can  provide  a  virtual layer of local  nonlinear processing  elements between synaptic in(cid:173) puts  and  the  final  output  at  the  cell  body,  analogous  to  a  hidden  layer  in  a  multi-layer  network.  In  this  paper,  an  abstract  model  neuron  is  in(cid:173) troduced,  called  a  clusteron,  which  incorporates  aspects  of the  dendritic  "cluster-sensitivity"  phenomenon seen  in these  detailed  biophysical mod(cid:173) eling  studies.  It is  shown,  using  a  clusteron,  that  a  Hebb-type  learning  rule  can  be  used  to  extract  higher-order  statistics  from  a  set  of  train(cid:173) ing patterns,  by manipulating the spatial ordering of synaptic connections  onto  the  dendritic  tree.  The  potential neurobiological  relevance  of these  higher-order statistics for  nonlinear pattern discrimination is then studied  within a  full  compartmental model  of a  neocortical  pyramidal cell,  using  a  training set  of 1000 high-dimensional sparse random patterns.
Application of Neural Network Methodology to the Modelling of the Yield Strength in a Steel Rolling Plate Mill
Authors: Ah Chung Tsoi
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: In this paper, a tree based neural network viz.  MARS (Friedman, 1991) for  the modelling of the yield strength of a steel rolling plate mill is  described.  The inputs  to  the  time series  model  are  temperature,  strain,  strain  rate,  and  interpass  time  and  the  output  is  the  corresponding  yield  stress.  It  is  found  that  the  MARS-based  model  reveals  which  variable's  functional  dependence  is  nonlinear,  and  significant.  The  results  are  compared  with  those  obta.ined  by  using  a  Kalman  filter  based  online  tuning method and  other  classification  methods,  e.g.  CART,  C4 .5,  Bayesian  classification.  It  is found  that the  MARS-based  method consistently  outperforms the other  methods.
Reverse TDNN: An Architecture For Trajectory Generation
Authors: Yann Le Cun Patrice Simard
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The backpropagation algorithm can be used for both recognition and gen(cid:173) eration of time trajectories. When used as a recognizer, it has been shown  that the performance of a network can be greatly improved by adding  structure to the architecture. The same is true in trajectory generation.  In particular a new architecture corresponding to a "reversed" TDNN is  proposed. Results show dramatic improvement of performance in the gen(cid:173) eration of hand-written characters. A combination of TDNN and reversed  TDNN for compact encoding is also suggested.
Principles of Risk Minimization for Learning Theory
Authors: V. Vapnik
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Learning is posed as a problem of function estimation, for which two princi(cid:173) ples of solution are considered: empirical risk minimization and structural  risk minimization. These two principles are applied to two different state(cid:173) ments of the function estimation problem: global and local. Systematic  improvements in prediction power are illustrated in application to zip-code  recognition.
Perturbing Hebbian Rules
Authors: Geoffrey Goodhill Peter Dayan
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Feedforward networks composed of units which compute a sigmoidal func(cid:173) tion of a weighted sum of their inputs have been much investigated. We  tested the approximation and estimation capabilities of networks using  functions more complex than sigmoids. Three classes of functions were  tested: polynomials, rational functions, and flexible Fourier series. Un(cid:173) like sigmoids, these classes can fit non-monotonic functions. They were  compared on three problems: prediction of Boston housing prices, the  sunspot count, and robot arm inverse dynamics. The complex units at(cid:173) tained clearly superior performance on the robot arm problem, which is  a highly non-monotonic, pure approximation problem. On the noisy and  only mildly nonlinear Boston housing and sunspot problems, differences  among the complex units were revealed; polynomials did poorly, whereas  rationals and flexible Fourier series were comparable to sigmoids.
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
Authors: Liyu Chen, Boqing Gong, Fei Sha Hexiang Hu
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence. In this paper, we consider the problem of learning to simultaneously transfer across both environments and tasks, probably more importantly, by learning from only sparse (environment, task) pairs out of all the possible combinations. We propose a novel compositional neural network architecture which depicts a meta rule for composing policies from  environment and task embeddings. Notably, one of the main challenges is to learn the embeddings jointly with the meta rule. We further propose new training methods to disentangle the embeddings, making them both distinctive signatures of the environments and tasks and effective building blocks for composing the policies. Experiments on GridWorld and THOR, of which the agent takes as input an egocentric view, show that our approach gives rise to high success rates on all the (environment, task) pairs after learning from only 40% of them.
Self-Supervised Generation of Spatial Audio for 360° Video
Authors: Nuno Nvasconcelos, Timothy Langlois, Oliver Wang Pedro Morgado
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360° video viewing, but spatial audio microphones are still rare in current 360° video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis from the audio and 360° video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360° videos uploaded with spatial audio. During training, ground truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach we show that it is possible to infer the spatial localization of sounds based only on a synchronized 360° video and the mono audio track.
On GANs and GMMs
Authors: Yair Weiss Eitan Richardson
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: A longstanding problem in machine learning is to find unsupervised methods that can learn the statistical structure of high dimensional signals. In recent years, GANs have gained much attention as a possible solution to the problem, and in particular have shown the ability to generate remarkably realistic high resolution sampled images. At the same time, many authors have pointed out that GANs may fail to model the full distribution ("mode collapse") and that using the learned models for anything other than generating samples may be very difficult.
In this paper, we examine the utility of GANs in learning statistical models of images by comparing them to perhaps the simplest statistical model, the Gaussian Mixture Model. First, we present a simple method to evaluate generative models based on relative proportions of samples that fall into predetermined bins. Unlike previous automatic methods for evaluating models, our method does not rely on an additional neural network nor does it require approximating intractable computations. Second, we compare the performance of GANs to GMMs trained on the same datasets. While GMMs have previously been shown to be successful in modeling small patches of images, we show how to train them on full sized images despite the high dimensionality. Our results show that GMMs can generate realistic samples (although less sharp than those of GANs) but also capture the full distribution, which GANs fail to do. Furthermore, GMMs allow efficient inference and explicit representation of the underlying statistical structure. Finally, we discuss how GMMs can be used to generate sharp images.
Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks
Authors: Hyo-Eun Kim Hyeonseob Nam
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Real-world image recognition is often challenged by the variability of visual styles including object textures, lighting conditions, filter effects, etc. Although these variations have been deemed to be implicitly handled by more training data and deeper networks, recent advances in image style transfer suggest that it is also possible to explicitly manipulate the style information. Extending this idea to general visual recognition problems, we present Batch-Instance Normalization (BIN) to explicitly normalize unnecessary styles from images. Considering certain style features play an essential role in discriminative tasks, BIN learns to selectively normalize only disturbing styles while preserving useful styles. The proposed normalization module is easily incorporated into existing network architectures such as Residual Networks, and surprisingly improves the recognition performance in various scenarios. Furthermore, experiments verify that BIN effectively adapts to completely different tasks like object classification and style transfer, by controlling the trade-off between preserving and removing style variations. BIN can be implemented with only a few lines of code using popular deep learning frameworks.
Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
Authors: Junhyuk Oh, Honglak Lee Sungryull Sohn
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce a new RL problem where the agent is required to generalize to a previously-unseen environment characterized by a subtask graph which describes a set of subtasks and their dependencies. Unlike existing hierarchical multitask RL approaches that explicitly describe what the agent should do at a high level, our problem only describes properties of subtasks and relationships among them, which requires the agent to perform complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural subtask graph solver (NSGS) which encodes the subtask graph using a recursive neural network embedding. To overcome the difficulty of training, we propose a novel non-parametric gradient-based policy, graph reward propagation, to pre-train our NSGS agent and further finetune it through actor-critic method. The experimental results on two 2D visual domains show that our agent can perform complex reasoning to find a near-optimal way of executing the subtask graph and generalize well to the unseen subtask graphs. In addition, we compare our agent with a Monte-Carlo tree search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of NSGS can be further improved by combining it with MCTS.
KDGAN: Knowledge Distillation with Generative Adversarial Networks
Authors: Rui Zhang, Yu Sun, Jianzhong Qi Xiaojie Wang
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.
Contour location via entropy reduction leveraging multiple information sources
Authors: Remi Lam, Karen Willcox Alexandre Marques
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce an algorithm to locate contours of functions that are expensive to evaluate. The problem of locating contours arises in many applications, including classification, constrained optimization, and  performance analysis of mechanical and dynamical systems (reliability, probability of failure, stability, etc.). Our algorithm locates contours using information from multiple sources, which are available in the form of relatively inexpensive, biased, and possibly noisy
 approximations to the original function. Considering multiple information sources can lead to significant cost savings. We also introduce the concept of contour entropy, a formal measure of uncertainty about the location of the zero contour of a function approximated by a statistical surrogate model. Our algorithm locates contours efficiently by maximizing the reduction of contour entropy per unit cost.
Contextual bandits with surrogate losses: Margin bounds and efficient algorithms
Authors: Akshay Krishnamurthy Dylan J Foster
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning. Using the ramp loss, we derive a new margin-based regret bound in terms of standard sequential complexity measures of a benchmark class of real-valued regression functions. Using the hinge loss, we derive an efficient algorithm with a $\sqrt{dT}$-type mistake bound against benchmark policies induced by $d$-dimensional regressors. Under realizability assumptions, our results also yield classical regret bounds.
Adaptive Sampling Towards Fast Graph Representation Learning
Authors: Tong Zhang, Yu Rong, Junzhou Huang Wenbing Huang
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge of adapting GCNs on large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to the uncontrollable neighborhood expansion across layers. In this paper, we accelerate the training of GCNs through developing an adaptive layer-wise sampling method. By constructing the network layer by layer in a top-down passway, we sample the lower layer conditioned on the top one, where the sampled neighborhoods are shared by different parent nodes and the over expansion is avoided owing to the fixed-size sampling. More importantly, the proposed sampler is adaptive and applicable for explicit variance reduction, which in turn enhances the training of our method. Furthermore, we propose a novel and economical approach to promote the message passing over distant nodes by applying skip connections.
Intensive experiments on several benchmarks verify the effectiveness of our method regarding the classification accuracy while enjoying faster convergence speed.
Analytic solution and stationary phase approximation for the Bayesian lasso and elastic net
Authors: Tom Michoel
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: The lasso and elastic net linear regression models impose a double-exponential prior distribution on the model parameters to achieve   regression shrinkage and variable selection,  allowing the inference of robust models from large data sets.  However, there has been limited success in deriving estimates for the full posterior distribution of regression coefficients in these models, due to a need to evaluate analytically intractable partition function integrals. Here, the Fourier transform is used to express these integrals as complex-valued oscillatory integrals over "regression frequencies". This results in an analytic expansion and stationary phase approximation for the partition functions of the Bayesian lasso and elastic net, where the non-differentiability of the double-exponential prior has so far eluded such an approach. Use of this approximation leads to highly accurate numerical estimates for the expectation values and marginal posterior distributions of the regression coefficients, and allows for Bayesian inference of much higher dimensional models than previously possible.
Identification and Estimation of Causal Effects from Dependent Data
Authors: Ilya Shpitser Eli Sherman
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: The assumption that data samples are independent and identically distributed (iid) is standard in many areas of statistics and machine learning. Nevertheless, in some settings, such as social networks, infectious disease modeling, and reasoning with spatial and temporal data, this assumption is false. An extensive literature exists on making causal inferences under the iid assumption [12, 8, 21, 16], but, as pointed out in [14], causal inference in non-iid contexts is challenging due to the combination of unobserved confounding bias and data dependence. In this paper we develop a general theory describing when causal inferences are possible in such scenarios. We use segregated graphs [15], a generalization of latent projection mixed graphs [23], to represent causal models of this type and provide a complete algorithm for non-parametric identification in these models. We then demonstrate how statistical inferences may be performed on causal parameters identified by this algorithm, even in cases where parts of the model exhibit full interference, meaning only a single sample is available for parts of the model [19]. We apply these techniques to a synthetic data set which considers the adoption of fake news articles given the social network structure, articles read by each person, and baseline demographics and socioeconomic covariates.
Streamlining Variational Inference for Constraint Satisfaction Problems
Authors: Tudor Achim, Stefano Ermon Aditya Grover
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Several algorithms for solving constraint satisfaction problems are based on survey propagation, a variational inference scheme used to obtain approximate marginal probability estimates for variable assignments. These marginals correspond to how frequently each variable is set to true among satisfying assignments, and are used to inform branching decisions during search; however, marginal estimates obtained via survey propagation are approximate and can be self-contradictory. We introduce a more general branching strategy based on streamlining constraints, which sidestep hard assignments to variables. We show that streamlined solvers consistently outperform decimation-based solvers on random k-SAT instances for several problem sizes, shrinking the gap between empirical performance and theoretical limits of satisfiability by 16.3% on average for k = 3, 4, 5, 6.