mtybilly/Instruct2DS huggingface dataset:
Network Model of State-Dependent Sequencing
Authors: Jeffrey P. Sutton, Adam N. Mamelak, J. Allan Hobson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A network model with temporal sequencing and state-dependent modula(cid:173) tory features is described. The model is motivated by neurocognitive data  characterizing different states of waking and sleeping. Computer studies  demonstrate how unique states of sequencing can exist within the same  network under different aminergic and cholinergic modulatory influences.  Relationships between state-dependent modulation, memory, sequencing  and learning are discussed.
Splines, Rational Functions and Neural Networks
Authors: Robert C. Williamson, Peter L. Bartlett
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Connections  between  spline  approximation,  approximation with  rational  functions,  and  feedforward  neural  networks  are  studied.  The  potential  improvement in  the  degree  of approximation in  going  from  single  to two  hidden layer networks is examined.  Some results of Birman and Solomjak  regarding the degree  of approximation achievable  when  knot positions are  chosen  on the basis of the probability distribution of examples rather than  the function  values  are extended.
Learning to Make Coherent Predictions in Domains with Discontinuities
Authors: Suzanna Becker, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  have  previously  described  an  unsupervised  learning  procedure  that  discovers  spatially coherent  propertit>_<;  of the  world  by  maximizing the  in(cid:173) formation  that  parameters  extracted  from  different  parts  of  the  sensory  input convey  about some  common  underlying cause.  When given  random  dot  stereograms  of curved  surfaces,  this  procedure  learns  to  extract  sur(cid:173) face  depth  because  that  is  the  property  that  is  coherent  across  space.  It  also  learns  how  to  interpolate  the  depth  at  one  location  from  the  depths  at  nearby  locations  (Becker  and  Hint.oll.  1992).  1n  this  paper,  we  pro(cid:173) pose  two  new  models  which  handle surfaces  with discontinuities.  The first  model  attempts  to  detect  cases  of discontinuities  and  reject  them.  The  second  model  develops  a  mixture of expert  interpolators.  It  learns  to  de(cid:173) tect  the  locations of discontinuities  and  to  invoke specialized,  asymmetric  interpolators  that  do  not cross  the  discontinuities.
Networks for the Separation of Sources that are Superimposed and Delayed
Authors: John C. Platt, Federico Faggin
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We have created new networks to unmix signals which have been  mixed either with time delays or via filtering. We first show that  a subset of the Herault-Jutten learning rules fulfills a principle of  minimum output power. We then apply this principle to extensions  of the Herault-Jutten network which have delays in the feedback  path. Our networks perform well on real speech and music signals  that have been mixed using time delays or filtering.
Burst Synchronization without Frequency Locking in a Completely Solvable Neural Network Model
Authors: Heinz Schuster, Christof Koch
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The dynamic behavior of a network model consisting of all-to-all excitatory  coupled binary neurons with global inhibition is studied analytically and  numerically. We prove that for random input signals, the output of the  network consists of synchronized bursts with apparently random intermis(cid:173) sions of noisy activity. Our results suggest that synchronous bursts can be  generated by a simple neuronal architecture which amplifies incoming coin(cid:173) cident signals. This synchronization process is accompanied by dampened  oscillations which, by themselves, however, do not play any constructive  role in this and can therefore be considered to be an epiphenomenon.
Incrementally Learning Time-varying Half-planes
Authors: Anthony Kuh, Thomas Petsche, Ronald L. Rivest
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present a distribution-free model for incremental learning when concepts vary  with time. Concepts are caused to change by an adversary while an incremental  learning algorithm attempts to track the changing concepts by minimizing the  error between the current target concept and the hypothesis. For a single half(cid:173) plane and the intersection of two half-planes, we show that the average mistake  rate depends on the maximum rate at which an adversary can modify the concept.  These theoretical predictions are verified with simulations of several learning  algorithms including back propagation.
Retinogeniculate Development: The Role of Competition and Correlated Retinal Activity
Authors: Ron Keesing, David G. Stork, Carla J. Shatz
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: During visual development, projections from retinal ganglion cells  (RGCs) to the lateral geniculate nucleus (LGN) in cat are refined to  produce ocular dominance layering and precise topographic mapping.  Normal development depends upon activity in RGCs, suggesting a key  role for activity-dependent synaptic plasticity. Recent experiments on  prenatal retina show that during early development, "waves" of activity  pass across RGCs (Meister, et aI., 1991). We provide the first  simulations to demonstrate that such retinal waves, in conjunction with  Hebbian synaptic competition and early arrival of contralateral axons,  can account for observed patterns of retinogeniculate projections in  normal and experimentally-treated animals.
A Neural Net Model for Adaptive Control of Saccadic Accuracy by Primate Cerebellum and Brainstem
Authors: Paul Dean, John E. W. Mayhew, Pat Langdon
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Accurate saccades require interaction between brainstem circuitry and the  cerebeJJum.  A model of this interaction is described, based on Kawato's  principle  of  feedback-error-Iearning.  In  the  model  a  part  of  the  brainstem  (the superior colliculus) acts as a simple feedback controJJer  with no knowledge of initial eye position, and provides an error signal  for the cerebeJJum to correct for eye-muscle nonIinearities.  This teaches  the cerebeJJum,  modelled as a CMAC, to adjust appropriately the gain  on the brainstem burst-generator's internal feedback loop and so alter the  size of burst sent to  the  motoneurons.  With  direction-only errors  the  system rapidly learns to make accurate horizontal eye movements from  any  starting position, and  adapts realistically to  subsequent simulated  eye-muscle weakening or displacement of the saccadic target.
Rule Induction through Integrated Symbolic and Subsymbolic Processing
Authors: Clayton McMillan, Michael Mozer, Paul Smolensky
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  describe a neural network, called RufeNet, that learns explicit, sym(cid:173) bolic  condition-action  rules  in  a  formal  string  manipulation  domain.  RuleNet  discovers  functional  categories  over  elements  of the  domain,  and,  at  various  points  during  learning,  extracts  rules  that  operate  on  these  categories.  The  rules  are  then  injected  back  into  RuleNet  and  training continues,  in  a  process called iterative projection.  By  incorpo(cid:173) rating rules in this way,  RuleNet exhibits enhanced learning and gener(cid:173) alization  performance  over  alternative  neural  net  approaches.  By  integrating symbolic  rule  learning  and  subsymbolic  category  learning,  RuleNet  has capabilities  that go beyond a  purely  symbolic system.  We  show  how  this  architecture  can be  applied to  the  problem  of case-role  assignment  in  natural  language  processing,  yielding a  novel  rule-based  solution.
A comparison between a neural network model for the formation of brain maps and experimental data
Authors: K. Obermayer, K. Schulten, G. G. Blasdel
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Recently, high resolution images of the simultaneous representation of  orientation preference, orientation selectivity and ocular dominance have  been obtained for large areas in monkey striate cortex by optical imaging  [1-3]. These data allow for the first time a "local" as well as "global"  description of the spatial patterns and provide strong evidence for corre(cid:173) lations between orientation selectivity and ocular dominance.  A quantitative analysis reveals that these correlations arise when a five(cid:173) dimensional feature space (two dimensions for retinotopic space, one each  for orientation preference, orientation specificity, and ocular dominance) is  mapped into the two available dimensions of cortex while locally preserving  topology. These results provide strong evidence for the concept of topology  preserving maps which have been suggested as a basic design principle of  striate cortex [4-7]. 
Monkey striate cortex contains a retinotopic map in which are embedded the highly  repetitive patterns of orientation selectivity and ocular dominance. The retinotopic  projection establishes a "global" order, while maps of variables describing other  stimulus features, in particular line orientation and ocularity, dominate cortical  organization locally. A large number of pattern models [8-12] as well as models  of development [6,7,13-21] have been proposed to describe the spatial structure of  these patterns and their development during ontogenesis. However, most models  have not been compared with experimental data in detail. There are two reasons  for this: (i) many model-studies were not elaborated enough to be experimentally  testable and (ii) a sufficient amount of experimental data obtained from large areas  of striate cortex was not available.  83 
84 
Obermayer, Schulten, and Blasdel 
Figure 1: Spatial pattern of orientation preference and ocular dominance in mon(cid:173) key striate cortex (left) compared with predictions of the SOFM-model (right). Iso(cid:173) orientation lines (gray) are drawn in intervals of 11.25° (left) and IS.00 (right), re(cid:173) spectively. Black lines indicate the borders (ws(rj = 0) of ocular dominance bands.  The areas enclosed by black rectangles mark corresponding elements of organization  in monkey striate cortex and in the simulation result (see text). Left: Data obtained  from a 3.1mm x 4.2mm patch of the striate cortex of an adult macaque (macaca  nemestrina) by optical imaging [1-3]. The region is located near the border with  area IS, close to midline. Right: Model-map generated by the SOFM-algorithm.  The figure displays a small section of a network of size N = d = 512. The param(cid:173) eters of the simulation were: € = 0.02, tTh = 5, vr,:x = 20.48, vrax = 15.36, 9 . 101  iterations, with retinotopic initial conditions and periodic boundary conditions. 
1 Orientation and ocular dominance columns in monkey
A Connectionist Learning Approach to Analyzing Linguistic Stress
Authors: Prahlad Gupta, David S. Touretzky
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We use connectionist modeling to develop an analysis of stress systems in terms  of ease  of learnability.  In traditional linguistic analyses,  learnability arguments  determine default parameter settings based on the feasibilty of logicall y deducing  correct settings from an initial state.  Our approach provides an empirical alter(cid:173) native to such arguments.  Based on perceptron learning experiments using data  from  nineteen  human  languages,  we  develop a  novel  characterization of stress  patterns in terms of six parameters.  These provide both a partial description of the  stress pattern itself and a prediction of its learnability, without invoking abstract  theoretical constructs  such  as  metrical  feet.  This  work  demonstrates  that ma(cid:173) chine learning methods can provide a fresh  approach to understanding linguistic  phenomena. 
1  LINGUISTIC STRESS 
The domain of stress systems in language is considered to have a relatively good linguistic  theory,  called metrical phonologyl.  In this  theory,  the stress  patterns of many  languages  can be described concisely, and characterized in terms of a set of linguistic "parameters,"  such as  bounded vs.  unbounded metrical feet,  left vs.  right dominant feet,  etc.2  In many  languages, stress tends to be placed on certain kinds of syllables rather than on others; the  former are termed heavy syllables, and the latter light syllables. Languages that distinguish 
lFor an overview of the theory, see [Goldsmith 90, chapter 4].  2See [Dresher 90]  for one such parameter scheme.
Stationarity of Synaptic Coupling Strength Between Neurons with Nonstationary Discharge Properties
Authors: Mark R. Sydorenko, Eric D. Young
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Based on a general non-stationary point process model, we computed estimates of  the synaptic coupling strength (efficacy) as a function of time after stimulus onset  between an inhibitory interneuron and its target postsynaptic cell in the feline dorsal  cochlear nucleus. The data consist of spike trains from pairs of neurons responding  to brief tone bursts recorded in vivo. Our results suggest that the synaptic efficacy is  non-stationary. Further. synaptic efficacy is shown to be inversely and  approximately linearly related to average presynaptic spike rate. A second-order  analysis suggests that the latter result is not due to non-linear interactions. Synaptic  efficacy is less strongly correlated with postsynaptic rate and the correlation is not  consistent across neural pairs.
Time-Warping Network: A Hybrid Framework for Speech Recognition
Authors: Esther Levin, Roberto Pieraccini, Enrico Bocchieri
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: interest  has  been  generated  regarding  speech  Recently.  much  recognition  systems  based  on  Hidden  Markov  Models  (HMMs)  and  neural  network  (NN)  hybrids.  Such  systems  attempt  to  combine  the  best  features  of  both  models:  the  temporal  structure  of HMMs  and  the  discriminative  power  of neural  networks.  In this  work  we  define  a  time-warping  (1W)  neuron  that extends  the  operation of the  fonnal  neuron  of a  back-propagation  network by  warping  the input pattern  to  match  it  optimally  to  its  weights.  We  show  that  a  single-layer  network  of TW  neurons  is  equivalent  to  a  Gaussian  density  HMM(cid:173) the  based  discriminative  power  of  this  system  by  using  back-propagation  discriminative  training.  and/or  by  generalizing  the  structure  of  the  recognizer  to  a  multi-layered  net  The  performance  of the  proposed  network  was  evaluated  on  a  highly  confusable,  isolated  word.  multi  speaker recognition  task.  The  results  indicate  that  not  only  does  the  recognition  performance  improve.  but the  separation  between  classes  to  set  up  a  rejection  criterion  to  is  enhanced  also,  allowing  us  improve the confidence of the system. 
recognition  system.  and  we  propose
Statistical Reliability of a Blowfly Movement-Sensitive Neuron
Authors: Rob de Ruyter van Steveninck, William Bialek
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We develop a model-independent method for characterizing the reliability  of neural responses to brief stimuli. This approach allows us to measure  the discriminability of similar stimuli, based on the real-time response of a  single neuron. Neurophysiological data were obtained from a movement(cid:173) sensitive neuron (HI) in the visual system of the blowfly Calliphom ery(cid:173) throcephala. Furthermore, recordings were made from blowfly photore(cid:173) ceptor cells to quantify the signal to noise ratios in the peripheral visual  system. As photoreceptors form the input to the visual system, the reli(cid:173) ability of their signals ultimately determines the reliability of any visual  discrimination task. For the case of movement detection, this limit can  be computed, and compared to the HI neuron's reliability. Under favor(cid:173) able conditions, the performance of the HI neuron closely approaches the  theoretical limit, which means that under these conditions the nervous  system adds little noise in the process of computing movement from the  correlations of signals in the photoreceptor array.
Networks with Learned Unit Response Functions
Authors: John Moody, Norman Yarvin
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Feedforward networks composed of units which compute a sigmoidal func(cid:173) tion  of a  weighted  sum of their  inputs  have  been  much investigated.  We  tested  the  approximation  and  estimation  capabilities  of networks  using  functions  more  complex  than  sigmoids.  Three  classes  of functions  were  tested:  polynomials,  rational  functions,  and  flexible  Fourier  series.  Un(cid:173) like  sigmoids,  these  classes  can  fit  non-monotonic functions.  They  were  compared  on  three  problems:  prediction  of  Boston  housing  prices,  the  sunspot  count,  and  robot  arm inverse  dynamics.  The  complex units  at(cid:173) tained  clearly superior  performance  on  the  robot  arm  problem,  which  is  a  highly  non-monotonic,  pure  approximation problem.  On  the  noisy  and  only  mildly  nonlinear  Boston  housing  and  sunspot  problems,  differences  among the  complex units  were  revealed;  polynomials did poorly,  whereas  rationals and flexible  Fourier series  were  comparable to sigmoids.
Adaptive Elastic Models for Hand-Printed Character Recognition
Authors: Geoffrey E. Hinton, Christopher K. I. Williams, Michael D. Revow
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Hand-printed digits can be modeled as splines that are governed by about  8 control points. For each known digit. the control points have preferred  ., home" locations, and deformations of the digit are generated by moving  the control points away from their home locations. Images of digits can be  produced by placing Gaussian ink generators uniformly along the spline.  Real images can be recognized by finding the digit model most likely to  have generated the data. For each digit model we use an elastic matching  algorithm to minimize an energy function that includes both the defor(cid:173) mation energy of the digit model and the log probability that the model  would generate the inked pixels in the image. The model with the lowest  total energy wins. If a uniform noise process is included in the model of  image generation, some of the inked pixels can be rejected as noise as a  digit model is fitting a poorly segmented image. The digit models learn  by modifying the home locations of the control points.
Modeling Applications with the Focused Gamma Net
Authors: José Carlos Príncipe, Bert de Vries, Jyh-Ming Kuo, Pedro Guedes de Oliveira
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The  focused  gamma  network  is  proposed  as  one  of  the  possible  implementations of the  gamma  neural  model.  The  focused  gamma  network is compared with the focused backpropagation network and  TDNN  for a time  series  prediction problem, and with  ADALINE in  a  system identification problem.
Towards Faster Stochastic Gradient Search
Authors: Christian Darken, John Moody
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Stochastic gradient descent is a general algorithm which includes LMS,  on-line backpropagation, and adaptive k-means clustering as special cases.  The standard choices of the learning rate 1] (both adaptive and fixed func(cid:173) tions of time) often perform quite poorly. In contrast, our recently pro(cid:173) posed class of "search then converge" learning rate schedules (Darken and  Moody, 1990) display the theoretically optimal asymptotic convergence rate  and a superior ability to escape from poor local minima. However, the user  is responsible for setting a key parameter. We propose here a new method(cid:173) ology for creating the first completely automatic adaptive learning rates  which achieve the optimal rate of convergence.
Active Exploration in Dynamic Environments
Authors: Sebastian B. Thrun, Knut Möller
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: \Vhenever an agent learns to control an unknown environment, two oppos(cid:173) ing principles have to be combined, namely: exploration (long-term opti(cid:173) mization) and exploitation (short-term optimization). Many real-valued  connectionist approaches to learning control realize exploration by ran(cid:173) domness in action selection. This might be disadvantageous when costs  are assigned to "negative experiences" . The basic idea presented in this  paper is to make an agent explore unknown regions in a more directed  manner. This is achieved by a so-called competence map, which is trained  to predict the controller's accuracy, and is used for guiding exploration.  Based on this, a bistable system enables smoothly switching attention  between two behaviors - exploration and exploitation - depending on ex(cid:173) pected costs and knowledge gain.  The appropriateness of this method is demonstrated by a simple robot  navigation task.
A Contrast Sensitive Silicon Retina with Reciprocal Synapses
Authors: Kwabena A. Boahen, Andreas G. Andreou
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The goal of perception is to extract invariant properties of the underly(cid:173) ing world. By computing contrast at edges, the retina reduces incident  light intensities spanning twelve decades to a twentyfold variation. In one  stroke, it solves the dynamic range problem and extracts relative reflec(cid:173) tivity, bringing us a step closer to the goal. We have built a contrast(cid:173) sensitive silicon retina that models all major synaptic interactions in the  outer-plexiform layer of the vertebrate retina using current-mode CMOS  circuits: namely, reciprocal synapses between cones and horizontal cells,  which produce the antagonistic center/surround receptive field, and cone  and horizontal cell gap junctions, which determine its size. The chip has  90 x 92 pixels on a 6.8 x 6.9mm die in 2/lm n-well technology and is fully  functional.
Markov Random Fields Can Bridge Levels of Abstraction
Authors: Paul R. Cooper, Peter N. Prokopowicz
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Network  vision  systems  must  make  inferences  from  evidential  informa(cid:173) tion across levels of representational abstraction, from low level invariants,  through  intermediate scene  segments,  to  high  level  behaviorally  relevant  object  descriptions.  This paper shows  that such  networks  can  be realized  as  Markov  Random  Fields  (MRFs).  We  show  first  how  to  construct  an  MRF  functionally  equivalent  to  a  Hough  transform  parameter  network,  thus establishing a  principled  probabilistic basis for  visual networks.  Sec(cid:173) ond,  we  show  that these  MRF  parameter networks  are  more capable and  flexible  than traditional methods.  In particular,  they  have a  well-defined  probabilistic  interpretation,  intrinsically  incorporate  feedback,  and  offer  richer  representations  and decision capabilities.
Information Measure Based Skeletonisation
Authors: Sowmya Ramachandran, Lorien Y. Pratt
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Automatic determination of proper neural network  topology by  trimming  over-sized  networks  is  an  important  area  of study,  which  has  previously  been  addressed  using  a  variety  of techniques.  In  this  paper,  we  present  Information  Measure  Based  Skeletonisation  (IMBS),  a  new  approach  to  this  problem where  superfluous  hidden  units  are  removed  based  on  their  information measure (1M).  This measure,  borrowed from decision  tree  in(cid:173) duction  techniques,  reflects  the  degree  to  which  the  hyperplane  formed  by  a  hidden  unit  discriminates  between  training  data  classes.  We  show  the results of applying IMBS  to three classification tasks and demonstrate  that it removes a  substantial number of hidden  units without significantly  affecting  network performance.
Interpretation of Artificial Neural Networks: Mapping Knowledge-Based Neural Networks into Rules
Authors: Geoffrey Towell, Jude W. Shavlik
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We propose and empirically evaluate a method for the extraction of expert(cid:173) comprehensible rules from trained neural networks. Our method operates in  the context of a three-step process for learning that uses rule-based domain  knowledge in combination with neural networks. Empirical tests using real(cid:173) worlds problems from molecular biology show that the rules our method extracts  from trained neural networks: closely reproduce the accuracy of the network  from which they came, are superior to the rules derived by a learning system that  directly refines symbolic rules, and are expert-comprehensible.
Competitive Anti-Hebbian Learning of Invariants
Authors: Nicol N. Schraudolph, Terrence J. Sejnowski
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Although the detection  of invariant structure in  a given set  of input patterns is  vital  to  many  recognition  tasks,  connectionist  learning  rules  tend  to  focus  on  directions of high variance (principal components).  The prediction paradigm is  often used to  reconcile this dichotomy; here we suggest a more direct approach  to  invariant learning based  on  an  anti-Hebbian learning rule.  An  unsupervised  tWO-layer  network implementing this  method in a competitive setting learns  to  extract coherent depth information from random-dot stereograms. 
1 
INTRODUCTION: LEARNING INVARIANT STRUCTURE 
Many connectionist learning algorithms share with principal component analysis (Jolliffe,  1986) the strategy of extracting the directions of highest variance from the input.  A single  Hebbian  neuron,  for  instance,  will come  to encode the input's first  principal component  (Oja and Karhunen, 1985); various forms of lateral interaction can be used to force a layer  of such  nodes to differentiate and span  the principal component subspace - cf.  (Sanger,  1989; Kung,  1990; Leen,  1991), and others. The same type of representation also develops  in the hidden layer of backpropagation autoassociator networks (Baldi and Hornik, 1989). 
as  the  case  may  be -
However,  the directions of highest variance  need not always be those that yield the most  information,  or - the information we  are  interested in  (Intrator,  1991).  In  fact,  it  is  sometimes  desirable to extract the invariant structure  of a stimulus  instead, learning to encode those aspects that vary the least.  The problem, then, is how to  achieve this within a connectionist framework that is so closely tied to the maximization of  variance.
3D Object Recognition Using Unsupervised Feature Extraction
Authors: Nathan Intrator, Joshua I. Gold, Heinrich H. Bülthoff, Shimon Edelman
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Intrator  (1990)  proposed  a  feature  extraction  method  that  is  related  to  recent  statistical  theory  (Huber,  1985;  Friedman,  1987),  and is  based  on  a  biologically motivated model of neuronal  plasticity  (Bienenstock  et  al.,  1982).  This method has  been recently applied to feature extraction in the  context of recognizing 3D objects from single 2D views  (Intrator and Gold,  1991).  Here  we  describe  experiments designed to analyze the nature of the  extracted features,  and their  relevance  to the theory and psychophysics of  object  recognition.
Propagation Filters in PDS Networks for Sequencing and Ambiguity Resolution
Authors: Ronald A. Sumida, Michael G. Dyer
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  present  a  Parallel  Distributed  Semantic  (PDS)  Network  architecture  that  addresses  the  problems  of sequencing  and  ambiguity  resolution  in  natural language understanding.  A PDS Network stores phrases and their  meanings  using  multiple  PDP  networks,  structured  in  the  form  of a  se(cid:173) mantic net.  A  mechanism called  Propagation  Filters  is  employed:  (1)  to  control  communication  between  networks,  (2)  to  properly  sequence  the  components of a  phrase, and (3)  to resolve ambiguities.  Simulation results  indicate that PDS Networks and Propagation Filters can successfully  rep(cid:173) resent  high-level knowledge, can be trained relatively quickly, and provide  for  parallel inferencing at the knowledge level.
Neural Network Routing for Random Multistage Interconnection Networks
Authors: Mark W. Goudreau, C. Lee Giles
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A routing scheme that uses  a neural network has been developed  that can  aid  in  establishing  point-to-point  communication  routes  through  multi(cid:173) stage interconnection  networks  (MINs).  The neural network  is  a  network  of the  type  that  was  examined  by  Hopfield  (Hopfield,  1984  and  1985).  In  this  work,  the  problem  of establishing  routes  through  random  MINs  (RMINs)  in a shared-memory, distributed computing system is  addressed.  The performance of the neural network routing scheme is  compared to two  more traditional approaches - exhaustive search  routing  and greedy  rout(cid:173) ing.  The results suggest  that a neural network  router may be  competitive  for  certain RMIN s.
Direction Selective Silicon Retina that uses Null Inhibition
Authors: Ronald G. Benson, Tobi Delbrück
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Biological retinas extract spatial and temporal features in an attempt to  reduce the complexity of performing visual tasks. We have built and tested  a silicon retina which encodes several useful temporal features found in ver(cid:173) tebrate retinas. The cells in our silicon retina are selective to direction,  highly sensitive to positive contrast changes around an ambient light level,  and tuned to a particular velocity. Inhibitory connections in the null di(cid:173) rection perform the direction selectivity we desire. This silicon retina is  on a 4.6 x 6.8mm die and consists of a 47 x 41 array of photoreceptors.
Software for ANN training on a Ring Array Processor
Authors: Phil Kohn, Jeff Bilmes, Nelson Morgan, James Beck
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Experimental research on Artificial Neural Network (ANN) algorithms requires  either writing variations on the same program or making one monolithic program  with many parameters and options. By using an object-oriented library, the size  of these experimental programs is reduced while making them easier to read,  write and modify. An efficient and flexible realization of this idea is Connection(cid:173) ist Layered Object-oriented Network Simulator (CLONES). CLONES runs on  UNIX1 workstations and on the 100-1000 MFLOP Ring Array Processor (RAP)  that we built with ANN algorithms in mind. In this report we describe CLONES  and show how it is implemented on the RAP.
Reverse TDNN: An Architecture For Trajectory Generation
Authors: Patrice Simard, Yann Le Cun
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The backpropagation algorithm can be used for both recognition and gen(cid:173) eration of time trajectories. When used as a recognizer, it has been shown  that the performance of a network can be greatly improved by adding  structure to the architecture. The same is true in trajectory generation.  In particular a new architecture corresponding to a "reversed" TDNN is  proposed. Results show dramatic improvement of performance in the gen(cid:173) eration of hand-written characters. A combination of TDNN and reversed  TDNN for compact encoding is also suggested.
Perturbing Hebbian Rules
Authors: Peter Dayan, Geoffrey Goodhill
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Feedforward networks composed of units which compute a sigmoidal func(cid:173) tion of a weighted sum of their inputs have been much investigated. We  tested the approximation and estimation capabilities of networks using  functions more complex than sigmoids. Three classes of functions were  tested: polynomials, rational functions, and flexible Fourier series. Un(cid:173) like sigmoids, these classes can fit non-monotonic functions. They were  compared on three problems: prediction of Boston housing prices, the  sunspot count, and robot arm inverse dynamics. The complex units at(cid:173) tained clearly superior performance on the robot arm problem, which is  a highly non-monotonic, pure approximation problem. On the noisy and  only mildly nonlinear Boston housing and sunspot problems, differences  among the complex units were revealed; polynomials did poorly, whereas  rationals and flexible Fourier series were comparable to sigmoids.
Synthesized Policies for Transfer and Adaptation across Tasks and Environments
Authors: Hexiang Hu, Liyu Chen, Boqing Gong, Fei Sha
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: The ability to transfer in reinforcement learning is key towards building an agent of general artificial intelligence. In this paper, we consider the problem of learning to simultaneously transfer across both environments and tasks, probably more importantly, by learning from only sparse (environment, task) pairs out of all the possible combinations. We propose a novel compositional neural network architecture which depicts a meta rule for composing policies from  environment and task embeddings. Notably, one of the main challenges is to learn the embeddings jointly with the meta rule. We further propose new training methods to disentangle the embeddings, making them both distinctive signatures of the environments and tasks and effective building blocks for composing the policies. Experiments on GridWorld and THOR, of which the agent takes as input an egocentric view, show that our approach gives rise to high success rates on all the (environment, task) pairs after learning from only 40% of them.
Self-Supervised Generation of Spatial Audio for 360° Video
Authors: Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, Oliver Wang
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360° video viewing, but spatial audio microphones are still rare in current 360° video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis from the audio and 360° video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360° videos uploaded with spatial audio. During training, ground truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach we show that it is possible to infer the spatial localization of sounds based only on a synchronized 360° video and the mono audio track.
On GANs and GMMs
Authors: Eitan Richardson, Yair Weiss
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: A longstanding problem in machine learning is to find unsupervised methods that can learn the statistical structure of high dimensional signals. In recent years, GANs have gained much attention as a possible solution to the problem, and in particular have shown the ability to generate remarkably realistic high resolution sampled images. At the same time, many authors have pointed out that GANs may fail to model the full distribution ("mode collapse") and that using the learned models for anything other than generating samples may be very difficult.
In this paper, we examine the utility of GANs in learning statistical models of images by comparing them to perhaps the simplest statistical model, the Gaussian Mixture Model. First, we present a simple method to evaluate generative models based on relative proportions of samples that fall into predetermined bins. Unlike previous automatic methods for evaluating models, our method does not rely on an additional neural network nor does it require approximating intractable computations. Second, we compare the performance of GANs to GMMs trained on the same datasets. While GMMs have previously been shown to be successful in modeling small patches of images, we show how to train them on full sized images despite the high dimensionality. Our results show that GMMs can generate realistic samples (although less sharp than those of GANs) but also capture the full distribution, which GANs fail to do. Furthermore, GMMs allow efficient inference and explicit representation of the underlying statistical structure. Finally, we discuss how GMMs can be used to generate sharp images.
Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks
Authors: Hyeonseob Nam, Hyo-Eun Kim
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Real-world image recognition is often challenged by the variability of visual styles including object textures, lighting conditions, filter effects, etc. Although these variations have been deemed to be implicitly handled by more training data and deeper networks, recent advances in image style transfer suggest that it is also possible to explicitly manipulate the style information. Extending this idea to general visual recognition problems, we present Batch-Instance Normalization (BIN) to explicitly normalize unnecessary styles from images. Considering certain style features play an essential role in discriminative tasks, BIN learns to selectively normalize only disturbing styles while preserving useful styles. The proposed normalization module is easily incorporated into existing network architectures such as Residual Networks, and surprisingly improves the recognition performance in various scenarios. Furthermore, experiments verify that BIN effectively adapts to completely different tasks like object classification and style transfer, by controlling the trade-off between preserving and removing style variations. BIN can be implemented with only a few lines of code using popular deep learning frameworks.
Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
Authors: Sungryull Sohn, Junhyuk Oh, Honglak Lee
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce a new RL problem where the agent is required to generalize to a previously-unseen environment characterized by a subtask graph which describes a set of subtasks and their dependencies. Unlike existing hierarchical multitask RL approaches that explicitly describe what the agent should do at a high level, our problem only describes properties of subtasks and relationships among them, which requires the agent to perform complex reasoning to find the optimal subtask to execute. To solve this problem, we propose a neural subtask graph solver (NSGS) which encodes the subtask graph using a recursive neural network embedding. To overcome the difficulty of training, we propose a novel non-parametric gradient-based policy, graph reward propagation, to pre-train our NSGS agent and further finetune it through actor-critic method. The experimental results on two 2D visual domains show that our agent can perform complex reasoning to find a near-optimal way of executing the subtask graph and generalize well to the unseen subtask graphs. In addition, we compare our agent with a Monte-Carlo tree search (MCTS) method showing that our method is much more efficient than MCTS, and the performance of NSGS can be further improved by combining it with MCTS.
KDGAN: Knowledge Distillation with Generative Adversarial Networks
Authors: Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.
Contour location via entropy reduction leveraging multiple information sources
Authors: Alexandre Marques, Remi Lam, Karen Willcox
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We introduce an algorithm to locate contours of functions that are expensive to evaluate. The problem of locating contours arises in many applications, including classification, constrained optimization, and  performance analysis of mechanical and dynamical systems (reliability, probability of failure, stability, etc.). Our algorithm locates contours using information from multiple sources, which are available in the form of relatively inexpensive, biased, and possibly noisy
 approximations to the original function. Considering multiple information sources can lead to significant cost savings. We also introduce the concept of contour entropy, a formal measure of uncertainty about the location of the zero contour of a function approximated by a statistical surrogate model. Our algorithm locates contours efficiently by maximizing the reduction of contour entropy per unit cost.
Contextual bandits with surrogate losses: Margin bounds and efficient algorithms
Authors: Dylan J Foster, Akshay Krishnamurthy
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: We use surrogate losses to obtain several new regret bounds and new algorithms for contextual bandit learning. Using the ramp loss, we derive a new margin-based regret bound in terms of standard sequential complexity measures of a benchmark class of real-valued regression functions. Using the hinge loss, we derive an efficient algorithm with a $\sqrt{dT}$-type mistake bound against benchmark policies induced by $d$-dimensional regressors. Under realizability assumptions, our results also yield classical regret bounds.
Adaptive Sampling Towards Fast Graph Representation Learning
Authors: Wenbing Huang, Tong Zhang, Yu Rong, Junzhou Huang
Published in: Advances in Neural Information Processing Systems 31  (NeurIPS 2018) (Main Conference Track), 2018

Abstract: Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge of adapting GCNs on large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to the uncontrollable neighborhood expansion across layers. In this paper, we accelerate the training of GCNs through developing an adaptive layer-wise sampling method. By constructing the network layer by layer in a top-down passway, we sample the lower layer conditioned on the top one, where the sampled neighborhoods are shared by different parent nodes and the over expansion is avoided owing to the fixed-size sampling. More importantly, the proposed sampler is adaptive and applicable for explicit variance reduction, which in turn enhances the training of our method. Furthermore, we propose a novel and economical approach to promote the message passing over distant nodes by applying skip connections.
Intensive experiments on several benchmarks verify the effectiveness of our method regarding the classification accuracy while enjoying faster convergence speed.