mtybilly/Instruct2DS huggingface dataset:
Analog Cochlear Model for Multiresolution Speech Analysis
Authors: Andreas G. Andreou, Moise H. Goldstein Jr. Weimin Liu
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper discusses the parameterization of speech by an analog cochlear  model.  The tradeoff between  time and  frequency  resolution  is  viewed  as  the  fundamental  difference  between  conventional  spectrographic  analysis  and cochlear signal processing for  broadband, rapid-changing signals.  The  model's response exhibits a  wavelet-like analysis in the scale domain that  preserves good temporal resolution; the frequency of each spectral compo(cid:173) nent  in  a  broadband signal  can be accurately  determined  from  the  inter(cid:173) peak  intervals  in  the  instantaneous  firing  rates  of auditory  fibers.  Such  properties  of the  cochlear  model  are  demonstrated  with  natural  speech  and synthetic complex signals.
A Boundary Hunting Radial Basis Function Classifier which Allocates Centers Constructively
Authors: Richard P Lippmann Eric I. Chang
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A  new  boundary  hunting  radial  basis  function  (BH-RBF)  classifier  which  allocates  RBF  centers  constructively  near  class  boundaries  is  described.  This  classifier creates complex  decision  boundaries  only  in  regions  where  confusions  occur  and  corresponding  RBF  outputs  are  similar.  A  predicted  square  error  measure  is  used  to  determine  how  many centers to add and to determine when to stop adding centers. Two  experiments are presented which demonstrate the advantages of the BH(cid:173) RBF classifier.  One uses artificial data with  two classes and  two  input  features  where each class contains  four clusters but only  one cluster is  near a decision region boundary. The other uses a large seismic database  with  seven  classes and  14  input features.  In  both  experiments  the  BH(cid:173) RBF classifier provides a lower error rate  with  fewer  centers  than  are  required  by  more  conventional  RBF,  Gaussian  mixture,  or  MLP  classifiers.
Learning to See Where and What: Training a Net to Make Saccades and Recognize Handwritten Characters
Authors: Mosfeq Rashid, David Chapman, James A. Pittman Gale Martin
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: to  integrated  segmentation  and  This  paper  describes  an  approach  recognition of hand-printed characters.  The approach, called Saccade,  integrates  ballistic  and  corrective  saccades  (eye  movements)  with  character recognition.  A single backpropagation net  is  trained to make  a classification decision on a character centered  in its input window, as  well as to estimate the distance of the current and next character from the  center of the input window.  The net  learns to accurately estimate these  distances  regardless  of variations  in  character  width,  spacing  between  characters,  writing  style  and  other factors.  During  testing,  the  system  uses  the  net~xtracted classification  and  distance  information,  along  with a set of jumping rules, to jump from character to character. 
The ability to read rests on multiple foundation skills.  In learning how to read, people learn  how to recognize individual characters centered in the visual  field.  They also learn how  to move their eyes along a line of text, sequentially centering the visual field on successive  characters.  We believe that the key to developing optical character recognition (OCR) sys(cid:173) tems that can mimic human reading capabilities, is to develop systems that can learn these  and other skills in an integrated fashion.  In this paper, we demonstrate that a backpropaga(cid:173) tion net can learn to naVigate along a line of handwritten characters, as  well as to recognize  the characters centered in its visual field.  The system, called Saccade, extends the current  state of the art in OCR technology by using a single classifier to accurately and efficiently  locate and recognize characters, regardless of whether they touch each other or are sepa(cid:173) rate.  The Saccade system was described briefly at the last NIPS conference (Martin & Ra(cid:173) shid,  1992).  In this  paper,  we  describe  it  mcx-e  fully  and report on results demonstrating  its accuracy and efficiency in recognizing handwritten digits. 
The Saccade system takes a cue from the ballistic and corrective saccades (eye movements)  of natural vision systems.  Natural saccades make it possible to efficiently move from one  informative area to another by jumping.  The eye  typically  initiates a ballistic saccade to
How Oscillatory Neuronal Responses Reflect Bistability and Switching of the Hidden Assembly Dynamics
Authors: H.-U. Bauer, J. Deppisch, T. Geisel K. Pawelzik
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A switching between apparently coherent (oscillatory) and stochastic  episodes of activity has been observed in responses from cat and monkey  visual cortex. We describe the dynamics of these phenomena in two paral(cid:173) lel approaches, a phenomenological and a rather microscopic one. On the  one hand we analyze neuronal responses in terms of a hidden state model  (HSM). The parameters of this model are extracted directly from exper(cid:173) imental spike trains. They characterize the underlying dynamics as well  as the coupling of individual neurons to the network. This phenomenolog(cid:173) ical model thus provides a new framework for the experimental analysis  of network dynamics. The application of this method to multi unit ac(cid:173) tivities from the visual cortex of the cat substantiates the existence of  oscillatory and stochastic states and quantifies the switching behaviour  in the assembly dynamics. On the other hand we start from the single  spiking neuron and derive a master equation for the time evolution of the  assembly state which we represent by a phase density. This phase density  dynamics (PDD) exhibits costability of two attractors, a limit cycle, and  a fixed point when synaptic interaction is nonlinear. External fluctuations  can switch the bistable system from one state to the other. Finally we  show, that the two approaches are mutually consistent and therefore both  explain the detailed time structure in the data.
Summed Weight Neuron Perturbation: An O(N) Improvement Over Weight Perturbation
Authors: Marwan Jabri Barry Flower
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The algorithm presented performs gradient descent on the weight space  of an Artificial Neural Network (ANN), using a finite difference to  approximate the gradient The method is novel in that it achieves a com(cid:173) putational complexity similar to that of Node Perturbation, O(N3), but  does not require access to the activity of hidden or internal neurons.  This is possible due to a stochastic relation between perturbations at the  weights and the neurons of an ANN. The algorithm is also similar to  Weight Perturbation in that it is optimal in terms of hardware require(cid:173) ments when used for the training ofVLSI implementations of ANN's.
Extended Regularization Methods for Nonconvergent Model Selection
Authors: F. Hergert, H. G. Zimmermann W. Finnoff
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Many techniques for model selection in the field of neural networks  correspond to well established statistical methods. The method  of 'stopped training', on the other hand, in which an oversized  network is trained until the error on a further validation set of ex(cid:173) amples deteriorates, then training is stopped, is a true innovation,  since model selection doesn't require convergence of the training  process.  In this paper we show that this performance can be significantly  enhanced by extending the 'non convergent model selection method'  of stopped training to include dynamic topology modifications  (dynamic weight pruning) and modified complexity penalty term  methods in which the weighting of the penalty term is adjusted  during the training process.
Synchronization and Grammatical Inference in an Oscillating Elman Net
Authors: Todd Troyer, Frank Eeckman Bill Baird
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have designed an architecture to span the gap between bio(cid:173) physics and cognitive science to address and explore issues of how  a discrete symbol processing system can arise from the continuum,  and how complex dynamics like oscillation and synchronization can  then be employed in its operation and affect its learning. We show  how a discrete-time recurrent "Elman" network architecture can  be constructed from recurrently connected oscillatory associative  memory modules described by continuous nonlinear ordinary dif(cid:173) ferential equations. The modules can learn connection weights be(cid:173) tween themselves which will cause the system to evolve under a  clocked "machine cycle" by a sequence of transitions of attractors  within the modules, much as a digital computer evolves by transi(cid:173) tions of its binary flip-flop attractors. The architecture thus em(cid:173) ploys the principle of "computing with attractors" used by macro(cid:173) scopic systems for reliable computation in the presence of noise. We  have specifically constructed a system which functions as a finite  state automaton that recognizes or generates the infinite set of six  symbol strings that are defined by a Reber grammar. It is a symbol  processing system, but with analog input and oscillatory subsym(cid:173) bolic representations. The time steps (machine cycles) of the sys(cid:173) tem are implemented by rhythmic variation (clocking) of a bifurca(cid:173) tion parameter. This holds input and "context" modules clamped  at their attractors while 'hidden and output modules change state,  then clamps hidden and output states while context modules are  released to load those states as the new context for the next cycle of  input. Superior noise immunity has been demonstrated for systems  with dynamic attractors over systems with static attractors, and  synchronization ("binding") between coupled oscillatory attractors  in different modules has been shown to be important for effecting  reliable transitions.
Attractor Neural Networks with Local Inhibition: from Statistical Physics to a Digitial Programmable Integrated Circuit
Authors: R. Zecchina E. Pasero
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Networks with local inhibition are shown to have enhanced compu(cid:173) tational performance with respect to the classical Hopfield-like net(cid:173) works. In particular the critical capacity of the network is increased  as well as its capability to store correlated patterns. Chaotic dy(cid:173) namic behaviour (exponentially long transients) of the devices in(cid:173) dicates the overloading of the associative memory. An implementa(cid:173) tion based on a programmable logic device is here presented. A 16  neurons circuit is implemented whit a XILINK 4020 device. The  peculiarity of this solution is the possibility to change parts of the  project (weights, transfer function or the whole architecture) with  a simple software download of the configuration into the XILINK  chip.
Metamorphosis Networks: An Alternative to Constructive Models
Authors: Michael Mozer Brian V. Bonnlander
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Given a set oft raining examples, determining the appropriate num(cid:173) ber  of  free  parameters  is  a  challenging  problem.  Constructive  learning algorithms attempt to solve this problem automatically by  adding  hidden  units,  and  therefore  free  parameters,  during  learn(cid:173) ing.  We  explore  an  alternative  class  of algorithms-called  meta(cid:173) morphosis  algorithms-in  which  the  number  of units  is  fixed,  but  the number of free  parameters gradually increases  during learning.  The architecture we investigate is composed of RBF units on a lat(cid:173) tice,  which  imposes  flexible  constraints  on  the  parameters  of the  network.  Virtues  of this  approach  include  variable  subset  selec(cid:173) tion,  robust  parameter  selection,  multiresolution  processing,  and  interpolation  of sparse training  data.
A Hybrid Linear/Nonlinear Approach to Channel Equalization Problems
Authors: John Pearson Wei-Tsih Lee
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Channel equalization problem is an important problem in high-speed  communications. The sequences of symbols transmitted are distorted by  neighboring symbols. Traditionally, the channel equalization problem is  considered as a channel-inversion operation. One problem of this  approach is that there is no direct correspondence between error proba(cid:173) bility and residual error produced by the channel inversion operation. In  this paper, the optimal equalizer design is formulated as a classification  problem. The optimal classifier can be constructed by Bayes decision  rule. In general it is nonlinear. An efficient hybrid linear/nonlinear  equalizer approach has been proposed to train the equalizer. The error  probability of new linear/nonlinear equalizer has been shown to be bet(cid:173) ter than a linear equalizer in an experimental channel.
Physiologically Based Speech Synthesis
Authors: Eric Vatikiotis-Bateson, Kiyoshi Honda, Yasuharu Koike, Mitsuo Kawato Makoto Hirayama
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This study demonstrates a  paradigm for  modeling speech  produc(cid:173) tion  based  on  neural  networks.  Using  physiological  data  from  speech  utterances,  a  neural  network  learns  the  forward  dynamics  relating  motor  commands  to  muscles  and  the  ensuing  articulator  behavior  that  allows  articulator trajectories  to be generated  from  motor commands constrained by phoneme input strings and global  performance parameters.  From these movement trajectories, a sec(cid:173) ond  neural  network generates  PARCOR parameters  that  are  then  used  to synthesize  the speech  acoustics.
Weight Space Probability Densities in Stochastic Learning: II. Transients and Basin Hopping Times
Authors: Todd K. Leen Genevieve B. Orr
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In  stochastic  learning,  weights  are  random  variables  whose  time  evolution  is  governed  by  a  Markov  process.  At  each  time-step,  n,  the weights  can be described by  a  probability density function  pew, n).  We summarize the theory of the time evolution of P, and  give  graphical  examples  of  the  time  evolution  that  contrast  the  behavior  of stochastic learning  with  true gradient  descent  (batch  learning).  Finally, we use the formalism to obtain predictions of the  time required for noise-induced hopping between basins of different  optima.  We  compare the theoretical predictions with simulations  of large  ensembles  of networks  for  simple  problems in  supervised  and unsupervised learning. 
1  Weight-Space Probability  Densities 
Despite the recent application of convergence theorems from stochastic approxima(cid:173) tion  theory to  neural  network  learning  (Oja 1982,  White  1989)  there remain  out(cid:173) standing questions about the search dynamics in stochastic learning.  For example,  the convergence theorems do  not  tell  us  to which  of several  optima the algorithm
Performance Through Consistency: MS-TDNN's for Large Vocabulary Continuous Speech Recognition
Authors: Alex Waibel Joe Tebelskis
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Connectionist Rpeech recognition systems are often handicapped by  an  inconsistency  between  training and  testing  criteria.  This  prob(cid:173) lem  is  addressed  by  the  Multi-State  Time Delay  Neural  Network  (MS-TDNN), a hierarchical phonf'mp and word classifier which uses  DTW  to  modulate  its  connectivit.y  pattern,  and  which  is  directly  trained  on  word-level  targets.  The  consistent  use  of word  accu(cid:173) racy  as  a  criterion  during  bot.h  t.raining  and  testing  leads  to  very  high  system  performance,  even  wif II  limited  training  dat.a.  Until  now,  the  MS-TDN N  has  been  appli('d  primarily  to  small  vocabu(cid:173) lary  recognition  and  word  spotting  tasks.  In  this  papf'f  we  apply  the architecture to large vocabulary continuous speech recognition,  and  demonstrate  that  our  MS-TDNN  outperforms  all  ot,hf'r  sys(cid:173) tems  that  have  been  tested  on  tht'  eMU  Conference  Registration  database.
Generic Analog Neural Computation - The EPSILON Chip
Authors: Donald J. Baxter, Alister Hamilton, Alan F. Murray, H. Martin Reekie Stephen Churcher
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: An analog CMOS VLSI neural processing chip has been designed and fabri(cid:173) cated. The device employs "pulse-stream" neural state signalljng, and is capa(cid:173) ble of computing some 360 million synaptic connections per secood. In addi(cid:173) tion to basic characterisation results. the performance of the chip in solving  "real-world" problems is also demonstrated.
Analog VLSI Implementation of Multi-dimensional Gradient Descent
Authors: Douglas Kerns, Kurt Fleischer, Alan H. Barr David B. Kirk
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We describe an analog VLSI implementation of a multi-dimensional  gradient estimation and descent technique for minimizing an on(cid:173) chip scalar function fO. The implementation uses noise injec(cid:173) tion and multiplicative correlation to estimate derivatives, as in  [Anderson, Kerns 92]. One intended application of this technique  is setting circuit parameters on-chip automatically, rather than  manually [Kirk 91]. Gradient descent optimization may be used  to adjust synapse weights for a backpropagation or other on-chip  learning implementation. The approach combines the features of  continuous multi-dimensional gradient descent and the potential  for an annealing style of optimization. We present data measured  from our analog VLSI implementation.
Improving Performance in Neural Networks Using a Boosting Algorithm
Authors: Robert Schapire, Patrice Simard Harris Drucker
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A boosting algorithm converts a learning machine with error rate less  than 50% to one with an arbitrarily low error rate. However, the  algorithm discussed here depends on having a large supply of  independent training samples. We show how to circumvent this  problem and generate an ensemble of learning machines whose  performance in optical character recognition problems is dramatically  improved over that of a single network. We report the effect of  boosting on four databases (all handwritten) consisting of 12,000 digits  from segmented ZIP codes from the United State Postal Service  (USPS) and the following from the National Institute of Standards and  Testing (NIST): 220,000 digits, 45,000 upper case alphas, and 45,000  lower case alphas. We use two performance measures: the raw error  rate (no rejects) and the reject rate required to achieve a 1% error rate  on the patterns not rejected. Boosting improved performance in some  cases by a factor of three.
An Object-Oriented Framework for the Simulation of Neural Nets
Authors: Th. Sudbrak, Ch. Tietz, F. Weber A. Linden
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The field of software simulators for neural networks has been ex(cid:173) panding very rapidly in the last years but their importance is still  being underestimated. They must provide increasing levels of as(cid:173) sistance for the design, simulation and analysis of neural networks.  With our object-oriented framework (SESAME) we intend to show  that very high degrees of transparency, manageability and flexibil(cid:173) ity for complex experiments can be obtained. SESAME's basic de(cid:173) sign philosophy is inspired by the natural way in which researchers  explain their computational models. Experiments are performed  with networks of building blocks, which can be extended very eas(cid:173) ily. Mechanisms have been integrated to facilitate the construction  and analysis of very complex architectures. Among these mech(cid:173) anisms are t.he automatic configuration of building blocks for an  experiment and multiple inheritance at run-time.
A Practice Strategy for Robot Learning Control
Authors: Terence D. Sanger
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: "Trajectory  Extension  Learning"  is  a  new  technique for  Learning  Control in Robots which assumes that there exists some parameter  of the desired trajectory that can be smoothly varied from a region  of easy  solvability of the dynamics to  a  region  of desired behavior  which may have more difficult dynamics.  By gradually varying the  parameter, practice movements remain near the desired path while  a  Neural Network learns to approximate the inverse dynamics.  For  example,  the average speed of motion might be varied, and the in(cid:173) verse dynamics can be  "bootstrapped" from  slow movements with  simpler dynamics to fast  movements.  This provides an example of  the more general  concept  of a  "Practice  Strategy"  in  which  a  se(cid:173) quence of intermediate tasks is  used to simplify learning a complex  task.  I  show  an example  of the  application  of this  idea to  a  real  2-joint  direct drive robot arm.
Modeling Consistency in a Speaker Independent Continuous Speech Recognition System
Authors: Nelson Morgan, Chuck Wooters, Victor Abrash, Michael Cohen, Horacio Franco Yochai Konig
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We  would like to  incorporate speaker-dependent consistencies, such  as  gender, in an otherwise speaker-independent speech recognition system.  In this paper we discuss a Gender Dependent Neural Network (GDNN)  which can  be tuned for each  gender,  while sharing most of the speaker  independent parameters. We use a classification network to help generate  gender-dependent phonetic probabilities for a statistical (HMM) recogni(cid:173) tion system.  The gender classification net predicts the gender with high  accuracy,  98.3% on a Resource Management test set.  However,  the in(cid:173) tegration of the GDNN into our hybrid HMM-neural network recognizer  provided an improvement in the recognition score that is not statistically  significant on a Resource Management test set.
Learning to categorize objects using temporal coherence
Authors: Suzanna Becker
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The invariance of an objects' identity as it transformed over time  provides a powerful cue for perceptual learning. We present an un(cid:173) supervised learning procedure which maximizes the mutual infor(cid:173) mation between the representations adopted by a feed-forward net(cid:173) work at consecutive time steps. We demonstrate that the network  can learn, entirely unsupervised, to classify an ensemble of several  patterns by observing pattern trajectories, even though there are  abrupt transitions from one object to another between trajecto(cid:173) ries. The same learning procedure should be widely applicable to  a variety of perceptual learning tasks.
Learning Curves, Model Selection and Complexity of Neural Networks
Authors: Shuji Yoshizawa, Shun-ichi Amari Noboru Murata
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Learning  curves  show  how  a  neural  network  is  improved  as  the  number  of  t.raiuing  examples  increases  and  how  it  is  related  to  the  network  complexity.  The  present  paper  clarifies  asymptotic  properties and  their relation of t.wo  learning curves, one concerning  the predictive loss  or generalization  loss  and  the other the training  loss.  The  result  gives  a  natural  definition  of the  complexity  of  a  neural  network.  Moreover,  it  provides  a  new  criterion  of  model  selection.
Optimal Depth Neural Networks for Multiplication and Related Problems
Authors: Vwani Roychowdhury Kai-Yeung Siu
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: An artificial neural network (ANN) is commonly modeled by a threshold  circuit, a network of interconnected processing units called linear threshold  gates. The depth of a network represents the number of unit delays or the  time for parallel computation. The SIze of a circuit is the number of gates  and measures the amount of hardware . It was known that traditional logic  circuits consisting of only unbounded fan-in AND, OR, NOT gates would  require at least O(log n/log log n) depth to compute common arithmetic  functions such as the product or the quotient of two n-bit numbers, unless  we allow the size (and fan-in) to increase exponentially (in n). We show in  this paper that ANNs can be much more powerful than traditional logic  circuits. In particular, we prove that that iterated addition can be com(cid:173) puted by depth-2 ANN, and multiplication and division can be computed  by depth-3 ANNs with polynomial size and polynomially bounded integer  weights, respectively. Moreover, it follows from known lower bound re(cid:173) sults that these ANNs are optimal in depth. We also indicate that these  techniques can be applied to construct polynomial-size depth-3 ANN for  powering, and depth-4 ANN for mUltiple product.
Time Warping Invariant Neural Networks
Authors: Hsing-Hen Chen, Yee-Chun Lee Guo-Zheng Sun
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We proposed a model of Time Warping Invariant Neural Networks (TWINN) 
to handle the time warped continuous signals. Although TWINN is a simple modifica(cid:173) tion of well known recurrent neural  network, analysis  has  shown  that TWINN com(cid:173) pletely removes time warping and is  able to handle difficult classification problem. It  is also shown that TWINN has certain advantages over the current available sequential  processing  schemes:  Dynamic  Programming(DP)[I],  Hidden  Markov  Model((cid:173) HMM)[2],  Time  Delayed  Neural  Networks(TDNN)  [3]  and  Neural  Network  Finite  Automata(NNFA)[4]. 
We also analyzed the time continuity employed in TWINN and pointed out that 
this  kind of structure can memorize longer input history compared with Neural Net(cid:173) work Finite Automata (NNFA).  This  may  help  to  understand  the  well  accepted fact  that for  learning  grammatical reference with NNF A one had to start with  very short  strings in training set. 
The  numerical  example  we  used  is  a  trajectory  classification  problem.  This 
problem, making a feature of variable sampling rates, having internal states, continu(cid:173) ous  dynamics,  heavily  time-warped  data  and  deformed  phase  space  trajectories,  is  shown to be difficult to other schemes. With TWINN this problem has been learned in  100 iterations. For benchmark we also trained the exact same problem with TDNN and  completely failed as expected.
Deriving Receptive Fields Using an Optimal Encoding Criterion
Authors: Ralph Linsker
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: An  information-theoretic  optimization  principle  ('infomax')  has  previously  been  used  for  unsupervised  learning  of statistical  reg(cid:173) ularities in an input ensemble.  The principle states that the input(cid:173) output mapping implemented by a  processing stage should be cho(cid:173) sen  so  as  to  maximize  the  average  mutual  information  between  input  and output  patterns,  subject  to constraints  and in  the  pres(cid:173) ence  of processing  noise.  In the present  work  I show  how  infomax,  when  applied  to  a  class  of nonlinear  input-output  mappings,  can  under  certain  conditions  generate  optimal filters  that  have  addi(cid:173) tional  useful  properties:  (1)  Output  activity  (for  each  input  pat(cid:173) tern)  tends  to  be  concentrated  among  a  relatively  small  number  (2)  The  filters  are  sensitive  to  higher-order  statistical  of  nodes.  structure  (beyond  pairwise correlations).  If the  input features  are  localized,  the  filters'  receptive  fields  tend  to  be  localized  as  well.  (3)  Multiresolution sets  of filters  with  subsampling at  low  spatial  frequencies - related to pyramid coding and wavelet representations  - emerge as favored  solutions for  certain types of input ensembles.
Connected Letter Recognition with a Multi-State Time Delay Neural Network
Authors: Alex Waibel Hermann Hild
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The  Multi-State  Time  Delay  Neural  Network  (MS-TDNN)  inte(cid:173) grates  a nonlinear time alignment procedure  (DTW) and the high(cid:173) accuracy  phoneme spotting capabilities of a TDNN into  a  connec(cid:173) tionist speech recognition system with word-level classification and  error  backpropagation.  We  present  an  MS-TDNN  for  recognizing  continuously  spelled  letters,  a  task  characterized  by  a  small  but  highly confusable vocabulary.  Our MS-TDNN  achieves  98.5/92.0%  word  accuracy  on  speaker  dependent/independent  tasks,  outper(cid:173) forming previously reported results on the same databases.  We pro(cid:173) pose training techniques  aimed at improving sentence  level  perfor(cid:173) mance, including free  alignment  across  word  boundaries,  word  du(cid:173) ration  modeling and error backpropagation on  the sentence  rather  than the word level.  Architectures  integrating submodules special(cid:173) ized  on  a  subset  of speakers  achieved further  improvements.
Input Reconstruction Reliability Estimation
Authors: Dean A. Pomerleau
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper describes a technique called Input Reconstruction Reliability Estimation  (IRRE) for determining the response reliability of a restricted class of multi-layer  perceptrons (MLPs). The technique uses a network's ability to accurately encode  the input pattern in its internal representation as a measure of its reliability. The  more accurately a network is able to reconstruct the input pattern from its internal  representation, the more reliable the network is considered to be. IRRE is provides  a good estimate of the reliability of MLPs trained for autonomous driving. Results  are presented in which the reliability estimates provided by IRRE are used to select  between networks trained for different driving situations.
Mapping Between Neural and Physical Activities of the Lobster Gastric Mill
Authors: Mary E. T. Boyle, Allen I. Selverston Kenji Doya
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A computer model of the musculoskeletal system of the lobster  gastric mill was constructed in order to provide a behavioral in(cid:173) terpretation of the rhythmic patterns obtained from isolated stom(cid:173) atogastric ganglion. The model was based on Hill's muscle model  and quasi-static approximation of the skeletal dynamics and could  simulate the change of chewing patterns by the effect of neuromod(cid:173) ulators. 
1 THE STOMATOGASTRIC NERVOUS SYSTEM 
The crustacean stomatogastric ganglion (STG) is a circuit of 30 neurons that con(cid:173) trols rhythmic movement of the foregut. It is one of the best elucidated neural  circuits. All the neurons and the synaptic connections between them are identi(cid:173) fied and the effects of neuromodulators on the oscillation patterns and neuronal  characteristics have been extensively studied (Selverston and Moulins 1987, H arris(cid:173) Warrick et al. 1992). However, STG's function as a controller of ingestive behavior  is not fully understood in part because of our poor understanding of the controlled  object: the musculoskeletal dynamics of the foregut. We constructed a mathemat(cid:173) ical model of the gastric mill, three teeth in the stomach, in order to predict motor  patterns from the neural oscillation patterns which are recorded from the isolated  ganglion. 
The animal we used was the Californian spiny lobster (Panulirus interruptus), which
A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization
Authors: Gert Cauwenberghs
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A  parallel  stochastic  algorithm  is  investigated  for  error-descent  learning  and  optimization in  deterministic  networks  of arbitrary  topology.  No  explicit  information  about  internal  network  struc(cid:173) ture is needed.  The method is  based on the model-free distributed  learning mechanism of Dembo and Kailath.  A modified parameter  update rule is proposed by which each individual parameter vector  perturbation contributes a  decrease in error.  A substantially faster  learning speed  is  hence  allowed.  Furthermore,  the  modified  algo(cid:173) rithm  supports  learning  time-varying  features  in  dynamical  net(cid:173) works.  We  analyze  the  convergence  and  scaling  properties  of the  algorithm,  and  present  simulation results  for  dynamic  trajectory  learning in recurrent  networks. 
1  Background and  Motivation 
We address general optimization tasks that require finding a set of constant param(cid:173) eter  values Pi  that minimize a  given error  functional £(p).  For supervised learning,  the error functional consists of some quantitative measure of the deviation between  a  desired  state  x T  and  the  actual state of a  network  x,  resulting  from  an input  y  and  the  parameters  p.  In  such  context  the  components  of p  consist  of the  con(cid:173) nection  strengths,  thresholds  and  other  adjustable  parameters  in  the  network.  A
Nets with Unreliable Hidden Nodes Learn Error-Correcting Codes
Authors: Paul W. Munro Stephen Judd
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In a multi-layered neural network, anyone of the hidden layers can be  viewed as computing a distributed representation of the input. Several  "encoder" experiments have shown that when the representation space is  small it can be fully used. But computing with such a representation  requires completely dependable nodes. In the case where the hidden  nodes are noisy and unreliable, we find that error correcting schemes  emerge simply by using noisy units during training; random errors in(cid:173) jected during backpropagation result in spreading representations apart.  Average and minimum distances increase with misfire probability, as  predicted by coding-theoretic considerations. Furthennore, the effect of  this noise is to protect the machine against permanent node failure,  thereby potentially extending the useful lifetime of the machine.
Unsmearing Visual Motion: Development of Long-Range Horizontal Intrinsic Connections
Authors: Jonathan A. Marshall Kevin E. Martin
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Human VlSlon systems integrate information nonlocally, across long  spatial ranges.  For example, a moving stimulus appears smeared  when viewed briefly (30 ms), yet sharp when viewed for a longer  exposure (100 ms) (Burr, 1980). This suggests that visual systems  combine information along a trajectory that matches the motion of  the stimulus. Our self-organizing neural network model shows how  developmental exposure to moving stimuli can direct the formation of  horizontal trajectory-specific motion integration pathways that unsmear  representations of moving stimuli. These results account for Burr's data  and can potentially also model ot.her phenomena, such as visual inertia.
Remote Sensing Image Analysis via a Texture Classification Neural Network
Authors: Rodney Goodman Hayit K. Greenspan
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: In this work we apply a texture classification network to remote sensing im(cid:173) age analysis. The goal is to extract the characteristics of the area depicted  in the input image, thus achieving a segmented map of the region. We have  recently proposed a combined neural network and rule-based framework  for texture recognition. The framework uses unsupervised and supervised  learning, and provides probability estimates for the output classes. We  describe the texture classification network and extend it to demonstrate  its application to the Landsat and Aerial image analysis domain .
Filter Selection Model for Generating Visual Motion Signals
Authors: Terrence J. Sejnowski Steven J. Nowlan
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Neurons  in  area  MT of primate visual  cortex  encode  the  velocity  of moving objects.  We present  a  model of how  MT cells  aggregate  responses  from  VI  to  form  such  a  velocity  representation.  Two  different  sets  of units,  with  local  receptive  fields,  receive  inputs  from motion energy filters.  One set of units forms estimates of local  motion, while the second set computes the utility of these estimates.  Outputs from  this second set  of units  "gate"  the outputs from  the  first  set  through  a  gain  control  mechanism.  This  active  process  for  selecting  only  a  subset  of local  motion  responses  to  integrate  into more global  responses  distinguishes  our  model from  previous  models of velocity estimation.  The model yields  accurate  velocity  estimates  in  synthetic  images  containing multiple moving  targets  of varying size,  luminance,  and spatial frequency  profile and  deals  well  with a  number of transparency  phenomena.
Non-Linear Dimensionality Reduction
Authors: Garrison W. Cottrell David DeMers
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: A method for creating a non-linear encoder-decoder for multidimensional data  with  compact  representations  is  presented.  The commonly  used  technique of  autoassociation is extended  to allow non-linear representations,  and an  objec(cid:173) tive  function  which  penalizes activations  of individual  hidden  units  is  shown  to  result in minimum dimensional encodings with respect to allowable error in  reconstruction.
Feudal Reinforcement Learning
Authors: Geoffrey E. Hinton Peter Dayan
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: One way to speed up reinforcement learning is to enable learning to  happen simultaneously at multiple resolutions in space and time.  This paper shows how to create a Q-Iearning managerial hierarchy  in which high level managers learn how to set tasks to their sub(cid:173) managers who, in turn, learn how to satisfy them.  Sub-managers  need  not initially understand  their managers' commands.  They  simply learn to maximise their reinforcement in the context of the  current command.  We illustrate the system using a simple maze task ..  As the system  learns  how to get around,  satisfying commands at the multiple  levels, it explores more efficiently than standard, flat,  Q-Iearning  and builds a more comprehensive map.
A Neural Network that Learns to Interpret Myocardial Planar Thallium Scintigrams
Authors: Jacob Erel, Henri Atlan Charles Rosenberg
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The planar thallium-201 myocardial perfusion scintigram is a widely used  diagnostic technique for detecting and estimating the risk of coronary  artery disease. Neural networks learned to interpret 100 thallium scinti(cid:173) grams as determined by individual expert ratings. Standard error back(cid:173) propagation was compared to standard LMS, and LMS combined with  one layer of RBF units. Using the "leave-one-out" method, generaliza(cid:173) tion was tested on all 100 cases. Training time was determined automati(cid:173) cally from cross-validation perfonnance. Best perfonnance was attained  by the RBF/LMS network with three hidden units per view and compares  favorably with human experts.
Learning Cellular Automaton Dynamics with Neural Networks
Authors: J A Hertz N. H. Wulff
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have trained networks of E - II units with short-range connec(cid:173) tions to simulate simple cellular automata that exhibit complex or  chaotic behaviour. Three levels of learning are possible (in decreas(cid:173) ing order of difficulty): learning the underlying automaton rule,  learning asymptotic dynamical behaviour, and learning to extrap(cid:173) olate the training history. The levels of learning achieved with and  without weight sharing for different automata provide new insight  into their dynamics.
Probability Estimation from a Database Using a Gibbs Energy Model
Authors: Rodney M. Goodman John W. Miller
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We present an algorithm for creating a neural network which pro(cid:173) duces accurate probability estimates as outputs. The network im(cid:173) plements a Gibbs probability distribution model of the training  database. This model is created by a new transformation relating  the joint probabilities of attributes in the database to the weights  (Gibbs potentials) of the distributed network model. The theory  of this transformation is presented together with experimental re(cid:173) sults. One advantage of this approach is the network weights are  prescribed without iterative gradient descent. Used as a classifier  the network tied or outperformed published results on a variety of  databases.
Word Space
Authors: Hinrich Schütze
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Representations  for  semantic  information about  words  are  neces(cid:173) sary  for  many applications of neural  networks  in natural language  processing.  This paper describes  an efficient,  corpus-based method  for  inducing distributed semantic  representations  for  a  large num(cid:173) ber  of words  (50,000)  from  lexical  coccurrence  statistics by means  of a  large-scale  linear  regression.  The  representations  are  success(cid:173) fully applied to word sense disambiguation using a nearest neighbor  method .
An Analog VLSI Chip for Radial Basis Functions
Authors: John C. Platt, David B. Kirk Janeen Anderson
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: We have designed, fabricated, and tested an analog VLSI chip  which computes radial basis functions in parallel. We have de(cid:173) veloped a synapse circuit that approximates a quadratic function.  We aggregate these circuits to form radial basis functions. These  radial basis functions are then averaged together using a follower  aggregator.
Predicting Complex Behavior in Sparse Asymmetric Networks
Authors: William B. Levy Ali A. Minai
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: Recurrent  networks  of threshold  elements  have  been  studied  inten(cid:173) sively  as  associative  memories and pattern-recognition devices. While  most  research  has  concentrated  on  fully-connected  symmetric  net(cid:173) works.  which  relax  to  stable  fixed  points.  asymmetric  networks  show  richer dynamical behavior. and can be used as  sequence  generators or  flexible  pattern-recognition  devices.  In  this  paper.  we  approach  the  problem of predicting  the  complex global behavior of a  class  of ran(cid:173) dom asymmetric networks in terms of network parameters.  These net(cid:173) works can show fixed-point.  cyclical or effectively aperiodic  behavior.  depending  on  parameter  values.  and  our approach  can  be used  to  set  parameters.  as  necessary.  to  obtain a  desired complexity of dynamics.  The  approach  also  provides  qualitative  insight  into  why  the  system  behaves as it does and suggests possible applications.
Destabilization and Route to Chaos in Neural Networks with Random Connectivity
Authors: Bruno Cessac, Mathias Quoy, Manuel Samuelides Bernard Doyon
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: The occurence of chaos  in recurrent neural  networks  is  supposed to  depend on the architecture and on the synaptic coupling strength. It is  studied here for a randomly diluted architecture.  By normalizing  the  variance  of synaptic  weights,  we  produce  a  bifurcation  parameter,  dependent on this variance and on the slope of the transfer function but  independent of the connectivity, that allows a sustained activity and the  occurence  of  chaos  when  reaching  a  critical  value.  Even  for  weak  connectivity  and small size, we find  numerical results in accordance  with the theoretical  ones  previously  established  for  fully  connected  infinite  sized  networks.  Moreover  the  route  towards  chaos  is  numerically checked to be a quasi-periodic one, whatever the type of the  first bifurcation is (Hopf bifurcation, pitchfork or flip).
Recognition-based Segmentation of On-Line Hand-printed Words
Authors: H. Weissman, I. Guyon, C. Nohl, D. Henderson M. Schenkel
Published in: Advances in Neural Information Processing Systems 5  (NIPS 1992) (Main Conference Track), 1992

Abstract: This paper reports on the performance of two methods for  recognition-based segmentation of strings of on-line hand-printed  capital Latin characters. The input strings consist of a time(cid:173) ordered sequence of X-Y coordinates, punctuated by pen-lifts. The  methods were designed to work in "run-on mode" where there is no  constraint on the spacing between characters. While both methods  use a neural network recognition engine and a graph-algorithmic  post-processor, their approaches to segmentation are quite differ(cid:173) ent. The first method, which we call IN SEC (for input segmen(cid:173) tation), uses a combination of heuristics to identify particular pen(cid:173) lifts as tentative segmentation points. The second method, which  we call OUTSEC (for output segmentation), relies on the empiri(cid:173) cally trained recognition engine for both recognizing characters and  identifying relevant segmentation points.