mtybilly/Instruct2DS huggingface dataset:
Exponentiated Gradient Algorithms for Large-margin Structured Classification
Authors: Michael Collins, Ben Taskar, David A. McAllester Peter L. Bartlett
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We consider the problem of structured classiﬁcation, where the task is to predict a label y from an input x, and y has meaningful internal struc- ture. Our framework includes supervised training of Markov random ﬁelds and weighted context-free grammars as special cases. We describe an algorithm that solves the large-margin optimization problem deﬁned in [12], using an exponential-family (Gibbs distribution) representation of structured objects. The algorithm is efﬁcient—even in cases where the number of labels y is exponential in size—provided that certain expecta- tions under Gibbs distributions can be calculated efﬁciently. The method for structured labels relies on a more general result, speciﬁcally the ap- plication of exponentiated gradient updates [7, 8] to quadratic programs.
Semi-Markov Conditional Random Fields for Information Extraction
Authors: William W. Cohen Sunita Sarawagi
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We describe semi-Markov conditional random ﬁelds (semi-CRFs), a con- ditionally trained version of semi-Markov chains. Intuitively, a semi- CRF on an input sequence x outputs a “segmentation” of x, in which labels are assigned to segments (i.e., subsequences) of x rather than to individual elements xi of x. Importantly, features for semi-CRFs can measure properties of segments, and transitions within a segment can be non-Markovian. In spite of this additional power, exact learning and inference algorithms for semi-CRFs are polynomial-time—often only a small constant factor slower than conventional CRFs. In experiments on ﬁve named entity recognition problems, semi-CRFs generally outper- form conventional CRFs.
Adaptive Manifold Learning
Authors: Zhenyue Zhang, Hongyuan Zha Jing Wang
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Recently, there have been several advances in the machine learning and           pattern recognition communities for developing manifold learning algo-           rithms to construct nonlinear low-dimensional manifolds from sample           data points embedded in high-dimensional spaces. In this paper, we de-           velop algorithms that address two key issues in manifold learning: 1)           the adaptive selection of the neighborhood sizes; and 2) better fitting the           local geometric structure to account for the variations in the curvature           of the manifold and its interplay with the sampling density of the data           set. We also illustrate the effectiveness of our methods on some synthetic           data sets.
Euclidean Embedding of Co-Occurrence Data
Authors: Gal Chechik, Fernando Pereira, Naftali Tishby Amir Globerson
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Embedding algorithms search for low dimensional structure in complex          data, but most algorithms only handle objects of a single type for which          pairwise distances are specified. This paper describes a method for em-          bedding objects of different types, such as images and text, into a single          common Euclidean space based on their co-occurrence statistics. The          joint distributions are modeled as exponentials of Euclidean distances in          the low-dimensional embedding space, which links the problem to con-          vex optimization over positive semidefinite matrices. The local struc-          ture of our embedding corresponds to the statistical correlations via ran-          dom walks in the Euclidean space. We quantify the performance of our          method on two text datasets, and show that it consistently and signifi-          cantly outperforms standard methods of statistical correspondence mod-          eling, such as multidimensional scaling and correspondence analysis.
Heuristics for Ordering Cue Search in Decision Making
Authors: Anja Dieckmann Peter M. Todd
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Simple lexicographic decision heuristics that consider cues one at a 
          time  in  a  particular  order  and  stop  searching  for  cues  as  soon  as  a 
          decision  can  be  made  have  been  shown  to  be  both  accurate  and 
          frugal  in  their  use  of  information.    But  much  of  the  simplicity  and 
          success  of  these  heuristics  comes  from  using  an  appropriate  cue 
          order.  For instance, the Take The Best heuristic uses validity order 
          for  cues,  which  requires  considerable  computation,  potentially 
          undermining  the  computational  advantages  of  the  simple  decision 
          mechanism.    But  many  cue  orders  can  achieve  good  decision 
          performance, and studies of sequential search for data records have 
          proposed  a  number  of  simple  ordering  rules  that  may  be  of  use  in 
          constructing  appropriate  decision  cue  orders  as  well.    Here  we 
          consider  a  range  of  simple  cue  ordering  mechanisms,  including 
          tallying, swapping, and move-to-front rules, and show that they can 
          find  cue  orders  that  lead  to  reasonable  accuracy  and  considerable 
          frugality when used with lexicographic decision heuristics.
Coarticulation in Markov Decision Processes
Authors: Robert Platt, Sridhar Mahadevan, Roderic Grupen Khashayar Rohanimanesh
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We investigate an approach for simultaneously committing to mul-         tiple activities, each modeled as a temporally extended action in         a semi-Markov decision process (SMDP). For each activity we de-         fine a set of admissible solutions consisting of the redundant set of         optimal policies, and those policies that ascend the optimal state-         value function associated with them. A plan is then generated by         merging them in such a way that the solutions to the subordinate         activities are realized in the set of admissible solutions satisfying         the superior activities. We present our theoretical results and em-         pirically evaluate our approach in a simulated domain.
Integrating Topics and Syntax
Authors: Mark Steyvers, David M. Blei, Joshua B. Tenenbaum Thomas L. Griffiths
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Statistical approaches to language learning typically focus on either          short-range syntactic dependencies or long-range semantic dependencies          between words. We present a generative model that uses both kinds of          dependencies, and can be used to simultaneously find syntactic classes          and semantic topics despite having no representation of syntax or seman-          tics beyond statistical dependency. This model is competitive on tasks          like part-of-speech tagging and document classification with models that          exclusively use short- and long-range dependencies respectively.
Object Classification from a Single Example Utilizing Class Relevance Metrics
Authors: Michael Fink
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We describe a framework for learning an object classifier from a single          example. This goal is achieved by emphasizing the relevant dimensions          for classification using available examples of related classes. Learning          to accurately classify objects from a single training example is often un-          feasible due to overfitting effects. However, if the instance representa-          tion provides that the distance between each two instances of the same          class is smaller than the distance between any two instances from dif-          ferent classes, then a nearest neighbor classifier could achieve perfect          performance with a single training example. We therefore suggest a two          stage strategy. First, learn a metric over the instances that achieves the          distance criterion mentioned above, from available examples of other          related classes. Then, using the single examples, define a nearest neigh-          bor classifier where distance is evaluated by the learned class relevance          metric. Finding a metric that emphasizes the relevant dimensions for          classification might not be possible when restricted to linear projections.          We therefore make use of a kernel based metric learning algorithm. Our          setting encodes object instances as sets of locality based descriptors and          adopts an appropriate image kernel for the class relevance metric learn-          ing. The proposed framework for learning from a single example is          demonstrated in a synthetic setting and on a character classification task.
A Large Deviation Bound for the Area Under the ROC Curve
Authors: Thore Graepel, Ralf Herbrich, Dan Roth Shivani Agarwal
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: The area under the ROC curve (AUC) has been advocated as an evalu- ation criterion for the bipartite ranking problem. We study large devi- ation properties of the AUC; in particular, we derive a distribution-free large deviation bound for the AUC which serves to bound the expected accuracy of a ranking function in terms of its empirical AUC on an inde- pendent test sequence. A comparison of our result with a corresponding large deviation result for the classiﬁcation error rate suggests that the test sample size required to obtain an -accurate estimate of the expected ac- curacy of a ranking function with δ-conﬁdence is larger than that required to obtain an -accurate estimate of the expected error rate of a classiﬁ- cation function with the same conﬁdence. A simple application of the union bound allows the large deviation bound to be extended to learned ranking functions chosen from ﬁnite function classes.
Adaptive Discriminative Generative Model and Its Applications
Authors: David A. Ross, Jongwoo Lim, Ming-Hsuan Yang Ruei-sung Lin
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: This paper presents an adaptive discriminative generative model that gen-          eralizes the conventional Fisher Linear Discriminant algorithm and ren-          ders a proper probabilistic interpretation. Within the context of object          tracking, we aim to find a discriminative generative model that best sep-          arates the target from the background. We present a computationally          efficient algorithm to constantly update this discriminative model as time          progresses. While most tracking algorithms operate on the premise that          the object appearance or ambient lighting condition does not significantly          change as time progresses, our method adapts a discriminative genera-          tive model to reflect appearance variation of the target and background,          thereby facilitating the tracking task in ever-changing environments. Nu-          merous experiments show that our method is able to learn a discrimina-          tive generative model for tracking target objects undergoing large pose          and lighting changes.
Modeling Conversational Dynamics as a Mixed-Memory Markov Process
Authors: Sumit Basu Tanzeem Choudhury
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: influences 
In  this  work,  we  quantitatively  investigate  the  ways  in  which  a  given  person  the  joint  turn-taking  behavior  in  a  conversation.  After  collecting  an  auditory  database  of  social  interactions  among  a  group  of twenty-three  people  via  wearable  sensors  (66  hours  of data  each  over  two  weeks),  we  apply  speech  and conversation detection methods to  the auditory streams.  These  methods  automatically  locate  the  conversations,  determine  their  participants,  and  mark  which  participant  was  speaking  when.  We  then  model  the  joint  turn-taking  behavior  as  a  Mixed-Memory  Markov  Model  [1]  that  combines  the  statistics  of the  individual  subjects'  self-transitions  and  the  partners '  cross-transitions.  The  mixture parameters in  this  model describe how much each person's  individual  behavior contributes  to  the joint turn-taking behavior of  the  pair.  By  estimating  these  parameters,  we  thus  estimate  how  much  influence  each participant  has  in  determining  the  joint turn(cid:173) this  measure  correlates  taking  behavior.  We  significantly  with  betweenness  centrality  [2],  an  independent  measure  of an  individual's  importance  in  a  social  network.  This  result  suggests  that  our  estimate  of  conversational  influence  is  predictive of social influence.
On-Chip Compensation of Device-Mismatch Effects in Analog VLSI Neural Networks
Authors: Seth Bridges, Chris Diorio Miguel Figueroa
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Device mismatch in VLSI degrades the accuracy of analog arithmetic           circuits and lowers the learning performance of large-scale neural net-           works implemented in this technology. We show compact, low-power           on-chip calibration techniques that compensate for device mismatch. Our           techniques enable large-scale analog VLSI neural networks with learn-           ing performance on the order of 10 bits. We demonstrate our techniques           on a 64-synapse linear perceptron learning with the Least-Mean-Squares           (LMS) algorithm, and fabricated in a 0.35m CMOS process.
Trait Selection for Assessing Beef Meat Quality Using Non-linear SVM
Authors: Gustavo F. Bayón, Jorge Díez, Oscar Luaces, Antonio Bahamonde, Carlos Sañudo Juan Coz
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: In this paper we show that it is possible to model sensory impressions          of consumers about beef meat. This is not a straightforward task; the          reason is that when we are aiming to induce a function that maps object          descriptions into ratings, we must consider that consumers' ratings are          just a way to express their preferences about the products presented in          the same testing session. Therefore, we had to use a special purpose          SVM polynomial kernel. The training data set used collects the ratings of          panels of experts and consumers; the meat was provided by 103 bovines          of 7 Spanish breeds with different carcass weights and aging periods.          Additionally, to gain insight into consumer preferences, we used feature          subset selection tools. The result is that aging is the most important trait          for improving consumers' appreciation of beef meat.
Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes
Authors: Michael I. Jordan, Matthew J. Beal, David M. Blei Yee W. Teh
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We propose the hierarchical Dirichlet process (HDP), a nonparametric Bayesian model for clustering problems involving multiple groups of data. Each group of data is modeled with a mixture, with the number of components being open-ended and inferred automatically by the model. Further, components can be shared across groups, allowing dependencies across groups to be modeled effectively as well as conferring generaliza- tion to new groups. Such grouped clustering problems occur often in practice, e.g. in the problem of topic discovery in document corpora. We report experimental results on three text corpora showing the effective and superior performance of the HDP over previous models.
Spike-timing Dependent Plasticity and Mutual Information Maximization for a Spiking Neuron Model
Authors: Jean-pascal Pfister, Kazuyuki Aihara, Wulfram Gerstner Taro Toyoizumi
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We derive an optimal learning rule in the sense of mutual information maximization for a spiking neuron model. Under the assumption of small ﬂuctuations of the input, we ﬁnd a spike-timing dependent plas- ticity (STDP) function which depends on the time course of excitatory postsynaptic potentials (EPSPs) and the autocorrelation function of the postsynaptic neuron. We show that the STDP function has both positive and negative phases. The positive phase is related to the shape of the EPSP while the negative phase is controlled by neuronal refractoriness.
Comparing Beliefs, Surveys, and Random Walks
Authors: Uri Gordon, Scott Kirkpatrick Erik Aurell
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Survey propagation is a powerful technique from statistical physics that          has been applied to solve the 3-SAT problem both in principle and in          practice. We give, using only probability arguments, a common deriva-          tion of survey propagation, belief propagation and several interesting hy-          brid methods. We then present numerical experiments which use WSAT          (a widely used random-walk based SAT solver) to quantify the complex-          ity of the 3-SAT formulae as a function of their parameters, both as ran-          domly generated and after simplication, guided by survey propagation.          Some properties of WSAT which have not previously been reported make          it an ideal tool for this purpose  its mean cost is proportional to the num-          ber of variables in the formula (at a xed ratio of clauses to variables) in          the easy-SAT regime and slightly beyond, and its behavior in the hard-          SAT regime appears to reect the underlying structure of the solution          space that has been predicted by replica symmetry-breaking arguments.          An analysis of the tradeoffs between the various methods of search for          satisfying assignments shows WSAT to be far more powerful than has          been appreciated, and suggests some interesting new directions for prac-          tical algorithm development.
English Alphabet Recognition with Telephone Speech
Authors: Ronald A. Cole, Krist Roginski Mark Fanty
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A recognition system is reported which recognizes names spelled over the  telephone with brief pauses between letters. The system uses separate  neural networks to locate segment boundaries and classify letters. The  letter scores are then used to search a database of names to find the best  scoring name. The speaker-independent classification rate for spoken let(cid:173) ters is 89%. The system retrieves the correct name, spelled with pauses  between letters, 91 % of the time from a database of 50,000 names.
Recognizing Overlapping Hand-Printed Characters by Centered-Object Integrated Segmentation and Recognition
Authors: Mosfeq Rashid Gale L. Martin
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper describes an approach, called centered object integrated seg(cid:173) mentation and recognition (COISR). for integrating object segmenta(cid:173) tion and recognition within a single neural network. The application  is hand-printed character recognition. 1\vo versions of the system are  described. One uses a backpropagation network that scans exhaus(cid:173) tively over a field of characters and is trained to recognize whether  it is centered over a single character or between characters. When  it is centered over a character, the net classifies the cnaracter. The  approach is tested on a dataset of hand-printed digits. Vel)' low errOr  rates are reported. The second version, COISR-SACCADE, avoids  the need for exhaustive scans. The net is trained as before. but also  is trained to compute ballistic 'eye' movements that enable the input  window to jump from one character to the next. 
The common model of visual processing includes multiple, independent stages. First,  flltering operations act on the raw image to segment or isolate and enhance to-be-re(cid:173) cognized clumps. These clumps are normalized for factors such as size, and sometimes  simplified further through feature extraction. The results are then fed to one or more  classifiers. The operations prior to classification simplify the recognition task. Object  segmentation restricts the number of features considered for classification to those as(cid:173) sociated with a single object, and enables normalization to be applied at the individual  object level. Without such pre-processing. recognition may be an intractable problem.  However, a weak point of this sequential stage model is that recognition and segmenta(cid:173) tion decisions are often inter-dependent. Not only does a correct recognition decision  depend on first making a correct segmentation decision, but a correct segmentation  decision often depends on first making a correct recognition decision. 
This is a particularly serious problem in character recognition applications. OCR sys(cid:173) tems use intervening white space and related features to segment a field of characters  into individual characters, so that classification can be accomplished one character at  a time. This approach fails when characters touch each other or when an individual  character is broken up by intervening white space. Some means of integrating the seg(cid:173) mentation and recognition stages is needed. 
This paper descnbes an approach, called centered object integrated segmentation and rec(cid:173) ognition (COISR), for integrating character segmentation and recognition within one 
• Also with Eastman Kodak Company
Adaptive Soft Weight Tying using Gaussian Mixtures
Authors: Geoffrey E. Hinton Steven J. Nowlan
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: One way  of simplifying neural  networks  so  they generalize  better is  to add  an  extra  t.erm  10  the  error  fUll ction  that  will  penalize  complexit.y.  \Ve  propose  a  new  penalt.y  t.erm  in  which  the  dist rihution  of  weight  values  is  modelled  as  a  mixture  of  multiple  gaussians .  C nder  this  model,  a  set  of  weights  is  simple  if  the  weights  can  be  clustered  into  subsets  so  that  the  weights  in  each  cluster  have  similar  values .  We  allow  the  parameters  of  the  mixture  model  to  adapt  at  t.he  same  time  as  t.he  network  learns.  Simulations demonstrate  that  this complexity  term  is  more  effective  than  previous  complexity terms.
Multi-State Time Delay Networks for Continuous Speech Recognition
Authors: Alex Waibel Patrick Haffner
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present the "Multi-State Time Delay Neural Network" (MS-TDNN) as  an  extension of the TDNN to robust word recognition. Unlike most other hybrid  methods. the MS-TDNN embeds an alignment search procedure into the con(cid:173) nectionist architecture. and allows for word level supervision. The resulting  system  has the ability to manage the sequential order of subword units. while  optimizing for the recognizer performance. In this paper we present extensive  new evaluations of this approach over speaker-dependent and speaker-indepen(cid:173) dent connected alphabet.
Neural Network - Gaussian Mixture Hybrid for Speech Recognition or Density Estimation
Authors: Renato De Mori, Giovanni Flammia, Ralf Kompe Yoshua Bengio
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The subject of this paper is the integration of multi-layered Artificial Neu(cid:173) ral Networks (ANN) with probability density functions such as Gaussian  mixtures found in continuous density Hidden Markov Models (HMM). In  the first part of this paper we present an ANN/HMM hybrid in which  all the parameters of the the system are simultaneously optimized with  respect to a single criterion. In the second part of this paper, we study  the relationship between the density of the inputs of the network and the  density of the outputs of the networks. A few experiments are presented  to explore how to perform density estimation with ANNs.
Threshold Network Learning in the Presence of Equivalences
Authors: John Shawe-Taylor
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This  paper  applies  the  theory  of Probably  Approximately  Correct  (PAC)  learning  to  multiple  output  feedforward  threshold  networks  in  which  the  weights  conform  to  certain  equivalences.  It is  shown that  the sample  size  for  reliable  learning  can  be  bounded  above  by  a  formula  similar  to  that  required  for  single  output  networks  with  no equivalences.  The best  previ(cid:173) ously obtained  bounds  are improved for  all  cases.
Node Splitting: A Constructive Algorithm for Feed-Forward Neural Networks
Authors: Mike Wynne-Jones
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A constructive algorithm is proposed for feed-forward neural networks,  which uses node-splitting in the hidden layers to build large networks from  smaller ones. The small network forms an approximate model of a set of  training data, and the split creates a larger more powerful network which is  initialised with the approximate solution already found. The insufficiency  of the smaller network in modelling the system which generated the data  leads to oscillation in those hidden nodes whose weight vectors cover re(cid:173) gions in the input space where more detail is required in the model. These  nodes are identified and split in two using principal component analysis,  allowing the new nodes t.o cover the two main modes of each oscillating  vector. Nodes are selected for splitting using principal component analysis  on the oscillating weight vectors, or by examining the Hessian matrix of  second derivatives of the network error with respect to the weight.s. The  second derivat.ive method can also be applied to the input layer, where it  provides a useful indication of t.he relative import.ances of parameters for  the classification t.ask. Node splitting in a standard Multi Layer Percep(cid:173) t.ron is equivalent to introducing a hinge in the decision boundary to allow  more detail to be learned. Initial results were promising, but further eval(cid:173) uation indicates that the long range effects of decision boundaries cause  the new nodes to slip back to the old node position, and nothing is gained.  This problem does not occur in networks of localised receptive fields such  as radial basis functions or gaussian mixtures, where the t.echnique appears  to work well.
Structural Risk Minimization for Character Recognition
Authors: V. Vapnik, B. Boser, L. Bottou, S. A. Solla I. Guyon
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The method of Structural Risk Minimization refers to tuning the capacity  of the classifier to the available amount of training data. This capac(cid:173) ity is influenced by several factors, including: (1) properties of the input  space, (2) nature and structure of the classifier, and (3) learning algorithm.  Actions based on these three factors are combined here to control the ca(cid:173) pacity of linear classifiers and improve generalization on the problem of  handwritten digit recognition. 
1 RISK MINIMIZATION AND CAPACITY 
1.1 EMPIRICAL RISK MINIMIZATION 
A common way of training a given classifier is to adjust the parameters w in the  classification function F( x, w) to minimize the training error Etrain, i.e. the fre(cid:173) quency of errors on a set of p training examples. Etrain estimates the expected risk  based on the empirical data provided by the p available examples. The method is  thus called Empirical Risk Minimization. But the classification function F(x, w*)  which minimizes the empirical risk does not necessarily minimize the generalization  error, i.e. the expected value of the risk over the full distribution of possible inputs  and their corresponding outputs. Such generalization error Egene cannot in general  be computed, but it can be estimated on a separate test set (Ete$t). Other ways of
Computer Recognition of Wave Location in Graphical Data by a Neural Network
Authors: Donald T. Freeman
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Five experiments were performed using several neural  network architectures to  identify  the  location  of a  wave  in  the  time  ordered  graphical  results  from  a  medical  test.  Baseline  results  from  the  first  experiment  found  correct  identification of the  target  wave in  85%  of cases  (n=20).  Other  experiments  investigated the effect of different architectures and preprocessing the raw data on  the results.  The methods used seem most appropriate for time oriented graphical  data  which  has  a clear starting point such  as  electrophoresis  Or  spectrometry  rather than continuous teSts such as ECGs and EEGs.
Improved Hidden Markov Model Speech Recognition Using Radial Basis Function Networks
Authors: Richard P Lippmann Elliot Singer
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A high performance speaker-independent isolated-word hybrid speech rec(cid:173) ognizer was developed which combines Hidden Markov Models (HMMs)  and Radial Basis Function (RBF) neural networks. In recognition ex(cid:173) periments using a speaker-independent E-set database, the hybrid rec(cid:173) ognizer had an error rate of 11.5% compared to 15.7% for the robust  unimodal Gaussian HMM recognizer upon which the hybrid system was  based. These results and additional experiments demonstrate that RBF  networks can be successfully incorporated in hybrid recognizers and sug(cid:173) gest that they may be capable of good performance with fewer parameters  than required by Gaussian mixture classifiers. A global parameter opti(cid:173) mization method designed to minimize the overall word error rather than  the frame recognition error failed to reduce the error rate. 
1 HMM/RBF HYBRID RECOGNIZER 
A hybrid isolated-word speech recognizer was developed which combines neural  network and Hidden Markov Model (HMM) approaches. The hybrid approach is  an attempt to capitalize on the superior static pattern classification performance of  neural network classifiers [6] while preserving the temporal alignment properties of  HMM Viterbi decoding. Our approach is unique when compared to other studies  [2, 5] in that we use Radial Basis Function (RBF) rather than multilayer sigmoidal  networks. RBF networks were chosen because their static pattern classification  performance is comparable to that of other networks and they can be trained rapidly  using a one-pass matrix inversion technique [8]. 
The hybrid HMM/RBF isolated-word recognizer is shown in Figure 1. For each  159 
160 
Singer and Lippmann
Connectionist Optimisation of Tied Mixture Hidden Markov Models
Authors: Nelson Morgan, Hervé Bourlard, Horacio Franco, Michael Cohen Steve Renals
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Issues  relating  to  the  estimation  of hidden  Markov  model  (HMM)  local  probabilities are  discussed.  In  particular  we  note  the  isomorphism of ra(cid:173) dial  basis  functions  (RBF)  networks  to  tied  mixture  density  modellingj  additionally  we  highlight  the  differences  between  these  methods  arising  from  the  different  training  criteria  employed.  We  present  a  method  in  which  connectionist  training  can  be  modified  to  resolve  these  differences  and  discuss  some preliminary experiments.  Finally,  we  discuss  some out(cid:173) standing problems with discriminative training.
Principled Architecture Selection for Neural Networks: Application to Corporate Bond Rating Prediction
Authors: Joachim Utans John Moody
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The notion of generalization ability can be defined precisely as the pre(cid:173) diction risk, the expected performance of an estimator in predicting new  observations. In this paper, we propose the prediction risk as a measure  of the generalization ability of multi-layer perceptron networks and use it  to select an optimal network architecture from a set of possible architec(cid:173) tures. We also propose a heuristic search strategy to explore the space of  possible architectures. The prediction risk is estimated from the available  data; here we estimate the prediction risk by v-fold cross-validation and  by asymptotic approximations of generalized cross-validation or Akaike's  final prediction error. We apply the technique to the problem of predicting  corporate bond ratings. This problem is very attractive as a case study,  since it is characterized by the limited availability of the data and by the  lack of a complete a priori model which could be used to impose a structure  to the network architecture. 
1 Generalization and Prediction Risk 
The notion of generalization ability can be defined precisely as the prediction risk,  the expected performance of an estimator is predicting new observations. Consider  a set of observations D = {(Xj, tj); j = 1 ... N} that are assumed to be generated  683 
684
Extracting and Learning an Unknown Grammar with Recurrent Neural Networks
Authors: C. B. Miller, D. Chen, G. Z. Sun, H. H. Chen, Y. C. Lee C. L. Giles
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Simple secood-order recurrent netwoIts are shown to readily learn sman brown  regular grammars when trained with positive and negative strings examples. We  show that similar methods are appropriate for learning unknown grammars from  examples of their strings. TIle training algorithm is an incremental real-time, re(cid:173) current learning (RTRL) method that computes the complete gradient and updates  the weights at the end of each string. After or during training. a dynamic clustering  algorithm extracts the production rules that the neural network has learned.. TIle  methods are illustrated by extracting rules from unknown deterministic regular  grammars. For many cases the extracted grammar outperforms the neural net from  which it was extracted in correctly classifying unseen strings.
Network activity determines spatio-temporal integration in single cells
Authors: Christof Koch, Rodney J. Douglas Öjvind Bernander
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Single nerve cells with static properties have traditionally been viewed  as the building blocks for networks that show emergent phenomena. In  contrast to this approach, we study here how the overall network activity  can control single cell parameters such as input resistance, as well as time  and space constants, parameters that are crucial for excitability and spatio(cid:173) temporal integration. Using detailed computer simulations of neocortical  pyramidal cells, we show that the spontaneous background firing of the  network provides a means for setting these parameters. The mechanism  for this control is through the large conductance change of the membrane  that is induced by both non-NMDA and NMDA excitatory and inhibitory  synapses activated by the spontaneous background activity.
Image Segmentation with Networks of Variable Scales
Authors: Craig R. Nohl, Jan Ben Hans Peter Graf
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  developed  a  neural  net  architecture  for  segmenting  complex  images, i.e.,  to localize two-dimensional geometrical shapes in a scene,  without  prior knowledge  of the  objects'  positions  and  sizes.  A  scale  variation is built into the network  to deal with varying sizes.  This algo(cid:173) rithm  has  been  applied  to  video  images  of railroad  cars,  to  find  their  identification  numbers.  Over  95%  of  the  characlers  were  located  correctly in a data base of 300 images, despile a large variation in light(cid:173) ing  conditions and  often a poor quality of the characters.  A part of the  network  is executed  on  a processor board containing an  analog  neural  net chip (Graf et aI.  1991). while the rest is  implemented as  a software  model on a workstation or a digital signal processor.
Decoding of Neuronal Signals in Visual Pattern Recognition
Authors: Barry J. Richmond, John A. Hertz, Lance M. Optican, Troels W. Kjær Emad N. Eskandar
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  have  investigated  the  properties of neurons  in  inferior  temporal  (IT)  cortex  in  monkeys  performing  a  pattern  matching  task.  Simple  back(cid:173) propagation  networks  were  trained  to  discriminate  the  various  stimulus  conditions on  the  basis  of the  measured  neuronal signal.  We  also trained  networks to predict the neuronal response waveforms from the spatial pat(cid:173) terns of the stimuli.  The  results  indicate  t.hat  IT neurons convey  tempo(cid:173) rally  encoded  information  about  both current  and  remembered  patterns,  as well  as  about  their  behavioral context.
Learning in Feedforward Networks with Nonsmooth Functions
Authors: T. Downs Nicholas J. Redding
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper is concerned with the problem of learning in networks where some  or all of the functions involved are not smooth. Examples of such networks are  those whose neural transfer functions are piecewise-linear and those whose error  function is defined in terms of the 100 norm.  Up to now, networks whose neural transfer functions are piecewise-linear have  received very little consideration in the literature, but the possibility of using an  error function defined in terms of the 100 norm has received some attention. In  this latter work, however, the problems that can occur when gradient methods are  used for non smooth error functions have not been addressed.  In this paper we draw upon some recent results from the field of nonsmooth  optimization (NSO) to present an algorithm for the non smooth case. Our moti(cid:173) vation for this work arose out of the fact that we have been able to show that,  in backpropagation, an error function based upon the 100 norm overcomes the  difficulties which can occur when using the 12 norm.
Constructing Proofs in Symmetric Networks
Authors: Gadi Pinkus
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper considers the problem of expressing predicate calculus in con(cid:173) nectionist networks that are based  on energy minimization.  Given a  first(cid:173) order-logic  knowledge  base and a  bound  k,  a  symmetric  network is  con(cid:173) structed  (like a  Boltzman  machine  or a  Hopfield  network)  that searches  for  a  proof for  a  given  query.  If a  resolution-based  proof of length  no  longer  than k  exists,  then the global  minima of the energy function  that  is  associated  with  the  network represent such  proofs.  The  network  that  is  generated  is  of size  cubic  in  the bound  k  and  linear in the  knowledge  size.  There are no restrictions on the type of logic  formulas  that can be  represented.  The network  is  inherently fault  tolerant and  can cope with  inconsistency and nonmonotonicity.
A Weighted Probabilistic Neural Network
Authors: David Montana
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The Probabilistic Neural Network (PNN) algorithm represents the likeli(cid:173) hood function of a given class as the sum of identical, isotropic Gaussians.  In practice, PNN is often an excellent pattern classifier, outperforming  other classifiers including backpropagation. However, it. is not. robust with  respect to affine transformations of feature space, and this can lead to  poor performance on certain data. We have derived an extension of PNN  called Weighted PNN (WPNN) which compensates for this flaw by allow(cid:173) ing anisotropic Gaussians, i.e. Gaussians whose covariance is not a mul(cid:173) tiple of the identity matrix. The covariance is optimized using a genetic  algorithm, some interesting features of which are its redundant, logarith(cid:173) mic encoding and large population size. Experimental results validate our  claims.
Shooting Craps in Search of an Optimal Strategy for Training Connectionist Pattern Classifiers
Authors: B. V. K. Vijaya Kumar J. B. Hampshire II
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We compare two strategies for training connectionist (as well as non(cid:173) connectionist) models for statistical pattern recognition. The probabilistic strat(cid:173) egy is based on the notion that Bayesian discrimination (i.e .• optimal classifica(cid:173) tion) is achieved when the classifier learns the a posteriori class distributions of  the random feature vector. The differential strategy is based on the notion that  the identity of the largest class a posteriori probability of the feature vector is  all that is needed to achieve Bayesian discrimination. Each strategy is directly  linked to a family of objective functions that can be used in the supervised training  procedure. We prove that the probabilistic strategy - linked with error measure  objective functions such as mean-squared-error and cross-entropy - typically  used to train classifiers necessarily requires larger training sets and more complex  classifier architectures than those needed to approximate the Bayesian discrim(cid:173) linked  inant function.  In contrast. we prove that the differential strategy - with classificationfigure-of-merit objective functions (CF'MmoIlO) [3] - requires  the minimum classifier functional complexity and the fewest training examples  necessary to approximate the Bayesian discriminant function with specified pre(cid:173) cision (measured in probability of error). We present our proofs in the context of  a game of chance in which an unfair C-sided die is tossed repeatedly. We show  that this rigged game of dice is a paradigm at the root of all statistical pattern  recognition tasks. and demonstrate how a simple extension of the concept leads  us to a general information-theoretic model of sample complexity for statistical  pattern recognition.
Multimodular Architecture for Remote Sensing Operations.
Authors: Carlos Mejia, Fouad Badran, Michel Crépon Sylvie Thiria
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper deals with an application of Neural Networks to satellite  remote sensing observations. Because of the complexity of the  application and the large amount of data, the problem cannot be solved  by using a single method. The solution we propose is to build multi(cid:173) modules NN architectures where several NN cooperate together. Such  system suffer from generic problem for whom we propose solutions.  They allow to reach accurate performances for multi-valued function  approximations and probability estimations. The results are compared  with six other methods which have been used for this problem. We  show that the methodology we have developed is general and can be  used for a large variety of applications.
Learning in the Vestibular System: Simulations of Vestibular Compensation Using Recurrent Back-Propagation
Authors: Thomas J. Anastasio
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Vestibular  compensation  is  the  process  whereby  normal  functioning  is  regained  following  destruction  of  one  member  of  the  pair  of  peripheral  vestibular  receptors.  Compensation  was  simulated  by  lesioning  a  dynamic  neural  network  model  of the  vestibulo~ular reflex  (VOR)  and  retraining  it  using  recurrent back-propagation.  The model reproduced the pattern of VOR  neuron activity  experimentally observed  in compensated animals,  but only  if  connections  heretofore  considered  uninvolved  were  allowed  to  be  plastic.  Because  the  model  incorporated  nonlinear  units,  it  was  able  to  reconcile  previously conflicting, linear analyses of experimental  results on the dynamic  properties of VOR neurons in normal and compensated animals. 
1 VESTIBULAR COMPENSATION 
Vestibular compensation is one of the oldest and most well studied paradigms in motor  learning.  Although it is neurophysiologically well described, the adaptive mechanisms  underlying  vestibular  compensation,  and  its  effects  on  the  dynamics  of vestibular  responses,  are still poorly understood.  The purpose of this study  is to gain insight into  the  compensatory  process  by  simulating  it  as  learning  in  a  recurrent  neural  network  model of the vestibulo-ocular reflex (VOR).
Repeat Until Bored: A Pattern Selection Strategy
Authors: Paul W. Munro
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: An alternative to the typical technique of selecting  training examples  independently from a fixed distribution is fonnulated and analyzed, in  which the current example is presented repeatedly until the error for that  item is reduced to  some criterion value,  ~; then,  another  item  is ran(cid:173) domly selected.  The convergence time can be dramatically increased or  decreased by this heuristic, depending on the task, and is very sensitive  to the value of ~.
Refining PID Controllers using Neural Networks
Authors: Jude W. Shavlik, W. Harmon Ray Gary M. Scott
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The KBANN approach uses neural networks to refine knowledge that can  be written in the form of simple propositional rules. We extend this idea  further by presenting the MANNCON algorithm by which the mathematical  equations governing a PID controller determine the topology and initial  weights of a network, which is further trained using backpropagation. We  apply this method to the task of controlling the outflow and temperature  of a water tank, producing statistically-significant gains in accuracy over  both a standard neural network approach and a non-learning PID con(cid:173) troller. Furthermore, using the PID knowledge to initialize the weights of  the network produces statistically less variation in testset accuracy when  compared to networks initialized with small random numbers.