mtybilly/Instruct2DS huggingface dataset:
Instance-Specific Bayesian Model Averaging for Classification
Authors: Shyam Visweswaran, Gregory F. Cooper
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Classification algorithms typically induce population-wide models  that are trained to perform well on average on expected future  instances. We introduce a Bayesian framework for learning  instance-specific models from data that are optimized to predict  well for a particular instance. Based on this framework, we present  a  that performs  selective model averaging over a restricted class of Bayesian  networks. On experimental evaluation, this algorithm shows  superior performance over model selection. We intend to apply  such instance-specific algorithms to improve the performance of  patient-specific predictive models induced from medical data. 
instance-specific algorithm called ISA
Exponentiated Gradient Algorithms for Large-margin Structured Classification
Authors: Peter L. Bartlett, Michael Collins, Ben Taskar, David A. McAllester
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We consider the problem of structured classiﬁcation, where the task is to predict a label y from an input x, and y has meaningful internal struc- ture. Our framework includes supervised training of Markov random ﬁelds and weighted context-free grammars as special cases. We describe an algorithm that solves the large-margin optimization problem deﬁned in [12], using an exponential-family (Gibbs distribution) representation of structured objects. The algorithm is efﬁcient—even in cases where the number of labels y is exponential in size—provided that certain expecta- tions under Gibbs distributions can be calculated efﬁciently. The method for structured labels relies on a more general result, speciﬁcally the ap- plication of exponentiated gradient updates [7, 8] to quadratic programs.
Semi-Markov Conditional Random Fields for Information Extraction
Authors: Sunita Sarawagi, William W. Cohen
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We describe semi-Markov conditional random ﬁelds (semi-CRFs), a con- ditionally trained version of semi-Markov chains. Intuitively, a semi- CRF on an input sequence x outputs a “segmentation” of x, in which labels are assigned to segments (i.e., subsequences) of x rather than to individual elements xi of x. Importantly, features for semi-CRFs can measure properties of segments, and transitions within a segment can be non-Markovian. In spite of this additional power, exact learning and inference algorithms for semi-CRFs are polynomial-time—often only a small constant factor slower than conventional CRFs. In experiments on ﬁve named entity recognition problems, semi-CRFs generally outper- form conventional CRFs.
Adaptive Manifold Learning
Authors: Jing Wang, Zhenyue Zhang, Hongyuan Zha
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Recently, there have been several advances in the machine learning and           pattern recognition communities for developing manifold learning algo-           rithms to construct nonlinear low-dimensional manifolds from sample           data points embedded in high-dimensional spaces. In this paper, we de-           velop algorithms that address two key issues in manifold learning: 1)           the adaptive selection of the neighborhood sizes; and 2) better fitting the           local geometric structure to account for the variations in the curvature           of the manifold and its interplay with the sampling density of the data           set. We also illustrate the effectiveness of our methods on some synthetic           data sets.
Euclidean Embedding of Co-Occurrence Data
Authors: Amir Globerson, Gal Chechik, Fernando Pereira, Naftali Tishby
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Embedding algorithms search for low dimensional structure in complex          data, but most algorithms only handle objects of a single type for which          pairwise distances are specified. This paper describes a method for em-          bedding objects of different types, such as images and text, into a single          common Euclidean space based on their co-occurrence statistics. The          joint distributions are modeled as exponentials of Euclidean distances in          the low-dimensional embedding space, which links the problem to con-          vex optimization over positive semidefinite matrices. The local struc-          ture of our embedding corresponds to the statistical correlations via ran-          dom walks in the Euclidean space. We quantify the performance of our          method on two text datasets, and show that it consistently and signifi-          cantly outperforms standard methods of statistical correspondence mod-          eling, such as multidimensional scaling and correspondence analysis.
Heuristics for Ordering Cue Search in Decision Making
Authors: Peter M. Todd, Anja Dieckmann
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Simple lexicographic decision heuristics that consider cues one at a 
          time  in  a  particular  order  and  stop  searching  for  cues  as  soon  as  a 
          decision  can  be  made  have  been  shown  to  be  both  accurate  and 
          frugal  in  their  use  of  information.    But  much  of  the  simplicity  and 
          success  of  these  heuristics  comes  from  using  an  appropriate  cue 
          order.  For instance, the Take The Best heuristic uses validity order 
          for  cues,  which  requires  considerable  computation,  potentially 
          undermining  the  computational  advantages  of  the  simple  decision 
          mechanism.    But  many  cue  orders  can  achieve  good  decision 
          performance, and studies of sequential search for data records have 
          proposed  a  number  of  simple  ordering  rules  that  may  be  of  use  in 
          constructing  appropriate  decision  cue  orders  as  well.    Here  we 
          consider  a  range  of  simple  cue  ordering  mechanisms,  including 
          tallying, swapping, and move-to-front rules, and show that they can 
          find  cue  orders  that  lead  to  reasonable  accuracy  and  considerable 
          frugality when used with lexicographic decision heuristics.
Coarticulation in Markov Decision Processes
Authors: Khashayar Rohanimanesh, Robert Platt, Sridhar Mahadevan, Roderic Grupen
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We investigate an approach for simultaneously committing to mul-         tiple activities, each modeled as a temporally extended action in         a semi-Markov decision process (SMDP). For each activity we de-         fine a set of admissible solutions consisting of the redundant set of         optimal policies, and those policies that ascend the optimal state-         value function associated with them. A plan is then generated by         merging them in such a way that the solutions to the subordinate         activities are realized in the set of admissible solutions satisfying         the superior activities. We present our theoretical results and em-         pirically evaluate our approach in a simulated domain.
Integrating Topics and Syntax
Authors: Thomas L. Griffiths, Mark Steyvers, David M. Blei, Joshua B. Tenenbaum
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Statistical approaches to language learning typically focus on either          short-range syntactic dependencies or long-range semantic dependencies          between words. We present a generative model that uses both kinds of          dependencies, and can be used to simultaneously find syntactic classes          and semantic topics despite having no representation of syntax or seman-          tics beyond statistical dependency. This model is competitive on tasks          like part-of-speech tagging and document classification with models that          exclusively use short- and long-range dependencies respectively.
A Large Deviation Bound for the Area Under the ROC Curve
Authors: Shivani Agarwal, Thore Graepel, Ralf Herbrich, Dan Roth
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: The area under the ROC curve (AUC) has been advocated as an evalu- ation criterion for the bipartite ranking problem. We study large devi- ation properties of the AUC; in particular, we derive a distribution-free large deviation bound for the AUC which serves to bound the expected accuracy of a ranking function in terms of its empirical AUC on an inde- pendent test sequence. A comparison of our result with a corresponding large deviation result for the classiﬁcation error rate suggests that the test sample size required to obtain an -accurate estimate of the expected ac- curacy of a ranking function with δ-conﬁdence is larger than that required to obtain an -accurate estimate of the expected error rate of a classiﬁ- cation function with the same conﬁdence. A simple application of the union bound allows the large deviation bound to be extended to learned ranking functions chosen from ﬁnite function classes.
Adaptive Discriminative Generative Model and Its Applications
Authors: Ruei-sung Lin, David A. Ross, Jongwoo Lim, Ming-Hsuan Yang
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: This paper presents an adaptive discriminative generative model that gen-          eralizes the conventional Fisher Linear Discriminant algorithm and ren-          ders a proper probabilistic interpretation. Within the context of object          tracking, we aim to find a discriminative generative model that best sep-          arates the target from the background. We present a computationally          efficient algorithm to constantly update this discriminative model as time          progresses. While most tracking algorithms operate on the premise that          the object appearance or ambient lighting condition does not significantly          change as time progresses, our method adapts a discriminative genera-          tive model to reflect appearance variation of the target and background,          thereby facilitating the tracking task in ever-changing environments. Nu-          merous experiments show that our method is able to learn a discrimina-          tive generative model for tracking target objects undergoing large pose          and lighting changes.
Modeling Conversational Dynamics as a Mixed-Memory Markov Process
Authors: Tanzeem Choudhury, Sumit Basu
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: influences 
In  this  work,  we  quantitatively  investigate  the  ways  in  which  a  given  person  the  joint  turn-taking  behavior  in  a  conversation.  After  collecting  an  auditory  database  of  social  interactions  among  a  group  of twenty-three  people  via  wearable  sensors  (66  hours  of data  each  over  two  weeks),  we  apply  speech  and conversation detection methods to  the auditory streams.  These  methods  automatically  locate  the  conversations,  determine  their  participants,  and  mark  which  participant  was  speaking  when.  We  then  model  the  joint  turn-taking  behavior  as  a  Mixed-Memory  Markov  Model  [1]  that  combines  the  statistics  of the  individual  subjects'  self-transitions  and  the  partners '  cross-transitions.  The  mixture parameters in  this  model describe how much each person's  individual  behavior contributes  to  the joint turn-taking behavior of  the  pair.  By  estimating  these  parameters,  we  thus  estimate  how  much  influence  each participant  has  in  determining  the  joint turn(cid:173) this  measure  correlates  taking  behavior.  We  significantly  with  betweenness  centrality  [2],  an  independent  measure  of an  individual's  importance  in  a  social  network.  This  result  suggests  that  our  estimate  of  conversational  influence  is  predictive of social influence.
On-Chip Compensation of Device-Mismatch Effects in Analog VLSI Neural Networks
Authors: Miguel Figueroa, Seth Bridges, Chris Diorio
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Device mismatch in VLSI degrades the accuracy of analog arithmetic           circuits and lowers the learning performance of large-scale neural net-           works implemented in this technology. We show compact, low-power           on-chip calibration techniques that compensate for device mismatch. Our           techniques enable large-scale analog VLSI neural networks with learn-           ing performance on the order of 10 bits. We demonstrate our techniques           on a 64-synapse linear perceptron learning with the Least-Mean-Squares           (LMS) algorithm, and fabricated in a 0.35m CMOS process.
Trait Selection for Assessing Beef Meat Quality Using Non-linear SVM
Authors: Juan Coz, Gustavo F. Bayón, Jorge Díez, Oscar Luaces, Antonio Bahamonde, Carlos Sañudo
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: In this paper we show that it is possible to model sensory impressions          of consumers about beef meat. This is not a straightforward task; the          reason is that when we are aiming to induce a function that maps object          descriptions into ratings, we must consider that consumers' ratings are          just a way to express their preferences about the products presented in          the same testing session. Therefore, we had to use a special purpose          SVM polynomial kernel. The training data set used collects the ratings of          panels of experts and consumers; the meat was provided by 103 bovines          of 7 Spanish breeds with different carcass weights and aging periods.          Additionally, to gain insight into consumer preferences, we used feature          subset selection tools. The result is that aging is the most important trait          for improving consumers' appreciation of beef meat.
Sharing Clusters among Related Groups: Hierarchical Dirichlet Processes
Authors: Yee W. Teh, Michael I. Jordan, Matthew J. Beal, David M. Blei
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We propose the hierarchical Dirichlet process (HDP), a nonparametric Bayesian model for clustering problems involving multiple groups of data. Each group of data is modeled with a mixture, with the number of components being open-ended and inferred automatically by the model. Further, components can be shared across groups, allowing dependencies across groups to be modeled effectively as well as conferring generaliza- tion to new groups. Such grouped clustering problems occur often in practice, e.g. in the problem of topic discovery in document corpora. We report experimental results on three text corpora showing the effective and superior performance of the HDP over previous models.
Spike-timing Dependent Plasticity and Mutual Information Maximization for a Spiking Neuron Model
Authors: Taro Toyoizumi, Jean-pascal Pfister, Kazuyuki Aihara, Wulfram Gerstner
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: We derive an optimal learning rule in the sense of mutual information maximization for a spiking neuron model. Under the assumption of small ﬂuctuations of the input, we ﬁnd a spike-timing dependent plas- ticity (STDP) function which depends on the time course of excitatory postsynaptic potentials (EPSPs) and the autocorrelation function of the postsynaptic neuron. We show that the STDP function has both positive and negative phases. The positive phase is related to the shape of the EPSP while the negative phase is controlled by neuronal refractoriness.
Comparing Beliefs, Surveys, and Random Walks
Authors: Erik Aurell, Uri Gordon, Scott Kirkpatrick
Published in: Advances in Neural Information Processing Systems 17  (NIPS 2004) (Main Conference Track), 2004

Abstract: Survey propagation is a powerful technique from statistical physics that          has been applied to solve the 3-SAT problem both in principle and in          practice. We give, using only probability arguments, a common deriva-          tion of survey propagation, belief propagation and several interesting hy-          brid methods. We then present numerical experiments which use WSAT          (a widely used random-walk based SAT solver) to quantify the complex-          ity of the 3-SAT formulae as a function of their parameters, both as ran-          domly generated and after simplication, guided by survey propagation.          Some properties of WSAT which have not previously been reported make          it an ideal tool for this purpose  its mean cost is proportional to the num-          ber of variables in the formula (at a xed ratio of clauses to variables) in          the easy-SAT regime and slightly beyond, and its behavior in the hard-          SAT regime appears to reect the underlying structure of the solution          space that has been predicted by replica symmetry-breaking arguments.          An analysis of the tradeoffs between the various methods of search for          satisfying assignments shows WSAT to be far more powerful than has          been appreciated, and suggests some interesting new directions for prac-          tical algorithm development.
English Alphabet Recognition with Telephone Speech
Authors: Mark Fanty, Ronald A. Cole, Krist Roginski
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A recognition system is reported which recognizes names spelled over the  telephone with brief pauses between letters. The system uses separate  neural networks to locate segment boundaries and classify letters. The  letter scores are then used to search a database of names to find the best  scoring name. The speaker-independent classification rate for spoken let(cid:173) ters is 89%. The system retrieves the correct name, spelled with pauses  between letters, 91 % of the time from a database of 50,000 names.
Recognizing Overlapping Hand-Printed Characters by Centered-Object Integrated Segmentation and Recognition
Authors: Gale L. Martin, Mosfeq Rashid
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper describes an approach, called centered object integrated seg(cid:173) mentation and recognition (COISR). for integrating object segmenta(cid:173) tion and recognition within a single neural network. The application  is hand-printed character recognition. 1\vo versions of the system are  described. One uses a backpropagation network that scans exhaus(cid:173) tively over a field of characters and is trained to recognize whether  it is centered over a single character or between characters. When  it is centered over a character, the net classifies the cnaracter. The  approach is tested on a dataset of hand-printed digits. Vel)' low errOr  rates are reported. The second version, COISR-SACCADE, avoids  the need for exhaustive scans. The net is trained as before. but also  is trained to compute ballistic 'eye' movements that enable the input  window to jump from one character to the next. 
The common model of visual processing includes multiple, independent stages. First,  flltering operations act on the raw image to segment or isolate and enhance to-be-re(cid:173) cognized clumps. These clumps are normalized for factors such as size, and sometimes  simplified further through feature extraction. The results are then fed to one or more  classifiers. The operations prior to classification simplify the recognition task. Object  segmentation restricts the number of features considered for classification to those as(cid:173) sociated with a single object, and enables normalization to be applied at the individual  object level. Without such pre-processing. recognition may be an intractable problem.  However, a weak point of this sequential stage model is that recognition and segmenta(cid:173) tion decisions are often inter-dependent. Not only does a correct recognition decision  depend on first making a correct segmentation decision, but a correct segmentation  decision often depends on first making a correct recognition decision. 
This is a particularly serious problem in character recognition applications. OCR sys(cid:173) tems use intervening white space and related features to segment a field of characters  into individual characters, so that classification can be accomplished one character at  a time. This approach fails when characters touch each other or when an individual  character is broken up by intervening white space. Some means of integrating the seg(cid:173) mentation and recognition stages is needed. 
This paper descnbes an approach, called centered object integrated segmentation and rec(cid:173) ognition (COISR), for integrating character segmentation and recognition within one 
• Also with Eastman Kodak Company
Adaptive Soft Weight Tying using Gaussian Mixtures
Authors: Steven J. Nowlan, Geoffrey E. Hinton
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: One way  of simplifying neural  networks  so  they generalize  better is  to add  an  extra  t.erm  10  the  error  fUll ction  that  will  penalize  complexit.y.  \Ve  propose  a  new  penalt.y  t.erm  in  which  the  dist rihution  of  weight  values  is  modelled  as  a  mixture  of  multiple  gaussians .  C nder  this  model,  a  set  of  weights  is  simple  if  the  weights  can  be  clustered  into  subsets  so  that  the  weights  in  each  cluster  have  similar  values .  We  allow  the  parameters  of  the  mixture  model  to  adapt  at  t.he  same  time  as  t.he  network  learns.  Simulations demonstrate  that  this complexity  term  is  more  effective  than  previous  complexity terms.
Multi-State Time Delay Networks for Continuous Speech Recognition
Authors: Patrick Haffner, Alex Waibel
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We present the "Multi-State Time Delay Neural Network" (MS-TDNN) as  an  extension of the TDNN to robust word recognition. Unlike most other hybrid  methods. the MS-TDNN embeds an alignment search procedure into the con(cid:173) nectionist architecture. and allows for word level supervision. The resulting  system  has the ability to manage the sequential order of subword units. while  optimizing for the recognizer performance. In this paper we present extensive  new evaluations of this approach over speaker-dependent and speaker-indepen(cid:173) dent connected alphabet.
Neural Network - Gaussian Mixture Hybrid for Speech Recognition or Density Estimation
Authors: Yoshua Bengio, Renato De Mori, Giovanni Flammia, Ralf Kompe
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The subject of this paper is the integration of multi-layered Artificial Neu(cid:173) ral Networks (ANN) with probability density functions such as Gaussian  mixtures found in continuous density Hidden Markov Models (HMM). In  the first part of this paper we present an ANN/HMM hybrid in which  all the parameters of the the system are simultaneously optimized with  respect to a single criterion. In the second part of this paper, we study  the relationship between the density of the inputs of the network and the  density of the outputs of the networks. A few experiments are presented  to explore how to perform density estimation with ANNs.
Structural Risk Minimization for Character Recognition
Authors: I. Guyon, V. Vapnik, B. Boser, L. Bottou, S. A. Solla
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The method of Structural Risk Minimization refers to tuning the capacity  of the classifier to the available amount of training data. This capac(cid:173) ity is influenced by several factors, including: (1) properties of the input  space, (2) nature and structure of the classifier, and (3) learning algorithm.  Actions based on these three factors are combined here to control the ca(cid:173) pacity of linear classifiers and improve generalization on the problem of  handwritten digit recognition. 
1 RISK MINIMIZATION AND CAPACITY 
1.1 EMPIRICAL RISK MINIMIZATION 
A common way of training a given classifier is to adjust the parameters w in the  classification function F( x, w) to minimize the training error Etrain, i.e. the fre(cid:173) quency of errors on a set of p training examples. Etrain estimates the expected risk  based on the empirical data provided by the p available examples. The method is  thus called Empirical Risk Minimization. But the classification function F(x, w*)  which minimizes the empirical risk does not necessarily minimize the generalization  error, i.e. the expected value of the risk over the full distribution of possible inputs  and their corresponding outputs. Such generalization error Egene cannot in general  be computed, but it can be estimated on a separate test set (Ete$t). Other ways of
Improved Hidden Markov Model Speech Recognition Using Radial Basis Function Networks
Authors: Elliot Singer, Richard P Lippmann
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A high performance speaker-independent isolated-word hybrid speech rec(cid:173) ognizer was developed which combines Hidden Markov Models (HMMs)  and Radial Basis Function (RBF) neural networks. In recognition ex(cid:173) periments using a speaker-independent E-set database, the hybrid rec(cid:173) ognizer had an error rate of 11.5% compared to 15.7% for the robust  unimodal Gaussian HMM recognizer upon which the hybrid system was  based. These results and additional experiments demonstrate that RBF  networks can be successfully incorporated in hybrid recognizers and sug(cid:173) gest that they may be capable of good performance with fewer parameters  than required by Gaussian mixture classifiers. A global parameter opti(cid:173) mization method designed to minimize the overall word error rather than  the frame recognition error failed to reduce the error rate. 
1 HMM/RBF HYBRID RECOGNIZER 
A hybrid isolated-word speech recognizer was developed which combines neural  network and Hidden Markov Model (HMM) approaches. The hybrid approach is  an attempt to capitalize on the superior static pattern classification performance of  neural network classifiers [6] while preserving the temporal alignment properties of  HMM Viterbi decoding. Our approach is unique when compared to other studies  [2, 5] in that we use Radial Basis Function (RBF) rather than multilayer sigmoidal  networks. RBF networks were chosen because their static pattern classification  performance is comparable to that of other networks and they can be trained rapidly  using a one-pass matrix inversion technique [8]. 
The hybrid HMM/RBF isolated-word recognizer is shown in Figure 1. For each  159 
160 
Singer and Lippmann
Connectionist Optimisation of Tied Mixture Hidden Markov Models
Authors: Steve Renals, Nelson Morgan, Hervé Bourlard, Horacio Franco, Michael Cohen
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Issues  relating  to  the  estimation  of hidden  Markov  model  (HMM)  local  probabilities are  discussed.  In  particular  we  note  the  isomorphism of ra(cid:173) dial  basis  functions  (RBF)  networks  to  tied  mixture  density  modellingj  additionally  we  highlight  the  differences  between  these  methods  arising  from  the  different  training  criteria  employed.  We  present  a  method  in  which  connectionist  training  can  be  modified  to  resolve  these  differences  and  discuss  some preliminary experiments.  Finally,  we  discuss  some out(cid:173) standing problems with discriminative training.
Principled Architecture Selection for Neural Networks: Application to Corporate Bond Rating Prediction
Authors: John Moody, Joachim Utans
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The notion of generalization ability can be defined precisely as the pre(cid:173) diction risk, the expected performance of an estimator in predicting new  observations. In this paper, we propose the prediction risk as a measure  of the generalization ability of multi-layer perceptron networks and use it  to select an optimal network architecture from a set of possible architec(cid:173) tures. We also propose a heuristic search strategy to explore the space of  possible architectures. The prediction risk is estimated from the available  data; here we estimate the prediction risk by v-fold cross-validation and  by asymptotic approximations of generalized cross-validation or Akaike's  final prediction error. We apply the technique to the problem of predicting  corporate bond ratings. This problem is very attractive as a case study,  since it is characterized by the limited availability of the data and by the  lack of a complete a priori model which could be used to impose a structure  to the network architecture. 
1 Generalization and Prediction Risk 
The notion of generalization ability can be defined precisely as the prediction risk,  the expected performance of an estimator is predicting new observations. Consider  a set of observations D = {(Xj, tj); j = 1 ... N} that are assumed to be generated  683 
684
Extracting and Learning an Unknown Grammar with Recurrent Neural Networks
Authors: C. L. Giles, C. B. Miller, D. Chen, G. Z. Sun, H. H. Chen, Y. C. Lee
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Simple secood-order recurrent netwoIts are shown to readily learn sman brown  regular grammars when trained with positive and negative strings examples. We  show that similar methods are appropriate for learning unknown grammars from  examples of their strings. TIle training algorithm is an incremental real-time, re(cid:173) current learning (RTRL) method that computes the complete gradient and updates  the weights at the end of each string. After or during training. a dynamic clustering  algorithm extracts the production rules that the neural network has learned.. TIle  methods are illustrated by extracting rules from unknown deterministic regular  grammars. For many cases the extracted grammar outperforms the neural net from  which it was extracted in correctly classifying unseen strings.
Network activity determines spatio-temporal integration in single cells
Authors: Öjvind Bernander, Christof Koch, Rodney J. Douglas
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Single nerve cells with static properties have traditionally been viewed  as the building blocks for networks that show emergent phenomena. In  contrast to this approach, we study here how the overall network activity  can control single cell parameters such as input resistance, as well as time  and space constants, parameters that are crucial for excitability and spatio(cid:173) temporal integration. Using detailed computer simulations of neocortical  pyramidal cells, we show that the spontaneous background firing of the  network provides a means for setting these parameters. The mechanism  for this control is through the large conductance change of the membrane  that is induced by both non-NMDA and NMDA excitatory and inhibitory  synapses activated by the spontaneous background activity.
Image Segmentation with Networks of Variable Scales
Authors: Hans Peter Graf, Craig R. Nohl, Jan Ben
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  developed  a  neural  net  architecture  for  segmenting  complex  images, i.e.,  to localize two-dimensional geometrical shapes in a scene,  without  prior knowledge  of the  objects'  positions  and  sizes.  A  scale  variation is built into the network  to deal with varying sizes.  This algo(cid:173) rithm  has  been  applied  to  video  images  of railroad  cars,  to  find  their  identification  numbers.  Over  95%  of  the  characlers  were  located  correctly in a data base of 300 images, despile a large variation in light(cid:173) ing  conditions and  often a poor quality of the characters.  A part of the  network  is executed  on  a processor board containing an  analog  neural  net chip (Graf et aI.  1991). while the rest is  implemented as  a software  model on a workstation or a digital signal processor.
Decoding of Neuronal Signals in Visual Pattern Recognition
Authors: Emad N. Eskandar, Barry J. Richmond, John A. Hertz, Lance M. Optican, Troels W. Kjær
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We  have  investigated  the  properties of neurons  in  inferior  temporal  (IT)  cortex  in  monkeys  performing  a  pattern  matching  task.  Simple  back(cid:173) propagation  networks  were  trained  to  discriminate  the  various  stimulus  conditions on  the  basis  of the  measured  neuronal signal.  We  also trained  networks to predict the neuronal response waveforms from the spatial pat(cid:173) terns of the stimuli.  The  results  indicate  t.hat  IT neurons convey  tempo(cid:173) rally  encoded  information  about  both current  and  remembered  patterns,  as well  as  about  their  behavioral context.
Learning in Feedforward Networks with Nonsmooth Functions
Authors: Nicholas J. Redding, T. Downs
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper is concerned with the problem of learning in networks where some  or all of the functions involved are not smooth. Examples of such networks are  those whose neural transfer functions are piecewise-linear and those whose error  function is defined in terms of the 100 norm.  Up to now, networks whose neural transfer functions are piecewise-linear have  received very little consideration in the literature, but the possibility of using an  error function defined in terms of the 100 norm has received some attention. In  this latter work, however, the problems that can occur when gradient methods are  used for non smooth error functions have not been addressed.  In this paper we draw upon some recent results from the field of nonsmooth  optimization (NSO) to present an algorithm for the non smooth case. Our moti(cid:173) vation for this work arose out of the fact that we have been able to show that,  in backpropagation, an error function based upon the 100 norm overcomes the  difficulties which can occur when using the 12 norm.
Shooting Craps in Search of an Optimal Strategy for Training Connectionist Pattern Classifiers
Authors: J. B. Hampshire II, B. V. K. Vijaya Kumar
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We compare two strategies for training connectionist (as well as non(cid:173) connectionist) models for statistical pattern recognition. The probabilistic strat(cid:173) egy is based on the notion that Bayesian discrimination (i.e .• optimal classifica(cid:173) tion) is achieved when the classifier learns the a posteriori class distributions of  the random feature vector. The differential strategy is based on the notion that  the identity of the largest class a posteriori probability of the feature vector is  all that is needed to achieve Bayesian discrimination. Each strategy is directly  linked to a family of objective functions that can be used in the supervised training  procedure. We prove that the probabilistic strategy - linked with error measure  objective functions such as mean-squared-error and cross-entropy - typically  used to train classifiers necessarily requires larger training sets and more complex  classifier architectures than those needed to approximate the Bayesian discrim(cid:173) linked  inant function.  In contrast. we prove that the differential strategy - with classificationfigure-of-merit objective functions (CF'MmoIlO) [3] - requires  the minimum classifier functional complexity and the fewest training examples  necessary to approximate the Bayesian discriminant function with specified pre(cid:173) cision (measured in probability of error). We present our proofs in the context of  a game of chance in which an unfair C-sided die is tossed repeatedly. We show  that this rigged game of dice is a paradigm at the root of all statistical pattern  recognition tasks. and demonstrate how a simple extension of the concept leads  us to a general information-theoretic model of sample complexity for statistical  pattern recognition.
Multimodular Architecture for Remote Sensing Operations.
Authors: Sylvie Thiria, Carlos Mejia, Fouad Badran, Michel Crépon
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper deals with an application of Neural Networks to satellite  remote sensing observations. Because of the complexity of the  application and the large amount of data, the problem cannot be solved  by using a single method. The solution we propose is to build multi(cid:173) modules NN architectures where several NN cooperate together. Such  system suffer from generic problem for whom we propose solutions.  They allow to reach accurate performances for multi-valued function  approximations and probability estimations. The results are compared  with six other methods which have been used for this problem. We  show that the methodology we have developed is general and can be  used for a large variety of applications.
Refining PID Controllers using Neural Networks
Authors: Gary M. Scott, Jude W. Shavlik, W. Harmon Ray
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: The KBANN approach uses neural networks to refine knowledge that can  be written in the form of simple propositional rules. We extend this idea  further by presenting the MANNCON algorithm by which the mathematical  equations governing a PID controller determine the topology and initial  weights of a network, which is further trained using backpropagation. We  apply this method to the task of controlling the outflow and temperature  of a water tank, producing statistically-significant gains in accuracy over  both a standard neural network approach and a non-learning PID con(cid:173) troller. Furthermore, using the PID knowledge to initialize the weights of  the network produces statistically less variation in testset accuracy when  compared to networks initialized with small random numbers.
A Cortico-Cerebellar Model that Learns to Generate Distributed Motor Commands to Control a Kinematic Arm
Authors: N. E. Berthier, S. P. Singh, A. G. Barto, J. C. Houk
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: A neurophysiologically-based model is presented that controls a simulated  kinematic arm during goal-directed reaches. The network generates a  quasi-feedforward motor command that is learned using training signals  generated by corrective movements. For each target, the network selects  and sets the output of a subset of pattern generators. During the move(cid:173) ment, feedback from proprioceptors turns off the pattern generators. The  task facing individual pattern generators is to recognize when the arm  reaches the target and to turn off. A distributed representation of the mo(cid:173) tor command that resembles population vectors seen in vivo was produced  naturally by these simulations.
Linear Operator for Object Recognition
Authors: Ronen Basri, Shimon Ullman
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Visual  object  recognition  involves  the  identification of images of 3-D  ob(cid:173) jects seen  from  arbitrary  viewpoints.  We  suggest  an  approach  to object  recognition  in  which  a  view  is  represented  as  a  collection  of points  given  by their location in the image.  An object is  modeled by a set of 2-D views  together  with  the  correspondence  between  the  views.  We show that  any  novel  view  of the  object  can  be expressed  as a  linear  combination  of the  stored  views.  Consequently,  we  build  a  linear operator that  distinguishes  between views of a specific object and views of other objects.  This opera(cid:173) tor can be implemented using neural network architectures with relatively  simple structures.
Some Approximation Properties of Projection Pursuit Learning Networks
Authors: Ying Zhao, Christopher G. Atkeson
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: This paper will address an important question in machine learning:  What  kind  of network architectures  work  better on  what  kind  of problems?  A  projection pursuit  learning  network  has  a  very similar  structure to a  one  hidden  layer  sigmoidal  neural  network.  A  general  method  based  on  a  continuous  version  of projection  pursuit  regression  is  developed  to  show  that  projection  pursuit  regression  works  better  on angular  smooth func(cid:173) tions  than on  Laplacian  smooth functions.  There exists  a  ridge  function  approximation  scheme  to  avoid  the  curse  of dimensionality  for  approxi(cid:173) mating functions  in  L2(¢d).
Unsupervised learning of distributions on binary vectors using two layer networks
Authors: Yoav Freund, David Haussler
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We study a particular type of Boltzmann machine with a bipartite graph structure called a harmo(cid:173) nium. Our interest is in using such a machine to model a probability distribution on binary input  vectors. We analyze the class of probability distributions that can be modeled by such machines.  showing that for each n ~ 1 this class includes arbitrarily good appwximations to any distribution  on the set of all n-vectors of binary inputs. We then present two learning algorithms for these  machines .. The first learning algorithm is the standard gradient ascent heuristic for computing  maximum likelihood estimates for the parameters (i.e. weights and thresholds) of the modeL Here  we give a closed form for this gradient that is significantly easier to compute than the corresponding  gradient for the general Boltzmann machine . The second learning algorithm is a greedy method  that creates the hidden units and computes their weights one at a time. This method is a variant  of the standard method for projection pursuit density estimation . We give experimental results for  these learning methods on synthetic data and natural data from the domain of handwritten digits.
Neural Computing with Small Weights
Authors: Kai-Yeung Siu, Jehoshua Bruck
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: An important issue in neural computation is the dynamic range of weights  in  the  neural  networks.  Many  experimental  results  on  learning  indicate  that  the  weights  in  the  networks  can  grow  prohibitively  large  with  the  size  of the  inputs.  Here  we  address  this  issue  by  studying  the  tradeoffs  between  the  depth  and  the  size  of weights  in  polynomial-size  networks  of linear  threshold  elements  (LTEs).  We  show  that  there  is  an  efficient  way  of simulating  a  network  of LTEs  with  large  weights  by  a  network  of LTEs  with  small weights.  In  particular,  we  prove  that  every  depth-d,  polynomial-size network  of LTEs  with  exponentially  large  integer  weights  can  be  simulated  by  a  depth-(2d + 1),  polynomial-size network  of LTEs  with  polynomially  bounded  integer  weights.  To  prove  these  results,  we  use  tools from  harmonic analysis of Boolean  functions.  Our  technique  is  quite  general,  it  provides  insights  to some other  problems.  For  example,  we  are  able to improve the  best  known  results  on  the  depth  of a  network  of linear  threshold  elements  that  computes the  COM PARI SO N,  SUM  and  PRO DU CT of two  n-bits  numbers,  and  the  MAX 1M U M  and  the  SORTING of n  n-bit numbers.
A Segment-Based Automatic Language Identification System
Authors: Yeshwant K. Muthusamy, Ronald A. Cole
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: We have developed  a four-language automatic language identification sys(cid:173) tem  for  high-quality  speech.  The  system  uses  a  neural  network-based  segmentation algorithm to segment speech  into seven  broad phonetic cat(cid:173) egories.  Phonetic  and prosodic features  computed on  these  categories  are  then  input to a  second  network that performs the language classification.  The system was trained  and tested on separate sets  of speakers of Ameri(cid:173) can English, Japanese, Mandarin Chinese and Tamil.  It currently performs  with  an accuracy  of 89.5% on  the utterances  of the test set.
A Topographic Product for the Optimization of Self-Organizing Feature Maps
Authors: Hans-Ulrich Bauer, Klaus Pawelzik, Theo Geisel
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Optimizing the performance of self-organizing feature maps like the Ko(cid:173) honen map involves the choice of the output space topology. We present  a topographic product which measures the preservation of neighborhood  relations as a criterion to optimize the output space topology of the map  with regard to the global dimensionality DA as well as to the dimensi(cid:173) ons in the individual directions. We test the topographic product method  not only on synthetic mapping examples, but also on speech data. In the  latter application our method suggests an output space dimensionality of  DA = 3, in coincidence with recent recognition results on the same data  set.
Learning How to Teach or Selecting Minimal Surface Data
Authors: Davi Geiger, Ricardo A. Marques Pereira
Published in: Advances in Neural Information Processing Systems 4  (NIPS 1991) (Main Conference Track), 1991

Abstract: Learning a  map from an input set  to an output set is similar to the prob(cid:173) lem of reconstructing  hypersurfaces  from sparse  data (Poggio and  Girosi,  1990).  In this framework,  we  discuss  the  problem of automatically select(cid:173) ing  "minimal"  surface  data.  The objective  is  to be  able to approximately  reconstruct  the  surface  from the selected  sparse  data.  We  show  that this  problem is  equivalent  to  the  one  of compressing  information by  data re(cid:173) moval and the one oflearning how to teach.  Our key step is to introduce a  process  that statistically selects  the data according  to the  model.  During  the process  of data selection  (learning how  to teach)  our system  (teacher)  is  capable  of predicting  the  new  surface,  the  approximated  one  provided  by  the  selected  data.  We  concentrate  on  piecewise  smooth surfaces,  e.g.  images,  and  use  mean field  techniques  to  obtain a  deterministic  network  that is shown to compress  image data. 
1  Learning and  surface  reconstruction 
Given a dense input data that represents a hypersurface, how could we automatically  select  very few  data points such  as  to be able to use  these fewer  data points (sparse  data)  to approximately reconstruct  the hypersurface  ?  We  will be  using the term surface  to refer  to hypersurface  (surface  in multidimen-