gaof23/citation_sentiment_corpus huggingface dataset:
context-free rules Charniak (1996) Collins (1996), Eisner (1996) context-free rules, headwords Charniak (1997) context-free rules, headwords, grandparent nodes Collins (2000) context-free rules, headwords, grandparent nodes/rules, bigrams, two-level rules, two-level bigrams, nonheadwords Bod (1992) all fragments within parse trees Scope of Statistical Dependencies Model Figure 4.
For non-recursive NPs, Collins (1997) does not use the probability function in (5), but instead substitutes P r (and, by analogy, P l ) by: P r (R i ;t(R i );l(R i )jP;R i1 ;t(R i1 );l(R i1 );d(i))(8) Here the head H is substituted by the sister R i1 (and L i1 ).
Prominent among these properties is the semi-free Language Size LR LP Source English 40,000 87.4% 88.1% (Collins, 1997) Chinese 3,484 69.0% 74.8% (Bikel and Chiang, 2000) Czech 19,000 80.0% (Collins et al. , 1999) Table 1: Results for the Collins (1997) model for various languages (dependency precision for Czech) wordorder, i.e., German wordorder is fixed in some respects, but variable in others.
controlled NP-traces (NPNP), we follow the standard technique of marking nodes dominating the empty element up to but not including the parent of the antecedent as defective (missing an argument) with a gap feature (Gazdar et al. , 1985; Collins, 1997).1 Furthermore, to make antecedent co-indexation possible with many types of EEs, we generalize Collins approach by enriching the annotation of non-terminals with the type of the EE in question (eg.
unlabeled R 100% 20/08/199605/08/1997 (351 days) 50% 20/08/199617/02/1997 (182 days) 10% 20/08/199624/09/1996 (36 days) labeled WSJ 50% sections 0012 (23412 sentences) 25% lines 1  292960 (11637 sentences) 5% lines 1  58284 (2304 sentences) 1% lines 1  11720 (500 sentences) 0.05% lines 1  611 (23 sentences) Table 1: Corpora used for the experiments: unlabeled Reuters (R) corpus for attachment statistics, labeled Penn treebank (WSJ) for training the Collins parser.
In the probabilistic LR model, probabilities are assigned to tree 696 Precision Recall F-score Time (min) Best-First Classifier-Based (this paper) 88.1 87.8 87.9 17 Deterministic (MaxEnt) (this paper) 85.4 84.8 85.1 < 1 Charniak & Johnson (2005) 91.3 90.6 91.0 Unk Bod (2003) 90.8 90.7 90.7 145* Charniak (2000) 89.5 89.6 89.5 23 Collins (1999) 88.3 88.1 88.2 39 Ratnaparkhi (1997) 87.5 86.3 86.9 Unk Tsuruoka & Tsujii (2005): deterministic 86.5 81.2 83.8 < 1* Tsuruoka & Tsujii (2005): search 86.8 85.0 85.9 2* Sagae & Lavie (2005) 86.0 86.1 86.0 11* Table 1: Summary of results on labeled precision and recall of constituents, and time required to parse the test set.
(owenOcogentex.com) 1 Introduction Dependency grammar has a long tradition in syntactic theory, dating back to at least Tesni~re's work from the thirties3 Recently, it has gained renewed attention as empirical methods in parsing are discovering the importance of relations between words (see, e.g., (Collins, 1997)), which is what dependency grammars model explicitly do, but context-free phrasestructure grammars do not.
The lexicalized PCFG that sits behind Model 2 of (Collins, 1997) has rules of the form P ~ LnLn-I"'" LIHRI"".Rn-IRn (1) S(will-MD) NP(AppI,~NNP) VP(wilI-MD) NNP I Apple MD VP (buy-VB) VB PRT(out-RP) NP(Microsoft--NNP) I \[ I buy RP NNP I I out Microsoft Figure 1: A sample sentence with parse tree.
Thus, over the past few years, along with advances in the use of learning and statistical methods for acquisition of full parsers (Collins, 1997; Charniak, 1997a; Charniak, 1997b; Ratnaparkhi, 1997), significant progress has been made on the use of statistical learning methods to recognize shallow parsing patterns  syntactic phrases or words that participate in a syntactic relationship (Church, 1988; Ramshaw and Marcus, 1995; Argamon et al. , 1998; Cardie and Pierce, 1998; Munoz et al. , 1999; Punyakanok and Roth, 2001; Buchholz et al. , 1999; Tjong Kim Sang and Buchholz, 2000).
So the sequence with a fork, which corresponds to only one nucleus is treated as a three word sequence in model C. Apart from this difference, model C directly relies on a combination of equations (10) and (12), namely conditioning by a80a7a81a49a82a9a12, a74a61a8a65a75a57a12 and a74a61a8a65a75a57a14a61a86, both the probability of generating a74a61a8a65a75 a47 and the one of generating a80a7a81a49a82 a47 . Thus, model C uses a reduced version of equation (12) and an extended version of 2Other models, as (Collins and Brooks, 1995; Merlo et al. , 1998) for PP-attachment resolution, or (Collins, 1997; Samuelsson, 2000) for probabilistic parsing, are somewhat related, but their supervised nature makes any direct comparison impossible.
2 Previous Work Other researchers have investigated the topic of automatic generation of abstracts, but the focus has been different, e.g., sentence extraction (Edmundson, 1969; Johnson et al, 1993; Kupiec et al. , 1995; Mann et al. , 1992; Teufel and Moens, 1997; Zechner, 1995), processing of structured templates (Paice and Jones, 1993), sentence compression (Hori et al. , 2002; Knight and Marcu, 2001; Grefenstette, 1998, Luhn, 1958), and generation of abstracts from multiple sources (Radev and McKeown, 1998).
To make the model more practical in parameter estimation, we assume the features in feature set FS are independent from each other, thus:   = FSFi AFiPAFSP ),|(),|(  (5) Under this PCFG+PF model, the goal of a parser is to choose a parse that maximizes the following score: )|,(maxarg)|( 1 AFS i i i n i T PSTScore  = = (6) Our model is thus a simplification of more sophisticated models which integrate PCFGs with features, such as those in Magerman(1995), Collins(1997) and Goodman(1997).
Function P R F Speed Partial Parsing 85.1 82.5 83.8 4500 wps Full Parsing 77.1 70.3 73.7 2100 wps Table 3: Performances of 1 st -level Partial Parsing and Full Parsing (wps: words per second) Table 3 shows that the performances of partial parsing and full parsing are quite low, compared to those of state-of-art partial parsing and full parsing for the English language (Zhou et al 2000a; Collins 1997).
As in (Collins, 1997), the parameter C8 D0 B4C4 CX B4D0D8 CX BND0DB CX B5CYC8BNC0BNDBBND8BNA1BNC4BVB5 is further smoothed as follows: C8 D0BD B4C4 CX CYC8BNC0BNDBBND8BNA1BNC4BVB5 A2 C8 D0BE B4D0D8 CX CYC8BNC0BNDBBND8BNA1BNC4BVBNC4 CX B5A2 C8 D0BF B4D0DB CX CYC8BNC0BNDBBND8BNA1BNC4BVBNC4 CX B4D0D8 CX B5B5 Note this smoothing is different from the syntactic counterpart.
(2005) 86.6 86.7 1.19 Klein and Manning (2003) 86.9 85.7 86.3 30.9 1.10 Charniak (1997) 87.4 87.5 1.00 Collins (1997) 88.6 88.1 0.91 Table 3: Comparison with other parsers (sentences of length  40) as head information.
Collins (1997)s parser and its reimplementation and extension by Bikel (2002) have by now been applied to a variety of languages: English (Collins, 1999), Czech (Collins et al. , 1999), German (Dubey and Keller, 2003), Spanish (Cowan and Collins, 2005), French (Arun and Keller, 2005), Chinese (Bikel, 2002) and, according to Dan Bikels web page, Arabic.
It is interesting to note that, while the study of how the granularity of context-free grammars (CFG) affects the performance of a parser (e.g. in the form 86 n1:IP [=] n2:NP [SUBJ=] n4:NR [=] GSC4ES JiangZemin n3:VP [=] n5:VV [=] ESDO interview n6:NP [OBJ=] n7:NR [ADJUNCT] AIC1 Thai n8:NN [=] D3D2 president f1             PRED ESDO SUBJ f2  PRED GSC4ESNTYPE proper NUM sg   OBJ f3       PRED D3D2 NTYPE common NUM sg ADJUNCT   f4  PRED AIC1NTYPE proper NUM sg                         : N  F (n1)=(n3)=(n5)=f1 (n2)=(n4)=f2 (n6)=(n8)=f3 (n7)=f4 Figure 1: Cand f-structures with  links for the sentence GSC4ESESDOAIC1D3D2 of grammar transforms (Johnson, 1998) and lexicalisation (Collins, 1997)) has attracted substantial attention, to our knowledge, there has been a lot less research on this subject for surface realisation, a process that is generally regarded as the reverse process of parsing.
Version of the System P R F Baseline 50.8 100 67.4 Discourse-new detection only 69 72 70 Hand-coded DT: partial 62 85 71.7 Hand-coded DT: total 77 77 77 ID3 75 75 75 Table 1: Overall results by Vieira and Poesio 2.2 Bean and Riloff Bean and Riloff (1999) developed a system for identifying discourse-new DDs1 that incorporates, in addition to syntax-based heuristics aimed at recognizing predicative and established DDs using postmodification heuristics similar to those used by Vieira and Poesio, additional techniques for mining from corpora unfamiliar DDs including proper names, larger situation, and semantically functional.
We trained log linear models with theperceptronalgorithm(Collins,2002)usingfea746 Markov order Classification Task 0 1 2 S1 (no multi-word constituent start) 96.7 96.9 96.9 E1 (no multi-word constituent end) 97.3 97.3 97.3 Table 2: Classification accuracy on development set for binary classes S1 and E1, for various Markov orders.
This algorithm is proved to converge (i.e. , there are no more updates) in the separable case (Collins, 2002a).1 Thatis,ifthereexistweightvectorU (with ||U|| = 1),  (> 0), and R (> 0) that satisfy: i,y  Y|xi| (xi,yi)U (xi,y)U  , i,y  Y|xi| ||(xi,yi)(xi,y)||  R, the number of updates is at most R2/2.
However, by examining the Algorithm 4.2: Perceptron with local and non-local features (parameters: n, Ca, Cl)   0 until no more updates do for i  1 to L do8 >> >> >> >> >> < >> >> >> >> >> : {yn} = n-bestyl(xi,y) y = argmaxy{yn}a(xi,y) y = 2nd-besty{yn}a(xi,y) if y = yi & a(xi,yi)a(xi,y)  Ca then  = + a(xi,yi)a(xi,y) (A) else if a(xi,yi)a(xi,y)  Ca then  = + a(xi,yi)a(xi,y) (A) else (B) 8> < >: if y1 = yi then (y1 represents the best in {yn})  = + l(xi,yi)l(xi,y1) else if l(xi,yi)l(xi,y2)  Cl then  = + l(xi,yi)l(xi,y2) proofs in Collins (2002a), we can see that the essential condition for convergence is that the weights are always updated using some y (= y) that satises: (xi,yi)(xi,y)  0 ( C in the case of a perceptron with a margin).
The definition of 2(x, h, m, c) is:  dir  cpos(xh)  cpos(xm)  cpos(xc)  dir  cpos(xh)  cpos(xc)  dir  cpos(xm)  cpos(xc)  dir  form(xh)  form(xc)  dir  form(xm)  form(xc)  dir  cpos(xh)  form(xc)  dir  cpos(xm)  form(xc)  dir  form(xh)  cpos(xc)  dir  form(xm)  cpos(xc) 3 Experiments and Results We report experiments with higher-order models for the ten languages in the multilingual track of the CoNLL-2007 shared task (Nivre et al. , 2007).1 In all experiments, we trained our models using the averaged perceptron (Freund and Schapire, 1999), following the extension of Collins (2002) for structured prediction problems.
Given an input training corpus of such derivations D = d1 dn, a vector feature function on derivations vectorF(d), and an initial weight vector vectorw, the perceptron performs two steps for each training example di  D:  Decode: d = argmaxdD(src(di)) parenleftBig vectorw vectorF(d) parenrightBig  Update: vectorw = vectorw + vectorF(di) vectorF(d) where D(src(d)) enumerates all possible derivations with the same source side as d. To improve generalization, the final feature vector is the average of all vectors found during learning (Collins, 2002).
0 10 20 30 40 5097.20 97.25 97.30 97.35 97.40 Iteration Accuracy on development data Every iterationEvery 4th iterationEvery 8th iteration Every 16th iterationOnce at the beginning No supervised data Figure 2: Dependence on the inclusion of the supervised training data (English) English Czech No supervised data 97.37 95.88 Once at the beginning 97.40 96.00 Every training iteration 97.44 96.21 Table 6: Dependence on the inclusion of the supervised training data 5.4 The morphological analyzers and the perceptron feature templates The whole experiment can be performed with the original perceptron feature set described in (Collins, 2002) instead of the feature set described in this article.
c2009 Association for Computational Linguistics Semi-supervised Training for the Averaged Perceptron POS Tagger Drahomra johanka Spoustova Jan Hajic Jan Raab Miroslav Spousta Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics, Charles University Prague, Czech Republic {johanka,hajic,raab,spousta}@ ufal.mff.cuni.cz Abstract This paper describes POS tagging experiments with semi-supervised training as an extension to the (supervised) averaged perceptron algorithm, first introduced for this task by (Collins, 2002).
To summarize, we can describe our system as follows: it is based on (Votrubec, 2006)s implementation of (Collins, 2002), which has been fed at each iteration by a different dataset consisting of the supervised and unsupervised part: precisely, by a concatenation of the manually tagged training data (WSJ portion of the PTB 3 for English, morphologically disambiguated data from PDT 2.0 for Czech) and a chunk of automatically tagged unsupervised data.
Previous authors have used numerous HMM-based models (Banko and Moore, 2004; Collins, 2002; Lee et al. , 2000; Thede and Harper, 1999) and other types of networks including maximum entropy models (Ratnaparkhi, 1996), conditional Markov models (Klein and Manning, 2002; McCallum et al. , 2000), conditional random elds (CRF) (Lafferty et al. , 2001), and cyclic dependency networks (Toutanova et al. , 2003).
Networks (Toutanova et al. , 2003) 97.24 Perceptron (Collins, 2002) 97.11 SVM (Gimenez and Marquez, 2003) 97.05 HMM (Brants, 2000) 96.48 Easiest-first 97.10 Full Bidirectional 97.15 Table 3: POS tagging accuracy on the test set (Sections 22-24 of the WSJ, 5462 sentences).
For instance, work has been done in Chinese using the Penn Chinese Treebank (Levy and Manning, 2003; Chiang and Bikel, 2002), in Czech using the Prague Dependency Treebank (Collins et al. , 1999), in French using the French Treebank (Arun and Keller, 2005), in German using the Negra Treebank (Dubey, 2005; Dubey and Keller, 2003), and in Spanish using the UAM Spanish Treebank (Moreno et al. , 2000).
The central question in learning is how to set the parameters a, given the training examples b x 1, y 1 ,x 2, y 2 , :::,x n, y n   . Logistic regression and boosting involve different algorithms and criteria for training the parameters a, but recent work (Friedman, Hastie, and Tibshirani 2000; Lafferty 1999; Duffy and Helmbold 1999; Mason, Bartlett, and Baxter 1999; Lebanon and Lafferty 2001; Collins, Schapire, and Singer 2002) has shown that the methods have strong similarities.
(2002) do not use a feature selection technique, employing instead an objective function which includes a Table 4 Values of Savings (a, b) for various values of a, b. ab Savings (a, b) 1100,000 2,692.7 110 48.6 11100 83.5 1011,000 280.0 1,00110,000 1,263.9 10,00150,000 2,920.2 50,001100,000 4,229.8 Collins and Koo Discriminative Reranking for NLP Gaussian prior on the parameter values, thereby penalizing parameter values which become too large: a C3  arg min a  LogLossa X k0:::m a 2 k 7 2 k  28 Closed-form updates under iterative scaling are not possible with this objective function; instead, optimization algorithms such as gradient descent or conjugate gradient methods are used to estimate parameter values.
Section 3 describes previous work (Friedman, Hastie, and Tibshirani 2000; Duffy and Helmbold 1999; Mason, Bartlett, and Baxter 1999; Lebanon and Lafferty 2001; Collins, Schapire, and Singer 2002) that derives connections between boosting and maximum-entropy models for the simpler case of classification problems; this work forms the basis for the reranking methods.
Results from Collins, Schapire and Singer (2002) show that under these definitions the following guarantee holds: LogLossUpda,k, BestWtk, a C20 BestLossk, a So it can be seen that the update from a to Upda, k, BestWtk, a is guaranteed to decrease LogLoss by at least  W  k q C0  W C0 k qC16C17 2 . From these results, the algorithms in Figures 3 and 4 could be altered to take the revised definitions of W  k and W C0 k into account.
MUC-6 347 30 204,071 6.8 Enron 833 143 204,423 3.0 Mgmt-Groups 631 128 104,662 3.7 Table 1: Summary of the corpora used in the experiments We used an implementation of Collins votedpercepton method for discriminatively training HMMs (henceforth, VP-HMM) (Collins, 2002) as well as CRF (Lafferty et al. , 2001) to learn a NER.
Previous approaches for training CRFs have either (1) opted for a training method that no longer maximizes the likelihood, (e.g. McCallum and Wellner (2004), Roth and Yih (2005)) 1, or (2) opted for a 1 Both McCallum and Wellner (2004) and Roth and Yih (2005) used the voted perceptron algorithm (Collins, 2002) to train intractable CRFs.
We can then state the following theorem (see (Collins, 2002) for a proof): Theorem 1 For any training sequence (xi;yi) that is separable with margin, for any value of T, then for the perceptron algorithm in figure 1 Ne R 2 2 where R is a constant such that 8i;8z 2 GEN(xi) jj (xi;yi) (xi;z)jj R. This theorem implies that if there is a parameter vector U which makes zero errors on the training set, then after a finite number of iterations the training algorithm will converge to parameter values with zero training error.
Discriminative learning methods, such as Maximum Entropy Markov Models (McCallum et al. , 2000), Projection Based Markov Models (Punyakanok and Roth, 2000), Conditional Random Fields (Lafferty et al. , 2001), Sequence AdaBoost (Altun et al. , 2003a), Sequence Perceptron (Collins, 2002), Hidden Markov Support Vector Machines (Altun et al. , 2003b) and Maximum-Margin Markov Networks (Taskar et al. , 2004), overcome the limitations of HMMs.
Averaged perceptron (Collins, 2002a), which has been successfully applied to several tagging and parsing reranking tasks (Collins, 2002c; Collins, 2002a), was employed for training rerank267 CLANG GEOQUERY P R F P R F SCISSOR 89.5 73.7 80.8 98.5 74.4 84.8 SCISSOR+ 87.0 78.0 82.3 95.5 77.2 85.4 Table 2: The performance of the baseline model SCISSOR+ compared with SCISSOR (with the best result in bold), where P = precision, R = recall, and F = F-measure.
The model is composed of three parts (Collins, 2002a): a set of candidate SAPTs GEN, which is the top n SAPTs of a sentence from SCISSOR; a function  that maps a sentence Inputs: A set of training examples (xi,yi ), i = 1n, where xi is a sentence, and yi is a candidate SAPT that has the highest similarity score with the gold-standard SAPT Initialization: Set W = 0 Algorithm: For t = 1T,i = 1n Calculate yi = argmaxyGEN(xi) (xi,y)  W If (yi negationslash= yi ) then W = W +(xi,yi )  (xi,yi) Output: The parameter vector W Figure 2: The perceptron training algorithm.
In comparison with shallow semantic analysis tasks, such as wordsense disambiguation (Ide and Jeaneronis, 1998) and semantic role labeling (Gildea and Jurafsky, 2002; Carreras and M`arquez, 2005), which only partially tackle this problem by identifying the meanings of target words or finding semantic roles of predicates, semantic parsing (Kate et al. , 2005; Ge and Mooney, 2005; Zettlemoyer and Collins, 2005) pursues a more ambitious goal  mapping natural language sentences to complete formal meaning representations (MRs), where the meaning of each part of a sentence is analyzed, including noun phrases, verb phrases, negation, quantifiers and so on.
Hidden Markov models (Rabiner, 1989) are one of the earliest structured learning algorithms, which have recently been followedbydiscriminativelearningapproachessuch as conditional random fields (CRFs) (Lafferty et al. , 2001; Sutton and McCallum, 2006), the structured perceptron (Collins, 2002) and its large-margin variants (Taskar et al. , 2003; Tsochantaridis et al. , 2004; McDonald et al. , 2005; Daume III et al. , 2006).
Denote the global feature vector for segmented sentence y with (y)  Rd, where d is the total number of features in the model; then Score(y) is computed by the dot product of vector (y) and a parameter vector   Rd, where i is the weight for the ith feature: Score(y) = (y) 841 Inputs: training examples (xi,yi) Initialization: set  = 0 Algorithm: for t = 1T, i = 1N calculate zi = argmaxyGEN(xi) (y) if zi negationslash= yi  =  + (yi)(zi) Outputs:  Figure 1: the perceptron learning algorithm, adapted from Collins (2002) The perceptron training algorithm is used to determine the weight values .
To facilitate comparisons with previous work (McDonald et al., 2005b; McDonald and Pereira, 2006), we used the training/development/test partition defined in the corpus and we also used the automatically-assigned part of speech tags provided in the corpus.10 Czech word clusters were derived from the raw text section of the PDT 1.0, which contains about 39 million words of newswire text.11 We trained the parsers using the averaged perceptron (Freund and Schapire, 1999; Collins, 2002), which represents a balance between strong performance and fast training times.
1 word w 2 word bigram w1w2 3 single-character word w 4 a word of length l with starting character c 5 a word of length l with ending character c 6 space-separated characters c1 and c2 7 character bigram c1c2 in any word 8 the first / last characters c1 / c2 of any word 9 word w immediately before character c 10 character c immediately before word w 11 the starting characters c1 and c2 of two consecutive words 12 the ending characters c1 and c2 of two consecutive words 13 a word of length l with previous word w 14 a word of length l with next word w Table 1: Feature templates for the baseline segmentor 2 The Baseline System We built a two-stage baseline system, using the perceptron segmentation model from our previous work (Zhang and Clark, 2007) and the perceptron POS tagging model from Collins (2002).
3.3 Perceptron learning of feature weights As we saw above, our model is a linear model with the global weight vector w acting as the coefficient vector, and hence various existing techniques can be exploited to optimize w. In this paper, we use the averaged perceptron learning (Collins, 2002; Freund and Schapire, 1999) to optimize w on a training corpus, so that the system assigns the highest score to the correct coordination tree among all possible trees for each training sentence.
Here we used the averaged perceptron (Collins, 2002), where the weight matrix used to classify the test data is the average of all of the matrices posited during training, i.e., a1 a62 a52 a49 a62 a49 a42a51a50a53a52 a1 a42 . 4.2 Multicomponent architecture Task specific and external training data are integrated with a two-component perceptron.
Finally, our modeling approaches follow the recent work on both local classifier-based modeling of complex learning problems (McCallum et al. 2000; Punyakanok and Roth 2001), as well as global discriminative approaches based on CRFs (Lafferty et al. 2001), SVM (Taskar et al. 2005), and the Perceptron algorithm (Collins 2002) that we used in our experiments.
3 Perceptron Training The parsing problem is to find a mapping from a set of sentences x ??X to a set of parses y ??Y. We assume that the mapping F is represented through a feature vector (x,y) ??Rd and a parameter vector  ??Rd in the following way (Collins, 2002): F(x) = argmax y?GEN(x) (x,y) (1) where GEN(x) denotes the set of possible parses for sentence x and (x,y)   = summationtexti ii(x,y) is the inner product.
1 Introduction In global linear models (GLMs) for structured prediction, (e.g., (Johnson et al., 1999; Lafferty et al., 2001; Collins, 2002; Altun et al., 2003; Taskar et al., 2004)), the optimal label y for an input x is y = arg max yY(x) w f(x,y) (1) where Y(x) is the set of possible labels for the input x; f(x,y)  Rd is a feature vector that represents the pair (x,y); and w is a parameter vector.
joint likelihood (JL) productdisplay i p parenleftBig xi,yi | vector parenrightBig conditional likelihood (CL) productdisplay i p parenleftBig yi | xi,vector parenrightBig classification accuracy (Juang and Katagiri, 1992) summationdisplay i (yi, y(xi)) expected classification accuracy (Klein and Manning, 2002) summationdisplay i p parenleftBig yi | xi,vector parenrightBig negated boosting loss (Collins, 2000)  summationdisplay i p parenleftBig yi | xi,vector parenrightBig1 margin (Crammer and Singer, 2001)  s.t. bardbl vectorbardbl  1;i,y negationslash= yi, vector  (vectorf(xi,yi )  vectorf(xi,y))   expected local accuracy (Altun et al. , 2003) productdisplay i productdisplay j p parenleftBig lscriptj(Y ) = lscriptj(yi ) | xi,vector parenrightBig Table 1: Various supervised training criteria.