gaof23/citation_sentiment_corpus huggingface dataset:
Following recent research about disambiguation models on linguistic grammars (Abney, 1997; Johnson et al. , 1999; Riezler et al. , 2002; Clark and Curran, 2003; Miyao et al. , 2003; Malouf and van Noord, 2004), we apply a log-linear model or maximum entropy model (Berger et al. , 1996) on HPSG derivations.
One is how to learn a statistical model to estimate the conditional probability    , and the other is how to generate confusion set C of a given query q 4.1 Maximum Entropy Model for Query Spelling Correction We take a feature-based approach to model the posterior probability     . Specifically we use the maximum entropy model (Berger et al. , 1996) for this task:     = exp     ,   =1 exp(     (,  ) =1 ) (2) where exp(     (, ) =1 ) is the normalization factor;   , is a feature function defined over query q and correction candidate c, while   is the corresponding feature weight.
AL has already been applied to several NLP tasks, such as document classification (Schohn and Cohn, 2000), POS tagging (Engelson and Dagan, 1996), chunking (Ngai and Yarowsky, 2000), statistical parsing (Thompson et al. , 1999; Hwa, 2000), and information extraction (Lewis and Catlett, 1994; Thompson et al. , 1999).
(1) Here, the candidate generator gen(s) enumerates candidates of destination (correct) strings, and the scorer P(t|s) denotes the conditional probability of the string t for the given s. The scorer was modeled by a noisy-channel model (Shannon, 1948; Brill and Moore, 2000; Ahmad and Kondrak, 2005) and maximum entropy framework (Berger et al., 1996; Li et al., 2006; Chen et al., 2007).
(14), where  i is the parameter to be estimated and f i (a, b) is a feature function corresponding to  i (Berger et al., 1996; Ratnaparkhi, 1997): P(E P |E G )  productdisplay i P(ep i |ep i1 ik ,eg i+k ik ) (11) P(C P |E G ,E P ) (12)  productdisplay i P(cp i |cp i1 ik ,eg, ep i+k ik ) P(C G |E G ,E P ,C P ) (13)  productdisplay i P(cg i |cg i1 ik ,eg, ep, cp i+k ik ) P(b|a)= exp( summationtext i  i f i (a, b)) summationtext b prime exp( summationtext i  i f i (a, b prime )) (14) f i (a, b) is a binary function returning TRUE or FALSE based on context a and output b. If f i (a, b)=1, its corresponding model parameter  i contributes toward conditional probability P(b|a) (Berger et al., 1996; Ratnaparkhi, 1997).
DTM2, introduced in (Ittycheriah and Roukos, 2007), expresses the phrase-based translation task in a unified log-linear probabilistic framework consisting of three components: (i) a prior conditional distribution P0(.|S), (ii) a number of feature functions i() that capture the translation and language model effects, and (iii) the weights of the features i that are estimated under MaxEnt (Berger et al., 1996), as in (1): P(T|S) = P0(T,J|S)Z expsummationdisplay i ii(T,J,S) (1) Here J is the skip reordering factor for the phrase pair captured by i() and represents the jump from the previous source word, and Z is the per source sentence normalization term.
The Maximum Entropy model (Berger et al., 1996; Ratnaparkhi, 1997; Abney, 1997) is a conditional model that assigns a probability to every possible parse  for a given sentence s. The model consists of a set of m feature functions fj() that describe properties of parses, together with their associated weights j. The denominator is a normalization term where Y (s) is the set of parses with yield s: p(|s;) = exp( summationtextm j=1 jfj())summationtext yY (s) exp( summationtextm j=1 jfj(y))) (1) The parameters (weights) j can be estimated efficiently by maximizing the regularized conditional likelihood of a training corpus (Johnson et al., 1999; van Noord and Malouf, 2005):  = argmax  logL()  summationtextm j=1  2j 22 (2) where L() is the likelihood of the training data.
We use a simple, single parameter distribution, with  = 8.0 throughout P(K|m,e) = P(K|m,l)  K Word-to-Phrase Alignment Alignment is a Markov process that specifies the lengths of phrases and their alignment with source words P(aK1,hK1,K1 |K,m,e) = Kproductdisplay k=1 P(ak,hk,k|ak1,k1,e) = Kproductdisplay k=1 p(ak|ak1,hk;l)d(hk)n(k;eak) The actual word-to-phrase alignment (ak) is a firstorder Markov process, as in HMM-based word-toword alignment (Vogel et al. , 1996).
660 2 Statistical Coreference Resolution Model Our coreference system uses a binary entity-mention model PL( je, m) (henceforth link model ) to score the action of linking a mention m to an entity e. In our implementation, the link model is computed as PL(L = 1je, m) max mprimee PL(L = 1je, mprime, m), (1) where mprime is one mention in entity e, and the basic model building block PL(L = 1je, mprime, m) is an exponential or maximum entropy model (Berger et al. , 1996): PL(Lje, mprime, m) = exp braceleftbig summationtext i igi(e, m prime, m, L)bracerightbig Z(e, mprime, m), (2) where Z(e, mprime, m) is a normalizing factor to ensure that PL( je, mprime, m) is a probability, fgi(e, mprime, m, L)g are features and fig are feature weights.
Given a set of features and a training corpus, the ME estimation process produces a model in which every feature fi has a weight i. From (Berger et al. , 1996), we can compute the conditional probability as: p(o|h) = 1Z(h)productdisplay i fi(h,o)i (2) Z(h) =summationdisplay o productdisplay i fi(h,o)i (3) The probability is given by multiplying the weights of active features (i.e. , those fi(h,o) = 1).
Feature selection methods have been proposed in the maximum-entropy literature by several authors (Ratnaparkhi, Roukos, and Ward 1994; Berger, Della Pietra, and Della Pietra 1996; Della Pietra, Della Pietra, and Lafferty 1997; Papineni, Roukos, and Ward 1997, 1998; McCallum 2003; Zhou et al. 2003; Riezler and Vasserman 2004).
6.4 Feature Selection Methods A number of previous papers (Berger, Della Pietra, and Della Pietra 1996; Ratnaparkhi 1998; Della Pietra, Della Pietra, and Lafferty 1997; McCallum 2003; Zhou et al. 2003; Riezler and Vasserman 2004) describe feature selection approaches for log-linear models applied to NLP problems.
The theory has been applied in probabilistic language modeling (Mark, Miller, and Grenander 1996; Mark et al. 1996; Johnson 1998), natural language processing (Berger, Della Pietra, and Della Pietra 1996; Della Pietra, Della Pietra, and Lafferty 1997), as well as computational vision (Zhu, Wu, and Mumford 1997).
The sequential classi cation approach can handle many correlated features, as demonstrated in work on maximum-entropy (McCallum et al. , 2000; Ratnaparkhi, 1996) and a variety of other linear classi ers, including winnow (Punyakanok and Roth, 2001), AdaBoost (Abney et al. , 1999), and support-vector machines (Kudo and Matsumoto, 2001).
Algorithm 1 The RRM Decoding Algorithm foreacha26a29a27a67a42 foreacha68 a1a20a23a69a10a11a10a12a10a45 a60 a48a22a70a26a22a71 a1a73a72a2a25 a57a38a50 a7 a56 a48a54a57 a64a74a30 a57 a31a33a26a17a34 a5a11a75 a60a77a76a74a76 a31a78a26a35a34a66a79a81a80a83a82a38a84a69a85a86a80a24a87a88a48 a60 a48 a70a26a61a71 Somewhat similarly, the MaxEnt algorithm has an associated set of weights a31a33a89 a48a54a57 a34a48a90a50 a7a53a52a54a52a54a52a15 a57a38a50 a7a58a52a54a52a54a52 a25, which are estimated during the training phase so as to maximize the likelihood of the data (Berger et al. , 1996).
The resulting model has an exponential form with free parameters a102 a91 a24a94a93 a8 a87 a24 a10a11a10a11a10 a24a46a95 . The parameter values which maximize the likelihood for a given training corpus can be computed with the socalled GIS algorithm (general iterative scaling) or its improved version IIS (Pietra et al. , 1997; Berger et al. , 1996).
We use maximum entropy model (Berger et al. , 1996) for both the mention-pair model (9) and the entity-mention model (8): a83a84a1a86a85a88a87 a43 a44 a71 a43 a16 a5a13a7 a55a35a34a23a36 a6a35a37 a6a39a38a40a6a42a41 a31a44a43a3a45a31 a6 a45a46a48a47a24a49 a50 a1 a43 a44 a71 a43 a16 a5 a71 (10) a83a84a1a4a85 a87 a55 a81 a71 a43 a16 a5a13a7 a55a35a34 a36 a6 a37 a6a39a38a40a6a42a41 a11a7a32 a45a31 a6 a45a46a48a47 a49 a50 a1 a55a39a81 a71 a43 a16 a5 a71 (11) wherea57 a16 a1a51a8 a71a52a8 a71a90a85a73a5 is a feature and a53 a16 is its weight; a50 a1a33a8 a71a54a8a5 is a normalizing factor to ensure that (10) or (11) is a probability.
We use maximum entropy modeling (Berger et al. , 1996) to directly model the conditional probability a17a19a18a20a2a21a15a23a22a24a26a25, where each a27a5a15 in a24a29a28a30a18a31a27a32a4a33a6a7a8a9a8a9a8a9a6a23a27a34a11a14a25 is an observation associated with the corresponding speaker a2 a15 . a27 a15 is represented here by only one variable for notational ease, but it possibly represents several lexical, durational, structural, and acoustic observations.
Speaker ranking accuracy Table 2 summarizes the accuracy of our statistical ranker on the test data with different feature sets: the performance is 89.39% when using all feature sets, and reaches 90.2% after applying Gaussian smoothing and using incremental feature selection as described in (Berger et al. , 1996) and implemented in the yasmetFS package.6 Note that restricting ourselves to only backward looking features decreases the performance significantly, as we can see in Table 2.
For this reason there is currently a great deal of interest in methods which incorporate syntactic information within statistical machine translation systems (e.g. , see (Alshawi, 1996; Wu, 1997; Yamada and Knight, 2001; Gildea, 2003; Melamed, 2004; Graehl and Knight, 2004; Och et al. , 2004; Xia and McCord, 2004)).
2.1.2 Research on Syntax-Based SMT A number of researchers (Alshawi, 1996; Wu, 1997; Yamada and Knight, 2001; Gildea, 2003; Melamed, 2004; Graehl and Knight, 2004; Galley et al. , 2004) have proposed models where the translation process involves syntactic representations of the source and/or target languages.
However, in order to cope with the prediction errors of the classi er, we approximate a74a51a18a77a76 a28 with an a80 -gram language model on sequences of the re ned tag labels: a38a58a39 a41 a81 a43a82a44a47a46a83a48a47a50a75a44a15a52 a53a9a54a49a84 a53a9a54a83a84a49a85a9a86a13a87a89a88a91a90 a55a57a56 a38a40a39 a81 a59a60a42a61 (2) a92 a44a47a46a83a48a47a50a75a44a15a52 a53a9a54 a84 a53a9a54a83a84a49a85a9a86a13a87a89a88a91a90 a93 a94a96a95 a55a57a56a98a97a66a99 a95 a59a100a27a61 (3) In order to estimate the conditional distribution a101 a18a20a19a15a21 a1 a68 a72 a28 we use the general technique of choosing the maximum entropy (maxent) distribution that properly estimates the average of each feature over the training data (Berger et al. , 1996).
p0(t|w) is calculated by ME models as follows (Berger et al. , 1996): p0(t|w)= 1Y(w) exp braceleftBigg Hsummationdisplay h=1 hgh(w,t) bracerightBigg, (20) 709 Language Features English Prefixes of 0 up to four characters, suffixes of 0 up to four characters, 0 contains Arabic numerals, 0 contains uppercase characters, 0 contains hyphens.
Following (Ratnaparkhi, 1996; Collins, 2002; Toutanova et al. , 2003; Tsuruoka and Tsujii, 2005), 765 Feature Sets Templates Error% A Ratnaparkhis 3.05 B A + [t0,t1],[t0,t1,t1],[t0,t1,t2] 2.92 C B + [t0,t2],[t0,t2],[t0,t2,w0],[t0,t1,w0],[t0,t1,w0], [t0,t2,w0], [t0,t2,t1,w0],[t0,t1,t1,w0],[t0,t1,t2,w0] 2.84 D C + [t0,w1,w0],[t0,w1,w0] 2.78 E D + [t0,X = prefix or suffix of w0],4 < |X|  9 2.72 Table 2: Experiments on the development data with beam width of 3 we cut the PTB into the training, development and test sets as shown in Table 1.
766 System Beam Error% (Ratnaparkhi, 1996) 5 3.37 (Tsuruoka and Tsujii, 2005) 1 2.90 (Collins, 2002) 2.89 Guided Learning, feature B 3 2.85 (Tsuruoka and Tsujii, 2005) all 2.85 (Gimenez and M`arquez, 2004) 2.84 (Toutanova et al. , 2003) 2.76 Guided Learning, feature E 1 2.73 Guided Learning, feature E 3 2.67 Table 4: Comparison with the previous works According to the experiments shown above, we build our best system by using feature set E with beam width B = 3.
Given a set of features and a training corpus, the MaxEnt estimation process produces a model in which every feature fi has a weight i. We can compute the conditional probability as (Berger et al., 1996): p(o|h) = 1Z(h) productdisplay i ifi(h,o) (1) Z(h) = summationdisplay o productdisplay i ifi(h,o) (2) The conditional probability of the outcome is the product of the weights of all active features, normalized over the products of all the features.
Formally, the approach we take can be thought of as a noisier channel, where an observed signal o gives rise to a set of source-language strings fprime  F(o) and we seek e = arg maxe max fprimeF(o) Pr(e,fprime|o) (2) = arg maxe max fprimeF(o) Pr(e)Pr(fprime|e,o) (3) = arg maxe max fprimeF(o) Pr(e)Pr(fprime|e)Pr(o|fprime).(4) Following Och and Ney (2002), we use the maximum entropy framework (Berger et al., 1996) to directly model the posterior Pr(e,fprime|o) with parameters tuned to minimize a loss function representing 1012 the quality only of the resulting translations.
4.1 Evaluation of Different Features and Models In pilot experiments on a subset of the features, we provide a comparison of HM-SVM with other two learning models, maximum entropy (MaxEnt) model (Berger et al., 1996) and SVM model (Kudo, 2001), to test the effectiveness of HMSVM on function labeling task, as well as the generality of our hypothesis on different learning 58 Table 3: Features used in each experiment round.
A conditional maximum entropy model q(xjw) for p has the parametric form (Berger et al. , 1996; Chi, 1998; Johnson et al. , 1999): q(xjw) = exp T f (x) y2Y(w) exp(T f (y)) (1) where  is a d-dimensional parameter vector and T f (x) is the inner product of the parameter vector and a feature vector.
Thus, we obtain the following second-order model: a36a39a38a41a40 a17 a5a7 a42a4 a5a7 a44 a8 a5a57 a15a27a58 a7 a36a39a38a41a40 a17a20a15a59a42a17 a15a41a49 a7 a7 a60 a4 a5a7 a44 a8 ma61a63a62a65a64a33a66 a5a57 a15a27a58 a7a68a67 a40 a17 a15 a42a17 a15a50a49 a7 a15a50a49a51a48 a60 a4 a15a27a47a55a48 a15a50a49a54a48 a44 a11 A well-founded framework for directly modeling the posterior probability a67 a40 a17 a15 a42a17 a15a50a49 a7 a15a50a49a54a48 a60 a4 a15a12a47a55a48 a15a50a49a54a48 a44 is maximum entropy (Berger et al. , 1996).
The goal of each selection stage is to select the feature f that maximizes the gain of the log likelihood, where the a and gain of f are derived through following steps: Let the log likelihood of the model be  -=  yx xZysump pL,, )(/|log()( ~ and the empirical expectation of feature f be  E  p (f)= p (x,y)f(x,y) x,y  With the approximation assumption in Berger et al (1996)s paper, the un-normalized component and the normalization factor of the model have the following recursive forms: )|()|( aa exysumxysum SfS =  | Z f + The approximate gain of the log likelihood is computed by  G Sf (a)L(p Sf a )-L(p S ) =- p (x)(logZ Sf,a (x) x  /Z S (x)) +aE  p (f) (1) The maximum approximate gain and its corresponding a are represented as: )(max),(~ a fS GfSL  =D maxarg f 3 A Fast Feature Selection Algorithm The inefficiency of the IFS algorithm is due to the following reasons.
Each component model takes the exponential form: a37a55a38a57a56 a51 a42a6a44a59a58a60a56 a61 a51a64a63a65a53a67a66 a53 a45a46a70 a71a16a72a21a73a75a74a77a76a79a78a81a80 a78a16a82a11a78 a38a83a44a59a58a60a56a84a61 a51a64a63a65a53a67a66 a53 a58a60a56 a51 a45a86a85 a87 a38a83a44a59a58a60a56a84a61 a51a64a63a65a53a67a66 a53 a45 a58 (2) where a87 a38a83a44a59a58a60a56 a61 a51a41a63a65a53a67a66 a53 a45 is a normalization term to ensure that a37a55a38a57a56 a51a42a6a44a88a58a60a56a62a61 a51a41a63a65a53a67a66 a53 a45 is a probability, a82a11a78 a38a83a44a59a58a60a56 a61 a51a64a63a65a53a67a66 a53 a58a60a56 a51 a45 is a feature function (often binary) and a80 a78 is the weight ofa82a21a78 . Given a set of features and a corpus of training data, there exist ef cient training algorithms (Darroch and Ratcliff, 1972; Berger et al. , 1996) to nd the optimal parameters a89 a80 a78a14a90 . The art of building a maximum entropy parser then reduces to choosing good features.
Our systems use both corpus-based and knowledge-based approaches: Maximum Entropy(ME) (Lau et al. , 1993; Berger et al. , 1996; Ratnaparkhi, 1998) is a corpus-based and supervised method based on linguistic features; ME is the core of a bootstrapping algorithm that we call re-training inspired  This paper has been partially supported by the Spanish Government (CICyT) under project number TIC-2003-7180 and the Valencia Government (OCyT) under project number CTIDIB-2002-151 by co-training (Blum and Mitchell, 1998); Relevant Domains (RD) (Montoyo et al. , 2003) is a resource built from WordNet Domains (Magnini and Cavaglia, 2000) that is used in an unsupervised method that assigns domain and sense labels; Specification Marks(SP) (Montoyo and Palomar, 2000) exploits the relations between synsets stored in WordNet (Miller et al. , 1993) and does not need any training corpora; Commutative Test (CT) (Nica et al. , 2003), based on the Sense Discriminators device derived from EWN (Vossen, 1998), disambiguates nouns inside their syntactic patterns, with the help of information extracted from raw corpus.
Figures 1 and 2 present best results in the learning experiments for the complete set of patterns used in the collocation approach, over two of our evaluation corpora.11 Type Positions Tags/Words Features Accuracy Precision Recall GIS 1 W 1254 0.97 0.96 0.98 IIS 1 T 136 0.95 0.96 0.94 NB 1 T 136 0.88 0.97 0.84 9 see Rish, 2001, Ratnaparkhi, 1997 and Berger et al, 1996 for a formal description of these algorithms.
It can be proven that the probability distribution p satisfying the above assumption is the one with the highest entropy, is unique and has the following expone ntial form (Berger et al. 1996): (1)  = = k j cajf jcZcap 1 ),( )( 1)|( a where Z(c) is a normalization factor, fj(a,c) are the values of k features of the pair (a,c) and correspond to the linguistic cues of c that are relevant to predict the outcome a. Features are extracted from the training data and define the constraints that the probabilistic model p must satisfy.
SEP/epsilon a/A# epsilon/# a/epsilon a/epsilon b/epsilon b/B UNK/epsilon c/C b/epsilon c/BC e/+E epsilon/+ d/epsilon d/epsilon epsilon/epsilon b/AB# b/A#B# e/+DE c/epsilon d/BCD e/+D+E Figure 1: Illustration of dictionary based segmentation finite state transducer 3.1 Bootstrapping In addition to the model based upon a dictionary of stems and words, we also experimented with models based upon character n-grams, similar to those used for Chinese segmentation (Sproat et al. , 1996).
Previous studies (Abney, 1997; Johnson et al. , 1999; Riezler et al. , 2000; Miyao et al. , 2003; Malouf and van Noord, 2004; Kaplan et al. , 2004; Miyao and Tsujii, 2005) defined a probabilistic model of unification-based grammars as a log-linear model or maximum entropy model (Berger et al. , 1996).
Previous studies (Abney, 1997; Johnson et al. , 1999; Riezler et al. , 2000; Malouf and van Noord, 2004; Kaplan et al. , 2004; Miyao and Tsujii, 2005) defined a probabilistic model of unification-based grammars including HPSG as a log-linear model or maximum entropy model (Berger et al. , 1996).
In the case of two orientation classes, cj,j is defined as: cj,j = braceleftbigg left, if j < j right, if j > j (4) Then, the reordering model has the form p(cj,j|fJ1,eI1,i,j) A well-founded framework for directly modeling the probability p(cj,j|fJ1,eI1,i,j) is maximum entropy (Berger et al. , 1996).
This was overcome by a probabilistic model which provides probabilities of discriminating a correct parse tree among candidates of parse trees in a log-linear model or maximum entropy model (Berger et al. , 1996) with many features for parse trees (Abney, 1997; Johnson et al. , 1999; Riezler et al. , 2000; Malouf and van Noord, 2004; Kaplan et al. , 2004; Miyao and Tsujii, 2005).
During the SRC stage, a Maximum entropy (Berger et al., 1996) classifier is used to predict the probabilities of a word in the sentence Language No-duplicated-roles Catalan arg0-agt, arg0-cau, arg1-pat, arg2-atr, arg2-loc Chinese A0, A1, A2, A3, A4, A5, Czech ACT, ADDR, CRIT, LOC, PAT, DIR3, COND English A0, A1, A2, A3, A4, A5, German A0, A1, A2, A3, A4, A5, Japanese DE, GA, TMP, WO Spanish arg0-agt, arg0-cau, arg1-pat, arg1-tem, arg2-atr, arg2-loc, arg2-null, arg4-des, argL-null, argMcau, argM-ext, argM-fin Table 1: No-duplicated-roles for different languages to be each semantic role.
Berger et al. 1996 presented a way of computing conditional maximum entropy models directly by modifying equation 6 as follows (now instead of w we will explicitly use (x, y) ): i ~Cx~) = ~ f~(~, y) * ~(~, y) ~ ~ .~(~, y) * ~(~) * pCy I ~) = p(xk) (9) x6X yEY xEX yEY where ~(x, y) is an empirical probability of a joint configuration (w) of certain instantiated factor I variables with certain instantiated behavior variables.
(1996) show that this model is a member of an exponential family with one parameter for each constraint, specifically a model of the form 1 ~ I~ (x,~) p(yl ) = E' in which z(x) = eZ, Y The parameters A1,  , An are Lagrange multipliers that impose the constraints corresponding to the chosen features fl, -,fnThe term Z(x) normalizes the probabilities by summing over all possible outcomes y. Berger et al.
, i.e.: (ll) Lj = ~ maz(zi(j, u)) i=I where xi(j,u)E Qi and max(xi(j,u)) is the highest score in the line of the matrix Qi which corresponds to the head word sense j. n is the number of modifiers of the head word h at the current tree level, and k i Lj = j~l Lj where k is the number of senses of the head word h. The reason why gj (I0) is calculated as a sum of the best scores (ll), rather than by using the traditional maximum likelihood estimate (Berger et al. , 1996)(Gah eta\[.
To determine the tree head-word we used a set of rules similar to that described by (Magerman, 1995)(Jelinek et al. , 1994) and also used by (Collins, 1996), which we modified in the following way:  The head of a prepositional phrase (PP-IN NP) was substituted by a function the name of which corresponds to the preposition, and its sole argument corresponds to the head of the noun phrase NP.
This allows us to compute the conditional probability as follows (Berger et al. , 1996): P(flh) = ~i~ '(h'I) (2) Z~(h) Z~(h) = ~I~I~ '(h'~) (a) ff i The maximum entropy estimation technique guarantees that for every feature gi, the expected value of gi according to the M.E. model will equal the empirical expectation of gi in the training corpus.
5.2 Results on the Newsblaster data We measured how well the models trained on DUC data perform with current news labeled using human 4http://newsblaster.cs.columbia.edu 5a20 (kappa) is a measure of inter-annotator agreement over and above what might be expected by pure chance (See Carletta (1996) for discussion of its use in NLP).a20a22a21a24a23 if there is perfect agreement between annotators anda20a25a21a27a26 if the annotators agree only as much as you would expect by chance.
For example, the coding manual for the Switchboard DAMSL dialogue act annotation scheme (Jurafsky, Shriberg, and Biasca 1997, page 2) states that kappa is used to assess labelling accuracy, and Di Eugenio and Glass (2004) relate reliability to the objectivity of decisions, whereas Carletta (1996) regards reliability as the degree to which we understand the judgments that annotators are asked to make.
He uses a specic reliability statistic, , for his measurements, but Carletta (1996) implicitly assumes kappa-like metrics are similar enough in practice for the rule of thumb to apply to them as well.A detailed discussion on the differences and similarities of these, and other, measures is provided by Krippendorff (2004); in this article we will use Cohens  (1960) to investigate the value of the 0.8 reliability cut-off for computational linguistics.
In Hirschberg and Nakatani (1996), average reliability (measured using the kappa coefficient discussed in Carletta \[1996\]) of segmentinitial labels among 3 coders on 9 monologues produced by the same speaker, labeled using text and speech, is.8 or above for both read and spontaneous speech; values of at least .8 are typically viewed as representing high reliability (see Section 3.2).