katebor/TableEval huggingface dataset:
Table 4: Results for small labeled training data. Given the performance with the full dataset, we show BERT trained only with the AL data.

Acc

Training dataset || ACP (6K) || Encoder || BERT

0.876

Training dataset || +AL || Encoder || BERT

0.886

Training dataset || ACP (6K) || Encoder || BiGRU

0.83

Training dataset || +AL+CA+CO || Encoder || BiGRU

0.879

Contrary to our expectations, supervised models (ACP) outperformed semi-supervised models (ACP+AL+CA+CO). This suggests that the training set of 0.6 million events is sufficiently large for training the models. For comparison, we trained the models with a subset (6,000 events) of the ACP dataset. As the results shown in Table 4 demonstrate, our method is effective when labeled data are small.
Table 4: The performance (BLEU) of base CMLM with different amounts of mask-predict iterations (T ) on WMT’14 EN-DE, bucketed by target sequence length (N). Decoding with (cid:96) = 1 length candidates.

T = 4

T = 10

T = N

1 ? N < 10

21.8

22.4

22.4

10 ? N < 20

24.6

25.9

26

20 ? N < 30

24.9

26.7

27.1

30 ? N < 40

24.9

26.7

27.6

40 ? N

25

27.5

28.1

Table 4 shows that increasing the number of decoding iterations (T) appears to mainly improve the performance on longer sequences. Having said that, the performance differences across length buckets are not very large, and it seems that even 4 mask-predict iterations are enough to produce decent translations for long sequences (40 ? N).
Table 2: Human evaluation results. The values in parentheses are standard deviations.

Average

Score

Method || Proposed

4.11

(0.99)

Method || Baseline

2.18

(1.17)

Table 2 shows the human evaluation results, where we compute the average score by first computing the average score of the three evaluators for each of the top-K outputs, then selecting the maximum of the K scores as the score of the method. The average shown in the table is the average of such maximum scores. The proposed method failed to generate any anagram for 3 instances. We therefore set the score of the proposed method to 1.0 for them. We can see that the proposed method yielded significantly higher human evaluation scores. Table 3 shows example anagrams generated by the proposed method and the baseline method. While the baseline method seems to employ some heuristics for deciding the word order, the combinations of selected words tend to form meaningless sentences. On the other hand, the proposed method tends to generate readable sentences. The anagrams shown in the table are listed in decreasing order of sentence probability. We can find that sentences with high probability are not always highly rated by the human evaluations. However, the normalized discounted cumulative gain (NDCG) score for the proposed method was 0.86. It suggests that using sentence probability works well for finding natural anagrams.
Table 9: Results of CRFs on CON dataset.

QC3

TA

MEb

56.67

63.29

B-LSTMp

65.15

66.93

MEe

59.94

59.55

CRF (LC-NO)

62.20

60.30

CRF (LC-LC)

62.35

60.30

CRF (LC-LC1)

65.94

61.58

CRF (LC-FC1)

61.18

60.00

CRF (FC-FC)

64.54

61.64

As baselines, we use three models: (i) MEb, (ii) Ba MaxEnt using BOW representation; LSTMp, which is now trained on the concatenated set of sentences from MRDA and CON training sets; and (iii) MEe, a MaxEnt using sentence embeddings extracted from the B-LSTMp, i.e., the sentence embeddings are used as feature vectors. We experiment with the CRF variants in Table 1. The CRFs are trained on the CON training set using the sentence embeddings that are extracted by applying the B-LSTMp model, as was done with MEe. Table 9 shows our results. We notice that CRFs generally outperform MEs in accuracy. This indicates that there are conversational dependencies between the sentences in a conversation. When we compare between CRF variants, we notice that the model that does not consider any link across comments perform the worst; see CRF (LC-NO). A simple linear chain connection between sentences in their temporal order does not improve much (CRF (LC-LC)), which indicates that the widely used linear chain CRF (Lafferty et al., 2001) is not the most appropriate model for capturing conversational dependencies in these conversations. The CRF (LC-LC1) is one of the best performing models and perform significantly better than B-LSTMp. This model considers linear chain connections between sentences inside comments and only to the first comment. Note that both QC3 and TA are forum sites, where participants in a conversation interact mostly with the person who posts the first comment asking for some information. This is interesting that our model can capture this aspect. Another interesting observation is that when we change the above model to consider relations with every sentence in the first comment (CRF (LC-FC1)), this degrades the performance. This could be due to the fact that the information seeking person first explains her situation, and then asks for the information. Others tend to respond to the requested information rather than to her situation. The CRF (FC-FC) also yields as good results as CRF (LC-LC1).
Table 4: Parsing results (LAS) on test sets for different sentence boundaries.

WSJ

Switchboard

WSJ*

GOLD

90.22

84.99

88.71

MARMOT

89.81

78.93

83.37

NOSYNTAX

89.95

80.30†

83.61

JOINT

89.71

79.97†

85.66†‡

JOINT-REPARSED

89.93

80.61†‡∗

85.38†‡

Table 4 shows the results on the test sets. For WSJ, where sentence segmentation is almost trivial, we see only minor drops in LAS between GOLD and the systems that use predicted sentence boundaries. Among the systems that use predicted boundaries, no differences are significant. For WSJ∗ and Switchboard the picture is much different. Compared to GOLD, all systems show considerable drops in accuracy which asserts that errors from the sentence boundary detection task propagate to the parser and worsen the parser accuracy. On Switchboard the parsers yield significantly better results than MARMOT. The best result is obtained after reparsing and this is also significantly better than any other system. Although there is a slight drop in accuracy between NOSYNTAX and JOINT, this difference is not significant. The results on WSJ∗ show that not only does syntax help to improve sentence segmentation, it does so to a degree that parsing results deteriorate when simpler sentence boundary detectors are used. Here, both JOINT and JOINT-REPARSED obtain significantly better parsing accuracies than the systems that do not have access to syntax during sentence boundary prediction. Although JOINT-REPARSED performs a bit worse, the difference compared to JOINT is not significant.
Table 4: Best F1 measures on the Accurat evaluation sets

EN-DE

EN-EL

EN-ET

EN-LT

EN-LV

EN-RO

EN-SL

SYSTEM || LEXACC || TEST SETS || 1:1

96.0

89.5

88.9

93.1

95.0

99.4

88.5

SYSTEM || STACC || TEST SETS || 1:1

96.7

88.0

92.0

96.1

96.6

98.8

89.5

SYSTEM || LEXACC || TEST SETS || 2:1

83.4

83.2

73.9

81.2

83.8

95.3

81.6

SYSTEM || STACC || TEST SETS || 2:1

89.2

83.2

79.9

86.9

88.2

95.3

82.3

SYSTEM || LEXACC || TEST SETS || 100:1

16.6

22.7

34.2

45.1

45.1

70.4

24.9

SYSTEM || STACC || TEST SETS || 100:1

33.7

37.3

42.5

56.0

56.2

75.7

35.3

Table 4 presents the results on the Accurat test sets for LEXACC and STACC using their respective optimal similarity thresholds. On the 21 test sets, the two systems were tied on two occasions, with STACC obtaining better results in 89.5% of the remaining cases. On the noisiest datasets, STACC was consistently and markedly better across language pairs.
Table 4: Performance of the basic ED model. ANN uses pre-trained word embeddings while ANNRandom uses randomly initialized embeddings.

Pre

Rec

F1

Methods || Nguyen’s CNN (2015)

71.8

66.4

69.0

Methods || Chen’s DMCNN (2015)

75.6

63.6

69.1

Methods || Liu’s Approach (2016)

75.3

64.4

69.4

Methods || ANN (ours)

79.5

60.7

68.8

Methods || ANN-Random (ours)

81.0

49.5

61.5

Table 4 shows the experimental results, from which we can see that the three-layer ANN model is surprisingly effective for event detection, which even yields competitive results compared with Nguyen’s CNN and Chen’s DMCNN. We believe the reason is that, compared with CNN and DMCNN, ANN focuses on capturing lexical features which have been proved much more important than sentence features for the ED task by (Chen et al., 2015).,. Moreover, our basic model achieves much higher precision than state-of-the-art approaches (79.5% vs. 75.6%). We also investigate the performance of the basic ED model without pre-trained word embeddings (denoted by ANN-Random). The result shows that randomly initialized word embeddings decrease the F1 score by 7.3 (61.5 vs. 68.8).
Table 5: Results of manual evaluations.

Precision (%)

Baselines || Methods || ANN

77.5

Baselines || Methods || SF

72.0

Baselines || Methods || RF

71.0

Baselines || Methods || SL

79.5

- || Methods || PSL-based Approach

81.0

Table 5 shows the results of manual evaluations. Through the comparison of ANN and SF, we can see that the application of H1 caused a loss of 5.5 point. It happens mainly because the performance of SF is very sensitive to the wrongly mapped frames. That is, if a frame is mismapped, then all sentences under it would be mislabeled as events. Thus, even a single mismapped frame could significantly hurt the performance. This result also proves that H1 is inappropriate to be used as a hard constraint. As H2 is only an extension of H1, RF performs similarly with SF. Moreover, SL obtains a gain of 2.0% improvement compared with ANN, which demonstrates that the ”same LU” hypothesis is very useful. Finally, with all the hypotheses, the PSL-based approach achieves the best performance, which demonstrates that our hypotheses are useful and it is an effective way to jointly utilize them as soft constraints through PSL for event detection in FN.
Table 2: Comparison of different methods on English event detection.

Trigger Identification || P

Trigger Identification || R

Trigger Identification || F

Trigger Classification || P

Trigger Classification || R

Trigger Classification || F

Model || MaxEnt

76.2

60.5

67.4

74.5

59.1

65.9

Model || Cross-Event

N/A

N/A

N/A

68.7

68.9

68.8

Model || Cross-Entity

N/A

N/A

N/A

72.9

64.3

68.3

Model || Joint Model

76.9

65.0

70.4

73.7

62.3

67.5

Model || PR

N/A

N/A

N/A

68.9

72.0

70.4

Model || CNN

80.4

67.7

73.5

75.6

63.6

69.1

Model || RNN

73.2

63.5

67.4

67.3

59.9

64.2

Model || LSTM

78.6

67.4

72.6

74.5

60.7

66.9

Model || Bi-LSTM

80.1

69.4

74.3

81.6

62.3

70.6

Model || HNN

80.8

71.5

75.9

84.6

64.9

73.4

Table 2 shows the overall performance of all methods on the ACE2005 English corpus. We can see that our approach significantly outperforms all previous methods. The better performance of HNN can be further explained by the following reasons: (1) Compared with feature based methods, such as MaxEnt, Cross-Event, Cross-Entity, and Joint Model, neural network based methods (including CNN, Bi-LSTM, HNN) performs better because they can make better use of word semantic information and avoid the errors propagated from NLP tools which may hinder the performance for event detection. (2) Moreover, Bi-LSTM can capture both preceding and following sequence information, which is much richer than dependency path. For example, in S2, the semantic of “court” can be delivered to release by a forward sequence in our approach. It is an important clue which can help to predict “release” as a trigger for “ReleaseParole”. For explicit feature based methods, they can not establish a relation between “court” and “release”, because they belong to different clauses, and there is no direct dependency path between them. While in our approach, the semantics of “court” can be delivered to release by a forward sequence. (3) Cross-entity system achieves higher recall because it uses not only sentence-level information but also document-level information. It utilizes event concordance to predict a local trigger’s event type based on cross-sentence inference. For example, an “attack” event is more likely to occur with “killed” or “die” event rather than “marry” event. However, this method heavily relies on lexical and syntactic features, thus the precision is lower than neural network based methods. (4) RNN and LSTM perform slightly worse than Bi-LSTM. An obvious reason is that RNN and LSTM only consider the preceding sequence information of the trigger, which may miss some important following clues. Considering S1 again, when extracting the trigger “releases”, both models will miss the following sequence “20 million euros to Iraq”. This may seriously hinder the performance of RNN and LSTM for event detection.
Table 4: Results on Spanish event detection.

Trigger Identification || P

Trigger Identification || R

Trigger Identification || F

Trigger Classification || P

Trigger Classification || R

Trigger Classification || F

Model || LSTM

62.2

52.9

57.2

56.9

32.6

41.6

Model || Bi-LSTM

76.2

63.1

68.7

61.5

42.2

50.1

Model || HNN

81.4

65.2

71.6

66.3

47.8

55.5

Table 4 presents the performance of our method on the Spanish ERE corpus. The results show that HNN approach performed better than LSTM and Bi-LSTM. It indicates that our proposed model could achieve the best performance in multiple languages than other neural network methods. We did not compare our system with other systems (Tanev et al., 2009), because they reported the results on a non-standard data set .
Table 5: SVM classification results across linguistic features (LF, bigrams here (Mukherjee et al., 2013b)), behavioral features (BF: RL, RD, MCS (Mukherjee et al., 2013b)); the SVM classification results by the intuitive method that finding the most similar existing review by edit distance ratio and take the found reviewers’ behavioral features as approximation (BF EditSim+LF), and results by the intuitive method that finding the most similar existing review by averaged pre-trained word embeddings (using Word2Vec) (BF W2Vsim+W2V); and the SVM classification results across the learnt review embeddings (RE), the learnt review’s rating embeddings (RRE), the learnt product’s average rating embeddings (PRE) by our model. Improvements of our model are statistically significant with p<0.005 based on paired t-test.

P || Hotel

P || Restaurant

R || Hotel

R || Restaurant

F1 || Hotel

F1 || Restaurant

A || Hotel

A || Restaurant

Features || LF

54.5

53.8

71.1

80.8

61.7

64.6

55.9

55.8

Features || LF+BF

63.4

58.1

52.6

61.2

57.5

59.6

61.1

58.5

Features || BF EditSim+LF

55.3

53.9

69.7

82.82

61.6

65.1

56.6

56.0

Features || BF W2Vsim+W2V

58.4

56.3

65.9

73.4

61.9

63.7

59.5

58.2

Features || Ours RE

62.1

58.4

68.3

75.1

65.1

65.7

63.3

60.8

Features || Ours RE+RRE+PRE

63.6

59.0

71.2

78.8

67.2

67.5

65.3

62.0

To illustrate the effectiveness of our model, we conduct experiments on the public datasets, and make comparison with the most effective traditional linguistic features, e.g., bigrams, and the three practicable traditional behavioral features (RL, RD, MCS (Mukherjee et al., 2013b)) referred in Section 3.2. The results are shown in Table 5. For our model, we set the dimension of embeddings to 100, the number of CNN filters to 100, Î¸ to 0.1, Z to 2. The hyper-parameters are tuned by grid search on the development dataset. The product and reviewer embeddings are randomly intialized from a uniform distribution (Socher et al., 2013). The word embeddings are initialized with 100-dimensions vectors pre-trained by the CBOW model (Word2Vec) (Mikolov et al., 2013). As Table 5 showed, our model observably performs better in detecting review spam for the cold-start task in both hotel and restaurant domains.
Table 2: The performance of predicting present keyphrases of various models on five benchmark datasets

Inspec || F1@5

Inspec || F1@10

Krapivin || F1@5

Krapivin || F1@10

NUS || F1@5

NUS || F1@10

SemEval || F1@5

SemEval || F1@10

KP20k || F1@5

KP20k || F1@10

Method || Tf-Idf

0.221

0.313

0.129

0.160

0.136

0.184

0.128

0.194

0.102

0.126

Method || TextRank

0.223

0.281

0.189

0.162

0.195

0.196

0.176

0.187

0.175

0.147

Method || SingleRank

0.214

0.306

0.189

0.162

0.140

0.173

0.135

0.176

0.096

0.119

Method || ExpandRank

0.210

0.304

0.081

0.126

0.132

0.164

0.139

0.170

N/A

N/A

Method || Maui

0.040

0.042

0.249

0.216

0.249

0.268

0.044

0.039

0.270

0.230

Method || KEA

0.098

0.126

0.110

0.152

0.069

0.084

0.025

0.026

0.171

0.154

Method || RNN

0.085

0.064

0.135

0.088

0.169

0.127

0.157

0.124

0.179

0.189

Method || CopyRNN

0.278

0.342

0.311

0.266

0.334

0.326

0.293

0.304

0.333

0.262

Table 2 provides the performances of the six baseline models, as well as our proposed models (i.e., RNN and CopyRNN). For each method, the table lists its F-measure at top 5 and top 10 predictions on the five datasets. The best scores are highlighted in bold and the underlines indicate the second best performances. The results show that the four unsupervised models (Tf-idf, TextRank, SingleRank and ExpandRank) have a robust performance across different datasets. The ExpandRank fails to return any result on the KP20k dataset, due to its high time complexity. The measures on NUS and SemEval here are higher than the ones reported in (Hasan and Ng, 2010) and (Kim et al., 2010), probably because we utilized the paper abstract instead of the full text for training, which may filter out some noisy information. The performance of the two supervised models (i.e., Maui and KEA) were unstable on some datasets, but Maui achieved the best performances on three datasets among all the baseline models. As for our proposed keyphrase prediction approaches, the RNN model with the attention mechanism did not perform as well as we expected. It might be because the RNN model is only concerned with finding the hidden semantics behind the text, which may tend to generate keyphrases or words that are too general and may not necessarily refer to the source text. In addition, we observe that 2.5% (70,891/2,780,316) of keyphrases in our dataset contain out-of-vocabulary words, which the RNN model is not able to recall, since the RNN model can only generate results with the 50,000 words in vocabulary. This indicates that a pure generative model may not fit the extraction task, and we need to further link back to the language usage within the source text. The CopyRNN model, by considering more contextual information, significantly outperforms not only the RNN model but also all baselines, exceeding the best baselines by more than 20% on average.
Table 4: Comparison of our proposed methods on Spanish-French and German-French translation tasks from the Europarl corpus. English is treated as the pivot language.

Es→ Fr || dev

Es→ Fr || test

De→ Fr || dev

De→ Fr || test

Method || sent-greedy

31.00

31.05

22.34

21.88

Method || sent-beam

31.57

31.64

24.95

24.39

Method || word-greedy

31.37

31.92

24.72

25.15

Method || word-beam

30.81

31.21

24.64

24.19

Method || word-sampling

33.65

33.86

26.99

27.03

Table 4 shows BLEU scores on the Europarl corpus of our proposed methods. For sentence-level approaches, the sent-beam method outperforms the sent-greedy method by +0.59 BLEU points over Spanish-French translation and +2.51 BLEU points over German-French translation on the test set. For word-level experiments, we observe that the word-sampling method performs much better than the other two methods: +1.94 BLEU points on Spanish-French translation and +1.88 BLEU points on German-French translation over the word-greedy method; +2.65 BLEU points on Spanish-French translation and +2.84 BLEU points on German-French translation over the word-beam method.
Table 1: Results on the CELEX dataset

13SIA

2PIE

2PKE

rP

Avg.

MED (Kann and Schutze 2016a)

83.9

95

87.6

84

87.62

NWFST (Rastogi et al. 2016)

86.8

94.8

87.9

81.1

87.65

LAT (Dreyer et al. 2008)

87.5

93.4

87.4

84.9

88.3

Soft

83.1

93.8

88

83.2

87

Hard

85.8

95.1

89.5

87.2

89.44

On the low resource setting (CELEX), our hard attention model significantly outperforms both the recent neural models of Kann and Schutze (2016a) (MED) and Rastogi et al. (2016) (NWFST) and the morphologically aware latent variable model of Dreyer et al. (2008) (LAT), as detailed in Table 1. In addition, it significantly outperforms our implementation of the soft attention model (Soft). It is also, to our knowledge, the first model that surpassed in overall accuracy the latent variable model on this dataset. We attribute our advantage over the soft attention models to the ability of the hard attention control mechanism to harness the monotonic alignments found in the data. The advantage over the FST models may be explained by our conditioning on the entire output history which is not available in those models.
Table 1: Performance of the reasoning module on manually formalized pre-university problems

Succeeded || Success %

Succeeded || Avg. Time

Failed || Timeout

Failed || Other

Dataset || DEV

75.3% (131/174)

10.5s

16.70%

8.10%

Dataset || TEST

78.2% (172/220)

16.2s

15.00%

6.80%

The reasoning module (i.e., the formula rewriting and the deduction with CAS and ATP) of the system has been extensively tested on a large collection of manually formalized pre-university math problems that includes more than 1,500 problems. It solves 70% of the them in the time limit of 10 minutes per problem. Table 1 shows the rate of successfully solved problems in the manually formalized version of the benchmark problems used in the current paper.
Table 1: Unlabeled dependency accuracy results with the 5x5 models and test sets. ∇ shows the slope of deterioration in parser performance.

Baseline

E05

E10

E15

E20

0%

91.43

91.12

90.87

90.61

90.29

5%

89.99

90

89.87

89.72

89.48

10%

87.84

87.99

88.07

88.14

88.04

15%

85.64

86.18

86.54

86.75

86.82

20%

84.12

84.78

85.28

85.5

85.76

Table 1 shows the result of unlabeled dependency accuracy (UAS). On the error-free test set (0%), the baseline (EF pipeline) outperforms other EREF models; the accuracy is lower when the parser is trained on noisier data. The difference among the models becomes small when the test set has 10% error injection rate. As the rate increases further, the trend of parser accuracy reverses. When the test set has 15% or higher noise, the E20 is the most accurate parser.
Table 3: Performance comparison. Examples/s are normalized by the number of GPUs used in the training job. FLOPs are computed assuming that source and target sequence length are both 50.

Examples/s

FLOPs

Params

Model || ConvS2S

80

15.7B

263.4M

Model || Trans. Base

160

6.2B

93.3M

Model || Trans. Big

50

31.2B

375.4M

Model || RNMT+

30

28.1B

378.9M

Table 3 summarizes training performance and model statistics. The Transformer Base model is the fastest model in terms of training speed. RNMT+ is slower to train than the Transformer
Big model on a per-GPU basis. However, sinc the RNMT+ model is quite stable, we were able to offset the lower per-GPU throughput with higher
concurrency by increasing the number of model replicas, and hence the overall time to convergence was not slowed down much. We also computed the number of floating point operations (FLOPs) in the modelfs forward path as well as the number of total parameters for all architectures (cf. Table 3). RNMT+ requires fewer FLOPs than the Transformer Big model, even though both models have a comparable number of parameters.