katebor/TableEval huggingface dataset:
Table 5: Human evaluation results on grammaticality (Gram), informativeness (Info), and relevance (Rel) of arguments. Our model with separate decoder and attention over keyphrases receives significantly better ratings in informativeness and relevance than seq2seq (one-way ANOVA, p < 0.005).

Gram

Info

Rel

System || RETRIEVAL

4.5 } 0.6

3.7 } 0.9

3.3 } 1.1

System || SEQ2SEQ

3.3 } 1.1

1.2 } 0.5

1.4 } 0.7

System || OUR MODEL

2.5 } 0.8

1.6 } 0.8

1.8 } 0.8

Table 5 shows that our model with separate decoder and attention over keyphrases produce significantly more informative and relevant arguments than seq2seq trained without evidence.8 . However, we also observe that human judges prefer the retrieved arguments over generation-based models, illustrating the gap between system arguments and human edited text.
Table 1: Parsing accuracy of the best existing grammar-free and -based models as well as our SHRG-based model. Results are copied from (Oepen et al., 2015; Peng et al., 2017a; Buys and Blunsom, 2017).

SDG

EDS

DMRS

Model || Data-driven || Grammar || NO

89.4

85.48

84.16

Model || ERG-based || Grammar || Unification

92.8

89.58

89.64

Model || SHRG-based || Grammar || Rewriting

-

90.39

89.51

Our parser achieves an accuracy of 90.35 for EDS and 89.51 for DMRS in terms of ELEMENTARY DEPENDENCY MATCH (EDM) which outperforms the best existing grammar-free model (Buys and Blunsom, 2017) by a significant margin (see Table 1). This marked result affirms the value of modeling the syntacto-semantic composition process for semantic parsing. On sentences that can be parsed by ERG-guided parsers, e.g. PET2 or ACE3 , significant accuracy gaps between ERG-guided parsers and data-driven parsers are repeatedly reported (see Table 1).
Table 2: Detection performance (trigger identification plus multi-class classification)

P (%)

R (%)

F (%)

Methods || MSEP-EMD

70.4

65

67.6

Methods || Cross-Event

68.8

68.9

68.8

Methods || Cross-Entity

72.9

64.3

68.3

Methods || Joint (Local+Global)

73.7

62.3

67.5

Methods || CNN

71.8

66.4

69

Methods || DM-CNN

75.6

63.6

69.1

Methods || NC-CNN

-

-

71.3

Methods || FB-RNN (GRU)

66.8

68

67.4

Methods || Bi-RNN (GRU)

66

73

69.3

Methods || ANNs (ACE+FN)

77.6

65.2

70.7

Methods || DM-CNN?(ACE+Wiki)

75.7

66

70.5

Methods || ANN-S2 (ACE+FN)

76.8

67.5

71.9

Methods || Hybrid: Bi-LSTM+CNN

84.6

64.9

73.4

Methods || SELF: Bi-LSTM+GAN

71.3

74.7

73

Table 2 shows the performance of multi-class classification. SELF achieves nearly the same F-score as Feng et al (2016)'s Hybrid, and outperforms the others. More importantly, SELF is the only one which obtains a performance higher than 70% for both precision and recall. Besides, by analyzing the experimental results, we have identified the following regularities:. ? Similar to the pattern classifiers that are based on hand-designed features, the CNN models enable higher precision to be obtained. However the recall is lower. ? The RNN models contribute to achieving a higher recall. However the precision is lower. ? Expansion of the training data set helps to increase the precision. Let us turn to the structurally more complicated models, SELF and Hybrid. SELF inherits the merits of the RNN models, classifying the events with higher recall. Besides, by the utilization of GAN, SELF has evolved from the traditional learning strategies, being capable of learning from GAN and getting rid of the mistakenly generated spurious features. So that it outperforms other RNNs, with improvements of no less than 4.5% precision and 1.7% recall.
Table 6: Results on MCNC task

Acc.(%)

Models || (Chambers and Jurafsky 2008)

30.92

Models || (Granroth-Wilding and Clark 2016)

43.28

Models || (Pichotta and Mooney 2016)

43.17

Models || (Wang et al. 2017)

46.67

Models || Our Results

48.83

Table 6 shows the comparisons of our results with the performance of several previous models, which were all trained with 1,500k event chains extracted from the NYT portion of the Gigaword corpus (Graff and Cieri 2003). Each event chain consists of a sequence of verbs sharing an actor within a news article. Except Chambers and Jurafsky (2008), other recent models utilized more and more sophisticated neural language models. Granroth-Wilding and Clark (2016) proposed a two layer neural network model that learns embeddings of event predicates and their arguments for predicting the next event. Pichotta and Mooney (2016) introduced a LSTM-based language model for event prediction. Wang et al.(2017) used dynamic memory as attention in LSTM for prediction. It is encouraging that by using event knowledge extracted from automatically identified narratives, we achieved the best event prediction performance, which is 2.2% higher than the best neural network model.
Table 5: Results for key-value knowledge retrieval (CN5Sel, 50 facts). Subj/Obj and integration. means: we attend over the fact subject (Key) and take the weighted fact object as value (Value). CN

NE || Dev

NE || Test

CN || Dev

CN || Test

Key/Value || Subj/Obj

76.65

71.52

71.85

67.64

Key/Value || Obj/Obj

76.7

71.28

71.25

67.48

Table 5 shows that for the NE dataset, the two strategies performequally well on the Dev set, whereas the Subj/Obj strategy works slightly better on the Test set. For Common Nouns, Subj/Obj is better.
Table 4: Human evaluation results on judging the homogeneity of latent actions in SMD.

Exp Agree

Worker κ

Match Rate

Model || DI-VAE

85.60%

0.52

71.30%

Model || DI-VST

93.30%

0.48

74.90%

Table 4 shows that both methods work well and DI-VST achieved better homogeneity than DI-VAE.
Table 4: Results on the human evaluation.

+2

+1

+0

kappa

Seq2Seq-att

29.32%

25.27%

45.41%

0.448

MMI-bidi

30.40%

24.85%

44.75%

0.471

MARM

20.11%

27.96%

51.93%

0.404

Seq2Seq+IDF

28.81%

23.87%

47.33%

0.418

SC-Seq2SeqNIWF s=1

42.47%

14.29%

43.24%

0.507

SC-Seq2SeqNIWF s=0.5

20.62%

40.16%

39.22%

0.451

SC-Seq2SeqNIWF s=0

14.34%

46.38%

39.28%

0.526

Table 4 shows the human evaluation results. We can observe that: (1) SC-Seq2SeqNIWF s=1 generates the most informative responses and interesting (labeled as g+2h) and the least general responses than all the baseline models. Meanwhile, SC-Seq2SeqNIWF s=0 generates the most general responses (labeled as g+1h); (2) MARM generates the most bad responses (labeled as g+0h), which indicates the drawbacks of the unknown latent responding mechanisms; (3) The kappa values of our models are all larger than 0.4, considered as gmoderate agreementh regarding quality of responses. The largest kappa value is achieved by SC-Seq2SeqNIWF s=0, which seems reasonable since it is easy to reach an agreement on general responses. Sign tests demonstrate the improvements of SC-Seq2SeqNIWF s=1 to the baseline models are statistically significant (p-value < 0.01). All the human judgement results again demonstrate the effectiveness of our controlled generation mechanism.
Table 2: Experiment results on ACE2005 and KBPEval2017. * indicates the result adapted from the original paper. For KBPEval2017, “Trigger Identification” corresponds to the “Span” metric and “Trigger Classification” corresponds to the “Type” metric reported in official evaluation.

ACE2005 || Trigger Identification || P

ACE2005 || Trigger Identification || R

ACE2005 || Trigger Identification || F1

ACE2005 || Trigger Classification || P

ACE2005 || Trigger Classification || R

ACE2005 || Trigger Classification || F1

KBPEval2017 || Trigger Identification || P

KBPEval2017 || Trigger Identification || R

KBPEval2017 || Trigger Identification || F1

KBPEval2017 || Trigger Classification || P

KBPEval2017 || Trigger Classification || R

KBPEval2017 || Trigger Classification || F1

Model || FBRNN(Char)

61.3

45.6

52.3

57.5

42.8

49.1

57.97

36.92

45.11

51.71

32.94

40.24

Model || DMCNN(Char)

60.1

61.6

60.9

57.1

58.5

57.8

53.67

49.92

51.73

50.03

46.53

48.22

Model || C-BiLSTM*

65.6

66.7

66.1

60.0

60.9

60.4

-

-

-

-

-

-

Model || FBRNN(Word)

64.1

63.7

63.9

59.9

59.6

59.7

65.10

46.86

54.50

60.05

43.22

50.27

Model || DMCNN(Word)

66.6

63.6

65.1

61.6

58.8

60.2

60.43

51.64

55.69

54.81

46.84

50.51

Model || HNN*

74.2

63.1

68.2

77.1

53.1

63.0

-

-

-

-

-

-

Model || Rich-C*

62.2

71.9

66.7

58.9

68.1

63.2

-

-

-

-

-

-

Model || KBP2017 Best*

-

-

-

-

-

-

67.76

45.92

54.74

62.69

42.48

50.64

Model || NPN(Concat)

76.5

59.8

67.1

72.8

56.9

63.9

64.58

50.31

56.56

59.14

46.07

51.80

Model || NPN(General)

71.5

63.2

67.1

67.3

59.6

63.2

63.67

51.32

56.83

57.78

46.58

51.57

Model || NPN(Task-specific)

64.8

73.8

69.0

60.9

69.3

64.8

64.32

53.16

58.21

57.63

47.63

52.15

Table 2 shows the results on ACE2005 and KBPEval2017. From this table, we can see that:. 1) NPNs steadily outperform all baselines significantly. Compared with baselines, NPN(Taskspecific) gains at least 1.6 (2.5%) and 1.5 (3.0%) F1-score improvements on trigger classification task on ACE2005 and KBPEval2017 respectively. 2) By exploiting compositional structures of triggers, our trigger nugget generator can effectively resolve the word-trigger mismatch problem. As shown in Table 2, NPN(Taskspecific) achieved significant F1-score improvements on trigger identification task on both datasets. It is notable that our method achieved a remarkable high recall on both datasets, which indicates that NPNs do detect a number of triggers which previous methods can not identify. 3) By summarizing information from both characters and words, the hybrid representation learning is effective for event detection. Comparing with corresponding characterbased methods3 , word-based methods achieved 2 to 3 F1-score improvements, which indicates that words can provide additional information for event detection. By combining character-level and word-level features, NPNs are able to perform character-based event detection meanwhile take word-level knowledge into consideration too.
Table 4: Results on the development, public test (Test-P) and hidden test (Test-H) sets. For each model, we report both accuracy and consistency.

Dev. || Acc.

Dev. || Con.

Test-P || Acc.

Test-P || Con.

Test-H || Acc.

Test-H || Con.

Model || MAJORITY

55.3

-

56.2

-

55.4

-

Model || MAXENT

68.0

-

67.7

-

67.8

-

Model || RULE

66.0

29.2

66.3

32.7

-

-

Model || SUP.

67.7

36.7

66.9

38.3

-

-

Model || SUP.+DISC

77.7

52.4

76.6

51.8

-

-

Model || WEAKSUP.

84.3

66.3

81.7

60.1

-

-

Model || W.+DISC

85.7

67.4

84.0

65.0

82.5

63.9

Table 4 describes our main results. Our weakly-supervised semantic parser with re-ranking (W.+DISC) obtains 84.0 accuracy and 65.0 consistency on the public test set and 82.5 accuracy and 63.9 on the hidden one, improving accuracy by 14.7 points compared to state-of-theart. The accuracy of the rule-based parser (RULE) is less than 2 points below MAXENT, showing that a semantic parsing approach is very suitable for this task. The supervised parser obtains better performance (especially in consistency), and with re-ranking reaches 76.6 accuracy, showing that generalizing from generated examples is better than memorizing manually-defined patterns. Our weakly-supervised parser significantly improves over SUP., reaching an accuracy of 81.7 before reranking, and 84.0 after re-ranking (on the public test set). Consistency results show an even crisper trend of improvement across the models.
Table 5: Human evaluation results for question generation. “Grammaticality”, “Making Sense” and “Answerability” are rated on a 1–5 scale (5 for the best, see the supplementary materials for a detailed rating scheme), “Average rank” is rated on a 1–3 scale (1 for the most preferred, ties are allowed.) Two-tailed t-test results are shown for our method compared to ContextNQG (stat. significance is indicated with ∗(p < 0.05), ∗∗(p < 0.01).)

Grammaticality

Making Sense

Answerability

Avg. rank

ContextNQG

3.793

3.836

3.892

1.768

CorefNQG

3.804*

3.847**

3.895*

1.762

Human

3.807

3.850

3.902

1.758

Table 5 shows the results of the human evaluation. Bold indicates top scores. We see that the original human questions are preferred over the two NQG ssystemsfoutputs, which is understandable given the examples in Figure 3. The humangenerated questions make more sense and correspond better with the provided answers, particularly when they require information in the preceding context. How exactly to capture the preceding context so as to ask better and more diverse questions is an interesting future direction for research. In terms of grammaticality, however, the neural models do quite well, achieving very close to human performance. In addition, we see that our method (CorefNQG) performs statistically significantly better across all metrics in comparison to the baseline model (ContextNQG), which has access to the entire preceding context in the passage.
Table 3: Human evaluations: Ranking of various systems. Rank 1st is best and rank 4th, worst. Numbers show the percentage of times a system gets ranked at a certain position.

1st

2nd

3rd

4th

Models || LEAD

0.15

0.17

0.47

0.21

Models || POINTERNET

0.16

0.05

0.31

0.48

Models || XNET

0.28

0.53

0.15

0.04

Models || HUMAN

0.41

0.25

0.07

0.27

The results of our human evaluation study are shown in Table 3. As one might imagine, HUMAN gets ranked 1st most of the time (41%). However, it is closely followed by XNET which ranked 1st 28% of the time. In comparison, POINTERNET and LEAD were mostly ranked at 3rd and 4th places. We also carried out pairwise comparisons between all models in Table 3 for their statistical significance using a one-way ANOVA with post-hoc Tukey HSD tests with (p < 0.01). It showed that XNET is significantly better than LEAD and POINTERNET, and it does not differ significantly from HUMAN. On the other hand, POINTERNET does not differ significantly from LEAD and it differs significantly from both XNET and HUMAN. The human evaluation results corroborates our empirical results in Table 1 and Table 2: XNET is better than LEAD and POINTERNET in producing informative and fluent summaries.
Table 6: Joint span detection and question generation results on the dense development set, using exact-match for both spans and questions.

P

R

F

Span + Local

37.8

43.7

40.6

Span + Seq. (tau = 0.5)

39.6

45.8

42.4

Table 6 shows precision and recall for joint span detection and question generation, using exact match for both. This metric is exceedingly hard, but it shows that almost 40% of predictions are exactly correct in both span and question.
Table 2: Performance of our approach on sentence ordering dataset from Barzilay and Lapata (2008).

Accidents

Earthquakes

- || Sequential CG

0.813

0.946

- || VLV-GM (2017)

0.77

0.931

- || HMM (2012)

0.822

0.938

- || HMM+Entity (2012)

0.842

0.911

- || HMM+Content (2012)

0.742

0.953

Discriminative approaches || DM (2017)

0.93

0.992

Discriminative approaches || Recursive (2014)

0.864

0.976

Discriminative approaches || Entity-Grid (2008)

0.904

0.872

Discriminative approaches || Graph (2013)

0.846

0.635

Table 2 shows performance of the method compared with other approaches for coherence prediction. We note that Sequential-CG performs competitively with the state-of-the-art for generative approaches for the task, while needing no other annotation. In comparison, the HMM based approaches use significant annotation and syntactic features. Sequential-CG also outperforms several discriminative approaches for the task.
Table 3: F1-score (%) of different passes from 1 to 5 on the test data sets. It shows that appropriate number of passes can boost the performance as well as avoid over-fitting of the model.

SE2

SE3

SE13

SE15

ALL

Pass || 1

71.6

70.3

67.0

72.5

70.3

Pass || 2

71.9

70.2

67.1

72.8

70.4

Pass || 3

72.2

70.5

67.2

72.6

70.6

Pass || 4

72.1

70.4

67.2

72.4

70.5

Pass || 5

72.0

70.4

67.1

71.5

70.3

Furthermore, Table 3 shows the effectiveness of multi-pass operation in the memory module. It shows that multiple passes operation performs better than one pass, though the improvement is not significant. The reason of this phenomenon is that for most target words, one main word sense accounts for the majority of their appearances. Therefore, in most circumstances, one-pass inference can lead to the correct word senses. Case studies in Table 2 show that the proposed multipass inference can help to recognize the infrequent senses like the third sense for word play. In Table 3, with the increasing number of passes, the F1-score increases. However, when the number of passes is larger than 3, the F1-score stops increasing or even decreases due to over-fitting. It shows that appropriate number of passes can boost the performance as well as avoid over-fitting of the model.
Table 2: Human evaluation results on a sample from Conceptual Captions.

GOOD (out of 3) || 1+

GOOD (out of 3) || 2+

GOOD (out of 3) || 3

Conceptual Captions

96.9%

90.3%

78.5%

The results are presented in Table 2 and show that, out of 3 annotations, over 90% of the captions receive a majority (2+) of GOOD judgments. This indicates that the Conceptual Captions pipeline, though involving extensive algorithmic processing, produces high-quality image captions.
Table 1: The proportion of images determined to be good representations of their corresponding word. In columns 2-5, we bucket the results by the word’s ground-truth concreteness, while column 6 shows the results over all words. The last row shows the number of words in each bucket of concreteness, and the number of words overall for each language.

Concreteness Ratings || 1-2

Concreteness Ratings || 2-3

Concreteness Ratings || 3-4

Concreteness Ratings || 4-5

Concreteness Ratings || Overall

Language || English

0.804

0.814

0.855

0.913

0.857

Language || French

0.622

0.653

0.706

0.828

0.721

Language || Indonesian

0.505

0.569

0.665

0.785

0.661

Language || Uzbek

0.568

0.53

0.594

0.683

0.601

Language || All

0.628

0.649

0.713

0.81

0.717

- || # Words

77

292

292

302

963

Table 1 shows the fraction of images that were judged to be good representations of the search word. It also demonstrates that as the concreteness of a word increases, the proportion of good images associated with that word increases as well. We further discuss the role of concreteness in Section 6.1. Overall, 85% of the English images, 72% of French, 66% of Indonesian, and 60% of Uzbek were judged to be good.
Table 3: Results on the triplets data

accuracy

Method || mean-vectors

0.65

Method || skip-thoughts-CS

0.615

Method || skip-thoughts-SICK

0.547

Method || triplets-sen

0.74

Table 3 indicates that our method, denoted by triplet-sen, clearly outperforms the other tested methods. Surprizingly, skip-thoughts-SICK is inferior to skip-thoughts-CS.
Table 3: Performance (AUCPR) of each noise reduction method; in bold are the best scores.

Original

Original+HITS

Original+LSA

Original+NMF

Original+Ensemble

System || CNN+ONE

0.18

0.183

0.173

0.178

0.181

System || CNN+ATT

0.234

0.235

0.235

0.233

0.236

System || PCNN+ONE

0.231

0.234

0.233

0.234

0.235

System || PCNN+ATT

0.248

0.253

0.25

0.252

0.255

Table 3 presents the performance of noise reduction methods. Recall that the K-means-based method achieves a high P@50 for the seed selection method. Our assumption is that each cluster may represent a set in which elements have similar semantic properties. However, we observed that as the number of relations is relatively high and there is no distinct definition between some relations in the distantly labeled data (e.g., the following three relations /location/country/capital, are quite /location/province/capital, and /location/us state/capital, we decided not to perform the K-means-based method for our noise reduction task. The performances of the HITS-based, LSA-based, and NMF-based noise reduction methods are presented in Table 3. We experimentally set the portion of retained data from the distantly labeled data to 90%, given that the performance can be affected if too many sentences are removed from the original data. We also perform experiments with an ensemble method that combines the HITS-based and LSA-based strategies to merge rankings from their outputs, with half of the triples coming from the LSA-based method and the other half from the HITS-based method. Table 3 indicates that our proposed methods improved the performance of all CNN and PCNN models. Our ensemble method achieved the best improvements for three out of four systems, except that the HITS-based method obtained the best score for CNN+ONE.
Table 1: Average alignment performance across images. MSFC provides the best recall and lowest AER, and modified k-means the best precision. In all cases, the alignment framework yields stronger results than either of the timing-based baselines.

MSFC || Precision

MSFC || Recall

MSFC || AER

Modified k-means || Precision

Modified k-means || Recall

Modified k-means || AER

Simultaneous

0.42

0.30

0.65

0.49

0.17

0.74

1-second delay

0.43

0.31

0.64

0.50

0.17

0.74

Alignment framework

0.43

0.50

0.54

0.56

0.31

0.60

Table 1 shows that the alignment framework performs better than either baseline. MSFC yields the highest recall and lowest AER with an absolute improvement of 0%, 19%, and 10% for precision, recall and AER, over the 1-second delay baseline. Modified k-means achieves higher precision with an absolute improvement of 6%, 14%, and 14%
over baseline.
Table 8: Results on math problems (§5.5).

UF1

LF1

Method || depccg

88.49

66.15

Method || + ELMo

89.32

70.74

Method || + Proposed

95.83

80.53

Results . Table 8 shows the F1 scores of depccg in the respective settings. Remarkably, we observe huge additive performance improvement. While, in terms of labeled F1, ELMo contributes about 4 points on top of the plain depccg, adding the new training set (converted from dependency trees) improves more than 10 points. Examining the resulting trees, we observe that the huge gain is primarily involved with expressions unique to math.
Table 4: Unimodal, Bimodal and Trimodal Results of HFFN. Here, L, A and V denotes language, acoustic and visual modalities, respectively.

CMU-MOSI || Acc

CMU-MOSI || F1

IEMOCAP || Acc

IEMOCAP || F1

Methods || L

78.59

78.52

81.46

81.54

Methods || A

48.14

48.3

38.08

38.17

Methods || V

56.97

57.48

34.52

29.15

Methods || L+A

78.06

78.29

80.38

80.6

Methods || L+V

79.39

79.38

80.05

80.26

Methods || A+V

55.17

55.76

55.17

55.79

Methods || L+A+V

80.19

80.34

82.37

82.42

4.3.2 Discussion on Modality Importance. To explore the underlying information of each modality, we carry out an experiment to compare the performance among unimodal, bimodal and trimodal models. For unimodal models, we can infer from Table 4 that language modality is the most predictive for emotion prediction, outperforming acoustic and visual modalities with significant margin. When coupled with acoustic and visual modalities, the trimodal HFFN performs best, whose result is 1% ~ 2% better than the language-HFFN, indicting that acoustic and visual modalities actually play auxiliary roles while language is dominant. However, in our model, when conducting outer product, all three modalities are treated equally, which is probably not the optimal choice. In the future, we aim to develop a fusion technique paying more attention to the language modality, while the other two modalities only serve as accessory sources of information.
Table 2: The overall mention detection results on the test set of OntoNotes. The F1 improvement is statistically significant under t-test with p < 0.05.

Prec.

Rec.

F1

Model || Our full model

89.6

82.2

85.7

Model || Lee et al. (2018)

86.2

83.7

84.9

Since mention detection is a subtask of coreference resolution, it is worthwhile to study the performance. Table 2 shows the mention detection results on the test set. Similar to coreference linking results, our model achieves higher precision and F1 score, which indicates that our model can significantly reduce false positive mentions while it can still ï¬nd a reasonable number of mentions.