katebor/TableEval huggingface dataset:
Table 2: Narrative reconstruction results in NYT dataset and post grouping results in Wikipedia conversation dataset.

AMI

ARI

NYT || LDA + DBSCAN

0.0627

0.0117

NYT || HDP + DBSCAN

0.0260

0.0203

NYT || HDHP

0.1768

0.0746

NYT || HD-GMHP (100D)

0.2479

0.1416

Wiki || W2V + DBSCAN

0.0055

0.0001

Wiki || HDHP

0.4240

0.3512

Wiki || HD-GMHP (100D)

0.5848

0.3834

7.5 Results. Table 2 shows the clustering accuracy of our method and the baseline methods in real world datasets. We average the results with five runs for each model. The highest value for each metric is indicated with boldface. From the results, we establish that our model outperforms the baseline methods in both the NYT narrative reconstruction task and the Wikipedia thread reconstruction task. For the NYT, to see the accuracy of our model in more detail, we compute and show the F-scores for the top ten most frequent labels and the micro and macro averages in table 3. To compute the F-score between the true labels and the recovered cluster labels, we select the cluster with the highest F-score as the corresponding cluster. From the results, we establish that our model performs better than the baseline model, HDHP.
Table 2: Experimental results on Sem2016 with two training settings TRAIN and TRAIN-ALL.

Sem2016 (TRAIN) || 3-way || Acc

Sem2016 (TRAIN) || 3-way || F1

Sem2016 (TRAIN) || Binary || Acc

Sem2016 (TRAIN) || Binary || F1

Sem2016 (TRAIN-ALL) || 3-way || Acc

Sem2016 (TRAIN-ALL) || 3-way || F1

Sem2016 (TRAIN-ALL) || Binary || Acc

Sem2016 (TRAIN-ALL) || Binary || F1

- || AVG || Acc

- || AVG || F1

NBOW-MLP

54.31

52.90

79.69

77.33

61.09

55.91

84.90

82.01

70.00

67.04

CNN

54.67

52.17

79.79

75.28

62.68

57.71

84.10

81.31

70.31

66.62

BiLSTM

55.57

52.33

81.90

77.12

63.26

60.31

85.89

84.14

71.66

68.50

AT-BiLSTM

56.95

54.53

80.09

73.93

64.20

61.64

86.77

83.67

72.00

68.44

Lexicon RNN

51.02

50.45

81.72

79.00

61.41

60.50

86.68

83.82

70.21

68.44

AGLR

59.01

56.67

82.79

80.10

66.62

64.36

87.16

84.98

73.90

71.53

4.2 Experimental Results. Table 1 and Table 2 report the results of our experiments. The results on TRAIN-ALL are higher than TRAIN for SemEval16 in lieu of the larger dataset. Firstly, we observe that our proposed AGLR outperforms all neural baselines on 3-way classification. The overall performance of AGLR achieves state-of-the-art performance. On average, AGLR outperforms Lexicon RNN and AT-BiLSTM by 1% âˆ’ 3% in terms of F1 score. We also observe that AGLR always improves AT-BiLSTM which ascertains the effectiveness of learning auxiliary lexicon embeddings. The key idea here is that the auxiliary lexicon embeddings provide a different view of the sentence which supports the network in making predictions. We also observe that Lexicon RNN does not handle 3-way classification well. Even though it has achieved good performance on binary classification, the performance on 3-way classification is lackluster (the performance of AGLR outperforms Lexicon RNN by up to 8% on SemEval16 TRAIN). This could also be attributed to the MSE based loss function. Conversely, by learning an auxiliary embedding (instead of a scalar score), our model becomes more flexible at the final layer and can be adapted to using a k softmax function. Finally, we observe that BiLSTM and AT-BiLSTM outperform Lexicon RNN on average with Lexicon RNN being slightly better on binary classification.
Table 4: BLEU scores on En-Jp test set. ”0” represents the normal translation results, and ”1” represents the teacher-forcing translation results.

left-to-right || 0

left-to-right || 1

right-to-left || 0

right-to-left || 1

left

7.90

9.91

7.45

8.95

right

8.70

11.52

9.24

10.59

Table 4 shows the BLEU score on the En-Jp test set. It can be observed that regardless of decoding direction (i.e., from left-to-right or from right-toleft) and with or without teacher forcing, the accuracy of the right half is always higher than that in the left half. This observation on Japanese is opposite to English, German and Chinese in Section3.1and3.2, and motivates us to investigate the differences between these languages.
Table 3: Translation results on the IKEA dataset

English → German || BLEU

English → German || METEOR

English → French || BLEU

English → French || METEOR

Method || LIUMCVC-Multi

59.9 ± 1.9

63.8 ± 0.4

58.4 ± 1.6

64.6 ± 1.8

Method || Text-Only NMT

61.9 ± 0.9

65.6 ± 0.9

65.2 ± 0.7

69.0 ± 0.2

Method || VAG-NMT

63.5 ± 1.2

65.7 ± 0.1

65.8 ± 1.2

68.9 ± 1.4

We then evaluate all models on the IKEA dataset. Table 3 shows the results. Our VAG-NMT has a higher value in BLEU and a comparable value in METEOR compared to the Text-only NMT baseline. Our VAG-NMT outperforms LIUMCVC's multimodal system by a large margin, which shows that the LIUMCVC's multimodal's good performance on Multi30K does not generalize to this real-world product dataset. The good performance may come from the visual attention mechanism that learns to focus on text segments that are related to the images. Such attention therefore teaches the decoder to apply the visual context to translate those words. This learned attention is especially useful for datasets with long sentences that have irrelevant text information with respect to the image.
Table 2: Performance of our approaches to QA-style sentiment classification in all domains.

Beauty || Macro-F1

Beauty || Accuracy

Shoe || Macro-F1

Shoe || Accuracy

Electronic || Macro-F1

Electronic || Accuracy

SVM

0.362

0.684

0.381

0.718

0.435

0.691

LSTM

0.499

0.712

0.520

0.754

0.562

0.715

Bi-LSTM

0.527

0.719

0.531

0.759

0.574

0.723

Bidirectional-Match

0.526

0.747

0.557

0.796

0.582

0.741

AtoQ-Match

0.543

0.745

0.602

0.792

0.567

0.754

QtoA-Match

0.573

0.751

0.647

0.807

0.608

0.752

Bidirectional-Match QA

0.583

0.760

0.666

0.815

0.617

0.764

HMN

0.598

0.776

0.683

0.827

0.640

0.779

Table 2 summarizes the experimental results of all the approaches above, and we can find that:. (1) All LSTM-based approaches are superior to SVM, indicating the effectiveness of neural network for this task. (2) The proposed approaches, with novel QA contextual representation, outperform the other baseline approaches. (3) When only employing QA bidirectional matching mechanism, Bidirectional-Match QA, which takes the sentence segmentation strategy, consistently outperforms Bidirectional-Match (without sentence segmentation) in all domains. It confirms our hypothesis that sentence segmentation helps to extract the sentiment matching information between the question and answer. (4) When comparing to QA unidirectional matching mechanism, Bidirectional-Match QA, which employs QA bidirectional matching mechanism, performs better than AtoQMatch and QtoA-Match. It confirms our hypothesis that both the question and answer information contribute to sentiment polarity of the QA text pair. (5) Impressively, the proposed approach HMN significantly outperforms all the other approaches in all domains (p-value<0.05 via t-test). It verifies the advantages of both QA bidirectional matching mechanism and selfmatching attention mechanism for this task.
Table 2: Performance on SNLI and transfer tasks of various sentence encoders. Test accuracies on SNLI, micro and macro averages of accuracies of dev set on transfer tasks are chosen as evaluation metrics.

SNLI

Micro

Macro

Model || BiLSTM-max

84.5

85.2

83.7

Model || AdaSent

83.4

82.0

80.9

Model || TBCNN

82.1

81.1

79.3

Model || DiSAN

85.6

84.7

83.4

Model || PSAN

86.1

85.7

84.5

Experiment results of our model and four baselines are shown in Table 2. Micro and macro accuracies are two composite indicators for evaluating transfer performance of tasks whose metric is classification accuracy. Macro accuracy is the proportion of true results in the population of instances from all tasks. Micro accuracy is the arithmetic mean of dev accuracies for each task. PSAN achieves the state-of-the-art performance with considerably fewer parameters, outperforming a RNN-based model, a CNN-based model, a fully attention-based model and a model that utilize syntactic information. Especially when compared with previous best model BiLSTM-Max, PSAN can outperform their model with only 5% of their parameter numbers, demonstrating the effectiveness of our model at extracting semantically important information from a sentence.
Table 3: Transfer test results for our model and various baselines. Classification accuracy is chosen as evaluation metric for datasets including MR, CR, SUBJ, MPQA, SST, TREC and SICK-E.

MR

CR

SUBJ

MPQA

SST

TREC

Model || BiLSTM-max

79.9

84.6

92.1

89.8

83.3

88.7

Model || AdaSent

77.0

82.0

89.9

87.2

82.3

85.6

Model || TBCNN

75.4

81.6

89.1

85.9

79.4

83.7

Model || DiSAN

79.7

84.1

92.2

89.5

82.9

88.3

Model || PSAN

80.0

84.2

91.9

89.9

83.8

89.1

In Table 3, we compare our model with baseline sentence encoders in each transfer task. PSAN can consistently outperform the baselines in almost every task considered.
Table 2: Results of independent training for intent detection in terms of error rate.

Error(%)

Methods || Recursive NN (Guo et al. 2014)

4.60

Methods || Boosting (Tur et al. 2010)

4.38

Methods || Boosting + Simplified sentences(Tur et al. 2011)

3.02

Methods || Attention Enc-Dec (Liu and Lane 2016a)

2.02

Methods || Our Model

2.69

4.4 Independent Learning. The results of separate training for slot filling and intent detection are reported in Table 1 and Table 2 respectively. On the independent slot filling task, we fixed the intent information as the ground truth labels in the dataset. But on the independent intent detection task, there is no interaction with slot labels. Table 2 compares the performance of our proposed model to previously reported results on intent detection task. Our model gives good performance in terms of classification error rate, but not as good as Attention Encoder-Decoder (with aligned inputs) method (Liu and Lane, 2016a). As their published state-of-the-art result described in (Liu and Lane, 2016a), their attention-based model is based on word-level embeddings. While in our model, we introduce character-level embeddings to improve the performance of joint learning. But independent learning for intent classification aims at capturing the global information of an utterance, not caring much about the details of specific word. The character-level embeddings introduced in our model bring very little hurt to independent learning of intent detection, as a trade-off in performance between both criterion.
Table 3: Results of joint training for slot filling and intent detection.

F1

Error(%)

Methods || Recursive NN (Guo et al.2014)

93.22

4.60

Methods || Recursive NN+Viterbi(Guo et al. 2014)

93.96

4.60

Methods || Attention Enc-Dec (Liu and Lane 2016a)

95.87

1.57

Methods || Attention BiRNN (Liu and Lane 2016a)

95.98

1.79

Methods || Our Model

96.52

1.23

Table 3 compares our joint model with reported results from previous works. We can see that our model achieves state-of-the-art results and outperforms previous best result by 0.54% in terms of F1-score on slot filling, and by 0.34% in terms of error rate on intent detection. This improvement is statistically significant. Besides, the joint learning achieves better results than separate learning. It can be interpreted that the two tasks are highly correlated and boost the performance each other. The slot filling task enables the model to learn more meaningful representations which give more supervisory signals for the learning of shared parameters. Similarly, intent is also useful to determine the slot label.
Table 3: Performances on Twitter corpus.

Twitter || ROUGE-1

Twitter || ROUGE-2

Twitter || BLEU

Twitter || METEOR

Models || Seq2Seq

30.43

14.61

30.54

12.80

Models || Residual LSTM

32.50

16.86

33.90

13.65

Models || Pointer-generator

38.31

21.22

40.37

17.62

Models || RL-ROUGE

40.16

22.99

42.73

18.89

Models || RbM-SL (ours)

41.87

24.23

44.67

19.97

Models || RbM-IRL (ours)

42.15

24.73

45.74

20.18

Table 3 shows the performances on the Twitter corpus. Our models again outperform the baselines in terms of all the evaluation measures. Note that RbM-IRL performs better than RbM-SL in this case. The reason might be that the evaluator of RbM-SL might not be effectively trained with the relatively small dataset, while RbM-IRL can leverage its advantage in learning of the evaluator with less data.
Table 2: User satisfaction test results. In total, we use the strategy mentioned in Section 4.2 to evaluate 400 randomly selected source news pages. Overall denotes the average score on these 400 samples.

AnnotatorA

AnnotatorB

Overall

Format || Text

3.67

3.75

3.71

Format || Pictorial

4.14

4.20

4.17

Table 2 shows our results for user satisfaction test. User ratings of pictorial summaries are 12.4% higher than text summaries. It shows that users prefer this way of presenting information. It also confirms our motivation for MSMO.
Table 3: Multi-task results of MCapsNet. In column Avg.△, we use BiLSTM as baseline and calculate the average improvements over it.

Dataset || MR

Dataset || SST-1

Dataset || SST-2

Dataset || Subj

Dataset || TREC

Dataset || AG’s

Dataset || Avg.△

BiLSTM

79.3

46.2

83.2

90.5

89.6

88.2

+0

MT-GRNN

-

49.2

87.7

89.3

93.8

-

+2.6

MT-RNN

-

49.6

87.9

94.1

91.8

-

+3.5

MT-DNN

82.1

48.1

87.3

93.9

92.2

91.8

+2.9

MT-CNN

81.6

49.0

86.9

93.6

91.8

91.9

+3.0

CapsNet-1

81.5

48.1

86.4

93.3

91.8

91.1

+2.5

CapsNet-2

82.4

48.7

87.8

93.6

92.9

92.3

+3.3

MCapsNet

83.5

49.7

88.6

94.5

94.2

93.8

+4.6

Multi-Task Performance. We simultaneously train our model McapsNet on six tasks in Table 1 and compare it with singletask scenario (Table 3). We can see that our multitask architecture clearly improves the performance over the single task models, which demonstrates the benefits of our multi-task architecture. As Table 3 shows, MCapsNet also outperforms the state-of-the-art multi-task learning models by at least 1.1%. This shows the advantages of our task routing algorithm, which can cluster the features for each task, instead of freely sharing the features among tasks.
Table 1: Performance (%CERR). Slash-separated pairs denote FSTs incapable/capable of inserting word boundaries, respectively; see Section 4. The -KN suffix denotes Kneser-Ney smoothing. The data from Section 3.2 is used for evaluating the modern-orthography language model perplexity, and “Corpus” evaluates test-set transliteration performance from the synthetic missionary text back to the original modern text.

LM perplexity || Valid.

LM perplexity || Test

Transliteration performance (%CERR) || Corpus

Transliteration performance (%CERR) || Newspaper 1

Transliteration performance (%CERR) || Newspaper 2

Transliteration model || FST-(C/Cwb)-7GRAM-KN

3.07

3.13

27.3%

50.1% / 38.7%

52.0% / 47.5%

Transliteration model || FST-(C/Cwb)-9GRAM-KN

2.95

3.02

26.6%

50.7% / 39.3%

52.5% / 47.2%

Transliteration model || FST-(C/Cwb)-11GRAM-KN

2.94

3.02

27.8%

53.9% / 41.3%

54.1% / 48.7%

Transliteration model || FST-RNNLM-(C/Cwb)

2.65

2.69

16.3%

47.2% / 34.3%

49.8% / 41.2%

Table 1 reports the results of the approaches we described in Section 4. Out of the Kneser-Ney n-gram models, we found that the FST-C-9GRAMKN and the version modelling word boundaries (FST-Cwb-9GRAM-KN) to perform best on the synthetic parallel corpus and newspapers, respectively. Cwb was not applied to the synthetic parallel corpus as we did not model word splitting. The hybrid models (FST-RNNLM) outperformed all FST-only approaches. Thus, we found the wordsplitting hybrid model (FST-RNNLM-Cwb) to be the best performing model overall (Table 1).
Table 4: Comparison of AUC Results

-

+SelfAtt

CCL-CT

Model || PCNN+ATT

0.341

0.368

0.381

Model || PCNN+ONE

0.325

0.352

0.38

Table 4 gives another comparison using AUC with all p-values less than 5e-02 from t-test evaluation. The results indicate that the larger AUC, the better performance. A simple ensemble model of two networks (AUC: 0.371) has a similar result as a single model (NetAtt, AUC: 0.368). The main purpose of adding an additional student network is to introduce conï¬‚icts and to build the collaborative curriculum learning scheme. With the CCL strategies, our models improve performances by removing 'hard' (noisy) entity pair bags during.
Table 3: Experimental results in exploring the shared syntactic order event detector.

Pre.

Rec.

F1

Model || CL_Trans_MLP

20.3

16.3

18.1

Model || CL_Trans_CNN

32.5

16.3

21.7

Model || CL_Trans_Hbrid

30.4

17.6

22.3

Model || CL_Trans_Self.

34.9

18.3

24

Model || CL_Trans_GCN (ours)

32

23.4

27

Model || CL_Trans_GCN_Self

32.1

23

26.8

Exploring the Syntactic Order Event Detector. We compare our syntactic order event detector (CL Trans GCN) with several event detectors, including 1) CL Trans MLP, which employs a feed-forward network as event detector; 2) CL Trans CNN, which uses CNNs as the event detector; 3) CL Trans Hbrid, which use a hybrid network (Feng et al., 2016) for event detection. We also compared our model with several variants including 4) CL Trans Self., which replaces the GCNs with a self-attention network, and 5) CL Trans GCN Self, which combines GCNs with a self-attention network. We train these models on the same translated English data. Table 3 shows the results. From the results, CL Trans MLP, CL Trans CNN, and CL Trans Hbrid behave poorly, as expected. The reason might be that these models usually employ order-sensitive structures (e.g., CNNs) for ED, which would suffer the word order inconsistency problem when trained on the translated data. CL Trans Self. yields relatively good performance. The reason might be that self-attention network could provide each word with a feature vector based on all the words of a sentence, which is also irrespective of the wordsâ€™ positions in a sentence. This could address the word order difference to some extent. Our syntactic order event detector yields the best performance.
Table 8: Results for each state change type. Performance on predicting creation and destruction are highest, partially due to the model’s ability to use verb semantics for these tasks.

Cat-1 || C

Cat-2 || M

Cat-3 || D

Cat-2 || C

Cat-3 || M

Cat-4 || D

Model || ETBERT

78.51

61.6

71.5

76.68

54.12

58.62

Model || ETGPT

79.82

56.27

73.83

77.24

50.82

56.27

For PROPARA, Table 5 shows that the model does not significantly outperform the SOTA models in state change detection (Cat-1). However, for those correctly detected events, the transformer model outperforms the previous models for detecting the exact step of state change (Cat-2), primarily based on verb semantics. We do a finer-grained study in Table 8 by breaking down the performance for the three state changes: creation (C), movement (M), and destruction (D), separately. Across the three state changes, the model suffers a loss of performance in the movement cases. This is owing to the fact that the movement cases require a deeper compositional and implicit event tracking. Also, a majority of errors leading to false negatives are due to the the formation of new sub-entities which are then mentioned with other names. For example, when talking about weak acid in "he water becomes a weak acid. the water dissolves limestone" the weak acid is also considered to move to the limestone.
Table 1: Comparison of multilingual sentence-image retrieval/matching (German-Image) and (English-Image) results on Multi30K. (Visual encoders:VGG† otherwise ResNet or Faster-RCNN(ResNet).) (Monolingual models*.)

German to Image || R@1

German to Image || R@5

German to Image || R10

Image to German || R@1

Image to German || R@5

Image to German || R10

English to Image || R@1

English to Image || R@5

English to Image || R10

Image to English || R@1

Image to English || R@5

Image to English || R10

Method || VSE†* (Kiros et al., 2014)

20.3

47.2

60.1

29.3

58.1

71.8

23.3

53.6

65.8

31.6

60.4

72.7

Method || OE†* (Vendrov et al., 2015)

21.0

48.5

60.4

26.8

57.5

70.9

25.8

56.5

67.8

34.8

63.7

74.8

Method || DAN* (Nam et al., 2017)

31.0

60.9

71.0

46.5

77.5

83.0

39.4

69.2

69.1

55.0

81.8

89.0

Method || VSE++* (Faghri et al., 2018)

31.3

62.2

70.9

47.5

78.5

84.5

39.6

69.1

79.8

53.1

82.1

87.5

Method || SCAN* (Lee et al., 2018)

35.7

64.9

74.6

52.3

81.8

88.5

45.8

74.4

83.0

61.8

87.5

93.7

Method || Pivot† (Gella et al., 2017)

22.5

49.3

61.7

28.2

61.9

73.4

26.2

56.4

68.4

33.8

62.8

75.2

Method || Ours† (Random, VGG19)

25.8

54.9

65.1

34.1

65.5

76.5

30.1

62.5

71.6

36.4

68.0

80.9

Method || Ours (Random, No diversity)

36.3

65.3

74.7

53.1

82.3

88.8

46.2

74.7

82.9

63.3

87.0

93.3

Method || Ours (Random)

39.2

67.5

76.7

55.0

84.7

91.2

48.7

77.2

85.0

66.4

88.3

93.4

Method || Ours (w/ FastText)

40.3

70.1

79.0

60.4

85.4

92.0

50.1

78.1

85.7

68.0

88.8

94.0

Method || Ours (w/ BERT)

40.7

70.5

78.8

56.5

84.6

91.3

48.9

78.3

85.8

66.5

89.1

94.1

Table 1 presents the results on the Multi30K testing set. The VSE baselines in the first five rows are trained with English and German descriptions independently. In contrast, PIVOT (Gella et al., 2017) and the proposed model are capable of handling multilingual input queries with single model. For a fair comparison with PIVOT, we also report the result of swapping Faster-RCNN with VGG as the visual feature encoder in our model. As can be seen, the proposed models successfully obtain state-of-the-art results, outperforming other baselines by a significant margin. GermanImage matching benefit more from joint training with English-Image pairs. The models with pretrained multilingual embeddings and contextualized embeddings achieve better performance in comparison to randomly initialized word embeddings, especially for German. One explanation is that the degradation from German singletons is alleviated by the multi-task training with English and the pre-trained embeddings. While the model with BERT performs better in English, FastText is preferred for German-Image matching.
Table 8: Evaluation on auxiliary tasks: Multitask models trained on both the translation and simplification dataset improves the performance for the task of English Simplification.

BLEU

SARI

PCC

Task || English Simplification || Simplify

55.76

41.7

0.736

Task || English Simplification || Translate and Simplify

56.47

41.3

0.730

Task || English Simplification || All Tasks

56.05

42.1

0.736

Task || Machine Translation || Translate

29.09

-

0.769

Task || Machine Translation || Translate and Simplify

27.33

-

0.647

Task || Machine Translation || All Tasks

27.63

-

0.658

Table 8 summarizes the results: the multitask model slightly outperforms a dedicated simplification model on English simplification, showing the benefits of the additional training data from other tasks. By contrast, on the resource-rich MT task, the standalone translation system performs better. This can be explained by the fact that the standalone system is only responsible for text translation, while the multi-task model is exposed to samples of more diverse complexity levels during training which damage its ability to preserve complexity.