katebor/TableEval huggingface dataset:
Table 3: Published results of other models on the SemEval2012 Task 2 dataset.

Accuracy

Correlation

Model || Rink and Harabagiu (2012)

0.394

0.229

Model || Mikolov et al. (2013b)

0.418

0.275

Model || Levy and Goldberg (2014)

0.452

-

Model || Zhila et al. (2013)

0.452

0.353

Model || Iacobacci et al. (2015)

0.459

0.358

Model || Turney (2013)

0.472

0.408

Model || VecOff

0.443

0.321

Model || LRA

0.415

0.264

Model || NLRA

0.453

0.36

Model || NLRA+VecOff

0.475

0.391

4.5 Comparison to other systems. We compared the results of our models to other published results. Table 3 displays those results. Rink and Harabagiu (2012) is the pattern-based model with naive Bayes. Mikolov et al. (2013b), Levy and Goldberg (2014), and Iacobacci et al. (2015) are the vector offset models. Zhila et al. (2013) is the model composed of various features. Turney (2013) extracts the statistical features of two word pairs from a word-context co-occurrence matrix and trains the classifier with additional semantic relational data to assign a relational similarity for two word pairs. NLRA+VecOff achieved a competitive performance to the state-of-the-art method of Turney (2013). Note that our method learns unsupervisedly and does not exploit additional resources, and the method of Turney (2013) cannot obtain the distributed representation of word pairs. A work similar to ours, Bollegala et al. (2015), represented lexico-syntactic patterns as the vector offset of co-occurring word pairs and updated the vector offsets of word pairs such that word pairs that co-occur in similar patterns have similar offsets. They evaluated their model on all 79 semantic relations of the dataset and achieved 0.449 accuracy. In their setting, NLRA+VecOff achieved 0.47 accuracy, outperforming their model.
Table 1: Automatic evaluation

PPL-V

PPL-T

distinct-1

distinct-2

Model || S2SA

8.41

9.05

0.0809

0.2110

Model || TAS2S

7.38

7.84

0.04759

0.1087

Model || SPMN500

7.04

7.93

0.06430

0.1734

Model || SPMN1000

6.28

7.72

0.07347

0.1909

Model || DPMN100

6.45

7.69

0.04350

0.1048

Automatic Evaluation:. Table 1 shows the automatic evaluation results. We see that both SPMN and DPMN obtain huge improvements over the two baselines in terms of PPL-V and PPL-T. Also, we observe that SPMN1000 outperforms SPMN500 in all the four automatic metrics. Note that each post has the same prototypes provided by SPMN. This is reasonable that the relevant response is more likely to be covered by storing more prototypes in SPMN. As for the DPMN, we can see that DPMN achieves the best performance with only 100 prototypes in terms of PPL-T, compared with the other 4 methods. This suggests that using a retrieval mechanism to incorporate the relevant responses brings more useful information for better response generation. Note that S2SA outperforms the others in terms of distinct-1 and distinct-2. Further human evaluation indicates that many responses generated by S2SA are irrelevant and meaningless, which could inevitably increase the distinct scores.
Table 4: Unsupervised POS tagging results on WSJ, with fastText vectors as the observed embeddings.

M-1

VM

System || Gaussian HMM

72.0

65.0

System || Ours (4 layers)

76.4

69.3

System || Ours (8 layers)

76.8

69.4

System || Ours (16 layers)

67.3

62.0

These results are summarized in Table 4 and Table 5. While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space.
Table 2: Results for simple and neural models on the test set of RecipeQA dataset.

Visual Cloze

Textual Cloze

Visual Coherence

Visual Ordering

Hasty Student

27.35

26.89

65.80

40.88

Impatient Reader (Text only)

-

28.03

-

-

Impatient Reader (Multimodal)

27.36

29.07

28.08

26.74

4.3 Baseline Results. We report the performance of the baseline models in Table 2 which indicates the ratio of correct answers against the total questions in the test. For the textual cloze, the comparison between text-only and multimodal Impatient Reader models shows that the additional visual modality helps the model to understand the question better and to provide more accurate answers. While for the cloze style questions, the Impatient Reader outperforms the Hasty student, for the visual coherence and visual ordering style questions Hasty student gives way better results. This demonstrates that better neural models are needed to be able to effectively deal with this kind of questions. Some qualitative examples are provided in the supplementary material.
Table 1: Accuracy scores on the Stanford Natural Language Inference (SNLI) and MultiNLI Mismatched (MNLI) tasks. DME=Dynamic Meta-Embeddings; CDME=Contextualized Dynamic Meta-Embeddings; *=multiple different embedding sets (see Section 4). Number of parameters included in parenthesis. Results averaged over five runs with different random seeds, using a BiLSTM-Max sentence encoder.

SNLI

MNLI

Model || InferSent (Conneau et al., 2017)

84.5

-

Model || NSE (Munkhdalai and Yu, 2017)

84.6

-

Model || G-TreeLSTM (Choi et al., 2017)

86.0

-

Model || SSE (Nie and Bansal, 2017)

86.1

73.6

Model || ReSan (Shen et al., 2018)

86.3

-

Model || GloVe BiLSTM-Max (8.6M)

85.2}.3

70.0}.5

Model || FastText BiLSTM-Max (8.6M)

85.2}.2

70.3}.3

Model || Naive baseline (9.8M)

85.6}.3

71.1}.2

Model || Naive baseline (61.3M)

86.0}.5

73.0}.2

Model || Unweighted DME (8.6M)

86.3}.4

74.4}.2

Model || DME (8.6M)

86.2}.2

74.4}.2

Model || CDME (8.6M)

86.4}.3

74.1}.2

Model || DME* (9.0M)

86.7}.2

74.3}.4

Model || CDME* (9.0M)

86.5}.2

74.9}.5

4.2 Results. Table 1 shows the results. We report accuracy scores averaged over five runs with different random seeds, together with their standard deviation, for the SNLI and MultiNLI datasets. We include two versions of the naive baseline: one with a 512dimensional BiLSTM encoder; and a bigger one with 2048 dimensions. Both naive baseline models outperform the single encoders that have only GloVe or FastText embeddings. This shows how including more than one embeddings can help performance. Next, we observe that the DME embeddings outperform the naive concatenation baselines, while having fewer parameters. Differences between the three DME variants are small and not significant, although we do note that we found the highest maximum performance for the contextualized version, which adds very few additional parameters. It is important to note that the imposition of weighting thus is not detrimental to performance, which means that DME and CDME provide additional interpretability without sacrificing performance. Finally, we obtain results for using the six different embedding types (marked *), and show that adding in more embeddings increases performance further. To our knowledge, these numbers constitute the state of the art within the model class of single sentence encoders on these tasks.
Table 2: Performance of different approaches on the WikiSQL dataset. The two evaluation metrics are logical form accuracy (Acclf ) and execution accuracy (Accex). The settings of the training data represent the proportion of supervised data we use.

Dev || Acclf

Dev || Accex

Test || Acclf

Test || Accex

Methods || Attentional Seq2Seq || Training Data || 100%

23.3%

37.0%

23.4%

35.9%

Methods || Aug.PntNet (Zhong et al. 2017) || Training Data || 100%

44.1%

53.8%

43.3%

53.3%

Methods || Aug.PntNet (re-implemented by us) || Training Data || 100%

51.5%

58.9%

52.1%

59.2%

Methods || Seq2SQL (Zhong et al. 2017) || Training Data || 100%

49.5%

60.8%

48.3%

59.4%

Methods || SQLNet (Xu et al. 2017) || Training Data || 100%

-

69.8%

-

68.0%

Methods || STAMP || Training Data || 30%

54.6%

69.7%

53.7%

68.9%

Methods || STAMP + QG || Training Data || 30%

61.6%

74.4%

61.2%

73.9%

Methods || STAMP || Training Data || 100%

61.5%

74.8%

60.7%

74.4%

Methods || STAMP + QG || Training Data || 100%

64.3%

76.5%

63.7%

75.5%

From Table 2, we can see that STAMP performs better than existing systems when trained on the full WikiSQL training dataset, achieving state-ofthe-art execution accuracy and logical form accuracy on WikiSQL. We further conduct experiments to demonstrate the effectiveness of our question generation driven approach. We run the entire pipeline (STAMP+QG) with different percentages of training data. The second column “Training Data” in Table 2 and the x-axis in Figure 3 represent the proportion of WikiSQL training data we use for training the QG model and semantic parser. That is to say, STAMP +QG with 30% means that we sample 30% WikiSQL training data to train the QG model, and then combine QG generated data and exactly the same 30% WikiSQL training data we sampled before to train the semantic parser. In this experiment, we sample five SQL queries for each table in the training data, resulting in 43.5K SQL queries. Applying the QG model on these SQL queries, we get 92.8K SQLquestion pairs. From Figure 3, we see that accuracy increases as the amount of supervised training data expands. Results show that QG empowers the STAMP model to achieve the same accuracy on WikiSQL dataset with 30% of the training data. Applying QG to the STAMP model under the full setting brings further improvements, resulting in new state-of-the-art accuracies.
Table 5: Results of using NP head alone for bridging anaphora resolution based on different word representation resources. Bold indicates statistically significant differences over the baselines (two-sided paired approximate randomization test, p < 0.01).

acc

GloVe GigaWiki14

21.42

GloVe Giga

21.87

embeddings PP

33.03

embeddings bridging

34.84

Table 5 lists the results for bridging anaphora resolution based on different word representation resources. We notice that there is not much difference between GloVe GigaWiki14 and GloVe Giga. We find that using embeddings PP achieves an accuracy of 33.03% on the ISNotes corpus, which outperforms the results based on GloVe GigaWiki14 and GloVe Giga by a large margin. Using embeddings bridging further improves the result by 1.8%. Although the improvement is not significant, we suspect that the representations for words without the suffix “ PP” in embeddings bridging are more accurate because they are trained on many more instances in the vanilla GloVe vectors (GloVe GigaWiki14).
Table 5: Comparison of our Syn-GCN model with (Marcheggiani and Titov, 2017), (He et al., 2018) and (Cai et al., 2018) on the English test set. ∆ F1 shows the absolute performance gap between syntax-agnostic and syntax-aware settings.

syntax-agnostic

syntax-aware

ΔF1

System || M&T (2017)

87.7

88.0

0.3

System || He et al. (2018)

88.7

89.5

0.8

System || Cai et al. (2018)

89.6

89.6

≈0.0

System || Our model

88.7

89.8

1.1

Effect of deep encoder. Table 5 reports F1 scores of our Syn-GCN model, Marcheggiani and Titov (2017), He et al. (2018) and Cai et al. (2018) on English test set in both syntax-agnostic and syntax-aware settings. The comparison shows that our framework is more effective for incorporating syntactic information by giving more performance improvement through introducing syntax over syntax-agnostic SRL than previous state-ofthe-art systems did.
Table 6: Human evaluation results. S-ocr refer to 150 sentences from the D-ocr and E-ocr represents the errors in S-ocr, similar for S-asr and Easr. R denotes the average recall of three students. Numbers in bold denotes the correctly-annotated results by students.

Stu1

Stu2

Stu3

R

S-ocr

84/150

100/150

75/150

57.3

E-ocr

104/170

121/170

100/170

72.0

S-asr

95/150

79/150

106/150

62.0

E-asr

341/393

179/393

356/393

74.3

Results. Table 6 shows the information of 300 sentences and the annotation results on them. The average recall in the table illustrates that three students have a higher recognition rate for errors from D-asr than that from D-ocr, which, to some extent, indicates that P-style errors are easier to be detected than V-style ones. Besides, we observe that three volunteers fail to identify around 36.9% errors on average, which may indicate that our generated sentences include some challenging errors which are likely to be made by human. Such errors are valuable for CSC since they are potential real cases in peopleâ€™s writing or typing.
Table 2: Performance of Clique-EBM with pruned edges in G.

WPT || P

WPT || R

WPT || F

WP3T || P

WP3T || R

WP3T || F

k || 5

90.46

92.27

91.36

83.52

80.48

81.97

k || 10

92.92

95.07

93.98

85.32

84.4

84.86

k || 15

94.85

96.14

95.49

87.67

86.38

87.02

k || 20

95.23

96.49

95.86

89.25

88.62

88.93

For any two words appearing in an exhaustive segmentation, we keep an edge only if both the words overlap within a distance of k characters. We experiment with k = 5, 10, 15 and 20. Hence, for K = 20, a word will form edges with all the words that fall within 20 characters to left and 20 characters to right. The average length of an input sequence in DCS10K is 40.79 characters. We do not modify our inference procedure in system 8 other than to take care of the possibility that a clique need not always be returned. Table 2 shows the results for different values of k. Interestingly, the results show a monotonic increase with the increase in context window, and the results with the entire context are still better than those with k = 20, even though only marginally. It is interesting to note that, keeping the entire context does not adversely affect the predictions as none of the pruned models outperforms System 8.
Table 2: Comparison of various bounding box regressors on Flickr30k Entities for different IoU thresholds. The number of parameters in Gr is also shown.

Params || -

IoU || 0.5

IoU || 0.6

IoU || 0.7

IoU || 0.8

IoU || 0.9

Architecture || w/o regression

-

65.21

53.19

35.70

14.32

1.88

Architecture || 300-16(-4096)

0.3M

64.14

57.66

48.22

33.04

9.29

Architecture || 300-64(-4096)

1.1M

63.87

57.43

49.05

33.84

10.55

Architecture || 300-256(-4096)

4.3M

63.84

57.70

48.71

33.87

10.05

Architecture || 300-1024(-4096)

17M

64.29

58.05

48.49

33.94

10.09

Architecture || 300(-256-4096)

4.5M

62.82

56.28

48.02

32.71

9.89

Architecture || 300-4096

1.2M

63.23

56.92

48.17

32.66

9.20

Table 2 shows the results with and without regressor. The regressor significantly improved the accuracy with high IoU thresholds, which demonstrates that the regressor improved the localization accuracy. In addition, the accuracy did not decrease as a result of sharing the hidden layer or reducing the number of units in the hidden layer. This suggests that the regressor lies in a very low-dimensional manifold because the regressor for one concept can be shared by many concepts (e.g., the person regressor can be used for man, woman, girl, boy, etc.). The number of parameters was significantly reduced by these tricks, to even fewer than in the linear transformation. The accuracy slightly decreased with a threshold of 0.5, because the regressor was not learned properly for the categories that did not frequently appear in the training data.
Table 1: Single-label classification (accuracy) results.

Commenting

Ogling

Groping

Model || Linear SVM

42.2

35.0

55.8

Model || Gaussian NB

46.8

74.7

66.0

Model || Logistic Reg.

61.4

78.0

69.1

Model || SVM

65.5

79.0

70.3

Model || CNN

80.9

82.2

86.0

Model || RNN

81.0

82.2

86.2

Model || CNN-RNN

81.6

84.1

86.5

5 Results. See Table 1 for single-label results on the selected harassment categories, where CNN-RNN was the best performing model compared to several nonneural and neural baselines.
Table 6: Results (presented in percentage) in terms of precision (P), recall (R) and F-measure (F1) on the test set for each label obtained by our HSLN-RNN model on the PubMed 20k dataset.

P

R

F1

Support

Label || Background

78.5

80.0

79.2

3077

Label || Objectives

74.2

69.9

72.0

2333

Label || Methods

95.0

97.7

96.3

9884

Label || Results

96.8

95.3

96.0

9713

Label || Conclusions

97.6

96.5

97.1

4571

Label || Total

92.6

92.7

92.6

29578

Table 6 and 7 detail the results of classification for each label in terms of performance scores (precision, recall and F1) and confusion matrix, respectively (for our HSLN-RNN model trained on the PubMed 20k dataset). These show that the classifier is very good at predicting the labels Methods, Results and Conclusions, whereas the greatest difficulty the classifier has is in distinguishing Background sections from Objectives sections. One fifth of Background sentences are incorrectly classified as Objectives, while around one forth of Objectives sentences are wrongly assigned to the label of Background. We conjecture this difficulty mainly comes from the fact that the difference between Background and Objectives sentences in terms of writing style is less obvious compared with the other sections of the abstract. Moreover, our model has some difficulty in telling Methods sentences apart from Results sentences.
Table 4: CV coherence scores for topics generated by various models. Higher is better. The best result in each column is in bold.

Snippets

TagMyNews

Twitter

Model || LDA

0.436

0.449

0.436

Model || BTM

0.435

0.463

0.435

Model || NTM

0.463

0.468

0.463

Model || TMN

0.487

0.499

0.468

4.2 Topic Coherence Comparison In Section 4.1, we find that TMN can significantly outperform comparison models on short text classification. In this section, we study whether jointly learning topic models and classification can be helpful in producing coherent and meaningful topics. We use the CV metric (RÂ¨oder et al., 2015) computed by Palmetto toolkit12 to evaluate the topic coherence, which has been shown to give the closest scores to human evaluation compared to other widely-used topic coherence metrics like NPMI (Bouma, 2009). Table 4 shows the comparison results of LDA, BTM, NTM, and TMN on the three English datasets. Note that we do not report CV scores for Chinese Weibo dataset as the Palmetto toolkit cannot process Chinese topics. As can be seen, TMN yields higher CV scores by large margins than all others in comparison. This indicates that jointly exploring classification would be effective in producing coherent topics. The reason is that the supervision from classification labels can guide unsupervised topic models in discovering meaningful and interpretable topics. We also observe that NTM produces better results than LDA and BTM, which implies the effectiveness of inducing topic models by neural networks.
Table 3: Comparison of dev set MRR of Ours(ConvE) and models without reward shaping and action dropout.

UMLS

Kinship

FB15k237

WN18RR

NELL995

Model || Ours(ConvE)

73.0

75.0

38.2

43.8

78.8

Model || -RS

67.7

66.5

35.1

45.7

78.4

Model || -AD

61.3

65.4

31.0

39.1

76.1

5.2 Ablation Study. We perform an ablation study where we remove reward shaping (−RS) and action dropout (−AD) from Ours(ConvE) and compare their MRRs to the whole model on the dev sets. As shown in Table 3, on most datasets, removing each component results in a significant performance drop. The exception is WN18RR, where removing the ConvE reward shaping module improves the performance. Removing reward shaping on NELL995 does not change the results significantly. In general, removing action dropout has a greater impact, suggesting that thorough exploration of the path space is important across datasets.
Table 3: MED accuracy on five randomly selected languages with 1, 2, and 4 sources and combined with paradigm transduction (“+PT”). Best results in bold.

SET1 || 1

SET1 || 2

SET1 || 4

SET1 || +PT

SET2 || 1

SET2 || 2

SET2 || 4

SET2 || +PT

SET3 || 1

SET3 || 2

SET3 || 4

SET3 || +PT

dutch

.00

.00

.00

.49

.04

.01

.00

.78

.43

.65

.72

.87

german

.00

.00

.00

.65

.00

.00

.01

.75

.44

.42

.59

.88

icelandic

.00

.00

.00

.41

.03

.02

.02

.50

.24

.33

.35

.77

spanish

.00

.00

.00

.92

.03

.09

.09

.98

.59

.63

.83

.99

welsh

.00

.00

.00

.91

.05

.14

.15

.97

.35

.53

.70

.99

The input to a multi-source model is the concatenation of all sources and corresponding tags. During training, we randomly sample (with repetition) one or three additional forms from the paradigm of each example. At test time, we sample the additional forms from the given partial paradigm; without repetition first, but repeating if not enough inflected forms are available. For autoencoding examples in the training data, we simply concatenate two or four copies of the source and the autoencoding tag. We randomly select five languages for this experiment. Table 3 shows that, for SET3, four sources (column header"4") are generally better than two sources ("2"), which in turn are better than one source ("1"); thus, as expected, making additional sources available in training improves results. We attribute one exception (German accuracy is .4391 for "1" and .4179 for "2") to the noisiness of the problem training sets in terms of number of paradigms are relatively small, even for SET3.
Table 2: Situation Entity Type Classification Results on the Training Set of MASC+Wiki with 10-Fold Cross-Validation. We report accuracy (Acc), macro-average F1-score (Macro) and class-wise F1 scores for STATE (STA), EVENT (EVE), REPORT (REP), GENERIC (GENI), GENERALIZING (GENA), QUESTION (QUE) and IMPERATIVE (IMP).

Macro

Acc

STA

EVE

REP

GENI

GENA

QUE

IMP

Model || Humans

78.6

79.6

82.8

80.5

81.5

75.1

45.8

90.7

93.6

Model || CRF (Friedrich et al. 2016)

71.2

76.4

80.6

78.6

78.9

68.3

29.4

84.4

75.3

Model || Clause-level Bi-LSTM

74.4

78.3

82.6

81.3

84.9

66.2

36.1

88.5

80.9

Model || Paragraph-level Model

77.6

81.2

84.3

82.1

85.3

76.4

43.2

90.8

81.2

Model || Paragraph-level Model+CRF

77.8

81.3

84.3

82.0

85.7

77.0

43.5

90.4

81.5

Table 4 shows cross-genre experimental results of our neural network models on the training set of MASC+Wiki by treating each genre as one crossvalidation fold. As we expected, both the macroaverage F1-score and class-wise F1 scores are lower compared with the results in Table 2 where in-genre data were used for model training as well. But the performance drop on the paragraph-level models is little, which clearly outperform the previous system (Friedrich et al., 2016) and the baseline model by a large margin. As shown in Table 5, benefited from modeling wider contexts and common SE label patterns, our full paragraphlevel model improves performance across almost all the genres. The high performance in the crossgenre setting demonstrates the robustness of our paragraph-level model across genres. Table 2 reports the cross-validation classification results. Consistently, our clause-level baseline model already outperforms the previous best model. By exploiting paragraph-wide contexts, the basic paragraph-level model obtains consistent performance improvements across all the classes compared with the baseline clause-level prediction model, especially for the classes GENERIC and GENERALIZING, where the improvements are significant. After using the CRF layer to fine-tune the predicted SE label sequence, slight performance improvements were observed on the four small classes. Overall, the full paragraphlevel neural network model achieves the best macro-average F1-score of 77.8% in predicting SE types, which not only outperforms all previous approaches but also reaches human-like performance on some classes.