katebor/TableEval huggingface dataset:
Table 5: Low-shot learning results for the Yahoo dataset, in error percentages. Transfer learning-based 3 of the total dataset, while the methods are trained on 1 other models are trained on the whole dataset.

Yahoo Binary

Model || RNN Baseline

35.29

Model || LSTM Baseline

32.41

Model || KimCNN Kim (2014)

14.25

Model || Self Attention Lin et al. (2017)

13.16

Model || RCNN Lai et al. (2015)

12.67

Model || BCN+ELMo

13.51

Model || ULMFiT

10.62

Model || BERTBASE

10.14

7.2 Low-Shot Learning . Low-shot learning refers to the practice of feeding a model with a small amount of training data, contrary to the normal practice of using a large amount of data. We chose the Yahoo dataset for this experiment due to its small size. Experimental results in Table 5 show that, with only 1 3 of the total dataset, ULMFiT and BERT perform better than task-specific models, while BCN+ELMo shows a comparable result. Clearly, this shows that the models have learned significantly during the transfer learning process.
Table 2: Results on recast MNLI and JOCI.

Log loss

Margin loss

Dataset || MNLI 1

93.6

93.4

Dataset || MNLI 2

87.9

87.9

Dataset || JOCI 1

86.6

86.9

Dataset || JOCI 2

76.6

78

Results on Recast MNLI and JOCI. Table 2 shows results on the recast MNLI and JOCI datasets. We find that for the two synthetic MNLI datasets, margin-loss performs similarly to cross entropy log-loss. Shifting to the JOCI datasets, with less extreme (contradiction / entailed) hypotheses, especially in the adversarial JOCI2 variant, marginloss outperforms log-loss.
Table 3: Experimental results on COPA test set.

Acc (%)

Method || PMI (Jabeen et al., 2014)

58.8

Method || PMI EX (Gordon et al., 2011)

65.4

Method || CS (Luo et al., 2016)

70.2

Method || CS MWP (Sasaki et al., 2017)

71.2

Method || BERTlog (ours)

73.4

Method || BERTmargin (ours)

75.4

Results on COPA. Table 3 shows our results on COPA. Compared with previous state-of-theart knowledge-driven baseline methods, a BERT model trained with a log-loss achieves better performance. When training the BERT model with a margin-loss instead of a log-loss, our method gets the new state-of-the-art result on the established COPA splits, with an accuracy of 75.4%.3 Analyses Table 4 shows some examples from the MNLI1, JOCI1 and COPA datasets, with scores 3 We exclude a blog-posted GPT result, which comes.
Table 2: Results on CQA dev-random-split with CoS-E used during training.

Accuracy (%)

Method || BERT (baseline)

63.8

Method || CoS-E-open-ended

65.5

Method || CAGE-reasoning

72.6

Table 2 shows results that compare a BERT baseline that uses only the CQA inputs and the same architecture but trained using inputs that contain explanations from CoS-E during training. The BERT baseline model reaches 64% accuracy and adding open-ended human explanations (CoS-E-open-ended) alongside the questions during training results in a 2% boost in accuracy. By generating explanations as described in Section 4.1, we can give the commonsense question answering model access to an explanation that is not conditioned on the ground truth. These explanations (CAGE-reasoning) can be provided during both training and validation and increases the accuracy to 72%.
Table 2: Comparison for nonlocal dependency identification on the test data.

Empty element detection (Fillers are ignored.) || pre.

Empty element detection (Fillers are ignored.) || rec.

Empty element detection (Fillers are ignored.) || F1

Nonlocal dependency identification || pre.

Nonlocal dependency identification || rec.

Nonlocal dependency identification || F1

Nonlocal dependency identification (Unindexed empty elements are excluded.) || pre.

Nonlocal dependency identification (Unindexed empty elements are excluded.) || rec.

Nonlocal dependency identification (Unindexed empty elements are excluded.) || F1

(Johnson, 2002)

85

74

79

73

63

68

–

–

–

(Dienes and Dubey, 2003)

–

–

–

81.5

68.7

74.6

–

–

–

(Campbell, 2004)

85.2

81.7

83.4

78.3

75.1

76.7

–

–

–

(Schmid, 2006)

86

82.3

84.1

–

–

–

81.7

73.5

77.4

(Cai et al., 2011)

90.1

79.5

84.5

–

–

–

–

–

–

(Hayashi and Nagata, 2016)

90.3

81.7

85.8

–

–

–

–

–

–

(Kato and Matsubara, 2016)

88.5

82.1

85.2

81.4

75.5

78.4

79.8

73.8

76.7

(Kummerfeld and Klein, 2017)

89.5

81.6

85.4

74.3

67.3

70.6

–

–

–

(Johnson, 2002)

93

83

88

80

70

75

–

–

–

(Campbell, 2004)

94.9

91.1

93

90.1

86.6

88.4

–

–

–

ours

92.6

87.7

90.1

88.1

83.4

85.7

88.4

81.1

84.6

ours (with ELMo)

94.2

90.3

92.3

89.9

86.2

88

90.4

84.1

87.2

ours (with BERT)

94.9

91.4

93.1

90.8

87.4

89

91.6

84.9

88.1

Table 2 summarizes the performances of our system and previous ones. These results demonstrate that our system significantly outperforms the previous methods in nonlocal dependency identification. Although the main reason for this is because of the performance of the K&K parser, the important point is that our proposed approximation method enables us to use the K&K parser for the nonlocal dependency identification task. The previous methods that introduce additional operations cannot adopt such parser directly. On the other hand, although post-processing approach can use any parser in pre-processing, our approach outperforms the post-processing approach, even if the pre-processing parser is assumed to always generate gold PTB trees.
Table 4: Results of Self-Play Evaluation.

Succ. (%)

#Turns

System || Retrieval

9.8

3.26

System || Retrieval-Stgy

67.2

6.56

System || Ours-PMI

47.4

5.12

System || Ours-Neural

51.6

4.29

System || Ours-Kernel

75

4.2

Table 4 shows the results of 500 simulations for each of the comparison systems. Our system with Kernel transition obtains the highest success rate, significantly improving over other approaches. The success rate of the base Retrieval agent is lower than 10%, which proves that a chitchat agent without a target-guided strategy can hardly accomplish our task. The Retrieval-Stgy agent has a relatively high success rate, while taking more turns (6.56) to accomplish this. This is partially due to the lack of coarse-grained keyword modeling and transition. We further note that, in the Kernel system, around 81% of predicted keywords eventually occur in the produced utterances, indicating that the predicted keywords have a great impact on the retrieval module.
Table 7: Comparison on the SCWS task. Setting (a) for WordCtx2Sense uses λ = 0.1 for all pairs, and setting (b) uses λ = 10−3 for pairs containing same target words and λ = 0.1 for all other pairs. Word2Sense, Word2Vec and Word2GM neglect context and compare target words. a numbers reported from (Mu et al., 2017) whose experimental setup we could replicate.

Pearson-coefficient

Method || WordCtx2Sense (a)

0.666

Method || WordCtx2Sense (b)

0.67

Method || Word2Sense

0.644

Method || Word2vec

0.651

Method || (Mu et al., 2017)

0.637

Method || (Huang et al., 2012)

0.657

Method || (Arora et al., 2018)

0.652

Method || Word2GM

0.655

Method || MSSG.300D.30K

0.679a

Method || MSSG.300D.6K

0.678a

Results . Table 7 shows that sense embeddings using context information perform better than all the existing models, except MSSG models (Neelakantan et al., 2015). Also, computing the embeddings of a word using the contextual information improves results by aprox. 0.025, compared to the case when words embeddings are used directly.
Table 2: Medical bilingual lexicon induction results showing the quality of the BWE based dictionaries using 1-best and 5-best translations.

Acc1

Acc5

freq || (Braune et al., 2018)

38.6

47.4

freq || EU+UFAL+orth

25.9

40.6

rare || (Braune et al., 2018)

26.3

28.2

rare || EU+UFAL+orth

17.5

28.8

We also compare the quality of our best dictionary (EU+UFAL+orth) to previous work by running bilingual lexicon induction using the test lexicons of Braune et al. (2018) containing frequent and rare medical words respectively. Accuracies of 1-best and 5-best translations in Table 2 show comparable word translation quality to previous work, although we do not employ any task specific steps in contrast to Braune et al. (2018). Note that our dictionary does not contain some of the rare words of the test lexicons which we ignore during evaluation.
Table 1: Automatic evaluation of sentiment analyzers.

H-M SentiCons

Model || Rule-Based (RB)

0.936

Model || Regression Model (RM)

0.846

Model || Domain Adversarial (DA)

0.747

Table 1 shows the automatic evaluation results of three sentiment analyzers. We find that: (1) The rule-based method RB performs the best. This accords with the fact that story endings in the ROC-Stories corpus are simple and have relatively obvious emotional words. (2) DA can not improve the performance of sentiment analysis in our task compared to RM. We hypothesize that is because the domains of labeled SST corpus and ROCStories corpus differ too much that affects the performance of domain adaptation.
Table 4: F1 score on the test set of HOTPOTQA distractor and full wiki setting. All numbers from the official leaderboard. All models except BiDAF are concurrent work (not published). DECOMPRC achieves the best result out of models reported to both distractor and full wiki setting.

Dist F1

Open F1

Model || DECOMPRC

69.63

40.65

Model || Cognitive Graph

-

48.87

Model || BERT Plus

69.76

-

Model || MultiQA

-

40.23

Model || DFGN+BERT

68.49

-

Model || QFE

68.06

38.06

Model || GRN

66.71

36.48

Model || BiDAF

59.02

Finally, Table 4 shows the F1 score on the test set for distractor setting and full wiki setting on the leaderboard. These include unpublished models that are concurrent to our work. DECOMPRC achieves the best result out of models that report both distractor and full wiki setting.
Table 1: Performance comparison on the private test set of HotpotQA in the distractor setting. Our DFGN is the second best result on the leaderboard before submission (on March 1st). The baseline model is from Yang et al. (2018) and the results with ∗ is unpublished. DFGN(Ours)† refers to the same model with a revised entity graph, whose entities are recognized by a BERT NER model. Note that the result of DFGN(Ours)† is submitted to the leaderboard during the review process of our paper.

Answer || EM

Answer || F1

Sup Fact || EM

Sup Fact || F1

Joint || EM

Joint || F1

Model || Baseline Model

45.60

59.02

20.32

64.49

10.83

40.16

Model || GRN∗

52.92

66.71

52.37

84.11

31.77

58.47

Model || DFGN(Ours)

55.17

68.49

49.85

81.06

31.87

58.23

Model || QFE∗

53.86

68.06

57.75

84.49

34.63

59.61

Model || DFGN(Ours)†

56.31

69.69

51.50

81.62

33.62

59.82

4.2 Main Results . We first present a comparison between baseline models and our DFGN2. Table 1 shows the performance of different models in the private test set of HotpotQA. From the table we can see that our model achieves the second best result on the leaderboard now3 (on March 1st). Besides, the answer performance and the joint performance of our model are competitive against state-of-the-art unpublished models. We also include the result of our model with a revised entity graph whose entities are recognized by a BERT NER model (Devlin et al., 2018). We fine-tune the pre-trained BERT model on the dataset of the CoNLLâ€™03 NER shared task (Sang and De Meulder, 2003) and use it to extract named entities from the input paragraphs. The results show that our model achieves a 1.5% gain in the joint F1-score with the entity graph built from a better entity recognizer.