katebor/TableEval huggingface dataset:
Table 3: Comparison on validation perplexity. BoW, TDNN and NABS are the baseline neural compression models with different encoders in Rush et al. (2015)

Labelled Data

Perplexity

Model || Bag-of-Word (BoW)

3.8M

43.6

Model || Convolutional (TDNN)

3.8M

35.9

Model || Attention-Based (NABS) (Rush et al.,,2015)

3.8M

27.1

Model || Forced-Attention (FSC)

3.8M

18.6

Model || Auto-encoding (ASC+FSC1)

3.8M

16.6

In Figure 3, we present the validation perplexity to compare the abilities of the three models to learn the compression languages. The ASC+FSC1(red) employs the same dataset for unlabelled and labelled training, while the ASC+FSC2(black) employs the full unlabelled dataset. Here, the joint ASC+FSC1 model obtains better perplexities than the single discriminative FSC model, but there is not much difference between ASC+FSC1 and ASC+FSC2 when the size of the labelled dataset grows. From the perspective of language modelling, the generative ASC model indeed helps the discriminative model learn to generate good summary sentences. Table 3 displays the validation perplexities of the benchmark models, where the joint ASC+FSC1 model trained on the full labelled and unlabelled datasets performs the best on modelling compression languages.
Table 4: Open-ended and multiple-choice (MC) results on VQA test set (trained on train+val set) compared with state-of-the-art: accuracy in %. See Sec. 4.4.

Test-dev || Open Ended || Y/N

Test-dev || Open Ended || No.

Test-dev || Open Ended || Other

Test-dev || Open Ended || All

Test-dev || MC || All

Test-standard || Open Ended || Y/N

Test-standard || Open Ended || No.

Test-standard || Open Ended || Other

Test-standard || Open Ended || All

Test-standard || MC || All

MCB

81.2

35.1

49.3

60.8

65.4

-

-

-

-

-

MCB + Genome

81.7

36.6

51.5

62.3

66.4

-

-

-

-

-

MCB + Att.

82.2

37.7

54.8

64.2

68.6

-

-

-

-

-

MCB + Att. + GloVe

82.5

37.6

55.6

64.7

69.1

-

-

-

-

-

MCB + Att. + Genome

81.7

38.2

57.0

65.1

69.5

-

-

-

-

-

MCB + Att. + GloVe + Genome

82.3

37.2

57.4

65.4

69.9

-

-

-

-

-

Ensemble of 7 Att. models

83.4

39.8

58.5

66.7

70.2

83.2

39.5

58.0

66.5

70.1

Naver Labs (challenge 2nd)

83.5

39.8

54.8

64.9

69.4

83.3

38.7

54.6

64.8

69.3

HieCoAtt (Lu et al. 2016)

79.7

38.7

51.7

61.8

65.8

-

-

-

62.1

66.1

DMN+ (Xiong et al. 2016)

80.5

36.8

48.3

60.3

-

-

-

-

60.4

-

FDA (Ilievski et al. 2016)

81.1

36.2

45.8

59.2

-

-

-

-

59.5

-

D-NMN (Andreas et al. 2016a)

81.1

38.6

45.5

59.4

-

-

-

-

59.4

-

AMA (Wu et al. 2016)

81.0

38.4

45.2

59.2

-

81.1

37.1

45.8

59.4

-

SAN (Yang et al. 2015)

79.3

36.6

46.1

58.7

-

-

-

-

58.9

-

NMN (Andreas et al. 2016b)

81.2

38.0

44.0

58.6

-

81.2

37.7

44.0

58.7

-

AYN (Malinowski et al. 2016)

78.4

36.4

46.3

58.4

-

78.2

36.3

46.3

58.4

-

SMem (Xu and Saenko, 2016)

80.9

37.3

43.1

58.0

-

80.9

37.5

43.5

58.2

-

VQA team (Antol et al. 2015)

80.5

36.8

43.1

57.8

62.7

80.6

36.5

43.7

58.2

63.1

DPPnet (Noh et al. 2015)

80.7

37.2

41.7

57.2

-

80.3

36.9

42.2

57.4

-

iBOWIMG (Zhou et al. 2015)

76.5

35.0

42.6

55.7

-

76.8

35.0

42.6

55.9

62.0

4.4 Comparison to State-of-the-Art. Table 4 compares our approach with the state-of-the-art on VQA test set. Our best single model uses MCB pooling with two attention maps. Additionally, we augment our training data with images and QA pairs from the Visual Genome dataset. We also concatenate the learned word embedding with pretrained GloVe vectors (Pennington et al. 2014). Each model in our ensemble of 7 models uses MCB with attention. Some of the models were trained with data from Visual Genome, and some were trained with two attention maps. This ensemble is 1.8 points above the next best approach on the VQA open-ended task and 0.8 points above the next best approach on the multiple-choice task (on Testdev). Even without ensembles, our “MCB + Genome + Att. + GloVe” model still outperforms the next best result by 0.5 points, with an accuracy of 65.4% versus 64.9% on the open-ended task (on Test-dev).
Table 1: Corpus size (length in token) and system performance by genre. News* used gold trees and is not included in total.

Sentences

Length

Yield

Precision

Genre || News*

100

19.3

142

78.9

Genre || News

100

19.3

144

70.8

Genre || Wiki

100

21.4

178

61.8

Genre || Web

100

19.2

165

49.1

Genre || Total

300

20.0

487

60.2

Results. From the whole corpus of 300 sentences, PropsDE extracted 487 tuples, yielding on average 1.6 per sentence with 2.9 arguments. 60% of them were labeled as correct. Table 1 shows that most extractions are made from Wikipedia articles, whereas the highest precision can be observed for newswire text. According to our expectations, web pages are most challenging, presumably due to noisier language. These differences between the genres can also be seen in the precision-yield curve (Figure 2).
Table 3: Results of instance-wise evaluation for anaphoric and cataphoric sets

Recall

Precision

F-score

Avg.P

Set || Anaphoric || Method || single-column CNN (w/ position vec.)

0.445

0.525

0.481

0.341

Set || Anaphoric || Method || MCNN (BASE)

0.591

0.33

0.424

0.367

Set || Anaphoric || Method || MCNN (BASE+SURFSEQ)

0.555

0.566

0.56

0.565

Set || Anaphoric || Method || MCNN (BASE+DEPTREE)

0.389

0.615

0.476

0.518

Set || Anaphoric || Method || MCNN (BASE+SURFSEQ+DEPTREE)

0.503

0.66

0.571

0.599

Set || Anaphoric || Method || MCNN (BASE+SURFSEQ+PREDCONTEXT)

0.535

0.611

0.57

0.581

Set || Anaphoric || Method || MCNN (BASE+DEPTREE+PREDCONTEXT)

0.33

0.699

0.449

0.528

Set || Anaphoric || Method || MCNN (Proposed)

0.492

0.673

0.569

0.602

Set || Cataphoric || Method || single-column CNN (w/ position vec.)

0.163

0.293

0.209

0.163

Set || Cataphoric || Method || MCNN (BASE)

0.171

0.13

0.148

0.099

Set || Cataphoric || Method || MCNN (BASE+SURFSEQ)

0.202

0.417

0.272

0.257

Set || Cataphoric || Method || MCNN (BASE+DEPTREE)

0.268

0.438

0.332

0.329

Set || Cataphoric || Method || MCNN (BASE+SURFSEQ+DEPTREE)

0.195

0.525

0.285

0.33

Set || Cataphoric || Method || MCNN (BASE+SURFSEQ+PREDCONTEXT)

0.258

0.406

0.316

0.276

Set || Cataphoric || Method || MCNN (BASE+DEPTREE+PREDCONTEXT)

0.24

0.488

0.322

0.341

Set || Cataphoric || Method || MCNN (Proposed)

0.251

0.522

0.339

0.337

The results in Table 3 show that our MCNN-based method achieved better average precision than the single-column CNN baseline except the method that uses only the BASE column set for the cataphoric case. The results also demonstrate that each column set consistently contributes to improving the average precision for both the anaphoric and cataphoric cases. However, Table 3 shows that the average precision for the cataphoric set remains low. As one future direction for further improvement, we need to explore clues for identifying cataphoric relations more accurately.
Table 1: METEOR results for different configuration of our model on STARdev, STARtest and CARTOON datasets.

STARdev

STARtest

CARTOON

Model || FULL || -

31.82

29.16

32.08

Model || FULL || -SEM

28.72

25.55

27.55

Model || FULL || -SYN

31.92

29.14

32.04

Table 1 reports METEOR, we notice that removing the semantic coherence scores in -SEM hurts the performance compared to FULL, this confirms our claim that semantic compatibility is crucial for building coherent stories. On the other hand, -SYN performs similarly to FULL. Closer inspection of the -SYN system’s output reveals a greater diversity in thematic elements as a result of the relaxed syntactic compatibility constraints. Hence it is more likely to have greater overlap with any of the reference rewrites, resulting in higher METEOR scores.
Table 2: Accuracy results on the test sets. Bold font marks the best performance for a dataset. * indicates statistical significance of improvement over lasso at p < 0.05 using micro sign test for one of our models LSI, GoW and word2vec (underlined).

no reg. || -

lasso || -

ridge || -

elastic || -

group lasso || LDA

group lasso || LSI

group lasso || sentence

group lasso || GoW

group lasso || word2vec

20NG || dataset || science

0.946

0.916

0.954

0.954

0.968

0.968*

0.942

0.967

0.968*

20NG || dataset || sports

0.908

0.907

0.925

0.92

0.959

0.964*

0.966

0.959*

0.946*

20NG || dataset || religion

0.894

0.876

0.895

0.89

0.918

0.907*

0.934

0.911*

0.916*

20NG || dataset || computer

0.846

0.843

0.869

0.856

0.891

0.885*

0.904

0.885*

0.911*

Sentiment || dataset || vote

0.606

0.643

0.616

0.622

0.658

0.653

0.656

0.640

0.651

Sentiment || dataset || movie

0.865

0.86

0.87

0.875

0.900

0.895

0.895

0.895

0.890

Sentiment || dataset || books

0.75

0.77

0.76

0.78

0.790

0.795

0.785

0.790

0.800

Sentiment || dataset || dvd

0.765

0.735

0.77

0.76

0.800

0.805*

0.785

0.795*

0.795*

Sentiment || dataset || electr.

0.79

0.8

0.8

0.825

0.800

0.815

0.805

0.820

0.815

Sentiment || dataset || kitch.

0.76

0.8

0.775

0.8

0.845

0.860*

0.855

0.840

0.855*

In Table 2 we report the results of our experiments on the aforementioned datasets, and we distinguish our proposed regularizers LSI, GoW, word2vec with underlining. Our results are inline and confirm that of (Yogatama and Smith, 2014a) showing the advantages of using structured regularizers in the text categorization task. The group based regularizers perform systematically better than the baseline ones.
Table 1: Accuracy Results on Wikipedia and Stack Exchange.

Wiki

SE

Model || Bag-of-Words

80.9%

64.6%

Model || Linguistic Features

82.6%

65.2%

Model || With Discovered Features

83.8%

65.7%

Model || CNN

85.8%

66.4%

Table 1 first presents our reproduced classification accuracy test results (two labels: positive or negative politeness) for the bag-of-words and linguistic features based models of Danescu-Niculescu-Mizil et al. (2013) (for our dataset splits) as well as the performance of our CNN model. As seen, without using any manually defined, theory-inspired linguistic features, the simple CNN model performs better than the feature-based methods. Next, we also show how the linguistic features baseline improves on adding our novelly discovered features (plus correcting some exising features), revealed via the analysis in Sec. 6. Thus, this reduces the gap in performance between the linguistic features baseline and the CNN, and in turn provides a quantitative reasoning for the success of the CNN model. More details in Sec. 6.
Table 3: Cross-validation performance on VUAMC. B is always significantly different from N (p < .001), and B ∪ N is always significantly different from both B and N (p < .001).

C

P

R

F

Genre || News || Features || B

1.0

0.475

0.742

0.576

Genre || News || Features || N

1.0

0.576

0.479

0.522

Genre || News || Features || B ∪ N

1.0

0.615

0.539

0.574

Genre || Academic || Features || B

0.6

0.489

0.733

0.568

Genre || Academic || Features || N

0.6

0.572

0.494

0.511

Genre || Academic || Features || B ∪ N

1.0

0.539

0.648

0.569

Genre || Conversation || Features || B

0.6

0.292

0.799

0.416

Genre || Conversation || Features || N

0.6

0.304

0.626

0.393

Genre || Conversation || Features || B ∪ N

1.0

0.299

0.731

0.406

Genre || Fiction || Features || B

0.6

0.349

0.695

0.460

Genre || Fiction || Features || N

0.6

0.430

0.418

0.421

Genre || Fiction || Features || B ∪ N

0.6

0.409

0.551

0.465

Finally, Table 3 shows the effect of the different feature sets on VUAMC used by Klebanov et al. (2014). We use the same 12-fold data split as Klebanov et al. (2014), and also in this case we perform a grid-search to optimize the meta-parameter C of the logistic regression classifier. The best value of C identified for each genre and feature set is shown in the column labeled C. On this data, N features alone are significantly outperformed by B (p < 0.01). On the other hand, for the genres “academic” and “fiction”, combining N and B features improves classification performance over B, and the difference is always statistically significant. Besides, the addition of N always leads to more balanced models, by compensating for the relatively lower precision of B. Due to the lack of a separate test set, as in the original setup by Klebanov et al. (2014), and to the high dimensionality of B’s lexicalized features, we cannot rule out over-fitting as an explanation for the relatively good performance of B on this benchmark. It should also be noted that the results reported in (Klebanov et al., 2014) are not the same, due to the mentioned differences in the implementation of the features and possibly other differences in the experimental setup (e.g., data filtering, pre-processing and meta-parameter optimization). In particular, our implementation of the B features performs better than reported by Klebanov et al. (2014) on all four genres, namely: 0.52 vs. 0.51 for “news”, 0.51 vs. 0.28 for “academic”, 0.39 vs. 0.28 for “conversation” and 0.42 vs. 0.33 for “fiction”.
Table 5: Comparison of various decoders using the same model from our full system (Global A∗). We report F1 with and without the backoff model, the percentage of sentences that the decoder can parse, and the time spent decoding relative to A∗.

Dev F1 || - backoff

Dev F1 || backoff

Relative Time || -

Decoder || Global A*

88.4

88.4 (99.8%)

1X

Decoder || Best-first

87.5

2.8 (6.7%)

293.4X

Decoder || 10-best reranking

87.9

87.9 (99.7%)

8.5X

Decoder || 100-best reranking

88.2

88.0 (99.4%)

72.3X

Decoder || 2-best beam search

88.2

85.7 (94.0%)

2.0X

Decoder || 4-best beam search

88.3

88.1 (99.2%)

6.7X

Decoder || 8-best beam search

88.2

86.8 (98.1%)

26.3X

Lastly, to show that our parser is both more accurate and efficient than other decoding methods, we decode our full model using best-first search, reranking, and beam search. Table 5 shows the F1 scores with and without the backoff model, the portion of the sentences that each decoder is able to parse, and the time spent decoding relative to the A* parser. In the best-first search comparison, we do not include the informative A* heuristic, and the parser completes very few parses before reaching computational limitsÑshowing the importance of heuristics in large search spaces. In the reranking comparison, we obtain n-best lists from the backoff model and rerank each result with the full model. In the beam search comparison, we use the approach from Clark et al. (2015) which greedily finds the top-n parses for each span in a bottom-up manner. Results indicate that both approximate methods are less accurate and slower than A*.
Table 3: SPE: Prediction Results

Tobacco

Alcohol

Drug

Methods || Unigram

0.663

0.672

0.644

Methods || LIWC

0.731

0.689

0.758

Methods || SVD

0.779

0.724

0.764

Methods || UserLDA

0.641

0.603

0.599

Methods || PostLDA_Word

0.733

0.617

0.628

Methods || PostLDA_Doc

0.768

0.687

0.721

Methods || Post-D-DM

0.536

0.622

0.52

Methods || User-D-DM

0.775

0.73

0.767

Methods || Post-D-DBOW

0.531

0.606

0.526

Methods || User-D-DBOW

0.802

0.768

0.819

We evaluate the performance of our models using 10-fold cross validation. The evaluation results shown in Table 3 are based on weighted ROC AUC of the best models. Among all the feature learning methods for Facebook status updates, User-D-DBOW performed the best. It significantly outperformed all the baseline systems that only rely on supervised training (p < 0.01 based on t-tests). It also significantly outperformed all the traditional feature learning methods such as LDA and SVD (p < 0.01 based on t-tests). Moreover, in terms of whether to treat all the posts by the same user as one big document or separate documents, LDA prefers one post one document (models with a "post" prefix) while all the document vector-based methods prefer one user one document (models with a User prefix). Moreover, to use post-level LDA to derive the SPE of a user, the document-based aggregation method (PostLDA_Doc) performed better than the wordbased method (PostLDA_Word).
Table 7: Results obtained by the best model on each category of the test set for the factual vs opinion argument classification task

P

R

F1

#argument per category

Category || fact

0.49

0.5

0.5

138

Category || opinion

0.88

0.87

0.88

575

Category || avg/total

0.81

0.79

0.8

713

Table 7 reports on the results obtained by the best configuration, i.e. LR + All features, per each category.
Table 4: Results for image-sentence ranking experiments on the COCO dataset. R@K denotes Recall@K (higher is better) and Med r is the median rank (lower is better). (†) taken from Kiros et al. (2015). (∗) taken from Karpathy and Fei-Fei (2015). (‡) taken from Mao et al. (2015).

R@1

Med r

R@2

Med r

Method || uni-skip

30.6

3

22.7

4

Method || bi-skip

32.7

3

24.2

4

Method || combine-skip

33.8

3

25.9

4

Our Results || hierarchical model+emb.

32.7

3

25.3

4

Our Results || composite model+emb.

33.8

3

25.7

4

Our Results || combine+emb.

34.4

3

26.6

4

Task-dependent methods || DVSA*

38.4

1

27.4

3

Task-dependent methods || m-RNN

41

2

29

3

Table 4 shows our results. Consistent with previous experiments, we empirically found that the encoder trained using the fixed word embedding performed better on this task, hence only results using this method are reported. As can be seen, we obtain the same median rank as in Kiros et al.(2015), indicating that our encoder is as competitive as the skip-thought vectors (Kiros et al., 2015). The performance gain between our encoder and the combine-skip model of Kiros et al.(2015) on the R@1 score is significant, which shows that the CNN encoder has more discriminative power on retrieving the most correct item than the skip-thought vector.
Table 4: Cognate clustering results on the Algonquian dataset (in %) with subsets of features.

Found Sets

Purity

Features || Phonetic only

52.0 (36.3)

70.2

Features || + Definitions

57.4 (41.7)

68.4

Features || + WordNet

61.9 (46.9)

68.1

Features || + Word Vectors

66.2 (51.3)

66.5

Table 4 shows the results. The phonetic features alone are sufficient to detect just over half of the cognate sets. Each successive variant substantially improves the recall at a cost of slightly lower precision. The full feature set yields a 27% relative increase in the number of found sets over the phonetic-only variant, with only a 5% drop in cluster purity.
Table 1: Results on NIST Chinese-to-English Translation Task. “*” indicates statistically significant better than “Sennrich-deponly” at p-value < 0.05 and “**” at p-value < 0.01. AVG = average BLEU scores for test sets.

Dev (MT02)

MT03

MT04

MT05

MT06

MT08

AVG

System || PBSMT

33.15

31.02

33.78

30.33

29.62

23.53

29.66

System || AttNMT

36.31

34.02

37.11

32.86

32.54

25.44

32.4

System || Sennrich-deponly

36.68

34.51

38.09

33.37

32.96

26.96

32.98

System || SDRNMT-1

36.88

34.98*

38.14

34.61

33.58

27.06

33.32

System || SDRNMT-2

37.34

35.91**

38.73*

34.18**

33.76**

27.64*

34.04

Table 1 shows the translation performances on test sets measured in BLEU score. The AttNMT significantly outperforms PBSMT by 2.74 BLEU points on average, indicating that it is a strong baseline NMT system. The baseline Sennrichdeponly improves the performance over the AttNMT by 0.58 BLEU points on average. This indicates that the proposed source dependency constraint is beneficial for improving the performance of NMT. Moreover, SDRNMT-1 gains improvements of 0.92 and 0.34 BLEU points on average than the AttNMT and Sennrich-deponly. These show that the proposed SDR can more effectively capture source dependency information than vector concatenation. Especially, the proposed SDRNMT2 outperforms the AttNMT and Sennrich-deponly on average by 1.64 and 1.03 BLEU points. These verify that the proposed double-context method is effective for word prediction.
Table 4: Segmentation results. Explicit bigrams are useful.

Accuracy

Size

Model || Zhang et al. (2016)

95.01

−

Model || Zhang et al. (2016)-combo

95.95

−

Model || Small FF 64 dim

94.24

846KB

Model || Small FF 256 dim

94.16

3.2MB

Model || Small FF 64 dim bigrams

95.18

2.0MB

The results in Table 4 show that spending our memory budget on small bigram embeddings is more effective than on larger character embeddings, in terms of both accuracy and model size. Our model featuring bigrams runs at 110KB of text per second, or 39k tokens/second.
Table 5: Performance on monolingual word similarity computation with seed lexicon size 6000.

Chinese (source) || WS-240

Chinese (source) || WS-297

English (target) || WS-353

English (target) || SL-999

Method || BiLex

60.36

62.17

60.46

27.22

Method || CLSP-WR

61.27

65.25

60.46

27.22

Method || CLSP-SE

60.84

65.62

62.47

28.79

Table 5 shows the results of monolingual word similarity computation on four datasets. From the table, we find that:. (1) Our models perform better than BiLex on both Chinese word similarity datasets. It signifies incorporating sememe information helps learn better monolingual embeddings;. (2) CLSP-WR model does not enhance English word similarity results but CLSPSE model does. It is because CLSP-WR model only post-processes Chinese word embeddings and keeps English word embeddings unchanged while CLSP-SE model undertakes bilingual alignment and sememe information incorporation together, which makes English word embeddings improve with Chinese word embeddings.
Table 4: Comparison with Transformer on FrenchEnglish translation task. The evaluation metric is caseinsensitive BLEU score.

Dev

Test

Method || Transformer

29.42

35.15

Method || this work

30.40

36.04

Table 4 shows that our model also outperforms Transformer by 0.89 BLEU points on French-English translation task.
Table 2: Performance of various models on NarrativeQA dataset (SS=sample size ≡ number of relevant sentences)

BLEU-1

BLEU-4

ROUGE-L

METEOR

Model || Seq2Seq

15.89

1.26

13.15

4.08

Model || ASR

23.20

6.39

22.26

7.77

Model || BiDAF

33.72

15.53

36.30

15.38

Model || MRU (Tay et al. 2018)

36.55

19.79

41.44

17.87

Model || Baseline 1 (SS=5)

30.22

14.43

34.40

13.36

Model || Baseline 2 (SS=5)

39.35

20.17

43.36

18.01

Model || ConZNet (SS=1)

28.97

16.70

36.02

12.39

Model || ConZNet (SS=3)

36.21

19.33

41.23

17.77

Model || ConZNet (SS=5)

42.76

22.49

46.67

19.24

Model || ConZNet (SS=7)

40.80

21.14

44.01

18.67

3.4 Results. Table 2 shows the performance of various models on NarrativeQA. It can be noted that our model with sample size 5 (choosing 5 relevant sentences) outperforms the best ROUGE-L score available so far by 12.62% compared to Tay et al. (2018). The low performance of Baseline 1 shows that the hybrid approach (ConZNet) for generating words from a fixed vocabulary as well as copying words from the document is better suited than span prediction models (Seq2Seq, ASR, BiDAF, MRU). To validate the importance of finding relevant sentences in contrast to using an entire document for answer generation, we experimented with sample sizes beyond 5. The performance of our model gradually dropped from sample size 7 onwards. This result shows evidence that only a few relevant sentences are sufficient to answer a question. We also experimented with various sample sizes to see the effect of intra sentence relations for answer generation. The performance of the model improved dramatically with sample sizes 3 and 5 compared to the sample size of 1. These results show that the importance of selecting multiple relevant sentences for generating an answer. In addition, the low performance of Baseline 2 indicates that just selecting multiple sentences is not enough, they should also be related to each other. This result points out that the self-attention mechanism in the Context zoom layer is an important component to identify related relevant sentences.