katebor/TableEval huggingface dataset:
Table 2: Results with traditional machine learning (Label Powerset)

Features || Word n-grams || FI

Features || Word n-grams || Fmacro

Features || Word n-grams || AccI

Features || Word n-grams || Fmicro

Features || Character n-grams || FI

Features || Character n-grams || Fmacro

Features || Character n-grams || AccI

Features || Character n-grams || Fmicro

Features || Averaged ELMo vectors || FI

Features || Averaged ELMo vectors || Fmacro

Features || Averaged ELMo vectors || AccI

Features || Averaged ELMo vectors || Fmicro

Features || Composite features || FI

Features || Composite features || Fmacro

Features || Composite features || AccI

Features || Composite features || Fmicro

Classifier || SVM

0.448

0.373

0.324

0.410

0.449

0.374

0.331

0.416

0.546

0.430

0.431

0.500

0.178

0.094

0.116

0.174

Classifier || LR

0.357

0.315

0.236

0.349

0.357

0.311

0.230

0.352

0.595

0.479

0.478

0.549

0.438

0.370

0.311

0.421

Classifier || RF

0.531

0.398

0.438

0.476

0.395

0.205

0.325

0.349

0.375

0.164

0.305

0.331

0.460

0.311

0.380

0.415

5.3 Results. Table 2 shows results produced using traditional ML methods (SVM, RF, and LR) across four different feature sets (word n-grams, character ngrams, averaged ELMo vectors, and composite features). We use Label Powerset for these methods, since the direct (non-transformative) formulation cannot be used with them. Among these combinations, logistic regression with averaged ELMo embeddings as features performs the best.
Table 3: Results of performance (AUROC) of up-to-date methods on identifying helpful reviews evaluated by the test sets of Amazon. (italic f onts∗: the best performance among the baseline approaches; bold fonts: the state-of-the-art performance of all the approaches)

Area under Receiver Operating Characteristic (AUROC) || STR

Area under Receiver Operating Characteristic (AUROC) || LEX

Area under Receiver Operating Characteristic (AUROC) || GALC

Area under Receiver Operating Characteristic (AUROC) || INQUIRER

Area under Receiver Operating Characteristic (AUROC) || FUSION (SVM)

Area under Receiver Operating Characteristic (AUROC) || FUSION (R.F.)

Area under Receiver Operating Characteristic (AUROC) || EG-CNN

Area under Receiver Operating Characteristic (AUROC) || MTNL

Area under Receiver Operating Characteristic (AUROC) || R2HP

Category (Amazon) || Clothing, Shoes & Jewelry

0.548

0.536

0.562

0.601*

0.579

0.55

0.583

0.589

0.623 (+0.022)

Category (Amazon) || Electronics

0.583

0.549

0.588

0.616*

0.577

0.58

0.611

0.613

0.661 (+0.045)

Category (Amazon) || Grocery & Gourmet Food

0.536

0.526

0.553

0.602

0.532

0.546

0.611

0.626*

0.657 (+0.031)

Category (Amazon) || Health & Personal Care

0.558

0.523

0.559

0.61

0.591

0.562

0.613

0.620*

0.683 (+0.063)

Category (Amazon) || Home & Kitchen

0.568

0.537

0.565

0.597

0.569

0.573

0.603

0.610*

0.646 (+0.036)

Category (Amazon) || Movies & TV

0.603

0.558

0.621

0.634

0.603

0.607

0.648

0.652*

0.713 (+0.061)

Category (Amazon) || Pet Supplies

0.56

0.542

0.585

0.603

0.548

0.558

0.58

0.629*

0.692 (+0.063)

Category (Amazon) || Tools & Home Improvement

0.584

0.558

0.58

0.592

0.575

0.586

0.617

0.624*

0.672 (+0.048)

Category (Amazon) || MACRO AVERAGE

0.568

0.541

0.577

0.607

0.572

0.57

0.608

0.620*

0.668 (+0.048)

Category (Amazon) || MICRO AVERAGE (Primary)

0.571

0.543

0.58

0.609

0.573

0.574

0.614

0.625*

0.675 (+0.049)

Identification of helpful reviews: As both the training and test sets are imbalanced, we adopt the Area under Receiver Operating Characteristic (AUROC) as the metric to evaluate the performance of all the approaches on helpful review identification. In line with Table 3, MTNL (Fan et al., 2018) achieves the best performance up-to-date on this classification task among the baseline approaches as it achieves the best performance on 12 of 14 categories in Amazon and Yelp datasets. R2HP surpasses MTNL on both datasets and obtains state-of-the-art (microaveraged) results of 67.5% AUROC (Amazon) and 75.1% AUROC (Yelp) with absolute improvements of 4.9% AUROC and 4.7% AUROC, respectively.
Table 3: Results for next utterance retrieval on the Ubuntu dialog corpus. This table shows previous work, and experimental results with two underlying architectures: a dual encoder model and Deep Attention Matching networks. The results shown in the DAM experiments section are performed with the open-sourced implementation of Zhou et al. (2018), which obtains slightly worse performance than they report. All bold-face results are statistically significant to p < 0.01.

MRR

R10@1

R2@1

Model Name || Previous Research || Dual Encoder (Lowe et al., 2015)

-

63.8

90.1

Model Name || Previous Research || MV-LSTM (Pang et al., 2016)

-

65.3

90.6

Model Name || Previous Research || Match-LSTM (Wang and Jiang, 2016)

65.3

90.4

Model Name || Previous Research || Multiview (Zhou et al., 2016)

-

66.2

90.8

Model Name || Previous Research || DL2R (Yan et al., 2016)

-

62.6

89.9

Model Name || Previous Research || SMN (Wu et al., 2016)

-

72.6

92.6

Model Name || Previous Research || DAM (Zhou et al., 2018)

-

76.7

93.8

Model Name || Dual Encoder Experiments || Dual Encoder (Lowe et al., 2015)

76.84

63.6

90.9

Model Name || Dual Encoder Experiments || Ensemble (5)

78.91

66.9

91.7

Model Name || Dual Encoder Experiments || Multi-Granularity (5)

80.10

68.7

91.9

Model Name || Deep Attention Matching Experiments || DAM (Zhou et al., 2018) (re-trained)

83.74

74.54

93.08

Model Name || Deep Attention Matching Experiments || Ensemble (5)

84.03

74.95

93.27

Model Name || Deep Attention Matching Experiments || Multi-Granularity (5)

84.26

75.3

93.45

MGT is applied on top of the dual encoder baseline (Lowe et al., 2015) and Deep Attention Matching networks (Zhou et al., 2018). The results shown in Table 3 show the performance of MGT using two different underlying architectures, as well as previous work. Across both base architectures, MGT outperforms ensembling. The primary difference between these two methods is that MGT explicitly ensures that several granularities of representation are learned. As such, these results reaffirm the hypothesis that learning multiple granularities of representation leads to more diverse models, and more general representations of dialog.
Table 4: Nested and overlapping event detection F1 (%) score performance on the CG task 2013 development set.

Nested

Overlapping

Flat

Overall F1 (%)

Model || TEES

42.7

34.49

56.81

52.16

Model || SBNN k = 8

45.24

36.92

60.5

54.36

Table 4 shows the performance comparison of the models on nested, overlapping and flat event detection. Our model yields higher F1-scores than TEES which can be attributed to its ability to maintain multiple beams and to detect events from all these beams during search.
Table 4: Performance comparison on the OntoNotes 5.0 Chinese Dataset.

Prec.

Rec.

F1

Model || Model || Pradhan et al. (2013)

78.2

66.45

71.85

Model || Model || Lattice LSTM (Z&Y, 2018)

76.34

77.01

76.67

Model || Model || BiLSTM-CRF (L = 0)

76.67

67.79

71.95

Model || Model || BiLSTM-CRF (L = 1)

78.45

74.59

76.47

Model || Model || BiLSTM-CRF (L = 2)

77.94

75.33

76.61

Model || Model || BiLSTM-CRF (L = 3)

76.17

75.23

75.7

Model || Model || BiLSTM-GCN-CRF

76.35

75.89

76.12

Model || Model || DGLSTM-CRF (L = 0)

76.91

70.65

73.65

Model || Model || DGLSTM-CRF (L = 1)

77.79

75.29

76.52

Model || Model || DGLSTM-CRF (L = 2)

77.4

77.41

77.4

Model || Model || DGLSTM-CRF (L = 3)

77.01

74.9

75.94

Model || Contextualized Word Representation || BiLSTM-CRF (L = 0) + ELMo

75.2

73.39

74.28

Model || Contextualized Word Representation || BiLSTM-CRF (L = 1) + ELMo

79.2

79.21

79.2

Model || Contextualized Word Representation || BiLSTM-CRF(L = 2) + ELMo

78.49

79.44

78.96

Model || Contextualized Word Representation || BiLSTM-CRF (L = 3) + ELMo

78.54

79.76

79.14

Model || Contextualized Word Representation || BiLSTM-GCN-CRF + ELMo

78.71

79.29

79

Model || Contextualized Word Representation || DGLSTM-CRF (L = 0) + ELMo

76.27

74.61

75.43

Model || Contextualized Word Representation || DGLSTM-CRF (L = 1) + ELMo

78.91

80.22

79.56

Model || Contextualized Word Representation || DGLSTM-CRF (L = 2) + ELMo

78.86

81

79.92

Model || Contextualized Word Representation || DGLSTM-CRF (L = 3) + ELMo

79.3

79.86

79.58

OntoNotes Chinese Table 4 shows the performance comparison on the Chinese datasets. We compare our models against the state-of-the-art NER model on this dataset, Lattice LSTM (Zhang and Yang, 2018). Our implementation of the strong BiLSTM-CRF baseline achieves comparable performance against the Lattice LSTM. Similar to the English dataset, our model with L = 0 significantly improves the performance compared to the BiLSTM-CRF (L = 0) model. Our DGLSTM-CRF model achieves the best performance with L = 2 and is consistently better (p < 0.02) than the strong BiLSTM-CRF baselines. As we can see from the table, the improvements of the DGLSTM-CRF model mainly come from recall (p < 0.001) compared to the BiLSTM model, especially in the scenario with word embeddings only. Empirically, we also found that those correctly retrieved entities of the DGLSTM-CRF (compared against the baseline) mostly correlate with the following dependency relations: “nn”, “nsubj”, “nummod”. However, DGLSTM-CRF achieves lower precisions against BiLSTM-CRF, which indicates that the DGLSTM-CRF model makes more false-positive predictions. The reason could be the relatively lower ratio of ST(%) as shown in Table 2, which means some of the entities do not form subtrees under the complete dependency trees. In such a scenario, the model may not correctly identify the boundary of the entities, which results in lower precision.
Table 2: Comparison of mean accuracy (%) on ARSC

Mean Acc

Model || Matching Networks (Vinyals et al., 2016)

65.73

Model || Prototypical Networks (Snell et al., 2017)

68.17

Model || Graph Network (Garcia and Bruna, 2017)

82.61

Model || Relation Network (Sung et al., 2018)

83.07

Model || SNAIL (Mishra et al., 2018)

82.57

Model || ROBUSTTC-FSL (Yu et al., 2018)

83.12

Model || Induction Networks (ours)

85.63

5.3 Experiment Results Overall Performance Experiment results on ARSC are presented in Table 2. The proposed Induction Networks achieves a 85.63% accuracy, outperforming the existing state-of-the-art model, ROBUSTTC-FSL, by a notable 3% improvement. We due the improvement to the fact that ROBUSTTC-FSL builds a general metric method by integrating several metrics at the sample level, which faces the difficulty of getting rid of the noise among different expressions in the same class. In addition to that, the task-clustering-based method used by ROBUSTTC-FSL must be found on the relevance matrix, which is inefficient when applied to real-world scenarios where the tasks change rapidly. Our Induction Networks, however, is trained in the meta-learning framework with more flexible generalization and its induction ability can hence be accumulated through different tasks. Ablation Study To analyze the effect of varying different components of the Induction Module and Relation Module, we further report the ablation experiments on the ARSC dataset as shown in Table 4. We can see that the best performance is achieved when we used 3 iterations, corresponding to the best result reported in Table 2 (more rounds of iterations did not further improve the performance), and the table shows the effectiveness of the routing component. We also changed the Induction Module with sum and self-attention and changed Relation Module with cosine distance. Changes in the performances validate the benefit of both the Relation Module and Induction Module. The Attention+Relation models the induction ability by self-attention mechanism, but the ability is limited by the learnt attention parameters. Conversely, the proposed dynamic routing induction method captures class-level information by automatically adjusting the coupling coefficients according to inputted support sets, which is more suitable for the few-shot learning task.
Table 2: Results of bilingual lexicon induction (accuracy % P@1) for similar and distant language pairs on the dataset BLI-1. Procrustes and MUSE represent the supervised and unsupervised model of Lample et al. (2018), VecMap is the supervised model of Artetxe et al. (2017). Word translations are retrieved by using CSLS. Bold face indicates the best result overall and italics indicate the best result between the two columns without refinement.

Supervised || VecMap

Supervised || Procrustes

Without refinement || MUSE

Without refinement || MUSEMethod

With refinement || Our

With refinement || Method

Similar language pairs || de-en

72.5

72

55.3

66.3

72.2

72.4

Similar language pairs || en-de

72.5

72.1

59.2

68.2

72.5

73.4

Similar language pairs || es-en

83.4

82.9

78.1

78

83.1

83.6

Similar language pairs || en-es

81.9

81.5

75.5

74.9

81.1

80.9

Similar language pairs || fr-en

81.5

82.2

72.2

75.6

81.2

83.3

Similar language pairs || en-fr

81.3

81.3

77.8

78.3

82.2

82.2

Similar language pairs || it-en

77.7

79.1

61

63.3

76.4

76.2

Similar language pairs || en-it

77.2

77.5

63.3

64.3

77.4

78.2

Metric || avg.

78.5

78.6

67.8

71.1

78.3

78.8

Distant language pairs || fi-en

55.9

56.2

0

42.8

28.6

53.5

Distant language pairs || en-fi

39.5

39.3

0

35.2

21

43

Distant language pairs || ru-en

60.5

61.3

43.8

54.3

49.9

58.9

Distant language pairs || en-ru

48

50.9

28.7

43.2

37.6

51.1

Distant language pairs || tr-en

55.7

58

12.2

42.1

25.5

54.7

Distant language pairs || en-tr

37.3

37.1

26.2

33.1

39.4

41.4

Distant language pairs || zh-en

45

41.2

26.9

32.1

30.8

42

Distant language pairs || en-zh

45.4

52.1

29.4

37.7

33

48

Metric || avg.

48.4

49.5

20.9

40.1

33.2

49.1

Comparison with unsupervised GANs . As we have mentioned before, the preliminary mappings trained by the method of Lample et al. (2018) perform well for some similar language pairs, such as Spanish to English and French to English. After refinement, their unsupervised GANs models can reach the same level as supervised models for these similar language pairs. However, the biggest drawback of standard GANs is that they exhibit poor performance for distant language pairs. The results from Table 2 clearly confirm this. For example, without refinement, the mapping trained by the unsupervised GAN method can only correctly predict 12% of the words from Turkish to English. Given that the quality of preliminary mappings can seriously affect the effect of refinement, the low-quality preliminary mappings for distant language pairs severely limits the improvements brought by post-refinement. Notice that the method of Lample et al. (2018) scores a null result for English to Finnish on both BLI-1 and BLI2, indicating that totally unsupervised adversarial training can yields rather unpredictable results.
Table 1: SEQ vs. CAMB system results on CWI

Macro || CAMB

F-Score || SEQ

Test Set || NEWS

0.8633

0.8763

Test Set || WIKINEWS

0.8317

0.854

Test Set || WIKIPEDIA

0.778

0.814

SEQ uses 300-dimensional GloVe embeddings as word representations (Pennington et al., 2014). The model is trained to predict the binary complexity of words as annotated in the dataset of Yimam et al. (2017). Training is performed over 20 iterations on randomly shuffled sentences from all genres included within the dataset. To test this novel architecture on CWI, we apply it to the CWI 2018 shared task test data (Yimam et al., 2018) and compare the results to the current state-of-theart (SOTA) CAMB system. Table 1 shows that the SEQ model outperforms the current SOTA system on all three text genres for binary CWI (statistically significant using McNemar's test, p=0.0016, chi-square=9.95).
Table 3: Performance comparisons of different methods for citation count prediction using two datasets. “↑" ( “↓") indicates that a larger (smaller) value corresponds to a better performance.

MAE

RMSE

OR(@30)

OR(@50)

Spearman Rank

Datasets || NIPS || LR

0.1776

0.1903

0.27

0.33

0.4776

Datasets || NIPS || KNN

0.1701

0.19

0.33

0.36

0.4848

Datasets || NIPS || SVR

0.1677

0.1856

0.33

0.4

0.5279

Datasets || NIPS || GBRT

0.1863

0.1974

0.23

0.34

0.531

Datasets || NIPS || Wide&Deep

0.147

0.1848

0.3

0.38

0.5351

Datasets || NIPS || MILAM

0.1426

0.1792

0.37

0.38

0.5458

Datasets || NIPS || Our model

0.1349

0.1726

0.4

0.42

0.5561

Datasets || ICLR || LR

0.2395

0.2723

0.4

0.7

0.1475

Datasets || ICLR || KNN

0.2293

0.2674

0.4

0.72

0.1874

Datasets || ICLR || SVR

0.2226

0.2578

0.4

0.7

0.1328

Datasets || ICLR || GBRT

0.2223

0.2607

0.43

0.7

0.1469

Datasets || ICLR || Wide&Deep

0.2182

0.2607

0.47

0.72

0.244

Datasets || ICLR || MILAM

0.2093

0.251

0.47

0.72

0.251

Datasets || ICLR || Our model

0.1866

0.2279

0.5

0.76

0.3026

Main Results. Table 3 presents the performance of different methods on citation count prediction. We can make the following observations. First, the four traditional baselines (LR, KNN, SVR and GBRT) perform worse than the two deep learning baselines (W&D, MILAM). These four baselines only utilize the wide features with traditional machine learning models. Second, MILAM performs consistently better than W&D, since it has designed a more elaborate architecture to model the review text. Finally, our model outperforms all the baselines with a substantial margin, especially for the ICLR dataset. Our model is able to integrate the wide features and learn the comprehensive representation for peer review text, which is the key of the performance the two improvement over baselines. Overall, datasets show the similar findings. In what follows, we will report the results on ICLR dataset due to space limit.
Table 1: Results of completion generation. We group MPC, character language model baseline, and two subword language models separately. +R implies the retrace algorithm. +M implies reranking with approximate marginalization. QPS stands for query per seconds. The higher the QPS, the better. The best results for each column related to accuracy are shown in bold for each segmentation algorithm (BPE and SR). SR model shows higher unseen PMRR scores (underlined). Our models are faster than the character baseline.

MRR Seen

MRR Unseen

MRR All

PMRR Seen

PMRR Unseen

PMRR All

MRL Seen

MRL Unseen

MRL All

Execution Speed (QPS) CPU

Execution Speed (QPS) GPU

Decode Length

Model || MPC

.570

.000

.290

.616

.095

.360

8.06

0.00

4.10

>100

>100

N/A

Model || Char

.458

.160

.311

.552

.372

.464

5.77

4.24

5.02

11.0 (1.0x)

16.5 (1.0x)

14.5

Model || BPE

.242

.085

.164

.305

.232

.269

0.49

0.54

0.51

24.2 (2.2x)

37.4 (2.3x)

7.1

Model || BPE+R1

.427

.156

.294

.517

.368

.444

5.28

3.98

4.64

15.8 (1.4x)

27.3 (1.7x)

11.8

Model || BPE+R2

.430

.157

.296

.520

.369

.446

5.44

4.01

4.74

15.5 (1.4x)

27.2 (1.6x)

12.2

Model || BPE+R∞

.431

.157

.296

.520

.369

.446

5.50

4.01

4.76

15.3 (1.4x)

26.9 (1.6x)

12.2

Model || SR

.422

.148

.288

.541

.379

.461

5.11

3.82

4.48

20.8 (1.9x)

40.1 (2.4x)

6.8

Model || SR+M

.424

.149

.289

.535

.373

.455

5.14

3.85

4.50

19.6 (1.8x)

40.0 (2.4x)

6.8

Model || SR+R∞

.423

.148

.289

.541

.378

.461

5.14

3.83

4.50

16.3 (1.5x)

29.6 (1.8x)

7.4

Model || SR+R∞+M

.427

.150

.291

.538

.375

.458

5.19

3.88

4.54

16.2 (1.5x)

28.7 (1.7x)

7.4

6.3 Decoding Results . We performed comprehensive experiments to analyze the performance of query auto-completion. Table 1 shows the generation result of MPC, the character baseline, and our model variants. For BPE models, we varied the maximum retrace step to 0 (without retrace algorithm), 1, 2, and âˆž (no limitation on retracing step size). For SR models, we compare decoding results without any techniques, with marginalization only, with retrace algorithm only, and with both. MPC is a very fast and remarkably strong baseline. It is worse than language models in the overall score (MRR, PMRR, and MRL), but better for previously seen queries. However, it is unable to predict unseen queries. Even with efficient data structures, MPC requires huge memory to keep statistics of all previous queries. As a practical view, combining frequency-based traditional method and neural language model approach can boost the accuracy and meet trade-off between the performance and computational costs. MRRs and PMRRs of our best methods are close to that of the character model with less than 0.02 point drop. Notably, the SR model has better generalization ability in that their PMRR for unseen queries is higher than that of the character model. In a real scenario, it is more critical because unseen queries come in increasingly as time goes by.
Table 4: Evaluation results on automatic metrics and human judgment. Human evaluation results are calculated by combining labels from the three judges. “Kappa” means Fleiss’ kappa. Numbers in bold mean that improvement over the best baseline is statistically significant.

METEOR

W-METEOR

Rouge_L

W-Rouge_L

CIDEr

W-CIDEr

BLEU-1

W-BLEU-1

Human

Kappa

Dataset || Tencent || Models || IR-T

0.107

0.086

0.254

0.217

0.018

0.014

0.495

0.470

2.43

0.64

Dataset || Tencent || Models || IR-TC

0.127

0.101

0.266

0.225

0.056

0.044

0.474

0.436

2.57

0.71

Dataset || Tencent || Models || Seq2seq

0.064

0.047

0.196

0.150

0.011

0.008

0.374

0.320

1.68

0.83

Dataset || Tencent || Models || Att

0.080

0.058

0.246

0.186

0.010

0.007

0.481

0.453

1.81

0.79

Dataset || Tencent || Models || Att-TC

0.114

0.082

0.299

0.223

0.023

0.017

0.602

0.551

2.26

0.69

Dataset || Tencent || Models || GANN

0.097

0.075

0.282

0.222

0.010

0.008

0.312

0.278

2.06

0.73

Dataset || Tencent || Models || DeepCom

0.181

0.138

0.317

0.250

0.029

0.023

0.721

0.656

3.58

0.65

Dataset || Yahoo || Models || IR-T

0.114

-

0.214

-

0.014

-

0.472

-

2.71

0.67

Dataset || Yahoo || Models || IR-TC

0.117

-

0.219

-

0.017

-

0.483

-

2.86

0.61

Dataset || Yahoo || Models || Seq2seq

0.061

-

0.203

-

0.011

-

0.365

-

2.26

0.68

Dataset || Yahoo || Models || Att

0.075

-

0.217

-

0.017

-

0.462

-

2.29

0.78

Dataset || Yahoo || Models || Att-TC

0.089

-

0.246

-

0.022

-

0.515

-

2.74

0.63

Dataset || Yahoo || Models || GANN

0.079

-

0.228

-

0.019

-

0.496

-

2.52

0.64

Dataset || Yahoo || Models || DeepCom

0.107

-

0.263

-

0.024

-

0.665

-

3.35

0.68

4.4 Evaluation Results . Table 4 reports evaluation results in terms of both automatic metrics and human annotations. On most automatic metrics, DeepCom outperforms baseline methods, and the improvement is statistically significant (t-test with p-value < 0.01). The improvement on BLEU-1 and W-BLEU-1 is much bigger than that on other metrics. This is because BLEU-1 only measures the proportion of matched unigrams out of the total number of unigrams in the generated comments. In human evaluation, although the absolute numbers are different from those reported in (Qin et al., 2018) due to the difference between human judgements, the overall trend is consistent. In human evaluation, the values of Fleiss' kappa over all models are more than 0:6, indicating substantial agreement among the annotators. Although built in a complicated structure, GANN does not bring much improvement over other baseline methods, which demonstrates that only using news titles is not enough in comment generation. IR-TC and Att-TC represent the best retrieval model and the best generation model among the baselines on both datasets, implying that news bodies, even used in a simple way, can provide useful information to comment generation.