katebor/TableEval huggingface dataset:
Table 2: Translation performance on one-to-two, one-to-three and one-to-four translation tasks. Rep denotes our proposed representor. Emb, Attn, and Dis represent our proposed language-sensitive methods to address multilingual translation. Note that the source language of all our experiments is English.

NMT Baselines

Multi-NMT Baselines Johnson et al.(2017)

Three-Stgy Wang et al.(2018)

Rep+Emb

Rep+Emb+Attn

Rep+Emb+Attn+Dis

One-to-Two || De

27.5

27.26

27.35

26.6

26.96

27.74

One-to-Two || Lv

16.28

16.32

16.38

15.37

15.87

16.79

One-to-Two || De

27.5

27.88

27.89

26.96

27.32

27.96

One-to-Two || Fi

16.83

16.47

16.7

15.78

16.58

16.89

One-to-Two || De

27.5

26.8

26.99

26.08

26.68

27.45

One-to-Two || Zh

26.04

25.54

25.78

24.48

25.33

26.17

One-to-Three || De

27.5

25.44

25.55

24.82

25.45

26.06

One-to-Three || Zh

26.04

24.87

25.63

24.12

24.93

26.12

One-to-Three || Fi

16.83

16.86

16.97

16.06

16.78

17.12

One-to-Three || De

27.5

25.98

26.12

24.88

25.8

26.42

One-to-Three || Lv

16.28

14.88

15.44

14.51

15.58

16.31

One-to-Three || Fi

16.83

16.94

17.05

16.15

16.79

17.22

One-to-Four || De

27.5

23.59

22.88

22.88

23.58

24.08

One-to-Four || Lv

16.28

15.57

16.02

15

16.21

16.57

One-to-Four || Zh

26.04

25.24

25.83

24.15

25.27

26.29

One-to-Four || Fi

16.83

13.45

14.12

12.99

14.11

15.03

The main results on the one-to-many translation scenario, including one-to-two, one-to-three and one-to-four translation tasks are reported in Table 2. We present a typical Multi-NMT adopting Johnson et al. (2017) method on Transformer as our Multi-NMT baselines model. Obviously, Multi-NMT Baselines cannot outperform NMT Baselines in all cases, among which four directions are comparable and twelve are worse. With respect to our proposed method, it is clear that our compact method consistently outperforms the baseline systems. Compared with another strong one-to-many translation model Three-Stgy proposed by Wang et al. (2018), our compact method can achieve better results as well. Moreover, our method can perform even better than individually trained systems in most cases (eleven out of sixteen cases). The results demonstrate the effectiveness of our method.
Table 1: Precision evaluated automatically for the top rated relation instances. † marks results reported in the original paper. ‡ marks our results using the OpenNRE implementation.

AUC

P@100

P@200

P@300

P@500

P@1000

P@2000

System || Mintz

0.107

52.3

50.2

45

39.7

33.6

23.4

System || PCNN+ATT

0.341

73

68

67.3

63.6

53.3

40

System || RESIDE

0.415

81.8

75.4

74.3

69.7

59.3

45

System || DISTRE

0.422

68

67

65.3

65

60.2

47.9

5.1 Held-out Evaluation. DISTRE with selective attention achieves a new state-of-the-art AUC value of 0.422. The precision-recall curve in Figure 4 shows that it outperforms RESIDE and PCNN+ATT at higher recall levels, while precision is lower for top predicted relation instances. The results of the PCNN+ATT model indicate that its performance is only better in the very beginning of the curve, but its precision drops early and only achieves an AUC value of 0.341. Similar, RESIDE performs better in the beginning but drops in precision after a recall-level of approximately 0.25. This suggests that our method yields a more balanced overall performance, which we believe is important in many real-world applications. Table 1 also shows detailed precision values measured at different points along the P-R curve. We again can observe that while DISTRE has lower precision for the top 500 predicted relation instances, it shows a state-of-the-art precision of 60.2% for the top 1000 and continues to perform higher for the remaining, much larger part of the predictions.
Table 2: Comparisons without post-processing of methods. Results reproduced from (Smith et al., 2017) for fair comparison. Left: Comparisons using the same expert dictionary as (Smith et al., 2017). Right: Comparisons using the pseudo-dictionary from (Smith et al., 2017).

English to Italian || @1

English to Italian || @5

English to Italian || @10

Italian to English || @1

Italian to English || @5

Italian to English || @10

English to Italian || @1

English to Italian || @5

English to Italian || @10

Italian to English || @1

Italian to English || @5

Italian to English || @10

Methods || Mikolov et. al.

33.8

48.3

53.9

24.9

41

47.4

1

2.8

3.9

2.5

6.4

9.1

Methods || CCA (Sklearn)

36.1

52.7

58.1

31

49.9

57

29.1

46.4

53

27

47

52.3

Methods || CCA

30.9

48.1

52.7

27.7

45.5

51

26.5

42.5

48.1

22.8

40.1

45.5

Methods || SVD

36.9

52.7

57.9

32.2

49.6

55.7

27.1

43.4

49.3

26.2

42.1

49

Methods || IBFA (Ours)

39.3

55.3

60.1

34.7

53.5

59.4

34.7

52.6

58.3

33.7

53.3

59.2

4.1.1 Detailed Comparison to Basic SVD. We present a more detailed comparison to the SVD method described in (Smith et al., 2017). We focus on methods in their base form, that is without post-processing techniques, i.e. cross-domain similarity local scaling (CSLS) (Lample et al., 2018) or inverted softmax (ISF) (Smithet al., 2017). Note that (Smith et al., 2017) used the scikit-learn 2 implementation of CCA, which uses an iterative estimation of partial least squares. This does not give the same results as the standard CCA procedure. In Table 2 we reproduce the results from (Smith et al., 2017) using the dictionaries and embeddings provided by (Dinu et al., 2014)3 and we compare our method (IBFA) using both the expert dictionaries from (Dinu et al., 2014) and the pseudo-dictionaries as constructed in (Smith et al., 2017). We significantly outperform both SVD and CCA, especially when using the pseudo-dictionaries.
Table 4: Human evaluations of ablation study.

Consistency

Novelty

Diversity

Coherence

Methods || Full model

3.84

3.24

3.16

3.61

Methods || w/o Adversarial

3.31

3.07

3.14

3.43

Methods || w/o Memory

3.53

2.73

2.77

3.19

Methods || w/o Dynamic

3.62

2.91

2.95

3.37

Adversarial training . Another conclusion is that adversarial training can better benefit the model to enhance the topic-consistency of the generated essay compared to memory mechanism. In detail, Table 4 shows that the consistency score given by humans for ablated versions without adversarial training and memory mechanism decline 0.53 and 0.31, respectively. The reason is that the discriminative signal in training not only evaluates the quality of the generated text, but also models its degree of association with the input topics, thus enhancing the topic-consistency.
Table 2: English dev set performance of joint span HPSG parsing. The converted means the corresponding dependency parsing results are from the corresponding constituent parse tree using head rules.

F1

UAS

LAS

Model || separate constituent

93.47

-

-

Model || converted dependency

93.47

95.06

93.81

Model || HPSG Parser joint span lambda = 1.0

93.67

-

-

Model || HPSG Parser joint span lambda = 0.0

-

95.82

94.43

Model || HPSG Parser joint span lambda = 0.5

93.78

95.92

94.49

Model || HPSG Parser converted dependency

93.78

95.69

94.45

When lambda is set to 0 and 1, our joint span HPSG parser works as the dependency-only parser and constituent-only parser respectively. Table 2 shows that even in such a work mode, our HPSG parser still outperforms the separate constituent parser in terms of either constituent and dependency parsing performance. As lambda is set to 0.5, our HPSG parser will give constituent and dependency structures at the same time, which are shown better than the work alone mode of either constituent or dependency parsing. Besides, the comparison also shows that the directly predicted dependencies from our model are slightly better than those converted from the predicted constituent parse trees.
Table 5: Averaged slot coherence results.

Ave Slot Coherence

Method || Nguyen et al. (2015)

0.1

Method || ODEE-F

0.1

Method || ODEE-FE

0.16

Method || ODEE-FER

0.18

Slot Coherence. Table 5 shows the comparison of averaged slot coherence results over all the slots in  the  schemas.   . Note  that  we  do  not  report  the slot coherence for the Clusteringmodel because it does not output the top-N head words in eachslot.  . The averaged slot coherence of ODEE-FER is the highest, which is consistent with the conclusion from Table 4.   . The averaged slot coherence of ODEE-F is comparable to that of Nguyen et al.(2015)(p= 0.3415),  which again demonstrates that  the  contextual  features  are  a  strong  alterna-tive to discrete features. The scores of ODEE-FE(p= 0.06) and ODEE-FER(p= 10^5) are both higher than that of ODEE-F, which proves that the latent event type is critical in ODEE.
Table 1: Performance of various models on EN-DE and EN-FR translation tasks.

BLEU || EN-DE

BLEU || EN-FR

Architecture || Vaswani et al. (2017) Transformer big

28.4

41

Architecture || Wu et al. (2018) Transformer big + sequence-loss

28.75

41.47

Architecture || Yang et al. (2018) Transformer big + localness

28.89

n/a

Architecture || this work Transformer big + hard-attention

29.29

42.26

Main Results . Table 1 shows the performance of various models on EN-DE and EN-FR translation tasks. These test set case sensitive BLEU scores are obtained using SacreBLEU toolkit1 (Post, 2018). The BLEU score difference between our hard-attention based Transformer model and the original soft-attention based Transformer model indicates the effectiveness of selecting a few relevant source tokens for each target token. The performance gap between our method and sequence loss based Transformer (Wu et al., 2018) shows that the improvements are indeed coming from the hard-attention mechanism. Our approach of incorporating hard-attention into decoder’s top selfattention layer to select relevant tokens yielded better results compared to the Localness SelfAttention (Yang et al., 2018) approach of incorporating localness bias only to lower self-attention layers. It can be noted that our model achieved 29.29 and 42.26 BLEU points on EN-DE and ENFR tasks respectively – surpassing the previously published models.
Table 5: Validation set performances of using different layers for attention relay.

ROUGE-1

ROUGE-2

ROUGE-L

Layer || 1

44.7

21.8

41.6

Layer || 2

44.7

22.3

41.7

Layer || 3

45

22

41.8

Layer || 4

44.9

22.1

41.3

Layer || 5

44.9

22.1

41.9

Layer || 6

45.1

22.3

42

Layers for Attention Relay. Transformer architecture used in our experiment is with six layers on both encoder and decoder. Attention relay can take place on each layer. Since each layer has eight heads for attention computation, we average the weights of all eight heads in the same layer. We study the attention relay effects on all six layers. The results in Table 5 show that relaying attention on the last layer achieves the best performance.
Table 3: BUCC results (F1) on the test set. We use the ratio function with maximum score retrieval and the filtering threshold optimized on the training set.

en-de

en-fr

en-ru

en-zh

Azpeitia et al. (2017)

83.7

79.5

-

-

Azpeitia et al. (2018)

85.5

81.5

81.3

77.5

Bouamor and Sajjad (2018)

-

76

-

-

Schwenk (2018)

76.9

75.8

73.8

71.6

Proposed method (Europarl)

95.6

92.9

-

-

Proposed method (UN)

-

-

92

92.6

Table 3 reports the results on the test set for both the Europarl and the UN model in comparison to previous work.9 . Our proposed system outperforms all previous methods by a large margin, obtaining improvements of 10-15 F1 points and showing very consistent performance across different languages, including distant ones.
Table 1: Baseline (Cordeiro et al., 2016) results on the reduced version of three gold-standard datasets ordered in decreasing overall performance along with the results of using only Poincar´e embedding.

RD-R

RD++-R

FD-R

Base. Model || W2V-CBOW

0.8045

0.6964

0.3405

Base. Model || W2V-SG

0.8034

0.6963

0.3396

Base. Model || GloVe

0.7604

0.6487

0.262

Base. Model || PPMI-SVD

0.7484

0.6468

0.2428

Base. Model || Poincare

0.6023

0.4765

0.2007

We compare the performance of the baseline models (Cordeiro et al., 2016) and Poincare embeddings as a single signal on the reduced version of the three gold standard datasets: RD-R (79 instances), RD++-R (146 instances), FD-R (780 instances) in order to closely examine the influence of Poincare embeddings. Table 1 shows the performance for all the baselines in terms of Spearman's rank correlation. We observe that W2V-CBOW model produces the best performance across all the three datasets and W2V-SG achieves the second-best performance. As noted in the table, the Poincare embeddings on their own perform worse than all the other baselines. Further, since our final model is based on an interpolation between Poincare embeddings and W2VCBOW, we also attempted interpolation between other four baseline models, but the best results were always close to the better of the two models, and are not reported here.
Table 8: Human Evaluation in WikiAnswers→Quora

Mean Rank

Agreement

Models || MTL+Copy

3.22

0.446

Models || Naïve DNPG

3.13

0.323

Models || Adapted DNPG

1.79

0.383

Models || Reference

1.48

0.338

MTL+Copy, DNPG and Adapted DNPG as well as the reference. The evaluators are asked to rank the candidates from 1 (best) to 4 (worst) by their readability, accuracy and surface dissimilarity to the input sentence. The detailed evaluation guide can be found in Appendix B. Table 8 shows the mean rank and inter-annotator agreement (Cohen's kappa) of each model. Adapted DNPG again significantly outperforms MTL+Copy by a large margin (p-value < 0.01). The performance of the original DNPG and MTL+Copy has no significant difference (p-value = 0.18). All of the interannotator agreement is regarded as fair or above.
Table 2: Evaluation results of co-teaching initialized with different networks.

Douban (Margin) || MAP

Douban (Margin) || MRR

Douban (Margin) || P@1

Douban (Margin) || R10@1

Douban (Margin) || R10@2

Douban (Margin) || R10@5

ECD (Curriculum) || MAP

ECD (Curriculum) || MRR

ECD (Curriculum) || P@1

ECD (Curriculum) || R10@1

ECD (Curriculum) || R10@2

ECD (Curriculum) || R10@5

SMN-Pre-training

0.527

0.57

0.396

0.236

0.392

0.734

0.662

0.742

0.598

0.302

0.464

0.757

SMN-Co-teaching

0.558

0.602

0.42

0.255

0.431

0.787

0.674

0.765

0.626

0.322

0.485

0.779

DAM-Pre-training

0.552

0.605

0.426

0.258

0.408

0.766

0.685

0.756

0.621

0.325

0.491

0.772

DAM-Co-teaching

0.57

0.617

0.438

0.27

0.455

0.781

0.696

0.775

0.652

0.341

0.499

0.784

Answer to Q2: as a case study of co-teaching with two networks in different capabilities, we initialize model A and model B with DAM and SMN respectively, and select teaching with dynamic margins for Douban and teaching with dynamic data curriculum for ECD (i.e., the best strategies for the two data sets when co-teaching is initialized with one network). Table 2 shows comparison between models before/after co-teaching. We find that co-teaching is still effective when starting from two networks, as both SMN and DAM get improved on the two data sets. Despite the improvement, it is still better to learn the two networks one by one, as co-teaching with two networks cannot bring more improvement than coteaching with one network, and the performance.
Table 1: Performance on the Chimera benchmark dataset with different numbers of context sentences, which is measured by Spearman correlation. Baseline results are from the corresponding papers.

2-shot

4-shot

6-shot

Methods || Word2vec

0.1459

0.2457

0.2498

Methods || FastText

0.1775

0.1738

0.1294

Methods || Additive

0.3627

0.3701

0.3595

Methods || Additive, no stop words

0.3376

0.3624

0.408

Methods || nonce2vec

0.332

0.3668

0.389

Methods || Ã la carte

0.3634

0.3844

0.3941

Methods || HiCE w/o Morph

0.371

0.3872

0.4277

Methods || HiCE + Morph

0.3796

0.3916

0.4253

Methods || HiCE + Morph + Fine-tune

0.1403

0.1837

0.3145

Methods || HiCE + Morph + MAML

0.3781

0.4053

0.4307

Methods || Oracle Embedding

0.416

0.4381

0.4427

Experimental Results . Table 1 lists the performance of HiCE and baselines with different numbers of context sentences. In particular, our method (HiCE+Morph+MAML)6 achieves the best performance among all the other baseline methods under most settings. Compared with the current state-of-the-art method, `a la carte, the relative improvements (i.e., the performance difference divided by the baseline performance) of HiCE are 4.0%, 5.4% and 9.3% in terms of 2,4,6shot learning, respectively. We also compare our results with that of the oracle embedding, which is the embeddings trained from DT , and used as ground-truth to train HiCE. This results can be regarded as an upper bound. As is shown, when the number of context sentences (K) is relatively large (i.e., K = 6), the performance of HiCE is on a par with the upper bound (Oracle Embedding) and the relative performance difference is merely 2.7%. This indicates the significance of using an advanced aggregation model.
Table 1: Discourse segmentation results. Superscript (cid:63) indicates the model is significantly superior to the WLYELMo model with a p-value < 0.01.

Precision

Recall

F1

Approach || Human Agreement || -

98.5

98.2

98.3

Approach || Baselines || SPADE (Soricut and Marcu, 2003)

83.8

86.8

85.2

Approach || Baselines || F&R (Fisher and Roark, 2007)

91.3

89.7

90.5

Approach || Baselines || JCN (Joty et al., 2012)

88

92.3

90.1

Approach || Baselines || SegBotglove (Li et al., 2018)

91.08±0.46

91.03 ±0.42

91.05 ±0.11

Approach || Baselines || WLYELMo (Wang et al., 2018)

92.04 ±0.43

94.41 ±0.53

93.21 ±0.33

Approach || Our Segmenter || Pointer Net (Glove)

90.55 ±0.33

92.29 ±0.09

91.41 ±0.21

Approach || Our Segmenter || Pointer Net (BERT)

92.05 ±0.44

95.03 ±0.28

93.51 ±0.16

Approach || Our Segmenter || Pointer Net (ELMo)

94.12±0.20

96.63±0.12

95.35±0.10

Approach || Our Segmenter || Pointer Net (ELMo) + Joint training

93.34±0.23

97.88±0.16

95.55±0.13

Results. Table 1 shows our segmentation results. As mentioned in Section 4, we implemented three key improvements on the top of (Li et al., 2018). Using encoder hidden states as decoder inputs and adopting dot product as the attention score function together gives 0.40%-7.29% relative improvement  in F1 over  the  first  four  baselines. Using ELMo,  our  segmenter  outperforms  all  the  baselines  in  all  three  measures. We  achieve  2.3%-11.9%, 2.4%-11.3% and 2.3%-12.3% relative improvements  inF1,  Recall  and  Precision,  respectively.   Jointly  training  with  the  parser  improves this further (95.55 F1). It is worthwhile to mention that our segmenter’s performance of 95.55 F1 is very close to the human agreement of 98.3 F1. ELMo, as a transfer learning method, provides notable improvements. A similar observation was reported in (Wang et al., 2018). Surprisingly, the results with BERT were not as good. We suspect this is due to BERT’s special tokenization.
Table 2: Parsing results with gold segmentation. Superscript (cid:63) indicates the model is significantly superior to the 2-Stage Parser with a p-value < 0.01.

Span

Nuclearity

Relation

Approach || Human Agreement || -

95.7

90.4

83

Approach || Baselines || SPADE (Soricut and Marcu, 2003)

93.5

85.8

67.6

Approach || Baselines || DCRF (Joty et al., 2012)

94.6

86.9

77.1

Approach || Baselines || DPLP (Ji and Eisenstein, 2014)

93.5

81.3

70.5

Approach || Baselines || 2-Stage Parser (Wang et al., 2017)

95.6

87.8

77.6

Approach || Our Parser || Stack Pointer (ELMo-medium)

96.37

89.04*

79.03*

Approach || Our Parser || Stack Pointer (ELMo-large)

96.86*

90.77*

81.12*

Approach || Our Parser || + Partial tree information

96.94*

90.89*

81.28*

Approach || Our Parser || + Joint training

97.44*

91.34*

81.70*

Thanks to the pointer network as the backbone of our model, we are able to train our segmenter and parser jointly by sharing the same encoder. The last row of Table 2 shows the results when we train the model jointly, and feed the parser with gold EDU segmentation during inference. The performance is improved further with joint training, achieving 97.44, 91.34, 81.70 F1 score, in Span, Nuclearity and Relation, respectively. The results accord with our assumption that discourse segmentation and parsing may benefit from each other. Our parser surpasses human agreement in span and nuclearity. We are also approaching human agreement in the most difficult task of relation labeling. We show a confusion matrix for the relation labels in Figure 5. We see that our model gets confused between relations that are semantically similar (e.g., CAUSE vs. EXPLANATION, COMPARISON vs. CONTRAST, and TEMPORAL vs. JOINT).
Table 4: Results on translating four languages to English for MLE, BLEU, SIMILE and Half. † denotes statistical significance (p < 0.05) over BLEU and ‡ denotes statistical significance over MLE. Statistical significance was computed using paired bootstrap resampling.

de-en || BLEU

de-en || SIM

cs-en || BLEU

cs-en || SIM

ru-en || BLEU

ru-en || SIM

tr-en || BLEU

tr-en || SIM

Model || MLE

27.52

74.96

17.02

67.18

17.92

70.24

14.47

63.52

Model || BLEU

27.95

86.93

17.29

81.92

17.92

84.63

15.00

80.30

Model || SIMILE

28.28

87.32

17.51

82.12

18.23

85.12

15.28

81.04

Model || Half

28.24

87.11

17.50

82.11

18.24

85.10

15.34

80.64

The results are shown in Table 4. From the table, we see that using SIMILE performs the best when using BLEU and SIM as evaluation metrics for  all  four  languages. It  is  interesting  that  using SIMILE in the cost leads to larger BLEU improvements than using BLEU alone,  the reasons for which we examine further in the following sections. It is important to emphasize that increasing BLEU was not the goal of our proposed method, human evaluations  were  our  target,  but  this  is  a welcome surprise.  Similarly, using BLEU as the cost function leads to large gains in SIM, though these gains are not as large as when using SIMILE in training.
Table 4: Ablation study results. Base model means that we does not use schema linking (SL), memory augmented pointer network (MEM) and the coarse-to-fine framework (CF) on IRNet.

IRNet

IRNet(BERT)

Technique || Base model

40.5%

53.90%

Technique || +SL

48.5%

60.30%

Technique || +SL + MEM

51.3%

60.60%

Technique || +SL + MEM + CF

53.2%

61.90%

Table 4 presents the ablation study results. It is clear that our base model significantly outperforms SyntaxSQLNet, SyntaxSQLNet(augment) and SyntaxSQLNet(BERT). Performing schema linking (‘+SL’) brings about 8.5% and 6.4% absolute improvement on IRNet and IRNet(BERT). Predicting columns in the WHERE clause is known to be challenging (Yavuz et al., 2018). The F1 score on the WHERE clause increases by 12.5% when IRNet performs schema linking. The significant improvement demonstrates the effectiveness of schema linking in addressing the lexical problem. Using the memory augmented pointer network (‘+MEM’) further improves the performance of IRNet and IRNet(BERT). We observe that the vanilla pointer network is prone to selecting same columns during synthesis. The number of examples suffering from this problem decreases by 70%, when using the memory augmented pointer network. At last, adopting the coarse-to-fine framework (‘+CF’) can further boost performance.