SlowGuess/NAACL_2022_OCR huggingface dataset:
5.1 Neural Language Modeling
Settings We use Wikitext-2 (WT2) and Wikitext-103 (WT103) (Merit et al., 2016) to test our method. We adopt the method proposed in (Gong et al., 2018) and (Gao et al., 2019) as our baseline algorithm. Gong et al. (2018) split the rare words with 80/20 rule and then trains an adversarial network as the  $\ell_{\mathrm{transfer}}$ . Gao et al. (2019) add a regularization  $\min_e\frac{1}{N}\sum_i^N\sum_{j,j\neq i}^N\frac{e_i^\top e_j}{\|e_i\|\|e_j\|}$ , to all the words. This can be regarded as  $\ell_{\mathrm{transfer}}$  whose source domain is a spherical uniform distribution and the target domain is all words.
The practical difference of our method with the baselines are: 1) We replace the target domain  $\mathcal{T}$  in the baselines with the more selective domain  $\mathcal{T}_{\eta}^{\prime}$  following (3). For (Gong et al., 2018), we regard words with the  $20\%$  highest frequency following
(Gong et al., 2018) as the source domain. For (Gao et al., 2019), we only apply Algorithm 1 and do not apply (3). 2) The update of each embedding vector  $e_i$  is based on an item-wise coefficient  $\lambda_i$  of the main and transfer loss that is adaptively decided as shown in Algorithm 1.
Model Description For WT2 datasets, we closely follow the regularization and optimization techniques introduced in AWD-LSTM (Merit et al., 2018b). For WT103, we use Quasi-Recurrent neural networks (QRNN)-based language models (Merit et al., 2018a; Bradbury et al., 2017) as our base model for efficiency.
WT2	# Dataset	Baseline	+ CosReg	+ Ours
Eval	68.6	67.4	65.5
Test	65.8	64.7	63.1
WT2	# Dataset	Baseline	+ FRAGE	+ Ours
Eval	68.6	66.5	64.3
Test	65.8	63.4	61.6
WT103	# Dataset	Baseline	+ FRAGE	+ Ours
Eval	32.0	31.3	30.5
Test	33.0	32.5	31.4
Cosine Similarity Calculation We construct the tri-gram co-occurrence matrix (co-occurrence counts with window size 3) with the top 1K frequent words, calculate the cosine similarity between rare and frequent words, rank the similarity and filter  $20\%$  items in the rare items.
Results Table 1 displays that for both FRAGE and CosReg loss, our method can improve the baseline performance on both evaluation and test sets. For example, on WT2 dataset, we boost the test perplexity with a large-marginal improvement, from 63.4 of FRAGE to 61.6. We refer readers to the appendix for sampled examples on WT103 test set.
Table 1: Perplexities measured on validation and test sets on WT2 and WT103. 'CosReg' refers Gao et al. (2019) and FRAGE refers to Gong et al. (2018).
WT2	# Dataset	Baseline	Select	Lexico	Both
Eval	68.6	68.1	65.1	64.3
Test	65.8	65.5	62.2	61.6
WT103	# Dataset	Baseline	Select	Lexico	Both
Eval	32.0	31.4	30.9	30.5
Test	33.0	32.5	32.1	31.4
Analyses We further construct ablation studies for understanding the benefits of each component in our method. We start from Table 2 in which we apply each technique in our method to the baseline. We notice that
1) for small-size vocabularies (e.g. WT2), 'select' only brings marginal improvements, while applying lexico optimization can improve the 63.4 test perplexity obtained by FRAGE to 62.2;
2) for WT103, whose vocabulary size is over 200K, both components of our method can gain substantial improvements over the baseline;
3) combining the two parts of our method together, we achieve the best performance on these two datasets.
Table 2: Ablation studies about our two techniques. Experiments are based on FRAGE.
Method	Head	Tail	Total
Baseline	21.7	313.2	33.0
+ FRAGE	21.8	297.2	32.5
+ Ours	21.5	275.6	31.4
Table 3: Perplexity of the high-frequency (head) and rare (tail) words on the test set of WT103 language modeling task.
As shown in Table 3, we substantially improve the performance on rare items without hurting the performance of high-frequency items.
Figure 2: Test perplexity and the value of transfer loss.
We analyze whether the adaptive coefficient strategy in lexico optimization outperforms the standard method with a fixed coefficient  $\lambda$  in (2). We compare our approach with grid search the coefficient  $\lambda$  for  $\ell_{\mathrm{transfer}}$ . The blue curves in Figure 2 show the performance as we vary  $\lambda$ . Each point on the curve denotes the  $\ell_{\mathrm{transfer}}$  value and the test perplexity for one  $\lambda$  value. We can see that our method yields strictly better results than any fixed  $\lambda$  in grid search.
5.2 Named Entity Recognition and Machine Translation
To verify whether our method can be useful for other NLP problems, we set up experiments on named entity recognition and machine translation. Settings For name entity recognition, we follow Li et al. (2020b), which transfers knowledge to rare entities with GAN. They use GAN to define the discrepancy measure in  $\ell_{\mathrm{transfer}}$  and apply the method to the CoNLL-03 dataset (Sang and De Meulder, 2003). We follow the BERT + CRF model used in
Method	BLEU
Transformer Base (Vaswani et al., 2017)	27.8
+ FRAGE (Gong et al., 2018)	28.3
+ Ours	28.6
Li et al. (2020b), implement the method based on the codebase Transformer (Wolf et al., 2020) by ourselves and yield a slightly worse score than the score reported in Li et al. (2020b). For machine translation, we do experiments on neural machine translation on the WMT2014 EN-DE (Bojar et al., 2014) dataset and adopt the method FRAGE in Gong et al. (2018). We follow the settings in Gong et al. (2018) and use the Transformer-based machine translation model (Vaswani et al., 2017) as our base model with Fairseq (Ott et al., 2019) as the codebase. We follow all the other settings the same as the language model case.
Cosine Similarity Calculation We calculate the cosine similarity following the settings in language model experiments.
Table 4: BLEU scores for the WMT2014 EN-DE machine translation.
Method	F1
BERT (Devlin et al., 2018) + CRF	91.0
+ DEI (Li et al., 2020b)	91.8
+ Ours	92.5
Table 5: Named entity recognition results of baselines and the proposed model on CoNLL-03.
Results As displayed in Table 4 and Table 5, we notice that 1) the transfer learning can improve the baseline performance 2) we can further improve the transfer learning loss. We enhance the BLEU score  ${}^{1}$  from 28.3 to 28.6 on WMT while boost the F1 score from 91.8 to 92.5 on CoNLL-03.
5.3 Recommendation Systems
In recommendation systems (RS), learning tail item embeddings is also an important topic (e.g. Yin et al., 2020; Zhang et al., 2021; Zhu et al., 2021). We extend our method to the click-through rate (CTR) prediction task to verify its generalizability. Settings We examine our method in CTR prediction tasks and create two different datasets, MovieLens 1M (Harper and Konstan, 2015) and Criteo (Cri). In CTR prediction, the user-item pair is represented as a discrete feature vector, and each
feature dimension is converted into a dense embedding. Passing a neural module, the combination of the dense embedding is finally converted into a dense feature and processed by other neural modules to give the final predictions. We apply transfer learning to the embedding of the features. For the simplicity of implementation, instead of following recent works (e.g. Zhang et al., 2021; Zhu et al., 2021), we follow our settings in the language model section and use an adversarial network (Tzeng et al., 2017) as the  $\ell_{\mathrm{transfer}}$ . It introduces an additional regularization term to the final training objective, which is the same as the language model experiments. We report AUC (Area Under the ROC Curve) to measure the performance of models.
Model Description We direct apply our method to DeepFM (Guo et al., 2017) and AutoInt (Song et al., 2019) models. DeepFM consists of an FM component and a deep component (e.g. MLP) which are integrated into a parallel structure. The input feature vectors are first converted into a dense embedding by a lookup table, and then pass the FM component and the deep component. Compared to DeepFM, instead of MLP, AutoInt (Song et al., 2019) uses a multi-head self-attentive neural network with residual connections to explicitly model the feature interactions.
Implementation Details We build our code upon the implementation given by Liu et al. (2021b)  $^{2}$ . Liu et al. (2021b) automatically creates models with the different number of parameters with learnable embedding sizes. This allows us to examine our method with flexible model sizes. We additionally introduce exponential moving average (EMA) (Booth et al., 2006) to reproduce the results of (Liu et al., 2021b).
Dataset Description Movielens 1M dataset encodes each example into a 7-dimension discrete feature vector, which represents timestamp, age, gender, occupation, age, zip code, years, and gender. Among these dimensions, zip code is encoded into 3439 different classes, and has a long tail frequency. Criteo contains 39 feature dimensions. We apply transfer loss to the zip code embeddings for Movielens 1M, and for all the feature dimensions with more than 100 values for Criteo.
Cosine Similarity Calculation We construct the side-information matrix to calculate cosine similarity. For each typical item, we count the times this
$^{1}$ https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl
$^{2}$ https://github.com/ssui-liu/learnable-embed-sizes-for-RecSys
item co-occurred with one of the top 1K items in one record and construct the 1K dimension feature vector.
Figure 3: The frequency distribution of all the items in the Movielens 1M and Criteo training datasets.
Deep FM	AutoInt
# P	B	+ T	+ Ours	B	+ T	+ Ours
10k	84.9	84.9	85.2	84.5	84.6	84.8
20k	85.0	84.9	85.2	84.7	84.8	84.8
30k	85.1	85.0	85.3	84.7	84.8	84.9
Table 6: Test AUC on the Movielens 1M.  $\# \mathrm{P}$  denotes the number of parameters; 'B' the baseline only trained with  $\ell_{\mathrm{main}}$ $^+\mathrm{T}$  denotes the transfer learning using the regularized loss (2).
Deep FM	AutoInt
# P	B	+ T	+ Ours	B	+ T	+ Ours
50k	79.7	79.8	80.0	79.3	79.5	79.7
20k	79.6	79.6	79.9	79.2	79.4	79.6
Results The results of our method and baseline are shown in Table 6 and Table 7. In these two tables, we examine our method with different architectures (DeepFM and AutoInt) and different model sizes. For all these results, we report the performance of the baseline method (training only with  $\ell_{\mathrm{main}}$ ), the transfer learning method (2) in which the coefficient is set to  $\lambda = 10^{-2}$ , and our method. From these results, we observe that:
1) our method never hurts the baseline performance, and sometimes can boost the results; simply applying transfer learning, on the other hand, requires a proper  $\lambda$  to avoid hurting the baseline performance;
2) compared to Movielens 1M, we can achieve more improvements on Criteo dataset. One reasonable explanation is that Criteo contains more long-tail items (see Figure 3) and therefore our method can be more helpful in this case.
Convergence Speed We empirically examine the convergence speed of both our method and the baseline method without  $\ell_{\mathrm{transfer}}$ . We demonstrate the test loss together with test AUC in Figure 4,
Figure 4: Using Moivelen 1M and  $10K$  parameter model, we show the test log loss and test AUC at different epochs for the baseline method and our method.
in which we train the  $10K$  parameters Deep FM model on the Movielens 1M dataset. We train the model with 90 epochs and plot the test log loss and test AUC every 5 epochs. We notice that compared to the baseline, our method has comparable convergence speed and finally boosts both test log loss (0.370 v.s. 0.368) and test AUC.
Table 7: Test AUC on Criteo with DeepFM and AutoInt.
SGD + 0.9 Momentum	SGD
# P	B	+ T	+ Ours	B	+ T	+ Ours
10k	84.9	84.7	85.1	84.2	84.2	84.3
20k	84.9	84.8	85.1	84.3	84.3	84.4
Table 8: Test AUC on Movielens 1M dataset with DeepFM model. The meanings of '#P', 'B', and ' $+$  T' are the same as Table 6.
Figure 5: The test AUC vs. epochs when we use SGD and SGD+Momentum to train the  $10K$ -parameter DeepFM model on the Movielens 1M dataset.
Different Optimizers The default optimizer in RS is Adam, and we test whether our method can be applied for different optimizers. We report the results in Table 8 and Figure 5. We observe that 1) SGD with momentum performs worse than Adam while SGD performs the worst, and 2) our method can improve the baseline performance with all different optimizers. It is possible to combine our method with the other recent advanced optimizers.
This work improves transfer learning for rare items embeddings in multiple tasks. In the future, we plan to 1) extend our method for more general methods and settings in the embedding learning problems, 2) apply our method to datasets with larger-scale vocabulary, and 3) develop an online version algorithm for selecting the target and source domain.
Acknowledgement The work is conducted in the statistical learning and AI group in the computer science department at UT Austin, which is supported in part by CAREER-1846421, SenSE-2037267, EAGER-2041327, and Office of Navy Research, and NSF AI Institute for Foundations of Machine Learning (IFML). We would like to thank the anonymous reviewers and the area chair for their thoughtful comments and efforts towards improving our manuscript.
Criteo. http://labs.criteo.com/downloads/.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Paul T Boggs and Jon W Tolle. 1995. Sequential quadratic programming. Acta numerica, 4:1-51.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, pages 12-58.
Eric Booth, Jeff Mount, and Joshua H Viers. 2006. Hydrologic variability of the columnnes river floodplain. San Francisco Estuary and Watershed Science, 4(2).
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural networks. International Conference on Learning Representations, ICLR.
Zhihong Chen, Rong Xiao, Chenliang Li, Gangfeng Ye, Haochuan Sun, and Hongbo Deng. 2020. Esam: Discriminative domain adaptation with non-displayed items to improve long-tail performance. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 579-588.
Edward Choi, Cao Xiao, Walter F Stewart, and Jimeng Sun. 2018. Mime: Multilevel medical embedding of electronic health records for predictive healthcare. arXiv preprint arXiv:1810.09593.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. 2009. Power-law distributions in empirical data. SIAM review, 51(4):661-703.
Stephan Dempe and Alain Zemkoho. 2020. Bilevel optimization. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Rosie Dunford, Quanrong Su, and Ekraj Tamang. 2014. The pareto principle.
Matthias Ehrgott. 1998. Discrete decision problems, multiple criteria optimization classes and lexicographic max-ordering. Trends in multicriteria decision making, pages 31-44.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and TieYan Liu. 2019. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. Frage: Frequency-agnostic word representation. arXiv preprint arXiv:1809.06858.
Chengyue Gong, Xingchao Liu, and Qiang Liu. 2021. Automatic and harmless regularization with constrained and lexicographic optimization: A dynamic barrier approach. Advances in Neural Information Processing Systems, 34.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: a factorization-machine based neural network for ctr prediction. arXiv preprint arXiv:1703.04247.
F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1-19.
Adrián Javaloy and Isabel Valera. 2021. Rotograd: Gradient homogenization in multi-task learning.
Katikapalli Subramanyam Kalyan and Sivanesan Sangeetha. 2020. Secnlp: A survey of embeddings in clinical natural language processing. Journal of biomedical informatics, 101:103323.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. 2019. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217.
David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94, pages 3-12. Springer.
Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. 2020a. Incremental potential contact: Intersection-and inversion-free, large-deformation dynamics. ACM transactions on graphics.
Yangming Li, Han Li, Kaisheng Yao, and Xiaolong Li. 2020b. Handling rare entities for neural sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6441-6451.
Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. 2020. Controllable pareto multi-task learning. arXiv preprint arXiv:2010.06313.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. 2021a. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34.
Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. 2021b. Learnable embedding sizes for recommender systems. arXiv preprint arXiv:2101.07577.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018a. An analysis of neural language modeling at multiple scales. arXiv preprint arXiv:1803.08240.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018b. Regularizing and optimizing LSTM language models. *ICLR*.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of NAACL-HLT* 2019: Demonstrations.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406-1415.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267.
Feng Qian, Chengyue Gong, Lu-chen Liu, Lei Sha, and Ming Zhang. 2017. Topic medical concept embedding: Multi-sense representation learning for medical concept. In 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 404-409. IEEE.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1161-1170.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. 2020. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662-11671.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. 2020. Long-tailed classification by keeping the good and removing the bad momentum causal effect. arXiv preprint arXiv:2009.12991.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. 2018. Distilled wasserstein learning for word embedding and topic modeling. arXiv preprint arXiv:1809.04705.
Jianwen Yin, Chenghao Liu, Weiqing Wang, Jianling Sun, and Steven CH Hoi. 2020. Learning transferrable parameters for long-tailed sequential user behavior modeling. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 359-367.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR), 52(1):1-38.
Yin Zhang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Lichan Hong, and Ed H Chi. 2021. A model of two tales: Dual transfer learning framework for improved long-tail item recommendation. In Proceedings of the Web Conference 2021, pages 2220-2231.
Yongchun Zhu, Ruobing Xie, Fuzhen Zhuang, Kaikai Ge, Ying Sun, Xu Zhang, Leyu Lin, and Juan Cao. 2021. Learning to warm up cold item embeddings for cold-start recommendation with meta scaling and shifting networks. arXiv preprint arXiv:2105.04790.
.1 Qualitative Analysis
#2 Context	A former basketball player, he grew up in <unk>, Indiana, where he starred on the <unk> Community High School basketball team, setting four school records. After high school, he attended DePauw University, where he played basketball and earned a degree in [MASK]
Baseline top-5	the . ; that engineer
Ours top-5	the . engineer science law
Reference	he attended DePauw University, where he played basketball and earned a degree in economics.
#2 Context	Bradley Kent " Brad " Stevens ( born October 22, 1976 ) is an American professional basketball head coach for the Boston Celtics of the [MASK]
Baseline top-5	. team ; the season
Ours top-5	. the team year NCAA
Reference	an American professional basketball head coach for the Boston Celtics of the NBA.
#2 Context	several of the tanks destined to be deployed to the Eighth Army in the Middle East for the North African Campaign were left in Britain when their cooling systems were determined to be unable to cope with the intense North [MASK]
Baseline top-5	. wind sea ; winds
Ours top-5	; . Britain its Africa
Reference	their cooling systems were determined to be unable to cope with the intense North African heat.
Table 9: Comparison of next-token prediction on WT103 data. '[MASK]' denotes the location to make prediction, while the reference displays the sentence in the corpus.
WT2	# Dataset	Baseline	+ CosReg	+ Ours
Eval	68.6	67.4	65.5±0.2
Test	65.8	64.7	63.1±0.3
WT2	# Dataset	Baseline	+ FRAGE	+ Ours
Eval	68.6	66.5	64.3±0.2
Test	65.8	63.4	61.6±0.1
WT103	# Dataset	Baseline	+ FRAGE	+ Ours
Eval	32.0	31.3	30.5±0.1
Test	33.0	32.5	31.4±0.1
In Table 9, we sample some examples from WT103 test set. 'Baseline' denotes the results of the model without transfer loss, and 'ours' denotes our method results. 'Reference' shows the ground-truth sentence sampled from the dataset.
.2 Full Results