SlowGuess/NAACL_2022_OCR huggingface dataset:
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020b. Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4678-4699, Stroudsburg, PA, USA. Association for Computational Linguistics.
Dingcheng Li, Hongliang Fei, Shaogang Ren, and Ping Li. 2021. A deep decomposable model for disentangling syntax and semantics in sentence representation. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 4300-4310, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. 2020c. Progressive Learning and Disentanglement of Hierarchical Representations. International Conference on Learning Representations.
Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004, Barcelona, Spain, pages 74-81.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 2020-Decem(NeurIPS):1-27.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. 2019. Learning under Concept Drift: A Review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346-2363.
Rebecca Marvin and Tal Linzen. 2020. Targeted syntactic evaluation of language models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, pages 1192-1202.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, volume 371, pages 311-318, Morristown, NJ, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-Theoretic Probing for Linguistic Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, Stroudsburg, PA, USA. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Shauli Ravfogel, Yanai Elazar, Jacob Goldberger, and Yoav Goldberg. 2020. Unsupervised distillation of syntactic information from contextualized word representations. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 91-106, Online. Association for Computational Linguistics.
Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. 2016. Unsupervised learning of 3D structure from images. In Advances in Neural Information Processing Systems, Nips, pages 5003-5011.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-80.
Djame Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez, Benoit Sagot, and Abhishek Srivastava. 2020. Building a User-Generated Content North-African Arabizi Treebank: Tackling Hell. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1139–1150. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. 35th International Conference on Machine Learning, ICML 2018, 10:7322-7330.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In 7th International Conference on Learning Representations, ICLR 2019, pages 1-14.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, 155(1-2):105-145.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. 2021. Unsupervised learning of disentangled speech content and style representation. Proceedings of the Annual Conference of the Interna
tional Speech Communication Association, INTERSPEECH, 4:3191-3195.
Jannis Vamvas and Rico Sennrich. 2021. On the Limits of Minimal Pairs in Contrastive Evaluation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 58-68, Stroudsburg, PA, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In NeurIPS, Nips.
Sarah Wiegrefe and Yuval Pinter. 2020. Attention is not explanation. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 11-20.
John Wieting and Kevin Gimpel. 2018. *ParanMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations*. ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), 1:451-462.
Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020. On Variational Learning of Controllable Representations for Text without Supervision. The 37th International Conference on Machine Learning (ICML 2020).
Xinyuan Zhang, Yi Yang, Siyang Yuan, Dinghan Shen, and Lawrence Carin. 2019. Syntax-Infused Variational Autoencoder for Text Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2069–2078, Stroudsburg, PA, USA. Association for Computational Linguistics.
A Results on the development set
We hereby display the scores on the development set. The encoder scores concerning the specialization of latent variables are in Table 6, while the transfer scores are in Table 7 for semantics, and Table 8 for syntax. The values on the development set concerning the comparison of QKVAE with VGVAE trained on various amounts of data is in Figure 2.
B Hyper-parameters
Hyper-parameter values The  $\beta$  weight on the KL divergence is set to 0.6 for  $z^c$  and to 0.3 for  $z^s$ , and the  $\lambda$  threshold for the Free-Bits strategy is set
zsem↑	zsyn↓
Supervised Models
VGVAE	99.0	16.4
SynPG	91.6	31.2
Unsupervised Models
Optimus	89.4	-
ADVAE	41.0	40.3
QKVAE	86.7	27.0
Table 6: The probability*100 that an embedding places a target sentence closer to its semantic source than it is to its syntactic source in the embedding space. (development set results)
sem_src	syn_src	target
M↑	PB↑	M↓	PB↓	M↑	PB↑
Control and Reference baselines
sem_src	100	1.0	7.4	0.13	27.4	0.82
syn_src	7.4	0.13	100	1.0	12.0	0.16
Optimus	13.00	0.35	13.4	0.34†	10.5	0.32
Supervised Models
VGVAE	18.3	0.58	15.2	0.17	23.0	0.57
SynPG	47.6	0.86	7.8	0.11	24.4	0.73
Unsupervised Models
ADVAE	9.0	0.20	8.1	0.17	7.7	0.19
QKVAE	13.4	0.36	11.3	0.19	12.9	0.35
Figure 2: Plotting STED w.r.t syn_ref and the PB cosine similarity w.r.t sem_ref for VGVAE with different amounts of labeled data and for QKVAE. Points are scaled proportionally to the amount of training data. The vertical and horizontal diameters of each ellipse are equal to the standard deviation of the associated data points and axes.
to 0.05. KL annealing is performed between steps 3K and 6K for  $z^{sem}$ , and between steps 7K and 20K for  $z^{syn}$ . The model is trained using Adafactor (Shazeer and Stern, 2018), a memory-efficient version of Adam (Kingma and Ba, 2015). Using a batch size of 64, we train for 40 epochs, which takes about 30 hours on a single Nvidia GEForce RTX 2080 GPU. We use 4 layers for both Transformer encoders and decoders. The encoders (resp. decoders) are initialized with parameters from the 4 first layers (resp. 4 last layers) of BART encoders (resp. decoders). In total, our model uses 236M parameters.
Manual Hyper-parameter search Given that the architecture for Transformer layers is fixed by BART, we mainly explored 3 parameters: number of latent variables  $L$ , number of Transformer layers, values for  $\beta$ . Our first experiments have shown that setting  $L$  to 8 or 16 does not yield good re
Table 7: Semantic transfer results (development set results)
sem_src	syn_src	target
STED↑	TMA2↓	TMA3↓	STED↓	TMA2↑	TMA3↑	STED↓	TMA2↑	TMA3↑
Control/Ceiling baselines
sem_src	0.0	100	100	11.9	46.4	6.8	10.9	47.0	7.3
syn_src	11.9	46.4	6.8	0.0	100	100	6.0	81.6	45.0
Optimus	9.7	58.2	20.6	9.2†	61.6†	22.6†	9.9	59.6	18.4
Supervised Models
VGVAE	11.9	45.4	6.8	3.2	84.2	58.2	6.7	77.6	39.0
SynPG	9.3	49.4	21.4	12.2	73.0	12.2	12.2	68.6	13.0
Unsupervised Models
ADVAE	10.1	53.4	18.6	9.8†	55.0†	17.4†	10.5	52.8	15.4
QKVAE	11.4	45.0	9.1	6.8	66.4	37.4	8.6	63.0	26.9
Table 8: Syntactic transfer results (development set results)
sults, which is probably due to the fact that a high  $L$  raises the search space for possible arrangements of values with keys, and consequently makes convergence harder. Concerning the number of layers, we observed that results with the full BART model (6 layers) have high variance over different runs. Reducing the number of layers to 4 solved this issue. In regards to  $\beta$ , we observed that it must be 0.6 or less for the model to produce adequate reconstructions and that it is beneficial to set it slightly lower for  $z^{syn}$  than for  $z^{sem}$  so as to absorb more syntactic information with  $z^{syn}$ .