SlowGuess/NAACL_2022_OCR huggingface dataset:
Baselines We compare our system to 4 previously published models, where 2 are supervised and 2 are unsupervised: i) VGVAE (Chen et al., 2020): a VAE-based paraphrase generation model with an LSTM architecture. This model is trained using paraphrase pairs and PoS Tags to separate syntax and semantics into two latent variables. This separation is used to separately specify semantics and syntax to the decoder in order to produce paraphrases; ii) SynPG (Huang and Chang, 2021): A paraphrase generation Seq2Seq model based on a Transformer architecture which also separately encodes syntax and semantics for the same purpose as VGVAE. This model is, however, trained using only source sentences with their syntactic parses, without paraphrases; iii) Optimus (Li et al., 2020b): A large-scale VAE based on a fusion between BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) with competitive performance on various NLP benchmarks; iv) ADVAE: This model is QKVAE without its syntactic variable. The size of its latent variable is set to 1536 to equal the total size of latent variables in QKVAE.
Official open-source instances $^{10}$  of the 4 models above are available, which ensures accurate comparisons. The off-the-shelf instances of VGVAE and SynPG are trained on ParaNMT with GloVe $^{11}$  (Pennington et al., 2014) embeddings. We fine-tune a pre-trained Optimus on our training set following instructions from the authors. Similar to our model, we initialize ADVAE with parameters from BART(Lewis et al., 2020) and train 5 instances of it on ParaNMT with  $L = 4$ .
5.2 Syntax and Semantics Separation in the Embedding Space
We first test whether  $z^{syn}$  and  $z^{sem}$  respectively specialize in syntax and semantics. A syntactic (resp. semantic) embedding should place syntactically (resp. semantically) similar sentences close
zsem↑	zsyn↓
Supervised Models
VGVAE	99.9	14.8
SynPG	93.4	26.5
Unsupervised Models
Optimus	91.8	-
ADVAE	39.5	40.0
QKVAE	89.2	26.4
Table 2: The probability*100 that an embedding places a target sentence closer to its semantic source than it is to its syntactic source in the embedding space. Arrows  $(\uparrow/\downarrow)$  indicate whether higher or lower scores are better.
to each other in the embedding space.
Using the (target, sem_src, syn_src) triplets, we calculate for each embedding the probability that target is closer to sem_src than it is to syn_src in the embedding space. For simplicity, we refer to the syntactic and semantic embeddings of all models as  $z^{syn}$  and  $z^{sem}$ . For Gaussian latent variables, we use the mean parameter as a representation (respectively the mean direction parameter from the von Mises-Fisher distribution of the semantic variable of VGVAE). We use an L2 distance for Gaussian variables and a cosine distance for the others. Since Optimus and ADVAE do not have separate embeddings for syntax and semantics  $i$  We take the whole embedding for Optimus; ii)For ADVAE, we measure the above probability on the development set for each latent variable  $z_{l}$  (Eq. 3). Then, we choose the latent variable that places target sentences closest to their sem_src (resp. syn_src) as a semantic (resp. syntactic) variable. The results are presented in Table 2.
Table 2 clearly shows for QKVAE, SynPG, and VGVAE that the syntactic (resp. semantic) variables lean towards positioning sentences in the embedding space according to their syntax (resp. semantics). Surprisingly, the syntactic variable of our model specializes in syntax (i.e. has low score) as much as that of SynPG. The generalist latent variable of Optimus seems to position sentences in the latent space according to their semantics. Accordingly, we place its score in the  $z^{sem}$  column. Interestingly, the variables in ADVAE have very close scores and score well below 50, which shows that the entire ADVAE embedding leans more towards syntax. This means that, without the key/value distinction in the Attention-based decoder, the variables specialize more in structure than in content.
$^{10}$ VGVAE: github.com/mingdachen/syntactic-template-generation/; SynPG: github.com/uclanlp/synpg; Optimus: github.com/ChunyuanLI/Optimus; ADVAE: github.com/ghazi-f/ADVAE
Gains could be observed with better embeddings for supervised models, but we stick to the original implementations.
sem_src	syn_src	target
STED↑	TMA2↓	TMA3↓	STED↓	TMA2↑	TMA3↑	STED↓	TMA2↑	TMA3↑
Control and Reference baselines
sem_src	0.0	100	100	13.0	40.3	4.8	12.0	39.6	7.0
syn_src	13.0	40.3	4.8	0.0	100	100	5.9	84.3	45.8
Optimus	11.6	50.0	15.9	9.2	61.6	23.6	10.2	58.9	21.8
Supervised Models
VGVAE	13.1	39.9	5.4	3.3	86.4	64.1	6.7	80.4	44.6
SynPG	11.7	41.9	18.0	13.5	74.1	10.5	13.1	69.1	13.3
Unsupervised Models
ADVAE	11.9	47.3	14.0	10.3	54.3†	19.2†	11.1	52.3	17.0
QKVAE	12.7	40.2	7.8	7.2	68.2	39.5	8.9	63.9	28.1
Table 3: Syntactic transfer results. STED is the Syntactic Tree Edit Distance, and TMA2/3 is the exact matching between constituency trees truncated at the  $2^{nd} / 3^{rd}$  level.
sem_src	syn_src	target
M↑	PB↑	M↓	PB↓	M↑	PB↑
Control and Reference baselines
sem_src	100	1.0	6.9	0.14	28.8
syn_src	6.9	0.14	100	1.0	12.1
Optimus	12.4	0.34	15.9	0.39	10.8
Supervised Models
VGVAE	17.6	0.58	15.3	0.18	24.9
SynPG	45.9	0.87	8.0	0.13	25.2
Unsupervised Models
ADVAE	8.0	0.19	8.3†	0.17	7.4
QKVAE	12.8	0.35	11.0	0.19	12.6
Table 4: Semantic transfer results.  $M$  is the Meteor score, and  $PB$  is the ParaBart cosine similarity.
5.3 Syntactic and Semantic Transfer
Similar to (Chen et al., 2020), we aim to produce sentences that take semantic content from sem_src sentences and syntax from syn_src sentences. For each of SynPG, VGVAE, and QKVAE we simply use the syntactic embedding of syn_src, and the semantic embedding of sem_src as inputs to the decoder to produce new sentences. Using the results of the specialization test in the previous experiment, we do the same for ADVAE by taking the 2 latent variables that lean most to semantics (resp. syntax) as semantic (resp. syntactic) variables. The output sentences are then scored in terms of syntactic and semantic similarity with sem_src, syn_src and target.
Control and reference baselines Beside model outputs, we also use our syntactic and semantic comparison metrics, explicitcd below, to compare syn_src and sem_src sentences to one another and to target sentences. Additionally, using Optimus, we embed sem_src and syn_src, take the dimensionwise average of both embeddings, and decode it. As VAEs are known to produce quality sentence interpolations (Bowman et al., 2016; Li et al., 2020b),
the scores for this sentence help contrast a naive fusion of features in the embedding space with a composition of well identified disentangled features.
Transfer metrics We measure the syntactic and semantic transfer from source sentences to output sentences. i) Semantics: For semantics, previous works (Chen et al., 2020; Huang and Chang, 2021) rely on lexical overlap measures such as BLEU (Papineni et al., 2001), ROUGE (Lin, 2004), and Meteor (Denkowski and Lavie, 2014). As will be shown in our results, the lexical overlap signal does not capture semantic transfer between sentences when this transfer is too weak to produce paraphrases. Therefore, we use Meteor  $(M)$  in conjunction with ParaBART (Huang et al., 2021) a model where BART (Lewis et al., 2020) is finetuned using syntactic information to produce neural representations that represent maximally semantics and minimally syntax. We measure the cosine similarity between sentences according to ParaBART embeddings  $(PB)$ . ii) Syntax: We use the script of (Chen et al., 2020) to produce a syntactic tree edit distance (STED) between the constituency trees of sentences, as was done to assess VGVAE. Additionally, following the evaluation procedure designed by Huang and Chang (2021) for SynPG, we measure the Template Matching Accuracy between sentences, where the template is the constituency tree truncated at the second level (TMA2). TMA2 is the percentage of sentence pairs where such templates match exactly. We extend this measure by also providing it at the third level (TMA3). Results are presented in Tables 3 and 4. In both Tables, the comparison scores between sentences and syn_src that are not significantly $^{12}$  different from the same
12We consider differences to be significant if their associated  $t$  -test yields a  $p$  -value  $<  0.01$
scores produced with regard to sem_src are marked with  $\dagger$
Sanity checks with metrics and baselines We notice in Table 4 that using Meteor as a semantic similarity measure results in various inconsistencies. For instance, paraphrases target have a higher Meteor score with the syntactic sources than with interpolations from Optimus. It can also be seen that the Meteor score between outputs from VGVAE and both syntactic and semantic sources are rather close  $^{13}$ . In contrast, ParaBART score behaves as expected across comparisons in Table 4. Consequently, we retain ParaBART score as a semantic similarity measure. In the following, we use the scores between sem_src, syn_src, and target (first two rows in Tables 4 and 3) as reference scores for unrelated sentences, paraphrase pairs, and syntactically similar sentences.
Comparing the supervised baselines VGVAE and SynPG greatly differ in scores. It can be seen that SynPG copies a lot of lexical items from its semantic input (high Meteor score) which allows for higher semantic similarity scores. However, Table 3 shows that SynPG transfers syntax from syn_src at a high level (high TMA2, but low TMA3). In contrast, VGVAE transfers syntax and semantics in a balanced way and achieves the best syntax transfer scores overall (lowest STED with syn_src and target).
Analysing the scores of QKVAE The semantic similarity scores  $PB$  of QKVAE outputs with target and sem_src are close to those of Optimus outputs. Although these scores are low compared to supervised models, they are notably higher than semantic similarity scores between unrelated sentences (e.g. syn_src and sem_src). However, in contrast to Optimus, QKVAE outputs display low PB scores with syn_src, which show that they draw very little semantic information from the syntactic sources. Concerning syntactic transfer in Table 3, QKVAE outputs share syntactic information with syn_src on all levels (low STED, and high TMA2 and TMA3). Our model is even competitive with SynPG on TMA2, and better on TMA3 and STED. As expected, the scores comparing QKVAE outputs to sem_src show that they share very little syntactic information. On the other hand, ADVAE shows poor transfer performance on syntax and semantics,
with only slight differences between scores w.r.t syn_src and scores w.r.t sem_src.
5.4 Comparing our Model to a Supervised Model with Less Data
Since VGVAE displays balanced syntactic and semantic transfer capabilities, we use it for this experiment where we train it on subsets of sizes in  $\{10K, 25K, 50K, 100K\}$  from its original training data. Our goal is to find out how much labeled data is needed for VGVAE to outperform our unsupervised model on both transfer metrics.
Figure 1: Plotting STED w.r.t syn_ref and the PB cosine similarity w.r.t sem_ref for VGVAE with different amounts of labeled data and for QKVAE. Points are scaled proportionally to the amount of training data. The vertical and horizontal diameters of each ellipse are equal to the standard deviation of the associated data points and axes.
In Figure 1, we plot for QKVAE and instances of VGVAE the STED of their outputs w.r.t syn_src and the  $PB$  of these outputs w.r.t sem_src. All values are averages over 5 runs, with standard deviations plotted as ellipses. Figure 1 shows that to outperform QKVAE on syntactic and semantic transfer, VGVAE needs more than 50K labeled samples.
In Table 5, we display example outputs of SynPG, VGVAE, and QKVAE along with their syntactic sources, semantic sources, and targets. We generally observed that the outputs of QKVAE range from paraphrases (line 6) to broadly related sentences (line 3). As was shown by our quantitative results, outputs from VAE-based models (VGVAE and QKVAE) share relatively few lexical items with the semantic input. This can be seen in the qualitative examples where they often swap words in the semantic source with closely related words (e.g.
This was not observed by Chen et al. (2020), as they only compared outputs from VGVAE to the target paraphrases.
sem_src	syn_src	SynPG	VGVAE	QKVAE	target
we have de-stroyed the 49th armored division.	concomitant usage is not recommended.	we have de-stroyed the 49th armored division.	armored division hasn't destroyed.	this military force will be destroyed.	49th armored division has been destroyed.
let the fire burn and put a piece of hot iron in it.	sing a song. sing a song for boys.	don't put the fire in it burn a hot piece of iron and fire.	burn the fire. put the iron on burns.	come on fire. get a fire on it.	keep this fire going. keep a piece of hot iron on it.
they took the lunch boxes?	have you given me your hands?	do they boxes took the lunch?	have they taken them your snacks?	have you heard of some lunch?	have they taken the lunch boxes?
does it have a coach?	that's a phone switcher, right?	how does it have a coach?	that's a coach coach, right?	that's a warden, huh?	it has a coach, no?
an old lady in a cemetery.	that is a bad time for a war.	there's a lady in an old cemetery.	that's an old lady in the cemetery.	this is a strange place for a woman.	there is an old lady in the cemetery.
don't be afraid.	there are still many places to go.	you don't be afraid.	there aren't be afraid to be.	there will be no need to worry.	there is no need to be afraid.
isn't there a door open?	the machines are still good, right?	a isn't open door there?	the doors aren't open, right?	the door will be open, okay?	there is a door open, right?
Table 5: Syntactic sources (syn_src), semantic sources (sem_src), the sentences produced when using them with different models, and the corresponding correct paraphrases (target).
"armored division" to "military force" in line 1, or "lunch boxes" to "snacks" in line 2). We attribute this quality to the smoothness of the latent space of VAEs which places coherent alternative lexical choices in the same vicinity. The examples above also show that our model is capable of capturing and transferring various syntactic characteristics such as the passive form (line 1), the presence of subject-verb inversion (lines 3, 4, and 7), or interjections (lines 4 and 6).
We presented QKVAE, an unsupervised model which disentangles syntax from semantics without syntactic or semantic information. Our experiments show that its latent variables effectively position sentences in the latent space according to these attributes. Additionally, we show that QKVAE displays clear signs of disentanglement in transfer experiments. Although the semantic transfer is moderate, syntactic transfer with QKVAE is competitive with SynPG, one of its supervised counterparts. Finally, we show that VGVAE, a supervised model, needs more than 50K samples to outperform QKVAE on both syntactic and semantic transfer.
We plan to extend this work in three directions: i) Finding ways to bias representations of each  $z_{l}^{sem}$  towards understandable concepts; ii) Applying QKVAE to non-textual data since it is data agnostic (e.g. to rearrange elements of a visual landscape.); iii) Investigating the behavior of QKVAE on other languages.
This work is supported by the PARSITI project grant (ANR-16-CE33-0021) given by the French National Research Agency (ANR), the Laboratoire d'excellence "Empirical Foundations of Linguistics" (ANR-10-LABX-0083), as well as the ONTORULE project. It was also granted access to the HPC resources of IDRIS under the allocation 20XX-AD011012112 made by GENCI.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. 2019. Generating Sentences from Disentangled Syntactic and Semantic Spaces. ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 6008-6019.
Melika Behjati and James Henderson. 2021. Inducing Meaningful Units from Character Sequences with Slot Attention. *Arxiv*.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings, pages 10-21.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for disentangling syntax and semantics in sentence representations. *NAACL HLT* 2019 - 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:2453-2464.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2020. Controllable paraphrase generation with a syntactic exemplar. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 1, pages 5972-5984.
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In 6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings.
Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li, and Lawrence Carin. 2020. Improving Disentangled Text Representation Learning with Information-Theoretic Guidance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7530-7541.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What Does BERT Look at? An Analysis of BERT's Attention. In Black-BoxNLP@ACL.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. *NAACL HLT* 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:4171-4186.
Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J O'Donnell, Yoshua Bengio, and Yue Zhang. 2020. Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 6611-6628.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference, pages 199-209.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. 2021. A Brief Review of Domain Adaptation. In Advances in Data Science and Information Engineering, pages 877-894.
Ghazi Felhi, Joseph Le Roux, and Djamé Seddah. 2021. Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles. In 1st CtrlGen: Controllable Generative Modeling in Language and Vision Workshop at NeurIPS 2021.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. *NAACL HLT* 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:4129-4138.
Irina Higgins, Loic Matthew, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. B-VAE: Learning basic visual concepts with a constrained variational framework. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, pages 1-22.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P Levy. 2020. A Systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725-1744.
James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang, 2021. Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1372-1379, Stroudsburg, PA, USA. Association for Computational Linguistics.
Kuan-hao Huang and Kai-wei Chang. 2021. Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1022-1033, Stroudsburg, PA, USA. Association for Computational Linguistics.
Drew A. Hudson and Christopher D. Manning. 2018. Compositional Attention Networks for Machine Reasoning. International Conference on Learning Representations, 333(6045):975-8.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 11-20.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2020. Disentangled representation learning for non-parallel text style transfer. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 424-434.
Yoon Kim, Alexander M Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1:1105-1117.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-4751.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings, MI, pages 1-14.
Jordan Kodner and Nitish Gupta. 2020. Overestimation of syntactic representation in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 1757-1762.
Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 4077-4091.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pretraining for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Stroudsburg, PA, USA. Association for Computational Linguistics.
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2020a. A surprisingly effective fix for deep latent variable modeling of text. In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3603-3614.