A Bilingual Generative Transformer for Semantic Sentence Embedding
Authors: John Wieting
Graham Neubig
Taylor Berg-Kirkpatrick
Venue:  (2020)
Keywords: sentence embedding
semantic similarity
multilingual
latent variables
vae

Abstract: Semantic sentence embedding models take natural language sentences and turn them into vectors, such that similar vectors indicate similarity in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model’s posterior to predict sentence embeddings for monolingual data at test time. Second, we use high- capacity transformers as both data generating distributions and inference networks – contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of se- mantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of test where simple word overlap is not a good indicator of similarity.

Under review as a conference paper at ICLR 2020
A BILINGUAL GENERATIVE TRANSFORMER FOR SEMANTIC SENTENCE EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Semantic sentence embedding models encode natural language sentences into vec- tors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic language-specific. We propose a deep latent variable model that attempts to perform source separation parallel sentences, isolating what they have in common in latent semantic vector, and explaining what left over with language-spede specifii latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and sentence embeddings for monolingual data at test time. Second. we use high- capacity transformers as both data generating distributions and inference networks contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-ar on standard suite of un- supervised semantic similarity evaluations. Further, we demonstrate that our ap- proach yields the largest gains on more difficult subsets these evaluations where simple word overlap is not good indicator of similarity.
our model's posterior predict
INTRODUCTION
Learning useful representations of language has been source of recent success in natural language processing (NLP).. Much work has been done on learning representations for words (Mikolov al.)
2013 PPnniigtonn et al. 2014) and sentences (Kiros et al. 2015 Conneau et al. 2017). More recently, deep neural architectures have been used to learn contextualized word embeddings (Peters cet al., 2018 Devlin et al. 2018) which have enabled state-of-the-ate results many tasks. We focus on learning semantic sentence embeddings in this paper, which play an important role in many downstream applications. Since they do not require any labelled data for fine-tuning sentence embeddings are useful for a variety of problems right out of the box. These include Semantic Textual Similarity (STS: [Ágirre al. (2012), mining bitext (Zweigenbaum et al., 2018 and paraphrase identification (Dolan et al. 22044). Semantic similarity measures also have downstream uses such as fine-tuning machine translation systems (Wieting et al. 2019a).
There are three main ingredients when designing a sentence embedding model: the architecture, the training data, and the objective function. Many architectures including LSTMs (Hill et al. 2016
Conneau et al., Schwenk & Douze 2017 Subramanian et al.. 2018), Transformers (Cer et al. 2018 Reimers Gurevych| 2019). and averaging models (Wieting al. 2016 Arora et al. 2017 have found success for learning sentence embeddings. The choice of training data and objective are intimately intertwined, and there are wide variety of options including next-sentence predic- tion (Kiros al. 2015). machine translation (Espana-Bonet et al.) 2017 Schwenk & Douze 2017 Schwenk 2018 Artetxe & Schwenk 2018). natural language inference (NLI) (Conneau et al, 2017). and multi-task objectives which include some of the previously mentioned objectives (Cer
ct al. 2018) as well as additional tasks like constituency parsing (Subramanian et al. 2018).
Surprisingly, despite ample testing of more powerful architectures, the best performing models for many sentence embedding tasks related to semantic similarity often use simple architectures that mostly agnostic the interactions between words. For instance, some of the top performing

---

Under review as a conference paper at ICLR 2020
techniques use word embedding averaging (Wieting et al. 2016a). character n-grams (Wieting et al. 2016b), and subword embedding averaging (Wieting et al. 2019b) to create representations. These simple approaches are competitive with much more complicated architectures on in in-ommiin data and generalize well to unseen domains, but are fundamentally limited by their inability to capture word order. Training these approaches generally relies on discriminative objectives defined on paraphrase data (Ganitkevitch et al., 2013] Wieting & Gimpel 2018) or bilingual data (Wieting et al. 2019b). The inclusion of latent variables in these models has also been explored (Chen et al. 2019).
Intuitively, bilingual data in particular is promising because it potentially offers useful signal for learning the underlying semantics of sentences. Within a translation pair, properties shared by both sentences are more likely semantic, while those that are divergent are more likely stylistic or language-specifie While previous work learning from bilingual data perhaps takes advantage of this fact implicitly, the focus of this paper is modelling this intuition explicitly, and to the best of our knowledge, this has not not been explored in prior work. Specifically, we propose deep generative model that is encouraged to perform source separation on parallel sentences, isolating what they have in common in a latent semantic embedding and explaining what is left over with tunguage--ppciftee latent vectors. At test time, we use inference networks (Kingma & Welling 2013) for approximating the model's 's posterior on the semantic and source source-separated latent variables to encode monolingual sentences. Finally, since our model and training objective are generative, our approach does not require knowledge of the distance metrics to be used during evaluation and it has the additional property of being able generate text.
In experiments, we evaluate our probabilistic source-separation approach on standard suite of STS evaluations. We demonstrate that the proposed approach is effective, most notably allowing the learning of high-capacity deep transformer architectures (Vaswani et al. (017) while still generaliz- ing to new domains, significantly outperforming variety of state-of-the art baselines. Further, we conduct a thorough analysis by identifying subsets of the STS evaluation where simple word overlap not able to accurately assess semantic similarity. On these most difficult instances, we find that our approach yields the largest gains, indicating that our system is modeling interactions between words to good effect. We also find that our model better handles cross-lingual semantic similarity than multilingual translation baseline approaches, indicating that stripping away language-specific information allows for better comparisons between sentences from different languages.
Finally, we analyze our model to uncover what information into the semantic and mmngaae--speciffe variables and the relationship between this encoded infor- mation and language distance to English. We find that the language-speciffe variables tend to explain more superficial or language-specific properties such as overall sentence length, amount and loca- tion of punctuation, and the gender of articles (if gender is present in the language), but semantic and syntactic information is more concentrated in the shared semantic variables, matching our intu- ition. Language distance has an effect as well, where languages that share common structures with English put more information into the semantic variables, while more distant languages put more information into the wanguage-specifne variables. Lastly, we show outputs generated from our model that exhibit its ability to do type of style transfer.
captured by the source separation
2
MODEL
Our proposed training objective leverages a generative model of parallel text in two languages (e.g. English (en) and French (fr)) that form a pair consisting of an English sentence Ten and a French sentence fr. Importantly, this generative process utilizes three underlying latent vectors: language- specific variation variables (language variables) 2fr and Zen. respectively for each side of the transla- tion, well as shared semantic variation variable (semantic variable) Zzem- In this section we will first describe the generative model for the text and latent variables. In the following section we will describe the inference procedure of Zeem given an input sentence, which corresponds to our core task of obtaining sentence embeddings useful for downstream tasks such as semantic similarity.
Further, by encouraging the model to perform this source separation, the learned semantic encoders will more crisply represent the underlying semantics, increasing performance on downstream se- mantic tasks.
In other words, don't assume cosine similarity a metric, though it does work well in our experiments.
2

---

Under review as a conference paper at ICLR 2020
Gaussian Priors
FOUD
The generative process of our model, the Bilingual Generative Transformer (BGT), is depicted in Figure [ and its putation graph is shown Figure 2 First, we sample latent variables (2/r: Pen, 2sem), where 24 RR from a mul- tivariate Gaussian prior N(, Ts). These vari- ables are then fed into a decoder that samples sentences; Ten is sampled conditioned on žeem and Sens while fr is sampled conditioned on Zsem and 2fr- Because sentences in both languages will use Zeem in generation, we expect that in a well-
Freneb-speeffte Latent Vector
Common Semantic
English-speeifie Latent Vactor
Latent Vector
Finguinti variation spectfic French
Lingaintic varkition sperific Fingliok
Zfr
Egem
renanite varsation
Sen
A
Freach Inference Net (Sectiet
Senartic Inforvnce Net (Sectias 3)
Esglish informnee (Section 3)
French Deceder (Section
English Deadda: (Section 7)
Xyr
X
Observed Sentenoe
Observed English Sentence
Translaign Pair
Figure 1: The generative process of model. Latent variables model- ing the linguistic variation in French and English, 2fr and žen as well latent variable modeling the common semantics, 2sem, drawn from a multivariate Gaussian prior. The observed text in each language is then conditioned its language-specifie variable and Zsem.
trained model this variable will encode semantic, syntactic, or stylistic information shared across both sentences, while fr and Zen will handle any language-speciff peculiarities or specific stylistic decisions that are less central to the sentence meaning and thus do not translate across sentences. In the following section, we further discuss how this is explicitly encouraged by the learning process.
Decoder Architecture. Many latent variable models for text use LSTMs (Hochreiter & Schmiid-
huber 1997) as their decoders (Yang et al. [017 Ziegler & Rush 2019 Ma et al. (019)). However, state-of-the-art models neural machine translation have seen increased performance and speed us- ing deep Transformer architectures. We also found in experiments (see Appendix [ for details) that Transformers led to increased performance in our setting, so they are used in our main model.
We use decoders in model, one for modelling p(Ifrlzsoms Zfr; 0) and one for modeling p(Cen Zsem; Zen; 0). These decoders are depicted on the right side of Figure 2 Each decoder takes two latent variables, language variable and a semantic variable. These variables are concatenated together prior to being used by the decoder for reconstruction. We explore four ways of using this latent vector: (1) Concatenate
the word embeddings (Word) (2) Use it as the initial hidden state (Hidden, LSTM only) (3) Use it as you would the attention context vector in the traditional sequence- to- sequence framework (Attention) and (4) Concatenate it to the hidden state immediately prior to computing the logits (Logit). Unlike Attention, there is no additional feedforward layer in this setting. We experimented with these four approaches, as well as combinations thereof, and report this analysis in Appendix A From these experiments, to the softmax, the better the performance on downstream tasks evaluating its semantic content. We hypothesise that this is due to better gradient propagation because the sentence embedding is now closer to the error signal. Since Attention and Logit performed best, we use these our Transformer experiments.
that the closer the sentence embedding
LEARNING AND INFERENCE
Our model is trained on a training set of parallel text consisting of N examples, X 2//n (aN N x. N N (12- 2/,m 21 zem)...... (2 zfr >.,)). We wish to maximize the likelihood the parameters of the two decoders 0 with respect to the observed X, marginalizing over the latent variables Z.
and Z is our collection of latent variables Z
p(X:))= /
p(X, 2:0
3

---

Under review as a conference paper at ICLR 2020
Inference Networks
Englls Agfresce Asr
M Har (9)
English Encodes
Latent Space
Generative Model
Aeroeus apersonaa punorder
Pl ())
Sewanttc Anfrrwcc Not
speration Derton
Megann
Makeo Rogk
concat
English Decoder
X
Sematzic Esssser
Ieput English and Fresch translation
Contostenate sompled luient and decade
Recoustruction input transiation pair
margo
Semanci Lacoder
pitz Eaacs ())
lyforence
Fia
X
French
Encodes
Figure 2: The computation graph for the variational lower bound used during training. The En- glish and French text are fed into their respective inference networks and the semantic inference network to ultimately produce the language variables frr and žen and semantic variable 2.em- Each language-specinn specific variable is then concatenated to 2.em and used by the decoder to reconstruct the input sentence pair.
Unfortunately, this integral is intractable due to the complex relationship between X and Z. How- ever, related latent variable models like variational autoencoders (VAEs (Kingma & Welling 2013) learn by optimizing variational lower bound on the log marginal likelihood. This surrogate objec- tive is called the evidence lower bound (ELBO) and introduces a variational approximation, q to the true posterior of the model p. The g distribution is parameterized by neural network with parameters Φ. ELBO can be written for our model as follows:
ELBO
(L(g(Z((:
This lower bound on the marginal can be optimized by gradient ascent by using the reparame- terization trick (Kingma & Welling, 2013). This trick allows for the expectation under to be approximated through sampling way that preserves backpropagation.
We make several independence assumptions for ((Zeems Zen; Zfrl Φ). Specifically, to match goal of source separation, we factor 9 g(asems Zen; 2 Ten: 2 fri Φ) ((sse Ten: Fyi (2)g((enllee. (9(2))/(1/)))))))>) ( Φ), with ø being the parameters of the encoders that make up the inference networks, defined in the next paragraph.
Lastly, we note that the KL term in our ELBO equation encourages explaining variation that is shared by translations with the shared semantic variable and explaining language-specifie variation with the corresponding language-speciff variables. Information shared by the two sentences will result in lower KL loss if encoded in the shared variable, otherwise that information will be replicated and the overall cost of encoding will increase.
Encoder Architecture. We use three inference networks as shown on the left side of Figure an English inference network to produce the English language variable, French inference network to produce the French language variable, and semantic inference network to produce the semantic variable. Just as in the decoder architecture, we use a Transformer for the encoders.
The semantic inference network is bilingual encoder that encodes each language. For each trans- lation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch. Since the semantic encoder is meant to capture language agnostic semantic information, its outputs for translation pair should be similar regardless of the language of the input sentence. We note that other operations are possible for combining the views each parallel sentence offers. For instance, could feed both sentences into the semantic encoder and pool their representations. However, in practice we find that alternating works well and leave further study of this to future work.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 BASELINE MODELS
We experiment with fourteen baseline models, covering both the most effective approaches for learn- ing sentence embeddings from the literature and ablations of our own BGT model. These baselines can be split into three groups as detailed below.
Models from the Literature (Trained on Different Data) We compare to well known sentence embedding models Infersent (onneau et al. 2017), GenSen (Subramanian et al. 2018 the Univer- sal Sentence Encoder (USE) (Cer al. 2018). as well BERT (Devlin al. 2018) We used the pretrained BERT model in two ways to create sentence embedding. The first way is to concatenate the hidden states for the CLS token the last four layers. The second way to concatenate the hidden states of all word tokens in the last four layers and mean pool these representations. Both methods result a 4096 dimension embedding. Finally, we compare to the newly released model, Sentence-Bes Bert (Reimers & Gurevych, 2019). This model is similar to Infersent (Conneau et al. 2017) in that it is trained on natural language inference data, SNLJ (Bowman et al. (005). How- ever, instead of using pretrained word embeddings, they fine-tune BERT in way to induce sentence embeddings
Models from the Literature (Trained on Our Data) These models are amenable to being trained in the exact same setting our own models they only require parallel text. These include the sentence piece averaging model, SP, from (Wieting et al. 2019b), which is among the best of the averaging models (i.e. compared to averaging only words character n-grams) well the LSTM model, BILSTM, from (Wieting & Gimpel| 2017]. These models use a contrastive loss with a margin. Following their settings, we fix the margin 0.4 and tune the number of batches to pool for selecting negative examples from {40, 60, 80, 100}. For both models, we set the dimension of the embeddings to 1024. For BILSTM, we train single layer bidirectional LSTM with hidden states of 512 dimensions. To create the sentence embedding, the forward and backward hidden states are concatenated and mean-pooled. Following (Wieting & Gimpel 2017), we shuffle the inputs with probability tuning from (0.3, 0.5}.
We also implicitly compare to previous machine translation approaches like (Espana-Bool Bonet et al.) 2017 [Schwenk & Douze 2017: Artetxe & Schwenk (0188) in Appendix[A]] where we explore differ- ent variations training LSTM sequence-to-seqqeeee models. We find that our translation baselines reported in the tables below (both LSTM and Transformer) outperform the architectures from these works due to using the Attention and Logit methods mentioned in Section 2 demonstrating that our baselines represent, even over-represent, the otate-of-the-arr for machine translation approaches.
BGT Ablations
Lastly, we compare to ablations of our model to better understand the benefits of parallel data, language-specific variables, the KI. loss term, and how much we gain from the more conventional translation baselines.
ENGLISHAE; English autoencoder on the English side of our en-fr data.
ENGLISHVAE: English variational autoencoder on the English side of our en-fr data.
ENGLISHTR ANS: Translation from en fr.
BILINGUALTRANS Translation from both en to fr and fr en where the encoding parameters are shared but each language has its own decoder.
BGT w/o LANG VARS: A model similar to BILINGUALTRANS. but it includes a prior over the embedding space and therefore KL loss term. This model differs from BGT since it does not have any language-specific variables.
BGT w/o PRIOR: Follows the same architecture as BGT, but without the priors and KL. loss term.
"Note that in all experiments using BERT, including Sentence- BERT, the large, uncased version is used. 2 *Most work evaluating accuracy on STS tasks has averaged the Pearson's r over each individual dataset for each year of the STS competition. However, [Reimers & Gurevych) (2019) computed Spearman's 's over concatenated datasets for each year of the STS competition. To be consistent with previous work, we re-ran their model and calculated results using the standard method, and thus our results are not the same as those reported Reimers & Gurevych (2019).
5

---

Under review as a conference paper at ICLR 2020
4.2 EXPERIMENTAL SETTINGS
The training data for our models a mixture of OpenSubtitles 2018 en-fr data and en-fr Gi- gaword data. To create our dataset, we combined the complete corpora of each dataset and then randomly selected 1,000,000 sentence pairs to be used for training with 10,000 used for validation. We use sent eentencepiece (Kudo & Richardson 2018) with vocabulary size of 20,000 to segment the sentences, and we chose sentence pairs whose sentences are between and 100 tokens each.
In designing the model architectures for the encoders and decoders, we experimented with Trans- formers and LSTMs. Due to better performance, we use a layer Transformer for each of the encoders and a single layer decoder for each of the decoders. This design decision empirically motivated as we found using a larger decoder was slower and worsened performance, but conversely, adding more encoder layers improved performance. More discussion of these trade-offs along with ablations and comparisons to LSTMs are included in Appendix C
For all of our models, set the dimension of the embeddings and hidden states for the encoders and decoders to 1024. Since we experiment with two different architectures 6 we follow two different optimization strategies. For training models with Transformers, we use Adam (Kingma & Ba 2014) with 31 0.9, 32 = 0.98, and e = 10 We use the same learning rate schedule as (Vaswani et al, 2017 i.e., the learning rate increases linearly for 4,000 steps to 5 x 10-1, after which it is decayed proportionally to the inverse square root of the number of steps. For training the LSTM models, we use Adam with a fixed learning rate 0.001. We train our models for 20 epochs.
For models incorporating a translation loss, we used label smoothed cross entropy (Szegedy et al., 2016 Pereyra et al.) 2017) with e = 0.1. For ENGLISHVAE. BGT and BILINGUALTRAN anneal the KL term that it increased linearly for 2 updates, which robustly gave good results in preliminarry experiments. We also found that in training BGT, combining its loss with the BILIN- GUALTRANS objective during training of both models increased performance, and this loss was summed with the BGT loss in all of our experiments. We note that this doesn't 't affect our claim of BGT being a generative model, as this loss is only used in a multi-task objective at training time, and we calculate the generation probabilities according to standard BGT at test time.
we
Lastly, in Appendix we illustrate that it is crucial train the Transformers with large batch sizes. Without this, the model can learn the goal task (such as translation) with reasonable accuracy, but the learned semantic embeddings are of poor quality until batch sizes approximately reach 25,000 tokens. Therefore, we use maximum batch size of 50,000 tokens in our ENGLISHTRANS, BILIN- GUALTRANS, and BGT w/o PRIOR, experiments and 25, 25,000 tokens in our BGT w/o LANGVARS and BGT experiments.
4.3 EVALUATION
Our primary evaluation are the 2012-2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al. 2012: 2013: 3 2014 2015 2016), where the goal is to accurately predict the de- gree to which two sentences have the same meaning as measured by human judges. The evaluation metric is Pearson's 's r with the gold labels.
Secondly, we evaluate on Hard STS, where we combine and filter the STS datasets in order make a more difficult evaluation. We hypothesize that these datasets contain many examples where their gold scores are easy to predict by either having similar structure and word choice and high score or dissimilar structure and word choice and low score. Therefore, we split the data using symmetric word error rate (SWER) finding sentence pairs with low SWER and low gold scores as well as sentence pairs with high SWER and high gold scores. This results in two datasets, Hard+ which have SWERs in the bottom 20% of all STS pairs and whose gold label is between and 1 and
https:///ww.statttt wwt/ http://rro///mmttmaaaaataam 10/training gia----igaa---aa gra..r.e.aaaa. tax
We use LSTMs in our ablations.
7 We define symmetric word error rate for sentences 81 and 82 as 1 WWER 81, 82) WWRR(22, 82), since word error rate (WER) is an asymmetric measure.
*STS scores are between 0 and 5.

---

Under review as a conference paper at ICLR 2020
Data
Sentence
Sentence 2
Gold Score
Hard+
Other ways are needed.
It is necessary to find other means. How long can I keep bread dough in the refriggraaoo?? It's a good idea do both.
4.5
Hard-
How long can you keep chocolate in the freezer?
1.0
Negation
It's good idea.
1.0
Table I: Examples from our Hard STS dataset and our negation split. The sentence pair in the first row has dissimilar structure and vocabulary yet a high gold score. The second sentence pair has similar structure and vocabulary and low gold score. The last sentence pair contains negation, where there is a not Sentence that causes otherwise similar sentences to have low semantic similarity.
Semantic Textual Similarity (STS)
Model
2012
2013
2014
2015
2016
Avg.
Hard+
Hard-
Avg.
BERT (CLS)
33.2
29.6
34.3
45.1
48.4
38.1
7.8
12.5
10.2
BERT (Mean)
48.8
46.5
54.0
59.2
63.4
54.4
3.1
24.1
13.6
Infersent
61.1
51.4
68.1
70.9
70.7
64.4
4.2
29.6
16.9
GenSen
60.7
50.8
64.1
73.3
66.0
63.0
24.2
6.3
15.3
USE
61.4
59.0
70.6
74.3
73.9
67.8
16.4
28.1
22.3
Sentence-BERG BERT
66.9
63.2
74.2
77.3
72.8
70.9
23.9
3.6
13.8
SP
68.4
60.3
75.1
78.7
76.8
71.9
19.1
29.8
24.5
BILSTM
67.9
56.4
74.5
78.2
75.9
70.6
18.5
23.2
20.9
ENGLISHAE
60.2
52.7
68.6
74.0
73.2
65.7
15.7
36.0
25.9
ENGLISH
59.5
54.0
67.3
74.6
74.1
65.9
16.8
42.7
29.8
ENGLISHTRANG
66.5
60.7
72.9
78.1
78.3
71.3
18.0
47.2
32.6
BILINGUALTRANS
67.1
61.0
73.3
78.0
77.8
71.4
20.0
48.2
34.1
BGT w/O LANGVARS
68.3
61.3
74.5
79.0
78.5
72.3
24.1
46.8
35.5
BGT W/o PRIOR
67.6
59.8
74.1
78.4
77.9
71.6
17.9
45.5
31.7
BGT
68.9
62.2
75.9
79.4
79.3
73.1
22.5
46.6
34.6
Results of models and models from prior work. The first six rows pretrained models from the literature, the next two rows are strong baselines trained on the same data as our models, and the last seven rows include model ablations and BGT, our final model. We show results, measured in Pearson's x 100, for each year of the STS tasks 2012-2016 and our two Hard STS datasets.
Table 2:
Hard- where the SWERs are in the top 20% of the gold scores are between 4 and 5. We also evaluate on split where negation was likely present in the example Examples are shown in Table
Lastly, we evaluate on STS in es and ar well as cross-lingual evaluations for en-es, en-ar, and en-tr. We use the datasets from SemEval 2017 (Cer al. (017). For this setting, we train BI..NGUAITTRNS and BGT on million examples from en-es, en and en-tr OpenSub- titles 2018 data.
4.4 RESULTS
The results on the STS and Hard STS are shown in Table 2 10 From the results, we see that BGT has the highest overall performance. does especially well compared prior work on the two Hard STS datasets.
We show further difficult splits Table 3 including negation split, beyond those used in Hard STS and compare the top two performing models in the STS task from Table We also show easier splits in the bottom the table.
From these results, we see that both positive examples that have little shared vocabulary and structure and negative examples with significant shared vocabulary and structure benefit significantly from using deeper architecture. Similarly, examples where negation occurs also benefit from our deeper model. These examples are difficult because more than just the identity of the words is needed to
We selected examples the negation split where one sentence contained not or and other not. We obtained values for STS 2 from prior works using SentEval (Conneau & Kiela (2018). Note that include all datasets for the 2013 competition, including SMT, which is included in SentEval.
7

---

Under review as a conference paper at ICLR 2020
Data Split
I 13,023
BGT
SP
All
75.3
74.1
Negation
705
73.1
68.7
Bottom 20% SWER, label E 0,
404
63.6
54.9
Bottom 10% SWER, label E 0,
72
47.1
22.5
Top 20% SWER, label E [3,
937
20.0
14.4
Top 10% SWER, label E [4,
159
18.1
10.8
Top 20% WER, label E 0, 2]
1380
51.5
49.9
Bottom 20% WER, label E [3, 5]
2079
43.0
42.2
Table Performance, measured in Pearson's 100, for different data splits of the STS data. The first row shows performance across all unique examples, the next row shows the negation split, and the last four rows show difficult examples filtered symmetric word error rate (SWER). The last two rows show relatively easy examples according to SWER.
Model
es ar-ar
en-es
en-tr
BIL.NGUAL
83.4
72.6
64.1
37.6
59.1
BGT w/o LANGVARS
81.7
72.8
72.6
73.4
74.8
BGT w/o PRIOR
84.5
73.2
68.0
66.5
70.9
BGT
85.7
74.9
75.6
73.5
74.9
Table 4: Performance measured in Pearson's paarson'sxx 100, on the SemEval 2017 STS task on the es-es, ar-ar, en-es, en-ar, and en-tr datasets.
determine the relationship of the sentences, and this is something that SP not equipped for since it is unable to model word order. The bottom two rows show easier examples where positive examples have high overlap and low SWER and vice versa for negative examples. Both models perform similarly on this data, with the BGT model having small edge consistent with the overall gap between these two models.
Lastly, in Table 4 we show the results of STS evaluations in es and a and cross-lingual evaluations for en-es, en-ar, and en-tr. From these results, we see that BGT has the best performance across all datasets, however the performance is significantly stronger than the BILINGALLTRAN and BGT w/o PRIOR baselines in the cross-lingual setting. Since BGT w/O LANG VARS also has significantly better performance on these tasks, most of this gain seems to be due to the prior have regularizing effect. However, BGT outperforms BGT w/o LANGVARS overall, and we hypothesize that the gap in performance between these two models is due to BGT being able to strip away the language-specific information the representations with its language-specific variables, allowing for the semantics of the sentences to be more directly compared.
ANALYSIS
We next analyze our BGT model by examining what elements of syntax and semantics the language and semantic variables capture relative both to each-other and to the sentence embeddings from the BILINGUALTRAN models. We also analyze how the choice of language and its lexical and syntactic distance from English affects the semantic and syntactic information captured by the semantic and ionguage-specifee encoders. Finally, we also show that our model is capable of sentence generation in type of style transfer, demonstrating its capabilities as generative model.
5.1 STS
We first show that the language variables are capturing little semantic information by evaluating the learned English language-specifie variable from BGT model our suite of semantic tasks. The results in Table 5 show that these encoders perform closer to random encoder than the seman- tic encoder from BGT. This is consistent with what we would expect see if they are capturing extraneous language-specific information.
5.2 PROBING
We probe our BGT semantic and language-specifts encoders, along with our BILINGUALTRANS encoders baseline, compare and contrast what aspects of syntax and semantics they
8

---

Under review as a conference paper at ICLR 2020
Semantic Textual Similarity (STS)
Model
2012
2013
2014
2015
2016
Hard+
Hard-
Random Encoder
51.4
34.6
52.7
52.3
49.7
4.8
17.9
English Language Encoder
44.4
41.7
53,8
62.4
61.7
15.3
26.5
Semantic Encoder
68.9
62.2
75.9
79.4
79.3
22.5
46.6
Table 5: STS performance the 2012-2016 datasets and our STS Hard datasets for randomly
initialized Transformer, the trained English language-specific encoder from BGT, and the trained
semantic encoder from BGT. Performance is measured in Pearson's r 100.
Lang. Model
STS
Num. Num.
Depth
Top Con.
Word
Len.
P. Num.
First
Gend.
BILINGUAL. RANS
71.2
78.0
76.5
28.2
65.9
80.2
74.0
56.9
88.3
53.0
Semantic Encoder
72.4
84.6
80.9
29.7
70.5
77.4
73.0
60.7
87.9
52.6
ir
Language Encoder
56.8
75.2
72.0
28.0
63.6
65.4
80.2
65.3
92.2
Language Encoder
56.5
BUI.INUAA..RRAN
70.5
84.5
82.1
29.7
68.5
79.2
77.7
63.4
90.1
54.3
Semantic Encoder
72.1
85.7
83.6
32.5
71.0
77.3
76.7
63.1
89.9
52.6
es
Language Encoder
55.8
75.7
73.7
29.1
63.9
63.3
80.2
64.2
92.7
es Language Encoder
54.7
BILINGUAL. RANS
70.2
77.6
74.5
28.1
67.0
77.5
72.3
57.5
89.0
Semantic Encoder
70.8
81.9
80.8
32.1
71.7
71.9
73.3
61.8
88.5
Language Encoder
58.9
76.2
73.1
28.4
60.7
71.2
79.8
63.4
92.4
BILINGUALTRRASS
70.7
78.5
74.9
28.
60.2
78.4
72.1
54.8
87.3
tr
Semantic Encoder
72.3
81.7
80.2
30.6
66.0
75.2
72.4
59.3
86.7
en Language Encoder
57.8
77.3
74.4
28.3
63.1
67.1
79.7
67.0
92.5
BILINGUALTRANG
71.0
66.4
64.6
25.4
54.1
76.0
67.6
53.8
87.8
ja
Semantic Encoder
71.9
68.0 O
66.8
27.5
58.9
70.1
68.7
52.9
86.6
en Language Encoder
60.6
77.6
76.4
28.0
64.6
70.0
80.4
62.8
92.0
Table Average STS performance for the 2012-2016 datasets, measured in Pearson's 's r x 100, fol- lowed by probing results on predicting number of subjects, number of objects, constituent tree depth, top constituent, word content, length, number of punctuation marks, the first punctuation mark, and whether the articles in the sentence are the correct gender. All probing results are measured in accuracy x 100.
learning relative to each other across five languages with various degrees of similarity with English. All models are trained on the OpenSubtitles 2018 corpus. We use the datasets from (Conneau al., 2018) for semantic tasks like number of subjects and number of objects, and syntactic tasks like tree depth, and top constituent. Additionally, we include predicting the word content and sentence length. We also add our own tasks to validate our intuitions about punctuation and sanguage-specingg information. the first of these, panctuation number, we train a classifier to predict the number of punctuation in a sentence. To make the task more challenging, we limit each label to have at most 20,000 examples split among training, validation, and testing data [2 In the second task, punctuation first, train classifier to predict the identity of the first punctuation mark in the sentence. In our last task, gender, we detect examples where the gender of the articles in the sentence is incorrect in French of Spanish. To create incorrect example, we switch articles from {le, la, un, une} for French and (el, la, los, las} for Spanish, with their (indefinite or definite for French and singular or plural for Spanish) counterpart with the opposite gender. This dataset was balanced so random chances gives 50% on the testing data. All tasks use 100,000 examples for training and 10,000 examples for validation and testing. The results of these experiments are shown in T
These results show that the source separation is effective stylistic and language-speeific informa- tion like length, punctuation and language-specific gender information are more concentrated in the language variables, while word content, semantic and syntactic information are more concentrated in the semantic encoder. The choice of language is also seen to be influential coders are capturing. When the languages are closely related to English, like in French and Spanish, the performance difference between the semantic and English language encoder is larger for word content, subject number, object number than for more distantly related languages like Arabic and
what these en-
Puncttuttion were taken from the set {'! $ + 12ehe labels are from I punctuation mark up to 10 marks with an additional label consolidating 11 more marks.
</::<>>>>?> (==>"(1'((--""

---

Under review as a conference paper at ICLR 2020
Turkish. In fact, word content performance is directly tied to how well the alphabets of the two lan- guages overlap. This relationship matches our intuition, because lexical information will be cheaper to encode in the semantic variable when it is shared between the languages. Similarly for the tasks of length, punctuation first, and punctuation number, the gap in performance between the two encoders also grows the languages become more distant from English. Lastly, the gap STS performance between the two encoders shrinks as the languages become more distant, which again is what we would expect, the language-specific encoders are forced to capture more information.
Japanese is an interesting case in these experiments, where the English language-specific encoder outperforms the semantic encoder on the semantic and syntactic probing tasks. Japanese is a very distant language to English both in its writing system and in its sentence structure (it is an SOV language, where English an SVO language). However, despite these difference, the semantic encoder strongly outperforms the English language-specific encoder, suggesting that the underlying meaning of the sentence much better captured by the semantic encoder.
5.3
GENERATION AND STYLE TRANSFER
Source
you know what i' 've seen?
Style
he said, "since when is going fishing" had anything do with fish?" he said, "what going do with mc since saw you?"
Output
Source
guys, that was the tech unit.
Style
is well, "capicci"
Output
is that what, "technician"?
Source
the pay no good, but it's money. do we know cause death?
Style
Output
do we have any money?
Source
we're we're always doing stupid things.
Style
right listen, like being exactly where am, all right, like being stupid, but am always here.
Output
Table 7: Style transfer generations from our learned BGT model. Source refers to the sentence fed into the semantic encoder, Style refers to the sentence fed into the English monguag--specffii encoder, and Output refers to the text generated by our model.
In this section, we qualitatively demonstrate the ability of our model generate sentences. We focus on style-transfer task where we have original seed sentences from which we calculate our semantic vector 2sem and language specific vector žen- Specifically, we feed in a Source sentence into the semantic encoder to obtain žxem: and another Style sentence into the English language- specific encoder to obtain žen- We then generate new sentence using these two latent variables. This can be seen has the semantics of the Source sentence and the style the Style sentence. We use our en- fr BGT model from Table 6 and show some examples in Table 7 All input sentences out en-fr OpenSubtitles data. From these examples, we see further evidence of the role of the semantic and language-specific encoders, where most of the semanties (e.g. topical word such as seen and tech in the Source sentence) are reflected in the output, but length and structure are more strongly influenced by the language-specific encoder.
type of style transfer where we expect the model to generate a sentence that
from held-
6 CONCLUSION
We propose Bilingual Generative Transformers, a model that uses parallel data to learn to perform source separation of common semantic information between two languages from language-specific information. We show that the model is able to accomplish this source separation through probing tasks and text generation in a style-transfer setting. We find that our model bests all baselines on semantic similarity tasks, with the largest gains coming from new challenge we propose Hard STS, designed to foil methods approximating semantic similarity as word overlap. We also find our model to be especially effective on gross-lingua semantic similarity, due to its stripping away of ianguage-specffii information allowing for the underlying semantics to be more directly compared. In future work, will explore generalizing this approach the multilingual setting.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre, SemEval-2014 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Com- putational Semantics-Voleee Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation. Association for Computational Linguisties, 2012.
I: Proceedings of the main conference and the shared task, and
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agire,, and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Se- mantics (* SEM), Volume I: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, volume 1, pp. 32-43, 2013.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Eval- uation (SemEval 2014), 2014.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Inigo Lopez-Gazpio Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), 2015.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, Ger- man Rigau, and Janyce Wiebe. SemEval-2016 task I: Semantic textual similarity, monolingual and cross- loosull evaluation. Proceedings of SemEval, pp. 497-511, 2016.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. In Proceedings of the International Conference on Learning Representations, 2017.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:/812. 2018.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642 642, Lisbon, Portugal, September 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio and Lucia Specia. SemEval-2011 Task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the Ith International Workshop on Semantic Evaluation (SemEval-201/ pp. 1-14, Vancouver, Canada, August 2017.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- stant, Mario Sunjjardo-Cespedee Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.1 11175, 2018
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. A multi-task approach for dis- entangling syntax and semantics in sentence representations. arXiv preprint arXiv: arXiv:1904.011 1904.0/173, 2019.
Alexis Conneau and Douwe Kiela, Senteval: An evaluation toolkit for universal sentence represen- tations. arXiv preprint 1803.0880.00544 2018.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680, Copenhagen, Denmark, September 2017.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint /805 1805.01070. 2018.
11

---

Under review as a conference paper at ICLR 2020
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv:1810.0480. 2018.
Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of COLING, 2004.
Cristina Espana- Espana-Bonet, Adám Csaba Varga, Alberto Barrón-Cedeinn and Josef van Genabith. An empirical analysis of nmt-derived interlingual embeddings and their use in parallel sentence iden- tification. IEEE Journal of Selected Topics in Signal Processing, 11(8):1340-1355 2017.
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burchh PPDB: The Paraphrase Database. In Proceedings of HLT-NAACI, 2013.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Kerg-Kirkpatrick. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv:/901.05:0 2019.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlahelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. drXiv:1312.6114,200 2013.
arXiv preprint
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems 28, pp. 3294-3302, 2015.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.006. 06226, 2018.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4273- 4283, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 1/D199//11 URL http::ttps//cc //ww acl www.wwww.boooo www..ergg/aatthooog///1111-1111
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre-
sentations of words and phrases and their compositionality, In Advances in Neural Information Processing Systems, 2013.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. Proceedings of Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp. 2227-2237, 2018.
Martin Popel and Ondiej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 110(1):43-70, 2018.
12

---

Under review as a conference paper at ICLR 2020
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using stamese bert- networks. arXiv preprint arXiv: 10084, 2019.
Holger Schwenk. Filtering and mining parallel data in a joint multilingual space. arXiv preprint arXiv: 09822, 2018.
Holger Schwenk and Matthijs Douze. Learning joint multilingual sentence representations with neural machine translation. arXiv preprint arXiv:1704.041:: 2017.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv: 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28266 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.074650 2018.
John Wieting and Kevin Gimpel. Revisiting recurrent networks for paraphrastic sentence embed- dings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 2078-2088 Vancouver, Canada, July 2017.
John Wieting and Kevin Gimpel. ParaNMT-50M: Pushing the limits of paraphrastic sentence em- beddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pp. 451-462. As- sociation for Computational Linguistics, 2018. URL pttt:://clwwbb.ot wwtp.eebbo..r/tt.toogooo.....o
P1e-100 1042
John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. arXiv preprint 19X1::190..1044,, 2019.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sen- tence embeddings. In Proceedings of the International Conference on Learning Representations, 2016a.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Charagram: Embedding words and sentences via character n-grams. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1504-1515, 2016b.
John Wieting, Taylor BirggKiikpatrick. Kevin Gimpel, and Graham Neubig, Beyond bleu: Training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4344-4355, 2019a.
John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpattick.. Simple and effective paraphrastic similarity from parallel translations. Proceedings of the 2019b.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th Interna- tional Conference on Machine Learning- Volume 70, pp. 3881 3881-3890. JMLR. org, 2017.
Improved variational
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In Interna- tional Conference on Machine Learning, pp. 7673-7682, 2019.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third buce shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 1 11th Workshop on Building and Using Comparable Corpora, pp. 39-42 2018
13

---

Under review as a conference paper at ICLR 2020
A LOCATION OF SENTENCE EMBEDDING IN DECODER FOR LEARNING REPRESENTATIONS
As mentioned in Section 2 we experimented with 4 ways to incorporate the sentence embedding into the decoder: Word, Hidden, Attention, and Logit. We also experimented with combinations of these 4 approaches. We evaluate these embeddings on the STS tasks and show the results, along with the time train the models epoch in Table
For these experiments, we train a single layer bidirectional LSTM (BiLSTM) ENGLISHTRANS model with embedding size set 1024 and hidden states 512 dimensions (in order to be roughly equivalent our Transformer models). To form the sentence embedding in this variant, we mean pool the hidden states for each time step. The cell states of decoder are initialized to the zero vector.
Architecture
STS
Time (s)
BiLSTM (Hidden)
54.3
1226
BiLSTM (Word)
67.2
1341
BiLSTM (Attention)
68.8
1481
BiLSTM (Logit)
69.4
1603
BiLSTM (Word + Hidden)
67.3
1377
BiLSTM (Word Hidden Attention)
68.3
1669
BiLSTM (Word + Hidden + Logit)
69.1
1655
BiLSTM (Word Hidden Attention + Logit)
68.9
1856
Table 8: Results for different ways of incorporating the sentence embedding in the decoder for BiLSTM on the Semantic Textual Similarity (STS) datasets, along with the time taken to train the model for epoch. Performance is measured in Pearson's x 100.
From this analysis, we see that the best performance is achieved with Logit, when the sentence embedding is place just prior the softmax. The performance is much better than Hidden Hid- den+ Word used in prior work. For instance, recently (Artetxe Schwenk 2018) used the Hid- Wen+Word strategy in learning multilingual sentence embeddings.
A.1 VAE TRAINING
We also found that incorporating the latent code of a VAE into the decoder using the Logit strategy increases the mutual information while having little effect the log likelihood. We trained two LSTM VAE models following the settings and aggressive training strategy in (He et al.) 2019). where LSTM model used the Hidden strategy and the other used the Hidden + Logit strategy. We trained the models on the en side of our en-fr fr data. We found that the mutual information increased form 0.89 to 2.46, while the approximate negative log likelihood, estimated by importance weighting, increased slightly from 53.3 to 54.0 when using Logit.
B RELATIONSHIP BETWEEN BATCH SIZE AND PERFORMANCE FOR TRANSFORMER AND LSTM
has been observed previously that the performance of Transformer models sensitive to batch size Popel & Bojar 2018). We found this to be especially true when training seguence-t-ssqqeene models to learn sentence embeddings. Figure 3 shows plots of the average 2012-2016 STS perfor- mance of the learned sentence embedding as batch size increases for both the BiLSTM and Trans- former. Initially, at a batch size of 2500 tokens, sentence embeddings learned are worse than random, though validation perplexity does decrease during this time. Performance rises batch size in- creases up to around 100,000 tokens. In contrast, the BiLSTM more robust to batch size, peaking much earlier around 25,000 tokens, and even degrading at higher batch sizes.
14

---

Under review as a conference paper at ICLR 2020
Performace Vs. Batch Size
Transformer
0.70
LSTM
0.65
STS Performance
0.60
0.55
0.50
50000
100000
150000
200000
250000
300000
Batch size (Max Tokens Per Batch)
Figure 3: The relationship between average performance for each year of the STS tasks 2012-2016 (Pearson's 's x 100) and batch size (maximum number of words per batch).
Architecture
STS
Time (s)
Transformer (5L/1L)
70.3
1767
Transformer (3L/IL)
70.1
1548
Transformer (IL/IL)
70.0
1244
Transformer (5L/5L)
69.8
2799
Table 9: Results the Semantic Textual Similarity (STS) datasets for different configurations of ENGLISHTRANS along with the time taken to train the model for epoch. (XL/YL.) means X layers were used in the encoder and Y layers the decoder. Performance is measured in Pearson's 100.
C MODEL ABLATIONS
In this section, vary the number of layers in the encoder and decoder in BGT w/o PRIOR. We that performance increases as the number of encoder layers increases, and also that large decoder hurts performance, allowing compared to those in Tablen that Transformers outperform BiLSTMS in these experiments.
training time by using single layer. These results can be
D CLASSIFICATION EXPERIMENTS
To explore our embeddings more detail, we evaluated them on the Quora Question Pairs datase 13 (QQP). This is paraphrase classification task, which is also part of GLUE (Wang al. 2018).
Since the test set is private, we deviated slightly from the standard evaluation protocol and split the development set into two halves of 20,215 examples each one half for model selection and the other for evaluation. We evaluated in two ways, cosine, where we score all pairs with cosine similarity and then find the threshold that gives the best accuracy, and logistic regression where use logistic regression. Its worth noting that the pretrained baseline models on this task were directly trained to produce the feature set used by the downstream classifier, while our embeddings trained without this supervision. They also tend to have larger dimensions which also gives them an advantage which is discussed in more detail in (Wieting Kiela 2019). The results are shown in Table 10 and show that BGT model outperforms the baseline models, SP, ENGLISHTRAN TRANS,
data. www.guurccoomm com rst Quora-uooo nataset Re Release- Question-Paae
15

---

Under review as a conference paper at ICLR 2020
Model
Dim
QQP (cosine)
QQP (logistic regression)
BERT (CLS)
4096
65.7
77.2
BERT (Mean)
4096
68.9
79.3
Infersent
4096
69.3
79.9
GenSen
4096
68.
80.9
USE
512
75.8
78.9
Sentence- BERT
1024
74.5
81.0
SP
1024
69.6
76.0
ENGLISHTRANS
1024
69.5
77.3
BGT w/o PRIOR
1024
69.4
77.1
BGT
1024
69.9
77.5
Table 10: Results the Quora Question Pairs (QQP) datasets for prior work, baselines and BGT model using two classification strategies. Performance is measured accuracy x 100.
and BILINGUALTRANS for both evaluations, and compares favorably to the pretrained models when evaluated using cosine similarity scores. The only models which perform better are USE which was trained on Quora data in unsupervised way and Sentence-BERT which uses BERT. Our models are not as strong when using classification for final predictions. This indicates that the embeddings learned by our approach may be most useful when downstream training possible though semi- supervised objectives that consider the downstream task might aid our approach, like the baselines, downstream training the goal.
16


Reviews:
[{"review_id": "rJxpiIYr5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a model that, given parallel bilingual data, separates the common semantics from the language-specific semantics on a sentence level. \n\nOverall the presentation is clear and the experiments show gains over the baselines. One major point of confusion however is that, while early on in the paper (introduction), it is stated repeatedly that one of the strengths of the proposed model is that it is sensitive to word order on a sentence level, this particular aspect of the model is neither evaluated nor analysed. Instead the empirical analysis focuses on sentence length, punctuation and semantics. The analysis of all these three aspects is superficial: for sentence length, it consists of  computing the sentence mean and median; for punctuation,  it consists of masking punctuation; and the last part just computes vectors of nouns only (and states that this is \"semantics\"). But there is no analysis per se.\n\nOverall, the paper presents a model and shows gains over baselines. The extent to which these gains are due to tuning as opposed to the inherent design of the model is not clear. The analysis is superficial.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJgs3cCaFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a bilingual generative model for sentence embedding based variational probabilistic framework. By separating a common latent variable from language-specific latent variables, the model is able to capture what's in common between parallel bilingual sentences and language-specific semantics. Experimental results show that the proposed model is able to produce sentence embeddings that reach higher correlation scores with human judgments on Semantic Textual Similarity tasks than previous models such as BERT. \n\nStrength: 1) the idea of separating common semantics and language-specific semantics in the latent space is pretty neat; 2) the writing is very clear and easy to follow; 3) the authors explore four approaches to use the latent vectors and four approaches to merge semantic vectors, makes the final choices reasonable.\n\nWeakness:\n\n1) Experiments: \n\nMy major concern is the fairness of the experiments.  The authors compare their model with many state-of-the-art models that could produce sentence embeddings. However, how they produce the sentence embeddings with existing models is not convincing. For example, why using the hidden states of the last four layers of BERT? Moreover, the proposed model is trained with parallel bilingual data, while the BERT model in comparison is monolingual.  Also, the proposed deep variational model is close to an auto-encoder framework. You can also train a bilingual encoder-decoder transformer model (perhaps with pre-trained BERT parameters) with auto-encoder objective using the same parallel data set.  It seems to be a more comparable model to me. \n\nAlthough the proposed model is based on variational framework, there's no comparison with previous neural variational models that learn encodings of texts as well such as https://arxiv.org/abs/1511.06038. \n\n2) Ablation study and analysis\n\nI really like the idea of separating common semantic latent variables with language-specific latent variables.  However, I expected to see more analysis or experimental results to show why it is better than a monolingual variational sentence embedding framework. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeqGhkAYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of constructing a sentence embedding using a generative transformer model which encodes semantic aspects and language-specific aspect separately. They use transformers to encode and decode sentence embedding, and the objective reconstructs input with a latent variables (language variables for each language and semantic language).  These latent variables are sampled from multivariate Gaussian prior, and the learning uses evidence lower bound (ELBO) for variational approximation of the joint distribution of latent variables and input. \n\nThe method is evaluated on two tasks: sentence similarity task and machine translation evaluation metric tasks. Both tasks evaluates how similar are two sequences, and the metric is correlation score with score’s from human judge. The model shows promising results on the first task, but weaker results on the second task, especially when compared against pretty naively built sentence embedding from BERT model. I’m not expert in sentence embedding literature, so a bit tricky to evaluate, but baselines seem strong and experimental results on semantic textual similarity task.\n\nIn terms of evaluation, I appreciated how they defined harder subset of the evaluation dataset and showed a larger improvements on those portions of the dataset. The The paper also includes analysis on what is captured by their language-specific latent vector and semantic latent vector. While I’m not totally convinced this distinction between language-specific characteristics and semantics of the sentence, it makes it easier to understand what’s going on in the model. \n\nOne of my question is, why not test this method in more popular benchmark such as MNLI or other classification tasks? MNLI evaluates how each sentence pair relates to one another, thus would be a good benchmark for sentence embeddings as well. Having to encode all the information about a sentence into a single vector will make these sentence embedding model weaker than other models which can do cross sentence attentions and etc, but I think that’s the genuine limitation of sentence embedding research and has to be clarified as such. I recommend discussing and clarifying these points. \n\nI’m a bit unclear how these sentence embeddings are translated into a score that decides the degree to which sentences have the same meaning. Is it just cosine similarity of two sentence embedding vectors?\n\nWhile the purpose of these references is to generate sentences instead of building a sentence embedding, the method is related and comparison and discussion would be worthwhile. \n\nGenerating Sentences from a Continuous Space\nSamuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio\nhttps://arxiv.org/abs/1511.06349\nToward Controlled Generation of Text\nZhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing\nhttps://arxiv.org/abs/1703.00955\n\n\nComments & Questions:\n- Methods using a large amount of unsupervised monolingual data shows very strong performance in a panoply of NLP tasks these days. If I understand correctly, this model is constrained by the amount of bitext — some analysis on this would be interesting. \n- Figure 1 mentions about “Section 3, 4” but I don’t think they are correct references? \n- BERT baseline seemed not to allow fine-tuning of the LM parameters. I think this makes the baseline significantly weaker? \n- It seems odd that only English semantic encoder is used to downstream application.\n- Does table 3 covers all the data? What proportion of the data is covered by each row?\n- Given the similarity of English and French, I’m not sure how “language-specific” such latent vectors are. It would be much more interesting analysis if it studies distant language pairs. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model for building sentence embeddings using a generative transformer model that encoders separately semantic aspects (that are common across languages)  and language-specific aspects. The authors evaluate their embeddings in a non-parametric way (i.e., on STS tasks by measuring cosine similarity) and find their method to outperform other sentence embeddings methods. The main concern that both reviewers (and myself) have about this work relates to its evaluation part. While the authors present a set of very interesting difficult evaluation and probing splits aiming at quantifying the linguistic behaviour of their model, it is unsatisfying the fact that the authors do not evaluate their model extensively in standard classification embedding benchmarks (e.g., as in GLUE). The authors comment: “[their model in producing embeddings] it isn’t as strong when using classification for final predictions. This indicates that the embeddings learned by our approach may be most useful when no downstream training is possible”. If this is true, why is it the case and isn’t it quite restrictive? I think this work is interesting with a nice analysis but the current empirical results are borderline  (yes, the model is better on STS, but this is quite limited of an idea compared to using these embeddings as features in a classification tasks). As such, I do not recommend this paper for acceptance but I do hope that authors will keep improving their method and will make it work in more general problems involving classification tasks.
Decision: Reject