Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents a model that, given parallel bilingual data, separates the common semantics from the language-specific semantics on a sentence level. 

Overall the presentation is clear and the experiments show gains over the baselines. One major point of confusion however is that, while early on in the paper (introduction), it is stated repeatedly that one of the strengths of the proposed model is that it is sensitive to word order on a sentence level, this particular aspect of the model is neither evaluated nor analysed. Instead the empirical analysis focuses on sentence length, punctuation and semantics. The analysis of all these three aspects is superficial: for sentence length, it consists of  computing the sentence mean and median; for punctuation,  it consists of masking punctuation; and the last part just computes vectors of nouns only (and states that this is "semantics"). But there is no analysis per se.

Overall, the paper presents a model and shows gains over baselines. The extent to which these gains are due to tuning as opposed to the inherent design of the model is not clear. The analysis is superficial.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper addresses the problem of constructing a sentence embedding using a generative transformer model which encodes semantic aspects and language-specific aspect separately. They use transformers to encode and decode sentence embedding, and the objective reconstructs input with a latent variables (language variables for each language and semantic language).  These latent variables are sampled from multivariate Gaussian prior, and the learning uses evidence lower bound (ELBO) for variational approximation of the joint distribution of latent variables and input. 

The method is evaluated on two tasks: sentence similarity task and machine translation evaluation metric tasks. Both tasks evaluates how similar are two sequences, and the metric is correlation score with score’s from human judge. The model shows promising results on the first task, but weaker results on the second task, especially when compared against pretty naively built sentence embedding from BERT model. I’m not expert in sentence embedding literature, so a bit tricky to evaluate, but baselines seem strong and experimental results on semantic textual similarity task.

In terms of evaluation, I appreciated how they defined harder subset of the evaluation dataset and showed a larger improvements on those portions of the dataset. The The paper also includes analysis on what is captured by their language-specific latent vector and semantic latent vector. While I’m not totally convinced this distinction between language-specific characteristics and semantics of the sentence, it makes it easier to understand what’s going on in the model. 

One of my question is, why not test this method in more popular benchmark such as MNLI or other classification tasks? MNLI evaluates how each sentence pair relates to one another, thus would be a good benchmark for sentence embeddings as well. Having to encode all the information about a sentence into a single vector will make these sentence embedding model weaker than other models which can do cross sentence attentions and etc, but I think that’s the genuine limitation of sentence embedding research and has to be clarified as such. I recommend discussing and clarifying these points. 

I’m a bit unclear how these sentence embeddings are translated into a score that decides the degree to which sentences have the same meaning. Is it just cosine similarity of two sentence embedding vectors?

While the purpose of these references is to generate sentences instead of building a sentence embedding, the method is related and comparison and discussion would be worthwhile. 

Generating Sentences from a Continuous Space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio
https://arxiv.org/abs/1511.06349
Toward Controlled Generation of Text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing
https://arxiv.org/abs/1703.00955


Comments & Questions:
- Methods using a large amount of unsupervised monolingual data shows very strong performance in a panoply of NLP tasks these days. If I understand correctly, this model is constrained by the amount of bitext — some analysis on this would be interesting. 
- Figure 1 mentions about “Section 3, 4” but I don’t think they are correct references? 
- BERT baseline seemed not to allow fine-tuning of the LM parameters. I think this makes the baseline significantly weaker? 
- It seems odd that only English semantic encoder is used to downstream application.
- Does table 3 covers all the data? What proportion of the data is covered by each row?
- Given the similarity of English and French, I’m not sure how “language-specific” such latent vectors are. It would be much more interesting analysis if it studies distant language pairs. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a bilingual generative model for sentence embedding based variational probabilistic framework. By separating a common latent variable from language-specific latent variables, the model is able to capture what's in common between parallel bilingual sentences and language-specific semantics. Experimental results show that the proposed model is able to produce sentence embeddings that reach higher correlation scores with human judgments on Semantic Textual Similarity tasks than previous models such as BERT. 

Strength: 1) the idea of separating common semantics and language-specific semantics in the latent space is pretty neat; 2) the writing is very clear and easy to follow; 3) the authors explore four approaches to use the latent vectors and four approaches to merge semantic vectors, makes the final choices reasonable.

Weakness:

1) Experiments: 

My major concern is the fairness of the experiments.  The authors compare their model with many state-of-the-art models that could produce sentence embeddings. However, how they produce the sentence embeddings with existing models is not convincing. For example, why using the hidden states of the last four layers of BERT? Moreover, the proposed model is trained with parallel bilingual data, while the BERT model in comparison is monolingual.  Also, the proposed deep variational model is close to an auto-encoder framework. You can also train a bilingual encoder-decoder transformer model (perhaps with pre-trained BERT parameters) with auto-encoder objective using the same parallel data set.  It seems to be a more comparable model to me. 

Although the proposed model is based on variational framework, there's no comparison with previous neural variational models that learn encodings of texts as well such as https://arxiv.org/abs/1511.06038. 

2) Ablation study and analysis

I really like the idea of separating common semantic latent variables with language-specific latent variables.  However, I expected to see more analysis or experimental results to show why it is better than a monolingual variational sentence embedding framework. 
review_title: Official Blind Review #2

Meta-review: This paper presents a model for building sentence embeddings using a generative transformer model that encoders separately semantic aspects (that are common across languages)  and language-specific aspects. The authors evaluate their embeddings in a non-parametric way (i.e., on STS tasks by measuring cosine similarity) and find their method to outperform other sentence embeddings methods. The main concern that both reviewers (and myself) have about this work relates to its evaluation part. While the authors present a set of very interesting difficult evaluation and probing splits aiming at quantifying the linguistic behaviour of their model, it is unsatisfying the fact that the authors do not evaluate their model extensively in standard classification embedding benchmarks (e.g., as in GLUE). The authors comment: “[their model in producing embeddings] it isn’t as strong when using classification for final predictions. This indicates that the embeddings learned by our approach may be most useful when no downstream training is possible”. If this is true, why is it the case and isn’t it quite restrictive? I think this work is interesting with a nice analysis but the current empirical results are borderline  (yes, the model is better on STS, but this is quite limited of an idea compared to using these embeddings as features in a classification tasks). As such, I do not recommend this paper for acceptance but I do hope that authors will keep improving their method and will make it work in more general problems involving classification tasks.
Decision: Reject