Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper describes a method to build sparse multilingual word vectors that is designed to scale easily to many languages. The key idea is to pick one language to be the source language, and then to build word embeddings and then a sparse dictionary + sparse coefficients  for that source language monolingually. Other target languages then first align their embedding to the source using a seed list of translations and standard techniques, and then determine their sparse coefficients based on the fixed source sparse dictionary. This latter process is a convex optimization, which improves efficiency and stability. The method is tested with an established correlation-based intrinsic metric (QVEC-CCA) as well as by using the multilingual embeddings to project systems for cross-lingual document classification, dependency parsing and natural language inference.

The core idea of this paper is substantially simpler than the method it compares to (BiSparse), which jointly optimizes dictionaries, coefficients and cross-lingual coefficient alignment. So, the question that immediately comes to mind for me is, how much accuracy am I giving up for improved scalability to multiple languages? This could be easily tested for the two language case, where BiSparse could be directly compared without modification to their proposed method. Instead, BiSparse is modified to fit scale to the multilingual setting (becoming MultiSparse), but since it shares the constraint that the source dictionary and coefficients are fixed, it has already lost a lot of the power of joint optimization. I think the paper would be stronger with a two-language experiment where we would expect the proposed method to lose to BiSparse, but we could begin to understand what has been given up for scalability.

I also wonder how relevant bilingual word embeddings are in a world of multilingual BERT and similar approaches. It would be interesting to know how cross-lingual embedding-in-context methods would do on the extrinsic evaluations in this paper, though I also acknowledge that this could be considered beyond the scope of the paper.

Otherwise, this is a fine paper. It is well written and easy to follow. The experiments look sane, and the inclusion of both intrinsic and extrinsic tasks makes them fairly convincing. I have only a few remaining nitpicks:

(1) As someone relatively unfamiliar with multilingual (as opposed to bilingual) word embedding research, it wasn’t clear to me how the experiments described here tested the multilinguality (as opposed to bilinguality) of the embeddings. It would be nice to provide an explanation for why (beyond the obvious efficiency gains) one couldn’t just do the necessary language pairs with bilingual methods for these experiments. And if one could perform the tests with bilingual methods, they should be included as baselines. 

(2) The discussion of MultiSpare hyper-parameter tuning appears in the Monolingual Experiments section, leading me to wonder what target languages were used for this tuning process.

(3) In the second-last paragraph of 3.2.2, there is a sentence fragment that ends in “nonetheless their multiCluster and multiCCA embedding spaces contain no embeddings for”

(4) The last paragraph before the Conclusion also feels like a fragment, or like two sentences have been spliced together: “We have detailed the differences to Upadhyay et al. (2018) extends the previous work by incorporating dependency relations into sparse coding.”

There are also several places where periods seem to have been left out (such as immediately before the above sentence).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper proposes a new approach for generating multilingual sparse representations. 
For generating such representations, the proposed approach solves a series of convex optimization problems, serially for each language. Compared to previous work for generating sparse cross-lingual representations which is applicable to a pair of languages, the proposed approach is applicable to an arbitrary number of languages.

The paper argues that these sparse representations can lead to better performance for downstream tasks and interpretability. This is demonstrated using experiments on QVEC-CCA (for interpretability analysis), NLI, cross-lingual document classification, and dependency parsing (downstream tasks).

Overall, the approach is well-motivated and performs well empirically. The experimental setup is also described in detail. 

Minor - I did not understand the benefit of MAMUS being "stable" across languages, as argued from Fig 1? The use of the word "cognitively" in the statement "representations determined by MAMUS behave in a cognitively more plausible manner" also seems a stretch to me.
 
One issue that the authors should discuss is whether such representations hold any extra value over contextual representations like multilingual Elmo, BERT etc. For instance, why would someone use MAMUS representations instead?
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes a method to generate sparse multilingual embeddings. The key idea is to build only one set of source basis embeddings, and then represent all multilingual embeddings as a linear combination of these source embeddings. I felt the paper is a nice extension of the Vyas 2016 paper. Compared to existing approaches, their method will be faster to train and will need less data (particularly useful for low resource languages). Since I am less aware of work in this area, I cannot comment on whether the evaluation is complete. Particularly, I wonder if there is a qualitative way to show interpretability of the sparse vectors created by the method. We currently only have QVEC-CCA numbers to judge interpretability.  A few more suggestions:

1. I got confused by the sentence ".. over a reduced number of parameters for each target language as it treats D_s as D_t .". Things got clear from the equations, but will be good to fix.

2. Figure 1 was very difficult to understand. Ideally your caption should be enough to understand the figure. 
review_title: Official Blind Review #2

Meta-review: This paper describes a new method for creating word embeddings that can operate on corpora from more than one language.  The algorithm is simple, but rivals more complex approaches.  

The reviewers were happy with this paper.  They were also impressed that the authors ran the requested multi-lingual BERT experiments, even though they did not show positive results. One reviewer did think that non-contextual word embeddings were of less interest to the NLP community, but thought your arguments for the computational efficiency were convincing.
Decision: Accept (Poster)