Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an approach to align word vector spaces based on a dictionary of pairs (e.g. translations) that retains "common geometric structure" (inner products within each vector space for a subset of word pairs). In essence the approach applies preprocessing to each vector space, finds a projection of one vector spaces to the (lower-dimensional) vector space minimising a combination of a reconstruction and transfer loss, and then aligns the two using weighted procrustes to resolve rotational symmetries.

The topic is of interest to the community and related papers on vector space alignment have been published at previous ICLRs. Essentially, this paper combines insights from a number of different works, with good results. However, overall the solution is relatively complex with a lot of moving parts.

Strengths:
* The paper is largely well written and easy to follow.
* The proposed method can be applied if vector spaces have differing dimensionality
* Results compare well to existing methods

Weaknesses:
* As the initial projection is not constrained to be orthogonal, the geometric structure within the projected vector space is altered (only relevant if dimensionality is the same). This is usually seen as a big advantage of procrustes-based solutions and not really discussed by the authors. How much does the solution found using this method deviate from an orthogonal solution?
* Introducing the isotropic preprocessing makes it difficult to compare the alignment procedures, as both word vector spaces are altered significantly (As shown in Arora where it significantly impacts performance on semantic similarity). The authors should explore the impact of this preprocessing also for other methods (e.g. procrustes).
* No comparison between weighted procrustes and regular procrustes to address the rotation symmetry for the inner product approach. Is it crucial to use the weighted variant here?
* Unclear how well this approach works in conjunction with retrieval strategies other than nearest neighbour - previous work found this to be a significant factor for performance.
* Only anecdotal evidence for comparison of SGD and low-rank approximation
* Related work: Mikolov 2013a did not propose a procrustes solution to the alignment problem. Instead, Artexte 2017 and Smith 2017 showed (independently) that procrustes provides an analytical solution to the optimisation problem proposed in Mikolov 2013 under the orthogonality constraint.

Smaller issues / suggestions:
* I found the "preview" of the technical approach in the introduction more confusing than clarifying, as some of the details mentioned later on where missing, e.g. that rows in $X_s$ and $X_t$ refer to the same word based on the dictionary.
* It would be great to get a more intuitive "feel" how the alignment that respects the "geometric structure" would be different to a procrustes alignment, e.g. by showing a few example translations
* The plural of corpus is corpora
* You assume dimensionality $d_2 < d_1$ but don't really state it anywhere?

Other (not considered in review):
* The CCA results are quite strong - If the authors are using the CCA implementation in scikit learn I would like to point out that it actually uses an iterative estimation of partial least squares, which works much better than a more naive implementation of CCA (as e.g. demonstrated in https://arxiv.org/abs/1905.05547)
review_title: Good results but a lot of moving parts

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary:**
This paper proposes a new approach for Bilingual Lexicon Induction (BLI). To do this, the paper proposes the use of a FIPP objective, which consists of the linear combination of reconstruction and transfer loss terms. It derives the Gram matrix that minimizes this loss, followed by a rank-constrained semi-definite approximation to obtain an aligned embedding whose Gram matrix is close to the one minimizing the FIPP. The proposed approach is deterministic and has been shown to be efficient. Experiments also demonstrate the method works well, both on BLI and on downstream tasks.

**Strengths:**
* The proposed method is interesting and elegant, and novel in the context of BLI. FIPP also performs well empirically, demonstrated both by its superior performance on BLI (as shown on with an evaluation on 28 XLing pairs) and on its comparable performance in the downstream CLDC and XNLI tasks. While the proposed approach does not do the best on the presented downstream tasks, it seems to perform well (reasonably comparably to the best) consistently, which none of the other methods seem to do.
* FIPP allows for BLI using embeddings of different dimensions, and performs well even when the embeddings are of different dimensions, as shown experimentally for en-de. To the best of my knowledge, no other method supports this.
* The proposed method is quite efficient because it can be represented as a few matrix operations. This has been demonstrated by comparing its runtimes with two other SotA approaches (VecMap and RCSLS).

**Comments/Questions:**
* One drawback is that the proposed method does not perform as well for the 1k case (Appendix). Some analysis about why that might be the case would have been nice. That being said, the same is the case for the other supervised methods as well (PROC-B aside). I also appreciate the authors' transparency in this regard.
* The authors present the downstream performance on 2 (CLDC, XNLI) of the 3 tasks (the other being CLIR) used in [5]. Is there a reason these 2 specific tasks were chosen, but not CLIR?
* The proposed method uses a weighted Procrustes objective to rotate $\widetilde{X}_s$, which is a minor detail, but whose use is, as far as I am aware, novel in the context of BLI. An ablation here to show how much the weighted Procrustes objective improves over the regular Procrustes transform (such as MUSE [4] does) would have been nice to see (Eg: a row "FIPP w/o WP w/ P" in Appendix Table 7).
* GeoMM [3] is very efficiently optimizable as well. Do the authors have an estimate about how FIPP performs compared to GeoMM, with respect to the average time they each take on CPU?
* The Appendix had some nice experimental details, additional experiments and insights. It would be good to add the Appendix to the end of the main paper (as opposed to separately) if possible.

**Minor points:**
* For XNLI, [1] should probably be cited along with [2], since XNLI was introduced by [1] based on the English-only MultiNLI corpus introduced by [2]
* Section 6.2 didn't seem to flow very well with the rest of Section 6 (which follows Section 5 nicely). Consider moving it to before Section 5 if possible (I'm not entirely certain where it would fit best, possibly as a new Section 3.3).

[1] Conneau, Alexis, et al. "XNLI: Evaluating cross-lingual sentence representations." arXiv preprint arXiv:1809.05053 (2018).
[2] Williams, Adina, Nikita Nangia, and Samuel R. Bowman. "A broad-coverage challenge corpus for sentence understanding through inference." arXiv preprint arXiv:1704.05426 (2017).
[3] Jawanpuria, Pratik, et al. "Learning multilingual word embeddings in latent metric space: a geometric approach." Transactions of the Association for Computational Linguistics 7 (2019): 107-120.
[4] Conneau, Alexis, et al. "Word translation without parallel data." arXiv preprint arXiv:1710.04087 (2017).
[5] Glavas, Goran, et al. "How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions." arXiv preprint arXiv:1902.00508 (2019).



============================================================

**Update:**

A few  suggestions on the latest version:

* Adding pointers from Table 1 and 2 to their full equivalents in the Appendix would be helpful.
* The Appendix ablations are now thorough and exhaustive, but parsing them and digesting what they represent is a little tricky. Adding a one-line summary (much like in Section 5.`1) would be very helpful here. For example, talking about in how many pairs FIPP+IP outperforms Proc.+IP,  FIPP+IN outperforms Proc.+IN, FIPP+IP outperforms FIPP+IN.

and a few comments:

* The sheer number of experiments that the authors have performed in general (and in particular in the relatively short time period of this rebuttal) is impressive, and is in my experience indicative of an extremely good, well-designed and easy to iterate upon framework/code-base. If my guess is correct, I urge the authors to release their code if possible, because I believe it would greatly help anyone working in this space (or even consuming BLI's output in a downstream task).
* It is also very heartening to see how much R1's suggestion helps improve the 1k case! However, because the efficiency and the lack of need of a GPU are big selling points, the drawback of adding self-supervision (10x the time, need for a GPU) might be a good caveat to add in Section 6.1 as opposed to keeping it till Appendix A.

Overall, I would like to thank the authors for their very detailed and thorough response,  and for taking into account so much of all the reviewers' feedback to make the paper clearer with much more comprehensive ablations. The paper, its techniques proposed and their performance and efficiency, and the detailed experiments it conducts will be helpful for both the field of BLI and other fields relying on it. In view of this, this paper is a clear accept in my opinion, and I raise my score from 7 to 8.
review_title: An interesting, effective and efficient method for Bilingual Lexicon Induction

final_score: rating: 6: Marginally above acceptance threshold
review_content: === After the revision and the author response ===
I would like to thank the authors for their very elaborate response. I acknowledge that I might have asked for too many experiments in my review, but this was mostly because I really wanted to understand the various aspects of the method (as one other reviewer mentions - it has too many 'moving parts') and ensure that all the comparisons are valid by comparing to the latest work. While it is not possible (due to time and computational restrictions) to run all the required experiments, I am still happy to raise my score. While the paper might not be so impactful, it might still be a pretty nice and quick-to-compute BLI baseline for any future developments in this area.
=== ===

This paper presents a new method to learn projection/mapping-based cross-lingual word embeddings, termed FIPP (Filtered Inner Product Projection), which shows some promising results in supervised settings with 5K dictionaries. The paper is clearly written and easy to follow, and the main results across three tasks (bilingual lexicon induction + 2 downstream tasks) following a standard evaluation setup demonstrate that the method can often outperform its competitors. Another advantage of the method is the efficiency of learning the alignment; the method is several orders of magnitude quicker than standard choices such as VecMap or RCSLS. Although the paper does provide a valuable contribution to the field of learning (static) cross-lingual word embeddings, I still have quite some concerns and additional questions related to this work, and I am not ready to accept this work in its current format.

*The authors mention (even in the abstract) that FIPP is applicable "even when the source and target embeddings are of differing dimensionalities." However, I don't see how previous approaches are not applicable in these setups - the linear or non-linear mapping can be formulated between embedding spaces of different dimensionalities in a general case. Therefore, I don't see why this is important to stress with FIPP. This is some inaccurate argumentation imo.

*The reconstruction loss obtained from the Gram matrices has not been used in bilingual settings, but this idea is directly borrowed from prior work on monolingual embeddings, so I do not see it as a methodological contribution. The second term of FIPP, however, reminds me of a recent work on instance-based mapping and learning local mapping functions, as proposed and validated by Glavas and Vulic (ACL 2020) and Nakashole et al. (several papers). However, the authors do not discuss this very relevant recent work nor provide any comparisons to these approaches, which seem closely related.
-- I would also suggest the authors to include some other very relevant recent models in their comparison. For instance, what about the following papers:
- https://arxiv.org/abs/2004.13889 (Mohiuddin et al.)
- https://doi.org/10.1162/coli_a_00374 (Mohiuddin and Joty, Computational Linguistics 2020)
- https://doi.org/10.18653/v1/p19-1018 (Patra et al., ACL 2019)
- https://www.aclweb.org/anthology/2020.acl-main.766/ (Anastasopoulos et al., ACL 2020)
- https://openreview.net/forum?id=S1l-C0NtwS (Wang et al., ICLR 2020)
- There are other recent papers...

This field is a rapidly evolving field, and comparing to the models which were SotA in 2019 is definitely not sufficient, especially if the good results are supposed to be the main contribution of the paper.

*Along the same line as the previous point, preprocessing of word embeddings can be quite crucial to the final performance, I wonder if the authors have also tried a well-performing preprocessing strategy from Zhang et al. (ACL 2019), also with the baselines. Would this change the ranking of the models in the comparison?

*Concerning the previous point, the paper lacks a comprehensive ablation study. Given that it does blend several different components (e.g., multi-loss, preprocessing regime, dictionary size), it is quite underwhelming to see that the authors did not conduct any experiments to further understand what components of the model are more crucial to the final performance, and how different choices of hparams affect the performance. Further, there are no experiments with different dictionary sizes and we're left wondering if the method would work with a smaller number of training examples (e.g., 500 examples or 1K examples as evaluated by Vulic et al., EMNLP 2019). What about typologically more dissimilar languages, etc.? What about self-learning strategies? Is it possible to improve the results with FIPP by running several iterations of the method? Have the authors tried that? How does preprocessing affect the final performance? What about the hyper-parameter epsilon? How is that hyperparameter related to language (dis)similarity?

-- In sum, a complete piece of work should provide answers to all these questions, and it is still quite difficult to judge whether this particular model will really advance the induction of cross-lingual word embeddings. Does it also rely on the approximate isomorphism assumption (Soogard et al., ACL 2018) and how does language distance relate to the transfer loss? 

*Notes on motivation: While working on static cross-lingual word embeddings is still an important research thread, the paper should provide a thorough discussion on why these models are still important in the context of the current mainstay of multilingual/cross-lingual NLP based on massively multilingual models such as multilingual BERT, XLM-R, multilingual T5, etc. An open question is what applications (besides bilingual dictionary induction) could still benefit from static cross-lingual word embeddings.

Minor comments:
- Mikolov et al. did not propose using the Orthogonal Procrustes solution per se - they solved the problem via SGD and did not impose any orthogonality constraint (which is needed to solve it analitically via the Procrustes method - see later work of Xing et al., and the survey paper of Ruder et al.)

- The title is not exactly accurate: 'multilingual embedding alignment' would suppose aligning more than 2 languages (3 and more), while the authors, as most of prior work, learn only bilingual word embeddings. Another question concerns the possibility to extend this idea to shared spaces with more than 2 languages. How feasible is this?
review_title: An interesting method, but the paper needs additional insights, experiments, and analyses (REVISED)

Meta-review: This paper proposes a method for bilingual lexicon induction.
The proposed method is efficient, it optimizes a reconstruction and transfer loss.
Extensive experiments are reported, and the methods provides improvements over prior work.
Overall, the paper brings together prior ideas in a useful way.
Decision: Accept (Poster)