Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new spectral normalization technique that improves the cross-lingual mapping of monolingual embeddings by rigid, orthogonal transformations. This is demonstrated by consistent gains on bilingual lexical induction and two other downstreeam cross-lingual tasks.

main_review: Strength:
+ The goal is clearly presented and the method is clean (and clear)
+ The paper is well-organized and clearly written
+ Experimental results show consistent improvement

Weaknesses:
- Limited to preprocessing for a specific cross-lingual embedding setup
- No uncertainty/confidence/error bars on experimental results, or significance testing [addressed]
- Some experiments may use a weak baseline [addressed]

The proposed normalization improves performance quite consistently overall and on average, even if it sometimes slightly degrades performance on few specific settings. This is an excellent sign that the spectral normalization does something reasonable and tends to help.
However in many experimeents, the compared baseline is simply no normalization, which is really not a very strong opposition. It is not surprising that some normalization helps vs. no normalization at all. In fact BLI results like Table 3 suggest that there is a much larger difference between any normalization and no normalization, than there is between the various normalizations tested. This raises the question of how the proposed SN compares to *other normalizations*, especially on downstream tasks.
In addition, the conclusion and Appendix E suggest that training joint cross-lingual embeddings compares favourably with the normalization+rigid transformation approach, yet only the latter is considered in the main paper.

In Figure 1: In addition to the desirable impact on effective condition numbers and single value gap on all languages, the iterative spectral normalization seems to have a massive impact on the effective rank for Japanese, taking it remarkably close to the ER for other languages (wherease the raw er was much lower according to the log scale). Any idea where this large effect comes from or whether this is due to the particular language or dataset?

Re. stability of language choice: experiments are run on different languages at different points in the paper. For example Hindi and Japanese are used in Figure 1, but not later. The choice of the four particular language pairs used in the cross-lingual natural language inference experiments is justified, however it is unclear why some BLI results use AR, DE and NL (Tab. 5), others include IT (Tab. 6).

Questions:

"significantly improves over the baseline" (p.7): What kind of significance testing was run?

"Normalizing contextual type-level embedding" (p.8): It is not clear what was done here. I understand some details are available in Xu&Koehn(2021), but this paper should arguably provide enough information to be self-contained and understandable.

In what way does the proposed spectral normalization "generalize previous approaches" (p.9)? Are there specific choices of settings for which it becomes equivalent to centering, length normalization or PCA removal? This is not obvious, especially since some of these have guarantees (eg on preserving angle or relative distances) that SN does not have.

Typos:
* P2,l14: "Euclidean of cosine" - presumably "or"
* P3,l2: what is "wlog"
* P6,l3: publically
* P7,l14: "with C+L" - presumably "without" is meant
* Tab3: $X_L_1$ should be $X_L_2$?
* Tab4,6,7,13,18,19: 5K, 3K, 1K should be 5k, 3k, 1k (lowercase)

[Thanks for the extensive reply to this and other reviewer's comments -- much appreciated]
summary_of_the_review: A simple, clean preprocessing method to help map word embeddings across languages. The article is clear and well organized, and extensive experiments produce consistent gains in a number of cross-lingual tasks. The experimental setup raises a few questions regarding the significance of these improvements.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes three new methods for normalizing word embeddings before cross-lingual alignment. The first method, SpecNorm, caps the singular values of the word embeddings to be twice the average (original) singular values. The second method does mean centering, SpecNorm, and L2 normalization. The third repeats this for a fixed number of iterations (5). Experiments on a wide variety of tasks generally show improvements.

main_review: SpecNorm seems like a very reasonable idea, but it takes a long time to finally present it on page 4 (out of 9). I also don’t think that pseudocode for SpecNorm is appropriate; it’s hard to understand, but the verbal explanation is far clearer.

Are you able to include some intuition for why capping the singular values is helpful? I gather that some of this is given by Dubossarsky et al. (2020), but to make this paper self-contained, it would be helpful to provide some intuition here as well.

I-C+SN+L is a more complicated method. How did you decide on these three normalizations, and their order? Why does iterating help? It would be nice if the method could be better motivated.

The five results tables compare four different sets of systems. It's especially confusing that sometimes SN means SpecNorm and sometimes it means I-C+SN+L. Besides being difficult to follow, it creates a feeling that the results are selectively reported. If possible, please include the same set of normalizations for all experiments.

A much more minor suggestion is that Table 4 should be transposed, so that the compared normalizations are the rows, as in all the other tables.
summary_of_the_review: The core SpecNorm method seems reasonable, but the iterative centering + SpecNorm + length normalization seems more ad hoc. Although the experiments are thorough and show improvements, I would like to see more principled motivation for method and/or deeper insight into why it works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an Iterative Spectral Normalization method to improve pairwise Cross-Lingual Word Embedding alignment. The work supposes that improvement to spectral properties in monolingual embedding spaces are sufficient to yield improvements to Bi-lingual Lexicon Induction, Cross-Lingual Document Classification and Cross-Lingual NLI. The main contributions are (a) the introduction of a simple, portable algorithm to improve pre-processing of embedding spaces for cross-lingual alignment and (b) empirical improvements in multiple domains of cross-lingual embedding oriented tasks. 

main_review: ### Strengths

The simplicity of the approach and combination with existing preprocessing approaches, is promising as a utility for future work. 

Empirical improvement on several extrinsic tasks across multiple domains identify a benefit to SN. Reasonable spectral property-based metrics are shown to improve (in the appropriate direction) when using this method as opposed to methods that do not consider such properties. 

The breadth of domains of study is commendable as the authors have clearly taken great care to illustrate that their method has wide applicability. The appendices show further breadth and the argument for the utility of the method is strongly presented. 

### Weaknesses

This method shows merit for the reasons given above, but I can describe my primary concern as this — you have proposed solution X for some method A, claiming that A is now improved due to a reduction in effect Z. But you do not demonstrably support that solution X influences effect Z or highlight if effect Z is a critical failure in A. Therefore, while empirical results are strong, it remains unclear what solution X does to improve A. Even if the influence of X on A is not well understood -- this should be delineated further to support that the empirical claims require further introspection.

The work does not make a self-contained case for what  spectral normalization is a solution towards. The problem is poorly defined, which leads to a gap in understanding the motivating strategy of the work. For example, in Section 3 the work states “As a result, if the spectral properties are extreme, it can help regularize them.” -- but it is not clear what it means to (i) regularize such properties or (ii) what this intends to do with respect to the embedding spaces. This could be improved if the problem hypothesis was clearer e.g. answering questions such as:

- Are spectral properties of monolingual embedding spaces malformed?

- Can this be demonstrably proved in this work so the problem to be solved is clear? 

- Are there confounding examples within embedding spaces to motivate this (e.g. unexpected versions of the common king-queen=man-woman analogies based on low-frequency words)? 

- What does spectral normalisation hope to intrinsically accomplish such that the findings can be better interpreted? 

Care is not taken to explicitly link together the spectral statistics to additional methods or to explain either what spectral metrics are informing the reader of in 2.2. It is somewhat of a foregone conclusion that spectrum-based metrics will improve with a spectrum-based method. While this section relies on the insight of Dubossarsky et al. (2020), there is little effort in interrogating the prior work and motivating why prior insight is directly applicable here as-is. At present, this must be taken at face value with little motivation for the succeeding sections. This caveat is somewhat improved in Appendix B, but I consider this insufficient without a clear hypothesis tying the motivation of this approach together. 

Overall, there is a lack of research questions in the work which leads to a difficulty in scoping the contribution. SN is clearly useful -- but why? We observe empirical benefits to several tasks, but the work does not source these benefits to specific components or processes within the method. This could be improved with intrinsic analysis (and some error analysis) identifying the effect of SN. This is hinted at in Section 4 where it is stated that “We hypothesize this is because it is somewhat uniformly stretching words along the major modes of variation” however this could be expanded to aid in understanding of the mechanics of the method.

### Unsupported claims

Every claim should be supported with evidence in literature or in the work. There are several instances of claims without proper argument. 

“Yet, previous work has clearly demonstrated that there exists significant overall structural similarity, and alignment seeks to make correspondences between those structures for translation and joint understanding.” - I would not consider the arguments of prior work as sufficient for motivation without a in-depth of explanation in this work. This work is not an appendix on prior publication. 

“effectively removes much of the clumping of words based on properties other than the pairwise similarity which encodes meaning.” - this term is not introduced or explained as a consequence or issue within CLWE prior art. It is unclear what this means, or how it is required to be mitigated for better CLWE.

“This buttresses the claim made by Xu & Koehn (2021) that improving the degree of isomorphism” - it is unclear where you demonstrate that you improve isomorphism with your method. If your iterative method subsumes methods already maintaining/improving isomorphism then it would be beneficial to demonstrate that such properties are maintained. If this is intended to be obvious, please state so.

“Our approach generalizes previous approaches” -- this is not proven (see previous comment), and little care is taken to show that SN does not interfere with complementary components like Length Normalization. 


#### Questions 
- In 2.1: what does “longer vectors” refer to here when you are explicitly working with fixed-length vectors? It could be clearer when reduction methods like PCA are cutting up embedding spaces and when frequency methods are used over others.

- What is the difference between matrix X in 2 and A in 2.2? It is unclear why these are re-defined. 

- Section 3 -- what is ‘critical information’ in reference to? It appears some assumption is made here, but this is unclear.

- Was a stopping condition explored in the iterative algorithm as opposed to a fixed number of iterations?

- It is stated that the vocabularies are trimmed to the top 200K words, but it would be interesting to see how SN-based methods influence the long tail of infrequent vocabulary. Was this considered?

- Were multi-lingual (L>2) alignments considered? 

Please address the unsupported claims and issues raised earlier:
- Is Isomorphism maintained and can you verify that your method subsumes prior methods with absolute certainty?
- What is meant by clumping?
- What are the motivating failings of prior work that has led to the SN method?
- What is the intended intuition and understanding of “regularizing an embedding space”?



#### Typos/Style (not intended as criticism)
- Page 3: why state "wlog" here?

- SN references both the iterative method and the SN method alone without clear discussion of the difference so should be more clearly separated when you are referring to each method. 

- Equation numbering gets lost in appendices

- There is repeated usage of embedded clauses without clear demarcation e.g. “This in some sense allows” -> “This, in some sense, allows”. This is generally more of a spoken pattern and could be omitted entirely.

- Procrustes alignment based methods are inconsistently referenced as a “problem”, “solution” or “alignment” with additional unclear italicization. These could be streamlined for consistency. 

- Imprecision in technical descriptions -> “terminate this iterative process after a few steps”. What is few here? This is additionally challenging in Section 4 “Alignment Algorithms”

- Inline equation labeling for 2.1-PCA Removal for “d” is lost.

- Figure 1 -- every key in the figure should be explained in the caption. It is not a given to recall what “I-C+L” means in contrast to “C+L”. 

- Inconsistent spelling of “Normalization”
summary_of_the_review: The recommendation is based on the lack of introspection into the method itself rather than the empirical improvement. The work could be improved if it was better understood what the method is doing and how this ties into performance improvements. 

I believe the direction of this work is novel, but overall there is a lack of appropriate setting which ultimately makes it unclear what problem SN solves, and how this fits into existing pre-processing approaches. The case for SN is not fluently well-motivated and there is little analysis to confidently support that SN is superior to prior art or generalises existing approaches. 

I support the merit of this method with empirical utility and wide applicability, however, I recommend that significant further revisions are required prior to publication in a later venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies learning cross-lingual word embeddings (CLWE) with alignment: mapping two pre-trained monolingual word embeddings to a shared space with a linear/orthogonal mapping. Previous work shows that normalization methods and the spectral properties of monolingual word embeddings have a big impact on alignment methods. Building on these results, the paper proposes a normalization method that regularizes the spectral properties of monolingual word embeddings. The method is then combined with mean centering and length normalization in an iterative process. Empirically, the proposed normalization method improves alignment and leads to better CLWE (measured by scores on bilingual lexicon induction, cross-lingual document classification, and natural language inference).
main_review: Strengths:
- The proposed method is simple to implement.
- The experiments are extensive and confirms that the proposed normalization method consistently improves cross-lingual word embeddings.

Weaknesses:
- The paper may have limited impact. While the paper does a good job improving alignment-based CLWE, these models are outperformed by multilingual transformers (e.g., multilingual BERT and XLM-R) in many applications. For example, the XNLI results reported in the paper are much worse than that from XLM-R. Therefore, it is unclear if the proposed method is useful for practitioners. One way to improve the impact is extending the method to contextualized embeddings, but this is non-trivial.
- The intuition behind the method is not immediately obvious. The paper may be clearer if there are more intuitive explanations of why the spectral statistics in section 2.2 helps alignment. It is also helpful to have a more explicit discussion of why the proposed method improves the spectral properties from section 2.2.
summary_of_the_review: The paper proposes a simple method for improving CLWE with good empirical results. However, the impact may be limited as multilingual transformers are replacing CLWE in many applications. Therefore, my recommendation is weak accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a normalisation method for cross-lingual text representations.  The goal is to normalise the monolingual embeddings based on spectral normalisation.  The main contributions are: novel method to normalise word embeddings, the proposed method includes different normalization approaches, and the proposed method improves performance on intrinsic and extrinsic evaluation tasks. The study shows that produced text representations keep their meaning and improve performance on downstream tasks.

main_review: Strengths
- Clear description of background knowledge and motivations needed to understand the proposed representation model.  
- Clear exposition of the proposed method.
- The authors perform a comprehensive comparison across different benchmarks and downstream tasks.

Questions to the Authors. Please address the following questions during the rebuttal:

- Please elaborate on how cross validation could affect the results in the cases where the scores between methods are close.
- Could you elaborate on the set-up for selecting the hyper-parameters for the proposed model? Given that is based on cross validation, how the model would behave on a low resource setting?
- Could a visualization of the learned representations be useful for the word similarity embedding tasks? Comparing with and without normalisation.
- A possible extra contribution could be to highlight  the single-cell task results on main text.

summary_of_the_review: I recommend acceptance given how clear the paper describes related work, motivations, and proposed model. The authors perform an exhaustive evaluation of the proposed model with different language downstream tasks. Moreover, the model outperforms related work on the downstream tasks.
review_title: null

Meta-review: The authors propose a normalization method for cross-lingual text representations. The goal is to normalize the monolingual embeddings based on spectral normalization. The study shows that produced text representations keep their meaning and improve performance on downstream tasks.

There is a disagreement among the reviewers. The main concern is whether the main contribution is an empirical study or a novel idea.  I think the authors well-addressed the concerns of most reviewers. The idea and empirical study are enough for publication for ICLR-2022.
Decision: Accept (Poster)