Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary: The authors attempt to investigate to what extent languages are hard to conditionally language-model. They do this by using some information theoretic measures. Claims:
- There are no statistically significant differences between source language representations, but there are significant difference between pairs of target language representations.
- There is no complexity that intrinsic to a language except its statistical properties concerning sequence length and vocabulary (unless word-based methods are used).
- They also observe phenomena such as Double Descent and erraticity.

----
Strengths:
- The Experiments are extensive.
- The relative similarity of source language representations is interesting and worth exploring further.

Weaknesses: 
- The diagrams are difficult to read
- The paper is hard to follow and would benefit from a clearer focus rather than the broad range of topics covered here. For example:
    - It is difficult to understand what the methods/terms (the information theoretic measure used, double descent) are - little time is spent explaining these.
    - Double descent is discussed in the paper but it is still made not clear why this is relevant in the paper.
    - Several portions of text are repeated - with some editing, space can be made to discuss concepts important to the paper
- The authors make recommendations for modeling (Eg. using char level or byte level models for certain models - which have been extensively studied for this): this is not followed up with any concrete results on translation/downstream tasks or pointing out relevant work.
review_title: Good premise; Unclear Paper Focus

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper is trying to answer an important question: How does representation play a role in carrying meanings? In doing so, the authors experimented with 6 languages in  3 + 5 kinds of representations. The authors concluded that the different performances among language pairs maybe be the result of word segmentation in different ways.

This is an interesting step towards understanding meaning representations, especially in languages that do not have an alphabet. However,  as much as I agree with some of the final conclusions, the soundness of the experiments appears to be in question.

My main concerns are about the additional experiments with Chinese.

The authors claimed that "On the character level, target language ZH (ZHtrg) shows a different learning pattern throughout." There are two types of character-level representations used: Wubi and Pinyin. Wubi was originally invented for professional typesetters so that they can type fast. The segmentation may not be correlated with the meaning of the word at all, as claimed by the papers cited by the authors. Pinyin, on the other hand, is highly ambiguous. One pinyin may representation dozens of words and the authors did not take tones into considerations at all. 

The author also mentioned that "After filtering length to 300 characters maximum per line in parallel for the 6 languages, we made 3 subsets of the data with 1 million lines each". Each language carries meaning differently and the information density is drastically different. 300 characters in Chinese carry much more information than 300 characters in English. This is an unfair comparison.

It would make a lot more sense if the authors treat each language differently because of their orthographic differences.
review_title: Interesting but raises questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper investigates whether languages are equally hard to Conditional-Language-Model (CLM). To do this, the authors perform controlled experiments by modeling text from parallel data from 6 typologically diverse languages. They pair the languages and perform experiments in 30 directions with Transformers, and compare 3 different unit representations: characters, bytes, and word-level (BPE). 

I appreciate the authors' effort for their systematic controlled experiments. However, I'm leaning towards rejecting this paper since I think some of the claims made in the paper are too strong and not really backed up by their experiments. 

Some comments:
* The term "Conditional-Language-Model" can be misleading, since this paper model a target language conditioned on a source language, so more like in a machine translation setting rather than standard language modeling setting where you can also condition on the previous history.
* I'm also not sure if comparing perplexity by conditioning on another different language (source language) is correct. The experiments would be clearer if done with standard LM with Transformers encoder model like BERT for example.
* At the end of Section 2, the authors mention about "generalizations", but I couldn't really find any discussion about this in the paper. Maybe this can be clarified?
* I found that claiming script bias in character models is too strong, if the experiment only shows bias in ZH (and this is expected since its character-level has different notion with languages with Latin script).
* Byte-level: I found that there is still some "erraticity" in Figure 3(b) especially when the data size increases (which is more practical in real world application), so this is not entirely resolved?
* I also think the summary in Section 1.2 stating that linguistic typological information is not necessary given "statistical properties concerning sequence length and vocabulary" is not necessarily valid since these two properties are the results from linguistic typology information.


Missing references:
1. From characters to words to in between: Do we capture morphology? Clara Vania and Adam Lopez. ACL 2017.
2. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. Barbara Plank, Anders Søgaard and Yoav Goldberg. ACL 2016. 
review_title: Interesting paper and experiments, but lack of evidence to support its claims.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper provides an empirical investigation of an important problem: the transferability of language modeling signals across languages in the transformer model. This is an important question because it can teach us both on the relations between languages and the properties of the transformer model (although it is not easy to tease the two effects apart).

This is a thorough paper with a large number of experiments and with interesting conclusions that nicely generalize the low level patterns observed in the experiments. These conclusions are likely to be useful for the research community as part of its on going investigation of language transfer and the transformer model.

I have several comments though:

1. The language of the paper is often very complicated. Just as a couple of examples: It was very hard for me to follow the abstract, the first paragraph, the (very long) sentence that start with "in order to eliminate" (1.1), item 3 in the list of contributions and this is just a partial list.  I ask that if the paper is accepted the authors will try to improve this aspect.

2. The writing is often over pedagogical and I often got the feeling that the authors try to educate their readers (but not in the positive sense of the word). I would try to avoid this style. 

3. This work seems highly relevant to the following paper:

"Towards Zero-shot Language Modeling." Edoardo Maria Ponti, Ivan Vulic,Ryan Cotterell, Roi Reichart and Anna Korhonen . EMNLP 2019

I think the discussion parts can gain form comparing the conclusions of the two papers, when relevant.


review_title: An interesting and thorough exaimination of an important problem. Writing is sometimes too complicated and some more high level analysis could help.

Meta-review: The paper study to what extent languages are hard to model by a conditional language-model based on information-theoretic measurements. 

Overall, the reviewers value the systematic and extensive controlled experiments present in the paper. However, the presentation of the paper makes it very hard to follow and reviewers all still complain that it is hard to understand the take-home message of the paper.  

Despite the reviewers also appreciate the authors' effort in improving the paper, submitting the revision, responding to the feedback, they still conclude that significant reorganizing and revising of the paper is needed before it can be published. 

In particular, the paper may be able to improve by backing up the empirical study with some linguistic phenomena or by a more careful rewriting in explaining and discussing the empirical results. 

Some other strong arguments such as "Our application of statistical comparisons as a fairness measure also serves as a novel rigorous method for the intrinsic evaluation of languages, resolving a decades-long debate on language complexity." may need to be carefully revised. In this particular example, it is unclear how this paper "resolve" the debate on language complexity by demonstrating a few experiments.  Several sentences like this one should be revised.  


Decision: Reject