Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the authors developed a clever twist to the traditional neural language modeling paradigm by building a model that operates on image patches of the text. It learns to output “masked” patches using an MSE loss. Through this, the model alleviates issues with fixed or non-transferable vocabulary. When fine-tuned on a suite of downstream syntactic and semantic tasks, PIXEL observes commendable performance often on par with BERT etc. or better for languages whose writing scripts are unfamiliar to BERT. The authors also show that the model can handle code-switching and works well in the face of character-level noise.
strength_and_weaknesses: Strength:
I commend the authors for the innovation here. Although a simple idea, it is highly effective and clever, and presents a very exciting alternative to BERT & friends. I look forward to future research on what these models encode and share across languages/writing systems.

Questions:
- It would be useful to have a more detailed discussion on the fertility of the pixel maps, ie., how many patches correspond to a word on average and the relationship to some of the tasks. For example, I am curious if there is a relationship between this and the question answering performance of PIXEL as the authors allude to as well.
- It would help to order the languages in the results tables based on [UNK]%. and insert a row showing the difference between BERT and pixel for each result table.
- Any hypotheses for why its performs much worse than BERT on COLA specifically?
- I understand the space and time constraints but I would be excited to see that perhaps training PIXEL on two very different script systems leads to big performance gains than traditional models and PIXEL trained on english only. Especially if the other language has a rich morphology for example.
- How hard was hyper-parameter tuning for this model? what do the loss curves look like?

clarity,_quality,_novelty_and_reproducibility: Clarity: Extremely clear and well written+motivated! I enjoyed reading the paper.
Quality: The authors evaluated many different aspects of this model and sufficiently demonstrated its capabilities.
Novelty: To the best of my knowledge, this is the first paper to use pixel-based language systems as language models.
Reproducibility: The authors will provide a public link with access to many aspects of the model it seems.
summary_of_the_review: Given the issues with vocabulary and tokenization in NLP, I think this paper offers a very innovative solution and I am curious to see how this research direction pans out.

NB:I would give this paper a 9/10 if the option existed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Pixel, a masked language modeling approach similar to BERT that first renders text as an image and then encodes the sequence of patches using a ViT. During pre-training, the setup resembles the masked autoencoder – spans of patches are masked, only unmasked patches are encoded and a decoder is used to predict the masked patches. During fine-tuning, the decoder is replaced with a classification head similar to BERT. Pixel is motivated by the need to remove the need for large vocabularies that increase model capacity via embedding layers and make prediction layers very expensive, while being able to generalize to a large set of scripts and languages.

Experiments include pre-training Pixel on the same data as BERT (Wiki + Books) and fine-tuning on a large set of tasks including pos tagging, dependency tagging, NER, GLUE/XNLI and extractive QA. Results show that while BERT often outperforms Pixel on English tasks and latin scripts, Pixel can transfer better to previously unseen scripts. They also conduct analysis on robustness to low-level orthographic attacks and code-switching.

strength_and_weaknesses: Strengths:
1. This paper takes on a new and interesting approach to a very well-established masked language model pre-training/fine-tuning paradigm. It is quite an interesting and refreshing setup.
2. The paper includes an extensive set of experiments, covering POS tagging, dependency parsing, extractive QA, GLUE, XNLI, NER.
3. Being able to non-trivially handle new scripts from only fine-tuning data is an impressive feature of the system.

Weaknesses:
1. It would have been really interesting to see the byte-based vocabulary baseline (something like ByT5). Since most of the experiments involve shorter sequences (right?), is length of the resulting byte-based sequences a concern? In some cases where the vocabulary clearly has very poor coverage, comparing against BERT seems like a slightly unfair comparison with BERT mostly fine-tuning on a sequence of unks. 
2. The discussion on patch span masking is a bit unclear. How was the resolution of the rendered text chosen? Within the same height and sequence length, how many words tend to end up in a patch? Or do patches often contain subwords? I may have missed these details but it seems relevant for understanding how things work. Were there any experiments that varied these parameters to clarify the decisions? A figure illustrating the approach with an example might also be helpful.
3. The robustness experiments currently lack decided conclusions/takeaways. It doesn’t seem like Pixel is firmly more robust than BERT. Also, the orthographic attack graphs might need error bars?
4. What if Pixel is provided inputs that are rendered vastly differently during fine-tuning? It would fail to generalize right? Or would the approach be to run OCR and get the text, render it with the Pixel text renderer and then predict? It might be worth adding a discussion that goes over some of these points, and more generally delves into design decisions that are crucial.

Nit: maybe the paper title should be masked language modeling with pixels. 

clarity,_quality,_novelty_and_reproducibility: The paper is generally comprehensive, but some details are hard to find (for instance, see previous section for suggestions). It might also be worth restructuring the results section so that it highlights conclusions a bit better – it’s a bit hard to parse right now.

This paper is novel and interesting, and the experiments are comprehensive. A bit more discussion on design decisions, ablations, qualitative analyses might enhance the quality further.

The paper states that code, checkpoints etc will be shared for reproducibility.

summary_of_the_review: Recommending weak accept (open to changing this after discussions) because the paper proposes an interesting approach to masked language modeling with a comprehensive set of experiments, but there is a slight lack of clarity in a few places.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors proposed a new methodology by rendering text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. The pre-trained step is learning reconstructing the pixels of masked patches instead of predicting a distribution over tokens.

In the empirical study, authors find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts.
strength_and_weaknesses: Strength

It is interesting to see authors propose to render the text to image for language modeling. 

I believe the robust to orthographic attacks and code-switching will be very useful in web security domains. For example, such attack pattern very common in phishing email as the fraudster try to bypass the classifier.

Weaknesses

clarity,_quality,_novelty_and_reproducibility: Clarity
Overall it is clear.

Quality and Novelty 
Overall the quality is great. The experiments are thorough and detailed. However, some ablation studies will be helpful, for example
I am curious why authors choose RGB channel? This seems not very useful in general text besides emojis. 
Also I am curious if the font of the text can influence the model performance?

Also what is the computation time for the proposed method (both training and inference) for the proposed method? 

Reproducibility
The experiment set up is clear. Authors promise to publish the implementation and model checkpoints and weights, so I believe it would be reproducible.
summary_of_the_review: Overall I feel this paper is novel and has strong real-world application. However, my minor concern is the novelty is limited given visual text representations is already proposed by Salesky et al. (2021). Therefore I would recommend "marginally above threshold".
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ## Overview
In this paper, PIXEL, a pixel-based encoder of language is introduced to circumvent the vocabulary bottleneck problem. PIXEL is built on ViT-MAE and is pretrained on the same corpus as BERT. Specifically, the authors proposed to render text onto blank images and patchify raw pixels into image tokens and masking 25% of the tokens. Following He et al. (2022), the author then trained PIXEL by using a decoder to reconstruct the masked pixels of the input images. The authors evaluated PIXEL on syntactic and semantic tasks in typologically diverse languages and demonstrated its robustness to orthographic attacks and linguistic code-switching. Although the downstream performance is falling short comparing with BERT and mBERT, the proposed method shows its novelty and potential by providing a new view of tackling the vocabulary bottleneck problem in both mono-lingual and multi-lingual setting. 


strength_and_weaknesses: ## Strength
- Avoid vocabulary bottleneck problem.
- Show more robustness to orthographic attacks and code-switching
- Perform better on non-Latin languages on syntactic tasks and semantic tasks comparing to Bert.
- Demonstrate the possibility of pixel-based language model without injecting a priori glyph knowledge.

Weaknesses
- Average performance on downstream tasks still lags far behind Bert and mBert.
- Data preprocessing (text rendering) is tricky and laborious and will cause inevitable latency in real application.
- The generated text pixels during pretraining is blurred and fuzzy which makes the model learning less explainable.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written.

Questions
- For word level tasks in Latin, how to you think that a word might be splitted into two patches?
- It would be interesting if the authors could provide some examples(cases) or any other possible ways to better illustrate how it avoids the vocabulary bottleneck problem (especially in non-Latin languages). 
- The authors claimed “PIXEL is more robust than BERT to orthographic attacks and linguistic code-switching”. However, “in lexical code-switching experiments, PIXEL performs on-par with BERT”. Also, in Table 5, it seems we cannot conclude that “PIXEL is more robust than BERT to linguistic code-switching”.
summary_of_the_review: Although it might be not that useful, I like this idea and it gives a new perspective  for text representation
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a novel approach of learning LMs from pixels and is essentially a vocab-free method for language. 

All reviewers liked the paper and think this approach is highly novel and interesting.

I strongly recommend acceptance. 

The AC likes the paper too. 
justification_for_why_not_higher_score: Already highest. I think this is an interesting paper deserving of an oral which would gain alot of attention (as it deserves)
justification_for_why_not_lower_score: This paper is highly innovative and interesting and could be pretty impactful down the line.
Decision: Accept: notable-top-5%