Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a challenging scenario for unsupervised word alignment where   human labeled word alignment data is not available and only a bilingual dataset for testing is given. Technically, it extends the alignment model proposed by Sabet et al. (2020) and iteratively updates the alignment model defined on top of a multi-lingual pre-trained language model. Specifically, it iteratively performs the follow two steps: 1. it employs the updated alignment model to generate some word alignment dataset with pseudo alignments for the given bilingual test dataset; 2. it fine-tunes the multi-lingual pre-trained language model through code-switch to updates the word alignment model. In this way, the multi-lingual pre-trianed model can be optimized for the word alignment task. Experiments on five datasets demonstrate the proposed approach delivers consistent improvements over the baseline Sabet et al. (2020).
strength_and_weaknesses: Strengths:

1. The proposed approach is appealing and easy to implement.
2. The experiment results shows that the proposed approach achieves good performance compared with the baseline.


Weaknesses:


1. Although the key idea for the proposed is explained clearly, some important details are unknown such that I have some key concerns on the proposed approach. Specifically, is the equation (4) updated at the sentence level or updated on the entire dataset? By the sentence level, I mean that the model is updated only for each sentence x and then the updated model is applied for inference on this sentence only. I guess it is updated on the entire dataset and then the natural question is how this approach works on a very small test dataset, for example, including one or two sentences. In addition, how many sentences does this method need in the test dataset to achieve good performance? These questions are important to show the superiority of the proposed method, but this paper does not study them.

2. In essence, the technical contribution of the proposed approach is limited in my opinion. At the high level, this approach is similar to Dou and Neubig (2021): both of them are extensions of Sabet et al. (2020); both employ the multi-lingual pretrained language model as the initialization of the word alignment model, and require a bilingual dataset to optimize another loss function which is aware of word alignment. The difference is that this paper only requires a relative small bilingual dataset (i.e., test dataset) whereas Dou and Neubig (2021) uses a large bilingual dataset. Theoretically, it is possible to use the similar idea of Dou and Neubig (2021) on the same scenario as a baseline. 

clarity,_quality,_novelty_and_reproducibility: The high-level idea of this paper is easy to understand, but some important details are not clear.
The novelty of the proposed method is not very good.
summary_of_the_review: Please strengths and weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies unsupervised word alignment using pre-trained multilingual LMs. The paper proposes an iterative method that repeats two steps: (1) generate word alignment using a LM, (2) using the word alignment, generate synthesis code-switched parallel text to fine-tune the LM. Empirically, this iterative method reduces alignment error rate for six language pairs.
strength_and_weaknesses: Strength:
- The proposed method is well-motivated and clear.
- Empirically, the method can effectively improve alignment error rate.

Weakness: 
- The biggest weakness of the proposed method is that it can be slow. Practitioners will need to decide whether using the method is worth the computation time and how to choose hyperparameters. Therefore, it would be very helpful if the paper can discuss computation time and add relevant metrics in the experiment result tables.
- The introduction suggests that the proposed method (or framework) can be applied to other NLP tasks, but the paper is restricted to only one task (word alignment). Ideally, the paper should have other tasks to show that this framework can really generalize to other tasks.
- The paper only experiments with up to 3 iterations. I am curious if the method can continually improve AER. Does the method converge after 3 iterations?
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear with a few typos. The paper has solid quality and is novel. I do not see any reproducibility problem.
summary_of_the_review: The paper proposes a novel method for unsupervised word alignment and shows good results. However, the paper does not discuss time complexity, which could be a key weakness of the method. Overall, I weakly recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work propose a simple fine tuning method for unsupervised word alignment based on multilingual contextualized pre-trained language model, e.g., BERT. This method is based on the similarity score of the word representations in two languages and employs fine tuning by 1) iteratively augmenting code-switched training data based on the previously predicted word alignment and 2) masking inputs. The proposed method allows a model to focus more on the paired word representation considering context. Experimental results show that the proposed method achieved gains when compared to the prior baselines.
strength_and_weaknesses: Strength

* It is a simple yet effective method to improve word alignment quality without any supervisions. 

Weakness

* It is not clear why the proposed method works. The author tries to prove it by the similarity of code-switched and masked sentences. However, the proposed method might prone to poor local optimum problem in the similar manner as done by an EM algorithm.

* The experimental procedure is a bit unclear in that that there exists no training data, but a model is simply fine-tuned on test data. Thus, it is not clear how the hyper-parameters are tuned, e.g., sampling probability, for a fair comparison.

* Given the iterative procedure for fine-tuning, it is not clear how the word alignment will be revised, e.g., whether the proposed method is trying to concentrate on more confident pairs or not. Probably it would be good to analyze the word alignment confidence in each iterations.
clarity,_quality,_novelty_and_reproducibility: This paper is easy to follow. However, it is a little bit unclear about how the hyperparameters were tuned given that a single dataset is employed for tuning and testing without a separate development (or validation) data.
summary_of_the_review: This work is interesting in employing code-switching during fine-tuning combined with masking. Experiments show gains when compared with baselines. However, it is not clear how the hyperparamters were tuned.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This proposes an  iterative self-supervised task-adaptive pretraining  framework with a focus on the task of word alignment. In other words, task adaptative pretraining and word alignments are done iteratively. The method works without labeled data.
Experiments shows a reduction of the AER with an XLM-R model.

strength_and_weaknesses: Strengths:
- the paper is well-written and illustrated
- the idea is very simple
- the application to word alignment is novel
- the method is somewhat reproducible 

Weaknesses:
- the paper lacks of substance. To me, it only sticks together some pieces from previous work (task-adaptative pre-training) and applies it to a particular task. 
- the need for code switching is never properly motivated in the paper. It makes sense to do code switching, but since this is a crucial point of the method it should be extensively motivated. Why does the method use code-switching? Code-switching is introduced very early, but briefly, in the paper, then there is a section about it in related work, and then it is presented as a component of the method. The relationship between task-adaptative pretraining and word alignments is clearly given and motivated, but we miss the links with code-switching.
- the analysis doesn't answer the questions that I expect the readers will have:
- How does this method increase the computational cost of the training? (it looks like a huge increase)
- How does this method compare to just pre-training XLM-R for a few more epochs? (I have no idea)
- What are the limits of the method?



clarity,_quality,_novelty_and_reproducibility: 
See "Strength And Weaknesses".
summary_of_the_review: In my opinion the proposed method lacks of originality. In short, I would described it as an application of task-adaptative pre-training to the task of word alignments. An extensive analysis and better motivation for code switiching would improve the paper, but it would still lack of substance for a top-tier conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers thought the paper is well-written and easy to understand. They questioned its novelty (mPaq, hkhx) and practicality due to increased training expense (KcR5, hkhx). They suggest more baselines, especially comparisons to similar code-switching based methods. Showing benefits for an end task other than word alignment may also be useful, perhaps even a task that has code-switching in itself.

The authors responded with experiments addressing comparisons to baseline (results are fairly similar to Dou 2021), and computational experiments addressing the additional time complexity. The reviewers are unconvinced (or did not revisit the paper).


justification_for_why_not_higher_score: None of the reviewers were excited about this paper and 3/4 felt negatively. Rejection is recommended based on reviewer consensus.
Decision: Reject