Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple similarity-based data augmentation technique to translate code written in one programming language to another. With empirical investigation, they show such apparently simple data augmentation closely matches the performance of the models trained on the manually annotated datasets. 
main_review: + The proposed technique can be useful for legacy code translation where annotated data is scarce. 
+ The empirical investigations are solid.
+ The paper is well-written and easy to understand.

- The authors did not provide any explanation/intuition/evidence why such a simple data augmentation technique works.

- The main problem with this approach is that the performance suffers significantly with CodeNet dataset. The reason behind similarity-based data augmentation will work is because for implementing similar functionalities often similar variable names are used. However, such an assumption may not be true for CodNet where developers did independent implementations. Thus, the dropping of performance for CodeNet is not surprising.  However, this raises questions about the usability of this technique.

- The contrast between performance with noise is useful, but again that shows how sensitive this technique is with noise and for some legacy code (e.g., COBOL --- a language authors use to motivate the study) the amount of noise might be high.



summary_of_the_review: The authors propose an approach of data augmentation for code translation for document similarities. However, the results show the approach suffered for independently developed code (CodeNet), which is the most realistic case. The approach is also susceptible to noise.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper mines noisy parallel datasets of code by calculating the similarity between two non-parallel sets of documents. The authors first show that the document similarity methods can indeed align parallel documents and find that the word movers distance (WMD) is the most effective one. Then, the authors show the high tolerance of models trained with noisy datasets. Based on the two findings, the authors finally apply the proposed method to a large, non-parallel code dataset, and observe a performance boost of using a noisy parallel dataset compared to randomly paired datasets.
main_review: **Strengths**
1. The paper is easy to follow. Although I am not super familiar with code translation, I can catch most of the points.
2. The proposed method is simple and easy to use.
3. The experiments are relatively thorough, covering different types of document similarity methods, programming languages, model architectures, and evaluation metrics. The results seem to be convincing.

**Weaknesses**
1. My main doubt is the results presented in "Section 5.3 RQ3: TRANSLATING BETWEEN A WIDER SET OF PROGRAMMING LANGUAGES". The authors report CA@5 scores here but in the other place, BLEU/CodeBLEU/EM is used. So, what is the intuition that uses different metrics? BLEU can also handle multi-reference evaluation. It would be nice if the authors could add these results in the author response.
2. The research is somewhat superficial. The authors only show the models can tolerate certain noises but do not propose any simple heuristics to alleviate the impact of noises. For example, penalizing the noises (i.e., the targets not belonging to the source) during model training (fine-tuning). [1] might inspire the authors.

[1] Wu, Lijun, Jinhua Zhu, Di He, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. "Machine translation with weakly paired documents." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4375-4384. 2019.

**Typos**  
Section 3 Proposed Method: curated -> created
summary_of_the_review: This paper is interesting, but the research is somewhat superficial.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use similarity metric to generate pseudo alignments between source/target program pairs. Generated pairs are utilized to train program translation model. The paper described a simple greedy method to align both codes, and experimented 5 types of similarity metric as its inner measure.
According to the experiments, the word mover's distance works notably well for this purpose, but other metric can also improve the translation accuracy significantly against random selection.
The other experiment investigating a performance curve by changing noise ratio in the ground-truth parallel corpus hypothesized that certain amount of alignment errors can be acceptable since the actual performance can be maintained.
The paper also challenged to construct translation systems between arbitrary pairs in 10 programming languages using the proposed framework and observed that the trained system works with certain accuracies.
main_review: Although I recognized that the main contribution of this paper is not proposing a sophisticated method to align samples, I thought the paper need more attention to design efficient strategy. The entire alignment algorithm takes O(D*f(D')) where f(D') is the complexity of GetSimilarDocuments(). Unless any assumption this takes O(DD') meaning that it tends to be infeasible by increasing the data. Also, the algorithm seems a simple greedy method and may be heavily affected by the sampling order.

According to Table 2 and 3, I observed that the pseudo-match accuracy introduced here does not reasonably reflect the real characteristics needed to be considered in code translation models since the BLEU of several systems compete with each other.

I recognized that Figure 1 shows every range of noise ratio, except around 100%, has certain gradient against the accuracy, and got the opposite conclusion "the less noise the better" against that the paper concluded "we expect a certain amount of noise, we can expect the models to perform reasonably well." The root cause of this is that the paper did not provide a reasonable criterion about acceptance of this experiment first.

As far as I saw the Figure 2 and 3, the accuracy is heavily reflected by the amount data available during training and some normalization should be needed to compare these metrics each other without biases. Figure 3 also show some remarkable characteristics in translating only into C++, but the paper did not mention about this point.

The motivation of the paper includes "modernizing legacy applications," but the experiments does not reflect this perspective. For example, the first section repeatedly referred to the COBOL applications, but we maybe not be able to reflect the experiments into this application due to large discrepancy between languages.
summary_of_the_review: - The alignment algorithm need to be improved. This is not critical: this is still acceptable for the first observation.
- Conclusion sounds sometimes not reasonable according to experiments. Some results are hard to discuss due to unnormalized data.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper constructs (weak) parallel corpora for code translation tasks using nearest neighbor sampling in the code-document-embedding space. A variety of similarity metrics / base LMs are evaluated.
main_review: In its current form, the problem statement and approach seem very trivial. First, the data sets seem rather unsophisticated and it is not clear what the pathway is from this method to say mining parallel corpora from github or some large-scale code repository websites. In its current form, it seems like this is preliminary work and with some more sophistication (in method + data sets) it is going to be a good future task. I suggest that this be submitted to an appropriate workshop and a future attempt be made to submit this to ICLR. In particular, good directions can include in-the-wild performance when tested on something outside of coding-problem data sets (especially since those are so well-organized by language and problem).


summary_of_the_review: The method and data set in this paper in its current form are rather trivial and more sophistication is required. The techniques seem highly tuned for mining well-organized data sets from coding-problem websites. A better and more improved model that can scale in the wild is recommended and would be a better fit for this venue.
review_title: null

Meta-review: Motivated by addressing the problem of lacking parallel training data for supervised code translation, this paper proposed to construct noisy parallel source code datasets using a document similarity based approach, and empirically evaluated its effectiveness for code translation tasks. 

The paper is in general well-written, easy to follow, and the method is simple and empirical results look positive. Some major concern by reviewers is that while the proposed method is simple and may be easy to use, the overall technical novelty/contribution is limited, e.g., there generally lacks of more thorough discussions on how to deal with the critical noise issue in a more robust or sophisticated way. In addition, there were also other concerns about the experimental issues, such as datasets, metrics, ablation analysis, usability, etc. 

Overall, the paper presents some preliminary positive results for an interesting research problem, but the overall technical novelty and contributions are incremental and the paper is not strong enough for the acceptance by this conference. Nonetheless, this work could be potentially valuable for the niche area of code translation research, and authors are encouraged to continue to improve this research with more thorough investigation for a future venue.
Decision: Reject