Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper seeks to solve program translation problem through code retrieval. It proposes an interactive code retrieval system, called iPTR to perform cross-language retrieval with minimum code pairs. The method extracts textual features and structural features from code and transform the features into target-language features through an autoencoder, which is trained through a uni-language manner. It further leverages users' correction to update feature representation and for new round of retrieval.

The problem is challenging, especially since the parallel data is scarse. I think the idea of utilizing both textual features and structural features is interesting. The idea of training autoencoder for each language and fuse encoder and decoder of different languages is also very interesting, though questionable as mentioned below.

Concerns:
- The major concern is that this paper does not made significant novel and technical algorithm/theory contribution, but rather like proposing a system. Therefore in my opinion the paper does not fit ICLR very well.

- Due to lack of parallel training corpus for query transformation model (QTM), the paper utilizes simple uni-language autoencoder training for each language, and take the encoder and decoder from the corresponding language respectively as the transformer model. This could cause significant issue if not taking seriously. The hidden space of different language can vary drastically, and hence the encoder output is very likely to not make any sense to decoder to perform the transformation. There should be a mechanism that encourages the hidden space of different languages to the similar.

-The paper tries to solve the program translation problem with retrieval method. In the experiment, what is the database to retrieve from? Is it that all training code form the database, or is it all training and testing code form the database? In practical situation, the desired translation is very likely to not exactly match the some code in the database. Therefore, I don't understand why the program accuracy of retrieval method can achieve so high if the database doesn't overlap with ground truth codes a lot.
review_title: Lacking significant algorithm/theory contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a retrieval-based approach for program translation. Existing ML/DL models for program translation typically design a decoder to directly generate the code in the target language. On the contrary, this work designs iPTR, which first computes a feature representation of the target code, then retrieves the most similar code in a large code corpus. Specifically, iPTR includes a query transformation model (QTM), which generates the feature representation of the target code, given the feature representation of the source code as the input. The feature representation includes the information of tokens in the code, and the paths in the syntax trees. The idea of using the paths is similar to the code2vec paper. QTM could be trained without parallel data between source and target codes. Specifically, encoders and decoders of different programming languages could be trained in a similar way to the training of autoencoders. This idea is similar to TransCoder. Meanwhile, they show that iPTR could be trained with active learning and user feedback, where they use active learning to acquire limited parallel training data, and user feedbacks are corrections of the wrong output code. They compare their approach to existing ML/DL program translation models, as well as code retrieval systems. They show that iPTR performs better than other baselines, even without active learning and feedbacks. Not surprisingly, active learning and incorporating user feedbacks further improve the performance.

Program translation is an important application, and the authors did a good job of evaluating on existing benchmarks and comparing with different types of baseline models. Intuitively, it makes sense that retrieval could at least provide more syntactically and semantically correct code, while synthesis models may struggle with generating coherent code. However, I have a couple of questions about the assumption of the task setup and the implementation of the algorithm, and I list them below.

1. It seems that the feature representation is summarized per complete code snippet for translation. Therefore, do the authors assume that the target code always exists in the retrieval corpus as a single piece of code? If this is the case, one clear limitation is that the retrieval approach could only search for existing code in the corpus, while the synthesis model could combine lines from different code snippets to construct the output code.

2. How do you simulate users for active learning and user feedback? For active learning, do you ask people to annotate the ground truth output programs for input programs, and then add them into the corpus for retrieval? For user feedback, in some parts of the paper you mention that the user modifies the extracted features of the code, but sometimes you also say that the user corrects the first wrong line. Could you provide some concrete examples of how you incorporate human annotations in training and inference loops?

3. More concrete examples of how to sample code for active learning could help. Now the description in Section 3.2.2 is not clear. For example, why N_s is 5 instead of 4? What is the benefit of "selecting program data where the largest disagreement occurs among those sampling strategies"? Do you compare among the entire training corpus?

4. What are the numbers of different paths and tokens for feature representation of code? Are text tokens representing terminal nodes or non-terminal codes? For example, are variable names included in tokens, or are they simply denoted as "identifier"? My understanding is that the feature vectors follow the bag-of-words representation, thus I don't quite understand why this simplified representation could work better than code2vec or other more advanced program encoders. The authors say that "theWord2vec and Code2vec variants of PTR cannot outperform PTR with our proposed feature representation based on structural features", but I think code2vec encodes very similar structural features in a potentially more concise way.

Writing comments: the paper requires proofreading, and there are a couple of typos. For example, (1) "to approximate measure the informativeness of the given program" on page 4: "approximate" should be "approximately". (2) "To reflect this in out feature representaion" on page 4, "out" should be "our".
review_title: Interesting work on program translation; some important details are unclear

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a program translation (retrieval) method by combining  syntax tree, transformer with specific encoder/decoder, and interactive signals from user. The novelty of each used techniques is limited, but they are reasonable for the code translation task with high accuracy, even for unsupervised version without interactive signals.

The experimental results are solid and demonstrate the power of representation of the proposed variant of syntax tree. The idea of AE/AD with specific language is quite simple, but it seems that the QTM successfully transfer the representation cross different languages. The performance gain achieved by interactive signals is reasonable.

Questions:
- there are other structural features of programming language, such as graph, which can provide rich information of the code snippets rather than syntax tree. 
- Is the proposed method a better code representation? Could it be applied to other downstream tasks after pre-training?
- Will the authors release their code to public?

review_title: Good paper

Meta-review: I found the setup for this paper a bit contrived. The tool is presented as a code translation tool, but it really functions more as a multi-language code search tool. The Idea is that one has a program in language A, and a database that contains the same program in language B, so one can translate from A to B simply by searching for the right program in the database. 

When evaluated as a language translation tool, it appears to outperform existing language translation schemes, but this is an unfair comparison, because iPTR is being given a database that contains the exact translation of the program in question. The performance is also compared with code search tools, but these are also apples-to-oranges comparisons, because the tools in question are operating from very high-level queries. A much more comparable baseline would be the Yogo tool  recently published in PLDI (https://dl.acm.org/doi/abs/10.1145/3385412.3386001), or for compiled languages you could compare against statistical similarity tools for binaries (https://dl.acm.org/doi/10.1145/2980983.2908126). 

The experiment in the appendix A5 is more fair to standard language translation, and it yields results that are much less impressive. I would be much more comfortable with this paper if it were written around this experiment, or alternatively if it were evaluated against a more comparable approach for semantic code search. 
Decision: Reject