Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors proposed a neural network based machine translation method between two programming languages. The model is based on both source/target syntax trees and performs an attentional encoder-decoder style network over the tree structure.

The new things in the paper are the task definition and using the tree-style network in both encoder and decoder. Although each structure of encoder/decoder/attention network is based on the application of some well-known components, unfortunately, the paper pays much space to describe them. On the other hand, the whole model structure looks to be easily generalized to other tree-to-tree tasks and might have some potential to contribute this kind of problems.

In experimental settings, there are many shortages of the description. First, it is unclear that what the linearization method of the syntax tree is, which could affect the final model accuracy. Second, it is also unclear what the method to generate train/dev/test data is. Are those generated completely randomly? If so, there could be many meaningless (e.g., inexecutable) programs in each dataset. What is the reasonableness of training such kind of data, or are they already avoided from the data? Third, the evaluation metrics "token/program accuracy" looks insufficient about measuring the correctness of the program because it has sensitivity about meaningless differences between identifier names and some local coding styles.

Authors also said that CoffeeScript has a succinct syntax and Javascript has a verbose one without any agreement about what the syntax complexity is. Since any CoffeeScript programs can be compiled into the corresponding Javascript programs, we should assume that CoffeeScript is the only subset of Javascript (without physical difference of syntax), and this translation task may never capture the whole tendency of Javascript. In addition, authors had generated the source CoffeeScript codes, which seems that this task is only one of "synthetic" task and no longer capture any real world's programs.
If authors were interested in the tendency of real program translation task, they should arrange the experiment by collecting parallel corpora between some unrelated programming languages using resources in the real world.

Global attention mechanism looks somewhat not suitable for this task. Probably we can suppress the range of each attention by introducing some prior knowledge about syntax trees (e.g., only paying attention to the descendants in a specific subtree).

Suggestion:
After capturing the motivation of the task, I suspect that the traditional tree-to-tree (also X-to-tree) "statistical" machine translation methods still can also work correctly in this task. The traditional methods are basically based on the rule matching, which constructs a target tree by selecting source/target subtree pairs and arranging them according to the actual connections between each subtree in the source tree. This behavior might be suitable to transform syntax trees while keeping their whole structure, and also be able to treat the OOV (e.g., identifier names) problem by a trivial modification. Although it is not necessary, it would like to apply those methods to this task as another baseline if authors are interested in.
review_title: New but trivial model, poor experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to translate source code from one programming language to another using
a neural network architecture that maps trees to trees. The encoder uses an upward pass of
a Tree LSTM to compute embeddings for each subtree of the input, and then the decoder 
constructs a tree top-down. As nodes are created in the decoder, a hidden state is passed
from parents to children via an LSTM (one for left children, one for right children), and
an attention mechanism allows nodes in the decoder to attend to subtrees in the encoder.

Experimentally, the model is applied to two synthetic datasets, where programs in the 
source domain are sampled from a PCFG and then translated to the target domain with a
hand-coded translator. The model is then trained on these pairs. Results show that the
nproposed approach outperforms sequence representations or serialized tree representations
of inputs and outputs.

Pros:

- Nice model which seems to perform well.

- Reasonably clear explanation.

A couple questions about the model:

- the encoder uses only bottom-up information to determine embeddings of subtrees. I wonder 
if top-down information would create embeddings with more useful information for the attention
in the decoder to pick up on.

- I would be interested to know more details about how the hand-coded translator works. Does
it work in a context-free, bottom-up fashion? That is, recursively translate two children nodes
and then compute the translation of the parent as a function of the parent node and
translations of the two children? If so, I wonder what is missing from the proposed model
that makes it unable to perfectly solve the first task?

Cons:

- Only evaluated on synthetic programs, and PCFGs are known to generate unrealistic programs, 
so we can only draw limited conclusions from the results.

- The paper overstates its novelty and doesn't properly deal with related work (see below)

The paper overstates its novelty and has done a poor job researching related work. 
Statements like "We are the first to consider employing neural network approaches 
towards tackling the problem [of translating between programming languages]" are
obviously not true (surely many people have *considered* it), and they're particularly
grating when the treatment of related  work is poor, as it is in this paper. For example, 
there are several papers that frame the code migration problem as one of statistical 
machine translation (see Sec 4.4 of [1] for a review and citations), but this paper 
makes no reference to them. Further, [2] uses distributed representations for the purpose 
of code migration, which I would call a "neural network approach," so there's not any 
sense that I can see in which this statement is true. The paper further says, "To the best 
of our knowledge, this is the first tree-to-tree neural network architecture in the 
literature." This is worded better, but it's definitely not the first tree-to-tree 
neural network. See, e.g., [3, 4, 5], one of which is cited, so I'm confused about 
this claim.

In total, the model seems clean and somewhat novel, but it has only been tested on 
unrealistic synthetic data, the framing with respect to related work is poor, and the
contributions are overstated.


[1] https://arxiv.org/abs/1709.06182
[2]  Trong Duc Nguyen, Anh Tuan Nguyen, and Tien N Nguyen. 2016b. Mapping API elements for code migration with
vector representations. In Proceedings of the International Conference on Software Engineering (ICSE).
[3] Socher, Richard, et al. "Semi-supervised recursive autoencoders for predicting sentiment distributions." Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 2011.
[4] https://arxiv.org/abs/1703.01925
[5] Parisotto, Emilio, et al. "Neuro-symbolic program synthesis." arXiv preprint arXiv:1611.01855 (2016).

review_title: Some interesting components, but overstates contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a tree-to-tree neural network for translating programs
written in one Programming language to another. The model uses soft attention
mechanism to locate relevant sub-trees in the source program tree when 
decoding to generate the desired target program tree. The model is evaluated
on two sets of datasets and the tree-to-tree model outperforms seq2tree and
seq2seq models significantly for the program translation problem.

This paper is the first to suggest the tree-to-tree network and an interesting
application of the network for the program translation problem. The evaluation
results demonstrate the benefits of having both tree-based encoder and decoder. 
The tree encoder, however, is based on the standard Tree-LSTM and the application
in this case is synthetic as the datasets are generated using a manual rule-based 
translation. 

Questions/Comments for authors:

The current examples are generated using a manually developed rule-based system. 
As the authors also mention it might be challenging to obtain the aligned examples
for training the model in practice. What is the intended use case then of 
training the model when the perfect rule-based system is already available?

How complex are the rules for translating the programs for the two datasets and what
type of logic is needed to write such rules? It would be great if the authors can 
provide the rules used to generate the dataset to better understand the complexity
of the translation task.

There are several important details missing regarding the baselines. For the 
seq2seq and seq2tree baseline models, are bidirectional LSTMs used for the encoder?
What type of attention mechanisms are used? Are the hyper-parameters presented in
Table 1 based on best training performance?

In section 4.3, it is mentioned that the current models are trained and tested on 
programs of length 20 and 50. Does the dataset contain programs of length upto 
20/50 or exactly of length 20/50? How is program length defined -- in terms of 
tree nodes or the number of lines in the program?

What happens if the models trained with programs upto length 20 are evaluated on 
programs of larger length say 40? It would be interesting to observe the 
generalization capabilities of all the different models.

There are two benefits of using the tree2tree model: i) use the grammar of the
language, and ii) use the structure of the tree for locating relevant sub-trees
(using attention). From the current evaluation results, the empirical benefit
of using the attention is not clear. How would the accuracies look when using 
the tree2tree model without attention or when attention vector e_t is set to the
hidden state h of the expanding node?

review_title: Review

Meta-review: the problem is interesting, and the approach is also interesting. however, the reviewers have found that this manuscript would benefit from more experiments, potentially involving some real data (even at least for evaluation) in addition to largely synthetic data sets used in the submission. i also agree with them and encourage authors to consider this option.
Decision: Invite to Workshop Track