Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a framework for problems where the output has some validity constraints, for e.g. the output must be a valid python program that must compile. These kind of problems arise naturally in settings such as pseudocode to program, and moreover there are many more unlabelled valid programs that are easily available (e.g. on Github) than there are labelled examples - i.e. paired pseudo-code, code examples. In this case, the authors propose the following framework of predict and de-noise: 1) train a de-noiser that learns to map synthetically noised versions of the un-labelled valid examples and 2) compose a predictor on the labelled examples with this de-noiser so that end predictions belong to the space of valid programs. The idea proposed in the paper is simple and intuitive, and the authors show that this approach leads to an improvement of 3-5% on the SPOC pseudo-code to code data-set. The authors also provide some theoretical justification why such a composition is the right thing to do.

Overall I like the idea in the paper, and such an approach has been used in NLP for various problems such as spelling correction etc. Besides the predict and de-noise framework another approach that is used in machine translation is the idea of using back-translation, which in the context of pseudo-code to code would look something like this: 1) train a sequence model A such as transformer on code to pseudo-code, 2) use the trained model A to generate paired training data for the unlabelled code (obtained e.g, from Github) by out-putting pseudo-code for it, and 3) train a final model on original labelled pseudo-code to code data plus the artificially generated pseudo-code to code data. I wonder how this would compare to the proposed method? 
review_title: Simple and intuitive idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a more data-efficient way to train generative models with constraints on the output; specifically they evaluate on image generation and pseudocode-to-code (SPoC) tasks. They train two separate models, a “predictor” and a “denoiser”, which they then compose: the output from the “predictor” is further processed by the “denoiser”. For the SPoC task they show an improvement of 3-5% over a simple transformer baseline.
 
The authors suggest a simple idea to make use of unlabelled data, should it be available. They use it to perturbate the unlabelled data and use the (perturbed example, example) pairs to train a denoising model. They argue that this should theoretically simplify the task of the predictor, and show improvements on several tasks. I believe that this is an interesting idea, and practically useful in the cases where data is sparse.
 
However, the results that they demonstrate do not seem very strong, and I would have liked to see this technique demonstrated on more competitive tasks to better gauge how well it works. The improvement of 3-5% they state seems like a low gain over a simple baseline, that may also be achievable with other techniques.
 
Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
 
I do recommend this paper to be accepted, because it clearly presents an interesting idea.
 
The recommendation is a “weak accept” though, because the experimental evidence for the technique is not convincing enough to me. I would have expected significant gains on a well understood task, clearly attributable to the technique.

review_title: Interesting idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a “predict-and-denoise” model for structured prediction, specifically for tasks where the output has to adhere to some constraints e.g. natural language, code etc. This framework allows leveraging of unlabelled output data to train the denoiser, which consequently allows the base predictor to be of low complexity that can potentially generalize with relatively fewer labelled data. The authors theoretically back their arguments basing their theory on a 2 layer ReLU model. The paper demonstrates the performance of this model on two tasks - font image generation, and pseudocode-to-code translation and shows improvement in performance over previous works. 

+ves :

- The paper is very well written and easy to follow. The motivation and the contributions are very clear, and the experimental section is also well detailed and organized. 
- To the best of my knowledge the framework of predict-and-denoise learned in a composed manner and using this framework to leverage unlabelled output data is a novel contribution of the paper. 
- The authors argue that this framework allows reduced complexity of the base predictor, backed theoretically for a 2 layer ReLU network. The authors have provided a detailed proof of their argument in the supplementary material, although I have not completely verified its correctness. 

Concerns :

- I believe that the experimental section currently lacks fair comparisons, especially in the task of pseudocode-to-code translation. The authors compare their method with other methods for leveraging unlabelled data such as pre-training and back-translation. The authors show that predict-and-denoise framework can be applied on top of these existing approaches, and yields consistent improvement.

  However, when comparing such combinations such as “pre-training/back-translation + composed” against pre-training/back-translation, the resulting performance is not  compared with an accordingly scaled base pre-training/back-translation model to have comparable number of parameters. With a very different number of parameters in the models being compared, it's hard to say where the performance benefit is coming from. 

- While the proposed method is complementary to approaches like pre-training and back translation, it will be helpful to also include comparisons such as “composed vs pre-training”, or “composed vs back-translation”. This will give an interesting comparison among these different ways of leveraging unlabelled output data. Again proper care needs to be taken about a comparable number of parameters. 

While I find the framework "predict-and-denoise" very interesting, I am not entirely convinced with its empirical performance reported in the current form and I have given my score accordingly. If the authors agree with my concerns, and can try to incorporate these changes during rebuttal, I will consider updating my score.  

review_title: new interesting framework to leverage unlabelled output data

Meta-review: This work is well written and easy to follow and proposes a novel framework to utilize unlabeled output data. The authors have also given a detailed proof that the denoiser reduces the required complexity of the predictor. However, ultimately the experimental results are somewhat weak and leave doubts as to how effective the approach is. More convincing experimental results such as significant improvements on a well understood task and acknowledging that the approach is mostly useful when combined with pre-training and back translation would improve the work.

Pros
- Well written.
- Technically novel approach to the problem of utilizing unlabeled output data.
- Interesting proof on the reduced complexity requirement for the predictor.

Cons:
- Experimental results are not convincing. Showing significant improvements on a well understood task would be more convincing.
- The approach is only really useful when combined with pre-training or back-translation.
Decision: Reject