Simplifying Models with Unlabeled Output Data
Authors: Sang Michael Xie
Tengyu Ma
Percy Liang
Venue:  (2021)
Keywords: semi-supervised learning
structured prediction

Abstract: We focus on prediction problems with high-dimensional outputs that are subject to output validity constraints, e.g. a pseudocode-to-code translation task where the code must compile. For these problems, labeled input-output pairs are expensive to obtain, but "unlabeled" outputs, i.e. outputs without corresponding inputs, are freely available and provide information about output validity (e.g. code on GitHub). In this paper, we present predict-and-denoise, a framework that can leverage unlabeled outputs. Specifically, we first train a denoiser to map possibly invalid outputs to valid outputs using synthetic perturbations of the unlabeled outputs. Second, we train a predictor composed with this fixed denoiser. We show theoretically that for a family of functions with a high-dimensional discrete valid output space, composing with a denoiser reduces the complexity of a 2-layer ReLU network needed to represent the function and that this complexity gap can be arbitrarily large. We evaluate the framework empirically on several datasets, including image generation from attributes and pseudocode-to-code translation. On the SPoC pseudocode-to-code dataset, our framework improves the proportion of code outputs that pass all test cases by 3-5% over a baseline Transformer.

Under review as conference paper at ICLR 2021
SIMPLIFYING MODELS WITH UNLABELED OUTPUT DATA
Anonymous authors Paper under double-blind review
ABSTRACT
We focus prediction problems with high-dimensional outputs that are subject to output validity constraints, c.g. a pscudocode-to code must compile. For these problems, labeled input-output pairs are expensive obtain, but "unlabeled" outputs, i.c. outputs without corresponding inputs, are freely available and provide information about output validity (c.g. code on GitHub). In this paper, we present predict-and and-denossd, framework that can leverage unlabeled outputs. Specifically, we first train a denoiser map possibly invalid outputs to valid outputs using synthetic perturbations of the unlabeled outputs. Second, train a predictor composed with this fixed denoiser. We show theoretically that for family of functions with nigh-dimensional discrete valid output space, composing with a denoiser reduces the complexity of 2-layer ReLU network needed represent the function and that this complexity gap can be arbitrarily large. We evaluate the framework empirically on several datasets, including image generation from attributes and pseudocode-to-cott dataset, our framework improves the proportion
translation task where the
translation.
On the SPoC pseudocode-to-es of code outputs that pass all test cases by 3-5% over a baseline Transformer.
INTRODUCTION
We study problems whose outputs have validity constraints. For example, in pscudocode-to-code translation, the output code must compile. Other examples include natural language translation and molecule generation, where outputs should be grammatically correct or chemically valid, respectively, State-of-the-an models typically learn the input-output mapping from expensively-obtainel labeled data Kulal 2019); Vaswani al. (2117): Méndez-Lueio al. (2020): Senior et al. (2020), which may not contain enough examples to learn complex validity structure on high-dimensional output spaces. However, there are often lots of "unlabeled"
outputts without corresponding input (e.g., GitHub has over 40 million public code repositories). How do we leverage these with a much smaller amount of labeled input-output pairs to improve accuracy and validity?
In this paper, we present predict-andddeniis which maps an input to a possibly invalid output, with a denoiser, which maps the possibly invalid output to valid output. We first train the denoiser on synthetic perturbations of unlabeled outputs. Second, train the base predictor composed with the fixed denoiser on the labeled data (Figure T left). By factorizing into two modules, base predictor and denoiser, the framework allows the base predictor to be simpler by offloading the complexity of modeling the output validity structure to the denoiser, which has the benefit of being trained on much more data.
framework in which we compose base predictor,
We aim to lay down a principled framework for using unlabeled outputs with theoretical justification for improving sample efficiency by reducing the complexity of the learned base predictor. Figure (middle.righ) shows pictorial example of staircase function where valid outputs are integers and requires a complex spline to represent. When composed with denoiser (which rounds to the nearest integer), a simple linear base predictor can represent the staircase function. We theoretically show that our framework reduces the complexity of a 2-layer ReLU network needed represent family of functions on a discrete valid output set in high-dimensions. This complexity gap can be arbitrarily large depending the stability of the target function being learned. We expect such lower complexity function to learnable with fewer samples, improving generalization.
Empirically, we show on image generation and two pseudocode-to- -o-code datasets (synthetic and SPoC Kulal et al. (2019)) that predict-and -nd-nnoose improves test performance across continuous and discrete output data modalities. of font images by learning a low-complexity base predictor to generate an abstract image while the denoiser sharpens the image. For pseudocode-to-code: we consider the more difficult program
image generation, our framework improves the clarity and styling

---

Under review as conference paper at ICLR 2021
Direct predictor (complex)
Direct Target
Base Compased
Ingest x eet x 10
Bice predictor [simple]
Qutput 5
3-
Denolser
X
1-
Labeled data [(input, outputi)
Many "untabeled" outputs
01
lounput, .utpptt
Figure 1: (Left) The predict-and-denoise framework: First, a denoiser is learned using synthetic perturbations of a large number of unlabeled outputs. Second, a base predictor composed with the denoiser is learned with labeled data. Composing with denoiser allows the base predictor to be simpler, improving generalization. (Middle) Univariate regression example where a staircase function requires a complex linear spline fit. (Right) A simple linear function can fit a staircase function when composed with denoiser which projects onto the valid outputs (the integers).
translation task rather than line-by-line translation (with compiler side information) studied by previous work|Kulal et al.] (2019):] (aaaaa/aaaaaaaaa and Liang (2200). Li20))(2000) We first study synthetic pseudocodc-to-codd dataset where the denoiser simplities the base predictor by helping with global type inference. On SPOC, a recent secudocodc-to-ood dataset on programming competition problems, we improve the proportion of correct programs by 3-5% points over baseline Transformer.
2 SETUP
We consider prediction problems from an input space X' (e.g.. pseudocode) to an output space )/ (e.g.. code) where there is unknown subset of valid outputs V J (e.g., code that compiles), where the true output always valid (in V). We have a labeled dataset (21,.9) ....)).....()) ".....(TTn..., where n. E and y E V and access to many unlabeled outputs (9m)... from V. We do not assume access to any black box function for testing validity (whether y E V not), allowing for general problems (e.g. language generation) where output validity is imprecisely characterized.
A predictor| f: X + y' from chosen hypothesis class Ht maps from inputs to the ambient output space. Our goal is to improve the predictor by leveraging information about the valid space V from the unlabeled examples {j; mm We leverage a denoiser y - V. which projects a possibly invalid output in ) and to the valid set V. We can use unlabeled outputs to learn approximate denoiser.
Base, composed, and direct predictors. Let ||-|| be a norm on H. Let II fuuse be a composed pre- dictor that is supposed to represent the target function f (that is, II foase =f* on X). In the context of a composed predictor, we call faase the base predictor. We compare against fuireat E Cargminges ew {ll.f]]: f()) = f E X). a minimum norm direct predictor which represents f*.
DENOISERS CAN REDUCE MODEL COMPLEXITY
In this section, we study direct and composed predictors from approximation standpoint and use complexity measures on predictors as surrogates for sample complexity. We aim to represent a target function f : X - V. We assume access to a denoiser II: ) - V which projects to the nearest valid output for an appropriate metric on the output space (breaking ties arbitrarily). In Section 3. give a simple example for when composing with a denoiser (II ftass) can drastically reduce the complexity of the learned predictor. Since Shace becomes easier to approximate, we may expect better generalization Bartlett al. (2017): Neyshabur al. (2017); Wei and Ma ( 2019). In Section 3.2 we theoretically show for two-layer ReLU networks that the complexity required to directly represent f" can be arbitrarily larger than representing with composed predictor depending the stability of f*
3.1
MOTIVATING EXAMPLE
Figure I shows staircase function I that requires a complex direct predictor fdirec but the
minimum norm base predictor fine has low complexity. For 0 < 8 << 1, let the input space
X' = be the integers, subset of the ambient output space y' - IR. The staircase function is f*(x) [x]
: [i f (1 5) /2 (1- 8)] be union of N disjoint intervals and the valid outputs y = Z
defined on X,, which rounds a linear function onto the integers. Following Savarese SaalareeeSSvvrreeeeeeeeee et al. (2019) we
define the norm of a univariate function f: R as
|| = 1 max ( (d' (+
(1)
2

---

Under review as conference paper at ICLR 2021
This norm measures the (lack of) stability of f. Complex functions will have higher norm.
Consider representing f* with linear splines, a family of piecewise linear functions. In linear splines. the norm in Equation becomes roughly the sum of absolute changes in slope between piecewise segments. If we represent f directly with linear spline fdineet, the norm of fairect has to be large due to the large number of slope changes:
[[areeet = (N 1)/8 (Figure left).
Suppose we have access to a denoiser (T()= = [y]. which projects onto V= Then a linear function hase composed with II can represent the staircase on X' reducing the norm to (Figure Saaureerrihhtt rrghtt) By
not requiring tuuse to represent the local complexity and discreteness f", the base predictor Shaee better captures the underlying globally linear structure of f*
3.2
ANALYSIS FOR 2-LAYER RELU NETWORKS
We extend more general hypothesis classes and high dimensional outputs. Our setting is motivated by the task of generating images of font characters from attributes, which we study empirically in Section 5.1 In font image generation, there is a discrete set of valid font images in the continuous ambient output space. Formally, we take the valid set V {vr .....Vyy } to be a discrete set over N output values in & and f is a piecewise constant function defined on N disjoint intervals X' = W [=)" (in ascending order), where there is a 6 >> 0 gap between each interval and the next. The target function f defined such that x E [=] then [*(x)= =
We study 2-layer ReLU networks, often studied as a first step towards understanding the expressivity of neural networks Neyshabur et al.] (2014); Saavarese et al. (2019; Eldan and Shamir (2166).
Following Savarese (2019), we define Ja aa as
h fo(x)= = Σ w
(taf (uff xx) +b,0 +0/20
on a E Rdd where we will take d= throughout. Here, - ma. is the element-wise ReLU nonlinearity. The parameters contain the hidden unit size h e N and all weights and biases. We let (1)) E Raxd denote the matrix with w/(") E Rd as rows and let ((1),(()) (2()) E R be vectors with E as elements respectively. We let θ denote this parameter space.
B( .b( un (2)
Measure of complexity. Following Savarese et al. (2019), the complexity of a network is associated with the squared Euclidean norm of the weights
C(0) - (2 I
The norm of f E is the minimum norm required to represent f:
|| = inf C(²) s.t. fg f.
dge
Savarese et al. 2019) showed that this norm is equivalent to Equation for univariate networks. Since these complexity measures typically appear in generalization bounds Bartlett et al. (2017); Neyshabur
ct al. 2017) we expect to improve generalization error by reducing these complexity measures.
Minimum complexity reduces with a denoiser. Given IT(y) E argmin ev //v" yll2 which is
projection onto V (breaking ties arbitrarily), we want to compare the norms of fairest that represents f" directly and the minimum norm base predictor that represents f":
Frase arggmin FEH
( [Il][[ = =f) (x)
Note that | || [| since fairec is a feasible solution. Thus composing cannot increase the norm.
Adjacent intervals measure stability. Our result depends crucially on the number of non-adjacent pairs of intervals in f*. Suppose the output dimension is & = 1. We define pair of interval indices (i,i+1) 1) as adjacent if there is no valid output value y e such that either y < y < y or <
Vi41 <y<ン,, y hold. The number of non-adjacent interval pairs characterizes the instability of f* Let [J] be the number non-adjacent pairs and // be the number of adjacent pairs, where +||| Our bound also depends on L min,ly; - Vi+ and U = max; ly; - v I. the min and max separation between valid points. For higher output dimensions (k> 1), let vi,, be the j-th output coordinate of the i-th valid point and let 1.I.,1,, be the analogous quantities for each output coordinate j [k].
3

---

Under review as conference paper at ICLR 2021
main
inb main
Instantiang var-T;
dering var 7y
ateing VaT. TV
2eed vuc_s -roe vieinn out
instantiale vat_5r
cinn po var 5y
var_o
cin vor by
AnE var_
add *ate_0" to the and var_i;
snt varL3 to nax of var_3 and var_9p
plus var_sy
san's nax cone 5, var $1A
vav
var_? "at-cLOTE
var PANTVAY b, vAr S12
11 outgut var.5 12 pein: 96
12
cout <6 vAY 52
12 cont 2a7.9)
11
ruturn
13
zuturn D:
Figure 2: (Left-Middle) Example pseudocode and code from the synthetie dataset. Since the pseudocode is ambiguous, variable types and whether to instantiate a variable must be inferred. (Right) Random corruption used to train denoiser from corrupted to valid code. The denoiser must infer the correct type of var_5 from other lines.
Theorem 1. Let the valid output space V C R be set over N multivariate output values 9 in V. Let f*: :RR 1R-+I +R be piecewise constant function defined on X' - where f* (x)= if x E [.] Let A, be the length of the smallest interval in X. Then
I Masee
Nmax, Lj
12
(4)
Σラ= (1)1+0 UI
See Appendix A for a proof. If | are sublinear in N and valid points are evenly spaced, then the gap is (2(15) which can be arbitrarily large for fixed output dimension d - and N - co. If any |J is linear in N (many non-ad adjacent intervals), then there is only a constant factor gap in the worst case. Overall, if f is stable with respect to its discrete output space, we can learn a simpler base predictor that still represents f when composed with the denoiser. Note that in practice, need to regularize the base predictor to find this low complexity solution.
PREDICT-AND-DENOIS FRAMEWORK
we assumed access to denoiser IT that maps output y E y to valid output ( E V, allowing the min-norm base predictor fine to have much lower complexity. In this section, we are not given a denoiser but instead have access to a large number of unlabeled outputs fi,-..,uu E V. We present models. In this framework, we first use self supe-sueee learning on the unlabeled outputs to learn an approximate denoiser 11, and then use II (which is now fixed) to learn composed predictor II fa. Here, fo is the learned base predictor with parameters 0.
In Section 3
framework for utilizing unlabeled output examples to simplify
(leftr-mdlt gives an example input-output pair in a pseudocode-to-co sediit-aan-dennois framework, the model could learn to make code translations on mostly local, line-by-line basis (a simpler solution) while relying on the denoiser correct types globally.
task. Using the
Learning the denoiser. Assume that as domain knowledge, we have a noising distribution g(Vl (0) over outputs given a valid output J. Figure (middle) gives an example of an output program in pseudoccodeto--ooe -oddee- translation task. Here, noising distribution may make random semantic and syntactic corruptions such as changing types or removing semicolons and parentheses (Figure Sequure2right right.. The denoising objective here is to recover the original code from corrupted code. More generally, given the noising distribution, we train a probabilistic model pa (ū (7') on output pairs (j7 .9) where y' q- f). We train the probabilistic model by maximizing the log- likeeiilioood
maximize Ey [ (oggs (7)l]]
(5)
using unlabeled output samples. The denoiser (T') argmax, Ps 8 is defined via the probabilistic model.
Learning the composed predictor. In this step, we fix the learned denoiser II and learn the composed predictor 11 o So on labeled examples. We train a probabilistic model po for the base predictor by optimizing
maximize, Ez.y (By'' (logs(y) AB
[x:)]-
The first term maximizes lower bound on the likeliiid of the composed predictor via p. and po (see Appendix DD). We optimize a lower bound since optimizing the log- likeliold directly requires computing an intractable partition function over the high- sigh-dimensional output space. The second term is the likeeiolloood of only po. We define the base predictor fo(x;) argmax,,PP Po (ar).
4

---

Under review as conference paper at ICLR 2021
4 6
C
A
Z
K
H
B
F
B
A
O
G H
H
A O
C
O
O
H
A
B
C
D
E
F
G
H
A
B
B
A
B
O
A
(a) Direct
(b) Composed
(c) Basc
3: Generated letters A-J for 10 randdolyy selected fonts. (a) The direct predictor makes blurry out- puts with many artifacts. (b) The composed predictor (base denoiser) makes clearer outputs with more distinct font patterns. (c) The improvement comes from leveraging output structure learned by the de- noiser. This allows the base predictor to produce blurrier outputs corresponding to lower norm model.
Since the learned 1 is imperfect, the hyperparameter À in the objective trades off between fitting the composition 11 fo and fitting fo directly to the data. For discrete output spaces, the first term in this objective involves expectation over discrete space of outputs. Depending the task, optimizing this objective may require R (1992) or a Gumbel-softme reparameterization Jang et al. (201)) Maddison et al. (2016). The direct predictor is only trained with the second term of our objective Er.y [ogpo(y x]]-
the model and
Choice of noising distribution. Learning the base predictor composed with the denoiser allows for some distribution mismatch between the errors of the base predictor and the noising distribution the denoiser is trained By learning in composed manner, the base predictor adapt the choice of noising distribution. In our experiments Appendix we find that predict-and-denois gives gains across variety of noising distributions.
EXPERIMENTS
We evaluate predict-and-denoit on image generation from given attributes and full-program mueedocode-to-code translation, showing its benefits on both continuous and discrete output spaces. In image generation, composed models generate clearer images with less artifacts with few labeled examples. For full full-program pseudocode-te to-code translation in SPoC Kulal et al. (2019), a recent pseudocode-code dataset, our framework improves the proportion of correctly generated programs by 3-5% points over a baseline Transformer and achieves comparable better results a line-by-line translation model from previous work Kulal et al. (2019).
5.1 IMAGE GENERATION FROM ATTRIBUTES
We evaluate prddict-and-denoie continuous. This task closely mirrors the theoretical setup, where the input wiwddimesiional (index of the font and character type) to a high-dimensional output (image). We also validate the theory from Section which suggested that regularization is required realize the complexity reduction of the minimum-norm base predictor fi Qualitatively, image samples from our composed predictor are clearer and has less artifacts.
font image generation, where the ambient output space is
Prediction task and denoising objective. We mapping two one-hot vectors corresponding to the character identity (out of 62 possible) and the font of the character to generate (out of 100 fonts) x 32 grayscale font images. Here, valid font images have cleanly defined lines and adhere to the font styling. We train using the pixel-wise squared error loss for all models and tune I.2 regularization strength on a validation set. To train the composed predictor, we set À in ( using only the composed loss. The denoising objective is to sharpen unlabeled font images distorted by Gaussian blur filter with randomly sampled radii in [0.2]. We also report gains with other noising functions (embossing, contrast perturbations) in Appendix
Data. We use dataset of 56k fonts originally scraped from the Internet [Bernhardsson (001)). Out of the 6200 labeled examples (62 characters x 100 fonts), split randomly into 2500 training examples, 100 validation examples, and 3600 test examples. The training examples contain a random subset of the characters for each font. The models must generate the unseen characters of each font with the correct font styling test-time. The denoiser uses additional unlabeled images for 50k other fonts.
5

---

Under review as conference paper at ICLR 2021
Direct
Test MSE
0.30
Composed
0.25
Direct
0.19 0.193
Composed
0.171
0.20
10-2
10-1
12
strangth
Figure 4: Test MSE on font image generation. (Left) Results when L2 regularization strength is tuned with the validation set. (Right) Varying 1.2 regularization strength (1e-6 to le-2) for direct and composed predictors. While similar at low regularization, increasing the regularization strength improves the composed predictor while hurting the direct predictor.
Models and metrics. The base predictor So and the direct predictor fdieeet are both 7-layer fully-connected networks (see Appendix(B). The denoiser HIg is 3-layer U-Net (ooneeeeneerrrgrr et al. (2015). We test image sample quality directly by computing the pixel-wise squared error with respect to ground truth test images.
Results. For regularization strength tuned on the validation set (Figure the composed predictor achieves an 11% reduction in test MSE compared to the best direct predictor test error. The direct predic- tor test MSE increases when its outputs are processed by the denoiser at test time. We visualize the pre- dicted images for some randomly- seseeml--seleced fonts for comparison (Figure The base predictor trained with 1.2 regularization outputs noisy gray images, suggesting that it has learned lower complexity model. In contrast, L2 regularization does not improve the direct predictor (Figure({rright)) since directly outputting clearly defined lines and transitions between black and white pixels requires a relatively high complexity model. Note that we study L2 regularization as motivated by theory, but we expect any reasonable regularization method to help. Indeed, we find that adding dropout to the composed model improves the MSE further to O. 0.. Additional results on varying labeled and unlabeled data size are in Appendix B
where the performance of the Composed model improves upon Direct on all instances.
5.2 PSEUDOCODE--T-COOM
We evaluate predict-and-denoise on pseudocode-to-coot translation, where the ambient output space is discrete. We evaluate on two aseudoccde--o-cous datasets (synthetic in Se.2ion[5.2.1 I.2.Iaadd and SPoC in Section 5.2.2 5.2.2 On SPOC, our framework improves the proportion of programs that pass all test cases by 3-5% points over baseline Transformer and has competitive or better results to line-by-line models Kulal et al. (201)).
Prediction task and denoising objective. We consider full-program pseudocode-to-code transla- tion, where inputs X' are human-genne -ouaan-geneatte pseudocode. The ambient output space y is all possible strings and the set of valid outputs V are strings that compile with the g++ compiler. In contrast to previous works which decompose the problem into line-by-line translation and use information from the compiler Kulal et al. (2019); Yasunaga and Liang (2020). we translate the entire program once without compiler access. Following Yasunaga and Liang (2020), the denoising objective for both pseudocode- co-eee datasets consists of repairing random semantic and syntactic corruptions of unlabeled code examples (see Appendix E)
Models and regularization. We use a Transformer Vaswani et al. (2017) for both the base predictor and the denoiser. In all models, we use combination of weight decay, dropout, attention dropout, and ReLU dropout as regularization. To train the composed predictor, we use x=1 to balance between the fitting the composed and direct objectives. During inference, we use a greedy decoding for simplicity (without beam search). Problem-specite optimizations such as beam search and querying a compiler during inference can improve the results further.
Pretraining models. In machine translation, a standard way to incorporate unlabeled outputs is to pretrain the encoder/decoder on monolingual data Ramachandran et al. (2018); (Skorokhodoy et al. 2018): Devlin et al. (2019). We consider pretrained predictor which is pretrained with the denoising objective on unlabeled code and then trained on labeled examples, utilizing a shared encoder/decoder vocabulary. We employ predict-and -ndeeni-nd-eeoie on top by initializing from the pretrained model Seneeraiind+Coopssse which provides complementary benefits beyond pretraining.
models. Back-translation methods the labeled data) applied on unlabeled outputs to generate additional synthetic inputs [Sennrich] ct al. (2016b). We employ aaddicc-anddeeooii on top by initializing from a wack-trnnslation model (Rack-translation + Composed), showing complementary benefits.
output to input model (learned on

---

Under review as conference paper at ICLR 2021
Complic
Exte Canreal
SPoC TESTP
Direct
75.5
Direct-caaed
Compused
75.1
Compile Err
Exec Err
Currel
Pretranne Composed
Direct
51.4
12.01
36.6
Composed
19.2
12.8
68.0
Backranskation Composed
712
Prettiined
11.2
10.8
73.0
Line-by-line
Kutaa etal. 2019
Prottained + Composed
9.0
4.8
87.6
SPoC TESTW
Backuranslation
0.8
10.0
$9.2
Direet
Backuranslation + Composed
1.2
7,0
91.8
Direes-scaled
Composed
36.8
Table 1: Results on synthetic pseudocode- to-code task. Proportion of generated code (%) resulting compilation error, execution-tine error, or correct code.
Pretiined
Pretrained Composed
46.1
Backtasssatiinn
Busktranslation Compused
42.4
15.3
42.1
Lino-by-line
Kulal etal.
2019
Table 2:
Results on SPOC pseudocode-to-code test sets, showing proportion of outcomes of generated code (%). Direct-scaled has the same number of total layers the composed model, but roughly double the number of d'trrinable parameters.
int rain II atzing vaz_5 "ata_21s
int vain 1 ( atzzna var_ *aterAA)
read var. Izon acdles
ser var to faloes
instantsate VaT_2:
aod *ate to the begiening of vaE_8
var falee;
14 2
vaz_3 feEe var_a:
vas LE907
bool Vos LEOR) if DaT terp
vaz_ "atsL var
UST 7 Falen;
71
cupput war_ Lo acdouty var_2; curput
LE var_2 in Lzuet
LE the values
and vet_. to value of var
ent var.2 to the value of War
and Var
14
11
12
13
10
if var 2 }
boot terp vaT
vaz_á vas_āy
12
13
1f ( var I
boot terp var 2)
ver_2 var_Br
16
vas_.r caut ds cout 44 war.5;
19
cout var_.;
10 cout cc var_5;
10
15
return Di
19
zetuzn 02 )
Figure 5: (Left-Middle) Example input and output of the base predictor on the synthetic dataset. (Right) Output of the denoiser, which instantiates var__ and corrects the type of temp.
Metrics. A generated program has three possible outcomes: compilation error, execution error, or correct. A program is correct if, executed outputs. We measure the proportion of programs that fall into these outcomes.
of input test cases, its outputs match the set of gold
5.2.1 SYNTHETIC DATASET
Pseudocode specifies local information but there are global consistency constraints to enforce (Figure 2). Modeling everything directly requires a complex model. With predict-and-denois the base predictor fo can do local translation while the denoiser IT enforces global constraints such as type correctness. To test this intuition, we generate a synthetic pscudocode-to-codee dataset where the pseudocode specifies all but the declaration types (see Figure
Dataset generation. The synthetic programs involve 1-4 variables (bools, ints, and strings) drawn from 10 possible variable names, which are first initialized (by reading stdin) and then processed by up to 1-5 random operations, including 3 unary operations per type and 2 binary operations ints. There are 100 possible integer values and 10 possible string values. We generate 1000 labeled examples and 20000 unlabeled code examples.
Results. Table shows the results for all models. The Pretrained+Compossee predictor improves the proportion of correct programs over direct training by 51% and over pretraining by 9.6%. We can also apply the learned denoiser to the outputs of the direct and pretrained predictors at test time, which reduces the improvement 29.6% and 7.6% respectively. Similarly, Backtranslationnoooossss improves upon direct training by 55.2% and over the strong backtranslation baseline by 2.6%. The Composed model without combining with pretraining or backtranslation still achieves a 31.4% increase over the direct model. but requires combining with pretraining or backtranslation to achieve the best per- formance. This suggests that wedict-and--enoirs offers a complementary benefit from using unlabeled output data. Results varying unlabeled and labeled data sizes, where the composed model improves over the baselines all instances, are Appendix[C] Figure an example input with the output of the base and composed predictors. With the denoiser, the base predictor does not have to output all the correct variable types. Here, the denoiser correctly instantiates va 5 and corrects the type of temp.
7

---

Under review as conference paper at ICLR 2021
Compile
Fsce
Crnnect
Direct-ccleed
42.4
38.2
Ruckmmnllation-ecalld
denoiser
23.8
Pretrained denuiser
Buckianddaion denoriser
Pretrained Composed
Backirundation Composed
Table 3: Results of scaled-up baselines and baselines with a denoiser on synthetic pseudocode-tt-code task. Proportion of generated code (%) resulting in a compilation error, execution-timg error, or correct code. Composed models are copied from Table
5.2.2 SPOC
Finally, we evaluate on the challenging SPoC wowudooooe-to-ooooe dataset Kulal et al. (2019), which contains code scraped from codef f r cess... com and pseudocode written by crowdsourced workers. Since we consider the fuol-program translation task instead of line-by-line as in previous works Kulal ett al. (2019); Yasunaga and Liang (2020), we filter out training examples where the code is longer than 1000 tokens after pre-processing. retaining over 95% (11 (11355/1 1925) of the training examples. We use the two given SPoC test sets, TESTP and TESTW. TESTP tests for generalization to unseen problems, while TESTW tests for generalization to pseudocode written by different workers. We report results on the full (unfiltered) test sets.
Denoising objective. We use random syntactic and semantic corruptions of additional 2800 unlabeled code examples from code forcesrr com as in |Yasunaga and Liang 2020). Previous program repair works Yasunaga and Liang (2020) utilize compiler error messages to guide the repair model. We only use code as input, and thus the task relatively difficult. We define pa in two parts. First, we train a binary classifier gy: y + {0,1} which detects if a program has an error (error is label 1), trained using the denoising dataset. For output y' if g. (v') 0 then we define p v/)=8(y)) to be delta distribution on y/. Otherwise, if (y())) =1, then ps y') Pu yy where pa is Transformer. The Transformer Pv is first pretrained using a linewise code repair dataset generated from unlabeled examples, then trained on full-program repair where the input program has one random corrupted line with probability 0.75. Thus, taking 3 (7,2), we have ly( = y if g- (v/) ) 0 and (1'()") ) argmax, (y) y otherwise.
On both test sets, predict-and-denoise (composed) models improve the proportion of correct code over the direct predictor by 3-5%, and applying reddet-aand-eenoise pretrained and backtranslation models improve them by about 1-2% (Table Predict-and-denoiss without pretraining or backtranslation still improved over pretraining. but improving over backtranslation requires the combining the complementary benefits of backtranslation and predict-an ded-oooss Applying the denoiser to the direct and pretrained models during test time did not improve their performance. Saakrrannlation--opppssseee
Results.
matches the top-1 performance of a line-by-line LSTM with attention- based copying Kulal et al. (2019) on TESTP and improve upon it by 11.4% on TESTW despite considering the more difficult full-progran generation task.
5.3 COMPARISONS TO SCALED-UF UP BASELINES
Although the composed model does not optimize the denoiser during joint training with the base predictor, the final composed model consists of roughly double the number of layers the baselines. Thus, we also present baseline results with the same number of layers as the composed model. We note that these scaled-up baselines have roughly double the amount of trainable parameters as the composed model, since the denoiser is fixed when training on labeled data.
3sows the results of scaled-up direct, pretrained, and backtranslation baselines on the synthetic code task. While the scaled direct predictor improves with respect to the unscaled direct predictor, scaling up worsens the pretrained and backtranslation baselines. Intruigingly, the pretraining becomes dramatically less effective with very large model, possibly due to the relatively large size of the model in comparison the unlabeled data (20k examples).
Table 2 gives results for a scaled-up direct predictor for the SPoC dataset. We find that while the compilation error rate decreases, scaling up slightly degrades the correct rate of its output programs.
5.4 COMPARISONS TO BASELINES WITH DENOISER
We also give comparisons to the baselines when their outputs are post- pstccesed by the same denoiser used by the composed model. This results in baselines with the exact same architecture and same
8

---

Under review as conference paper at ICLR 2021
number of trainable parameters. The only difference is that the Composed model trains the base predictor jointly with the fixed denoiser.
Table 3 shows the results of the baselines with denoiser. While the improvement between the composed models and the baselines decreases, predict-and-denee gives gains of 29.6%. 7.6%, and 3.2% above direct+denoiser, pretrained+denoiser and backtranslation--dnniie respectively. We find that using the denoiser does not improve SPOC baseline results in general. These experiments highlight the importance of training jointly with the fixed denoiser.
6 RELATED WORK
Semi-superviser learning. Like semi-supervised learning, predict-and seiiet--nd-deenne menddeooise leverages large amounts of unlabeled data. However, semi-supervised learning works typically use unlabeled input data Tarvainen and Volpola 2017); Miyato et al. 2: Shu et al. (2018); Berthelot et al. (2019). whereas we have "unlabeled" outputs. In classification. unlabeled outputs can help with handling label shift Lipton al. 2018); Azizzadenesheli al. (2019), but otherwise there very little output structure. If both unlabeled inputs and outputs are available, our method is complementary with semi-supervised methods.
Denoising autoencoding. Denoising autoencoders (DAE) are classical building blocks for unsupervised deep representation learning Vincent et al. (2008 2010). Recently, DAFs have been considered on the input side combat adversarial robustness by attempting to clean the adversarial example first using invariances learned from unlabeled data Gu and Rigazio 2015); WWon and Kolter (2020).
We consider DAEs for learning invariances and structure the output space instead of inputs.
Machine translation. Machine translation methods use monolingual data in both the source and tar- get languages to improve their Sennrich et al. (2))(2) (2016b): 6b):nC et al. (206)). Pretraining methods use language modeling on monolingual data to initialize the encoder and decoder Ramachandran et al. Skorokkoooo et al. (2018): Devlin et al. (2019). Back-translation methods generate additional synthetic parallel examples by training the backwards (target to source) problem Seenniihh et al. (2016b). seddict-anddenoote gives complementary gains on top of pretraining and back mack-rranslation.
Semantic parsing and structured prediction. Some recent semantic parsing works have explicitly provided output constraints using abstract syntax trees (AST) and enforcing type constraints Yin and Neubig (201 y Krishnamurthy et al. (2017) 7); Xiao et al. (2016); Dong and Lapata (2016). Krishna murthy et al. (207) note that enforcing type constraints during training not only prevents invalid outputs but also improves generalization, supporting our results. While these methods are useful when the valid- ity structure is known and well-defined, we focus on extracting unknown structure from unlabeled out- puts. Structured prediction spans applications including speech speechZZangg Zhang and Wu 2013), vision Mueller (2013), and medical paggnnosis[Jaannaathe and Yu (2016). Many approaches use graphical models (on top of neural models) for enforcing validity, e.g. HMMs and CRFs in OCR and sequence tagging Kassel (1995); ); [Huang et al. (2015). These approaches typically require carefully engineering the graphical model to integrate with a neural component and do not consider the simplicity benefits composition.
CONCLUSION
Many tasks machine learning are longer classification or regression but require generating outputs with rich structure (images, text, music, proteins, etc.), for which unpaired outputs are very common. We introduce the predict-and-denoise framework, in which we compose predictor with a denoiser trained on unpaired outputs. Open questions include whether we can train a more differentiable way for discrete output spaces and how to choose the best denoising objective for given prediction task.

---

Under review as conference paper at ICLR 2021
REFERENCES
K. Arizzadenesheli, A. Liu, F. Yang, and Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations (ICLR), 2019.
P. Bartlett D. Foster, and M. Telgarsky. Spectrally-normaaizze margin bounds for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
E. Bernhardsson. Analyzing 50k fonts using deep neural networks, 2016.
D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel. Mixmatch: A holistic approach to supe-supervised learning. arXiv, 2019.
Y. Cheng, W. Xu, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. Semi-supervised learning for neural machine translation. In Association for Computational Linguistics (ACL), 2016.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 2019.
L. Dong and M. Lapata. Language to logical form with neural attention. In Association for Computational Linguistics (ACL), 2016.
R. Eldan and O. Shamir. The power of depth for feedforward neural networks. In Conference on Learning Theory (COLT), 2016.
S. Gu and L. Rigazio. Towards deep neural network architectures robust to adversarial examples. In International Conference Learning Representations Workshop (ICLR), 2015.
Guzmán, P. Chen, M. Ott, J. Pino, G. Lample, P. Kochn, V. Chaudhary, and M. Ranzato. Two new evaluation datasets for low-resource machine translation: Nepali-English and sinhala-English. arXiv, 2019.
7. Huang, W. Xu, and K. Yu. Bidirectional LSTM-CRF models for sequence tagging. arXiv, 2015.
A. Jagannatha and H. Yu. Structured prediction models for RNN based sequence labeling in clinical text. In Empirical Methods in Natural Language Processing (EMNLP), 2016.
E. Jang, S. Gu, and B. Poole. Categorical reparameterization with Gumbel-softmax, arXiv preprint arXiv: 1611011611.01144 2017.
R. H. Kassel. A comparison of approaches on-line handwritten character recognition. PhD thesis. Massachusetts Institute of Technology, 1995.
J. Krishnamurthy, P. Dasigi, and M. Gardner. Neural semantic parsing with type constraints for semi-structured tables. In Empirical Methods in Natural Language Processing (EMNLP), 2017.
Kudo and J. Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing, In Empirical Methods in Natural Language Processing (EMNLP),2015
S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang. Spoc: Search-based pseudocode to code. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Z. C. Lipton. Y. Wang, and A. J. Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning (ICMI), 2018.
C. J. Maddison, A. Mnih, and Y. W. Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 16X 2016.
T. Miyato, S. Maeda, S. Ishii, and M. Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
A. Mueller. Methods for Learning Structured Prediction in Semantic Segmentation of Natural Images. PhD thesis, University of Bonn, 2013.
10

---

Under review as conference paper at ICLR 2021
O. Méndez-Lucio, B. Baillif, D. Clevert, D. Rouquié, and J. Wichard. De novo generation of hit-like molecules from gene expression signatures using artificial intelligence. Nature Communications, 11, 2020.
B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv, 2014.
B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ramachandran, P. Liu, and Q Le. Unsupervised pretraining for sequence sequence learning. arXiv, 2018.
O. Ronneberger, Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. arXiv, 2015.
P. Savarese, I. Evron, D. Soudry, and N. Srebro. How do infinite width bounded norm networks look in function space? arXiv, 2019.
A. W. Senior, R. Evans, J. Jumper. J. Kirkpatrick, L. Sifre, Green, C. Qin, A. Zídek, A. W. R. Nelson, A. Bridgland, H. Penedones, S. Petersen, K. Simonyan, S. Crossan, P. Kohli, D. T. Jones, D. Silver, K. Kavukcuoglu, and D. Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577, 2020.
R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL), 2016a.
R. Sennrich, B. Haddow, and A. Birch. Improving neural machine translation models with monolingual data. arXiv, 2016b.
R. Shu, H. H. Bui, H. Narui, and S. Ermon. A DIRT-T approcah to unsupervised domain adaptation. In International Conference on Learning Representations (ICLR),, 2018.
I. Skorokhodov, A. Rykachevskiy, D. Emelyanenko, Slotin, and A. Ponkratov. Semi-supervised neural machine translation with language models. In AMTA 2018 Workshop on Technologies for MT of Low Resource Languages (LoResMT), 2018.
A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pages 1195-1204, 2017.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03766 2017.
P. Vincent, H. Larochelle, Y. Bengio, and P. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), 2008.
Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research (JMLR), 11, 2010.
C. Wei and T. Ma. Data-dependent sample complexity of deep neural networks via Lipschitz augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
C. Wei and T. Ma. Improved sample complexities for deep networks and robust classification via an all-layer margin. In International Conference on Learning Representations (ICLR), 2020.
R. Williams. Simple statistical grodiint- algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256, 1992.
E. Wong and J. Z. Kolter. Learning perturbation sets for robust machine learning. arXiv, 2020.
C. Xiao, M. Dymetman, and C. Gardent. Sequence-based structured prediction for semantic parsing.
In Association for Computational Linguistics (ACL), 2016.
M. Yasunaga and P. Liang. Graph based, self-supervised program repair from diagnostic feedback. In International Conference Machine Learning (ICMI.), 2020.
11

---

Under review as conference paper at ICLR 2021
P. Yin and G. Neubig. A syntactic neural model for ceneral-purpose code generation. In Association for Computational Linguistics (ACL), pages 440 450, 2017.
X. Zhang and J. Wu, Denoising deep neural networks based voice activity detection. arXiv, 2013.
12


Reviews:
[{"review_id": "bUN-Pzesp6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a “predict-and-denoise” model for structured prediction, specifically for tasks where the output has to adhere to some constraints e.g. natural language, code etc. This framework allows leveraging of unlabelled output data to train the denoiser, which consequently allows the base predictor to be of low complexity that can potentially generalize with relatively fewer labelled data. The authors theoretically back their arguments basing their theory on a 2 layer ReLU model. The paper demonstrates the performance of this model on two tasks - font image generation, and pseudocode-to-code translation and shows improvement in performance over previous works. \n\n+ves :\n\n- The paper is very well written and easy to follow. The motivation and the contributions are very clear, and the experimental section is also well detailed and organized. \n- To the best of my knowledge the framework of predict-and-denoise learned in a composed manner and using this framework to leverage unlabelled output data is a novel contribution of the paper. \n- The authors argue that this framework allows reduced complexity of the base predictor, backed theoretically for a 2 layer ReLU network. The authors have provided a detailed proof of their argument in the supplementary material, although I have not completely verified its correctness. \n\nConcerns :\n\n- I believe that the experimental section currently lacks fair comparisons, especially in the task of pseudocode-to-code translation. The authors compare their method with other methods for leveraging unlabelled data such as pre-training and back-translation. The authors show that predict-and-denoise framework can be applied on top of these existing approaches, and yields consistent improvement.\n\n  However, when comparing such combinations such as “pre-training/back-translation + composed” against pre-training/back-translation, the resulting performance is not  compared with an accordingly scaled base pre-training/back-translation model to have comparable number of parameters. With a very different number of parameters in the models being compared, it's hard to say where the performance benefit is coming from. \n\n- While the proposed method is complementary to approaches like pre-training and back translation, it will be helpful to also include comparisons such as “composed vs pre-training”, or “composed vs back-translation”. This will give an interesting comparison among these different ways of leveraging unlabelled output data. Again proper care needs to be taken about a comparable number of parameters. \n\nWhile I find the framework \"predict-and-denoise\" very interesting, I am not entirely convinced with its empirical performance reported in the current form and I have given my score accordingly. If the authors agree with my concerns, and can try to incorporate these changes during rebuttal, I will consider updating my score.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R625utgPjP6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a more data-efficient way to train generative models with constraints on the output; specifically they evaluate on image generation and pseudocode-to-code (SPoC) tasks. They train two separate models, a “predictor” and a “denoiser”, which they then compose: the output from the “predictor” is further processed by the “denoiser”. For the SPoC task they show an improvement of 3-5% over a simple transformer baseline.\n \nThe authors suggest a simple idea to make use of unlabelled data, should it be available. They use it to perturbate the unlabelled data and use the (perturbed example, example) pairs to train a denoising model. They argue that this should theoretically simplify the task of the predictor, and show improvements on several tasks. I believe that this is an interesting idea, and practically useful in the cases where data is sparse.\n \nHowever, the results that they demonstrate do not seem very strong, and I would have liked to see this technique demonstrated on more competitive tasks to better gauge how well it works. The improvement of 3-5% they state seems like a low gain over a simple baseline, that may also be achievable with other techniques.\n \nClearly state your recommendation (accept or reject) with one or two key reasons for this choice.\n \nI do recommend this paper to be accepted, because it clearly presents an interesting idea.\n \nThe recommendation is a “weak accept” though, because the experimental evidence for the technique is not convincing enough to me. I would have expected significant gains on a well understood task, clearly attributable to the technique.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "375bqYlqfTl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for problems where the output has some validity constraints, for e.g. the output must be a valid python program that must compile. These kind of problems arise naturally in settings such as pseudocode to program, and moreover there are many more unlabelled valid programs that are easily available (e.g. on Github) than there are labelled examples - i.e. paired pseudo-code, code examples. In this case, the authors propose the following framework of predict and de-noise: 1) train a de-noiser that learns to map synthetically noised versions of the un-labelled valid examples and 2) compose a predictor on the labelled examples with this de-noiser so that end predictions belong to the space of valid programs. The idea proposed in the paper is simple and intuitive, and the authors show that this approach leads to an improvement of 3-5% on the SPOC pseudo-code to code data-set. The authors also provide some theoretical justification why such a composition is the right thing to do.\n\nOverall I like the idea in the paper, and such an approach has been used in NLP for various problems such as spelling correction etc. Besides the predict and de-noise framework another approach that is used in machine translation is the idea of using back-translation, which in the context of pseudo-code to code would look something like this: 1) train a sequence model A such as transformer on code to pseudo-code, 2) use the trained model A to generate paired training data for the unlabelled code (obtained e.g, from Github) by out-putting pseudo-code for it, and 3) train a final model on original labelled pseudo-code to code data plus the artificially generated pseudo-code to code data. I wonder how this would compare to the proposed method? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work is well written and easy to follow and proposes a novel framework to utilize unlabeled output data. The authors have also given a detailed proof that the denoiser reduces the required complexity of the predictor. However, ultimately the experimental results are somewhat weak and leave doubts as to how effective the approach is. More convincing experimental results such as significant improvements on a well understood task and acknowledging that the approach is mostly useful when combined with pre-training and back translation would improve the work.

Pros
- Well written.
- Technically novel approach to the problem of utilizing unlabeled output data.
- Interesting proof on the reduced complexity requirement for the predictor.

Cons:
- Experimental results are not convincing. Showing significant improvements on a well understood task would be more convincing.
- The approach is only really useful when combined with pre-training or back-translation.
Decision: Reject