Simplifying Models with Unlabeled Output Data

1 INTRODUCTION . We study problems whose outputs have validity constraints . For example , in pseudocode-to-code translation , the output code must compile . Other examples include natural language translation and molecule generation , where outputs should be grammatically correct or chemically valid , respectively . State-of-the-art models typically learn the input-output mapping from expensively-obtained labeled data Kulal et al . ( 2019 ) ; Vaswani et al . ( 2017 ) ; Méndez-Lucio et al . ( 2020 ) ; Senior et al . ( 2020 ) , which may not contain enough examples to learn a complex validity structure on high-dimensional output spaces . However , there are often lots of “ unlabeled ” outputs—outputs without a corresponding input ( e.g. , GitHub has over 40 million public code repositories ) . How do we leverage these with a much smaller amount of labeled input-output pairs to improve accuracy and validity ? In this paper , we present predict-and-denoise , a framework in which we compose a base predictor , which maps an input to a possibly invalid output , with a denoiser , which maps the possibly invalid output to a valid output . We first train the denoiser on synthetic perturbations of unlabeled outputs . Second , we train the base predictor composed with the fixed denoiser on the labeled data ( Figure 1 left ) . By factorizing into two modules , base predictor and denoiser , the framework allows the base predictor to be simpler by offloading the complexity of modeling the output validity structure to the denoiser , which has the benefit of being trained on much more data . We aim to lay down a principled framework for using unlabeled outputs with theoretical justification for improving sample efficiency by reducing the complexity of the learned base predictor . Figure 1 ( middle , right ) shows a pictorial example of a staircase function where valid outputs are integers and requires a complex spline to represent . When composed with a denoiser ( which rounds to the nearest integer ) , a simple linear base predictor can represent the staircase function . We theoretically show that our framework reduces the complexity of a 2-layer ReLU network needed to represent a family of functions on a discrete valid output set in high-dimensions . This complexity gap can be arbitrarily large depending on the stability of the target function being learned . We expect such a lower complexity function to be learnable with fewer samples , improving generalization . Empirically , we show on image generation and two pseudocode-to-code datasets ( synthetic and SPOC Kulal et al . ( 2019 ) ) that predict-and-denoise improves test performance across continuous and discrete output data modalities . In image generation , our framework improves the clarity and styling of font images by learning a low-complexity base predictor to generate an abstract image while the denoiser sharpens the image . For pseudocode-to-code , we consider the more difficult full-program translation task rather than line-by-line translation ( with compiler side information ) studied by previous work Kulal et al . ( 2019 ) ; Yasunaga and Liang ( 2020 ) . We first study a synthetic pseudocode-to-code dataset where the denoiser simplifies the base predictor by helping with global type inference . On SPOC , a recent pseudocode-to-code dataset on programming competition problems , we improve the proportion of correct programs by 3-5 % points over a baseline Transformer . 2 SETUP . We consider prediction problems from an input spaceX ( e.g. , pseudocode ) to an output space Y ( e.g. , code ) where there is an unknown subset of valid outputs V⊆Y ( e.g. , code that compiles ) , where the true output is always valid ( in V ) . We have a labeled dataset ( x1 , y1 ) , ... , ( xn , yn ) where xi ∈X and yi∈V and access to many unlabeled outputs ( ỹ1 , ... , ỹm ) from V . We do not assume access to any black box function for testing validity ( whether y∈V or not ) , allowing for general problems ( e.g . language generation ) where output validity is imprecisely characterized . A predictor f : X →Y from a chosen hypothesis class H maps from inputs to the ambient output space . Our goal is to improve the predictor by leveraging information about the valid space V from the unlabeled examples { ỹi } mi=1 . We leverage a denoiser Π : Y→V , which projects a possibly invalid output inY and to the valid set V . We can use unlabeled outputs to learn an approximate denoiser . Base , composed , and direct predictors . Let ‖·‖ be a norm onH . Let Π◦fbase be a composed predictor that is supposed to represent the target function f ? ( that is , Π◦fbase =f ? onX ) . In the context of a composed predictor , we call fbase the base predictor . We compare against fdirect∈argminf∈H { ‖f‖ : f ( x ) =f ? ( x ) , x∈X } , a minimum norm direct predictor which represents f ? . 3 DENOISERS CAN REDUCE MODEL COMPLEXITY . In this section , we study direct and composed predictors from an approximation standpoint and use complexity measures on predictors as surrogates for sample complexity . We aim to represent a target function f ? : X →V . We assume access to a denoiser Π : Y→V which projects to the nearest valid output for an appropriate metric on the output space ( breaking ties arbitrarily ) . In Section 3.1 , we give a simple example for when composing with a denoiser ( Π ◦ fbase ) can drastically reduce the complexity of the learned predictor . Since fbase becomes easier to approximate , we may expect better generalization Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) ; Wei and Ma ( 2020 ; 2019 ) . In Section 3.2 , we theoretically show for two-layer ReLU networks that the complexity required to directly represent f ? can be arbitrarily larger than representing with a composed predictor depending on the stability of f ? . 3.1 MOTIVATING EXAMPLE . Figure 1 shows a staircase function f ? that requires a complex direct predictor fdirect but the minimum norm base predictor f∗base has low complexity . For 0 < δ < 1 , let the input space X = ] Ni=1 [ i− ( 1−δ ) /2 , i+ ( 1−δ ) /2 ] be a union ofN disjoint intervals and the valid outputs V =Z be the integers , a subset of the ambient output space Y =R . The staircase function is f ? ( x ) = bxe defined onX , which rounds a linear function onto the integers . Following Savarese et al . ( 2019 ) , we define the norm of a univariate function f : R→R as ‖f‖= 1 2 max ( ∫ ∞ −∞ |f ′′ ( x ) |2dx , |f ′ ( −∞ ) +f ′ ( +∞ ) | ) . ( 1 ) This norm measures the ( lack of ) stability of f . Complex functions will have a higher norm . Consider representing f ? with linear splines , a family of piecewise linear functions . In linear splines , the norm in Equation ( 1 ) becomes roughly the sum of absolute changes in slope between piecewise segments . If we represent f ? directly with a linear spline fdirect , the norm of fdirect has to be large due to the large number of slope changes : ‖fdirect‖= ( N−1 ) /δ ( Figure 1 left ) . Suppose we have access to a denoiser Π ( y ) =bye , which projects onto V=Z . Then a linear function f∗base composed with Π can represent the staircase onX , reducing the norm to 1 ( Figure 1 right ) . By not requiring f∗base to represent the local complexity and discreteness in f ? , the base predictor f∗base better captures the underlying globally linear structure of f ? . 3.2 ANALYSIS FOR 2-LAYER RELU NETWORKS . We extend to more general hypothesis classes and high dimensional outputs . Our setting is motivated by the task of generating images of font characters from attributes , which we study empirically in Section 5.1 . In font image generation , there is a discrete set of valid font images in the continuous ambient output space . Formally , we take the valid set V= { y∗1 , ... , y∗N } to be a discrete set overN output values in Rk and f ? is a piecewise constant function defined onN disjoint intervalsX = ] Ni=1 [ xli , xui ] ( in ascending order ) , where there is a δ > 0 gap between each interval and the next . The target function f ? is defined such that if x∈ [ xli , xui ] , then f ? ( x ) =y∗i . We study 2-layer ReLU networks , often studied as a first step towards understanding the expressivity of neural networks Neyshabur et al . ( 2014 ) ; Savarese et al . ( 2019 ) ; Eldan and Shamir ( 2016 ) . Following Savarese et al . ( 2019 ) , we define fθ∈H as fθ ( x ) = h∑ l=1 w ( 2 ) l [ 〈w ( 1 ) l , x〉+b ( 1 ) l ] + +b ( 2 ) l on x∈Rd , where we will take d= 1 throughout . Here , [ x ] + = max ( x,0 ) is the element-wise ReLU nonlinearity . The parameters θ contain the hidden unit size h∈N and all weights and biases . We let W ( 1 ) ∈Rh×d denote the matrix with w ( 1 ) l ∈Rd as rows and let b ( 1 ) , b ( 2 ) , w ( 2 ) ∈Rh be vectors with b ( 1 ) l , b ( 2 ) l , w ( 2 ) l ∈R as elements respectively . We let Θ denote this parameter space . Measure of complexity . Following Savarese et al . ( 2019 ) , the complexity of a network is associated with the squared Euclidean norm of the weights C ( θ ) = 1 2 ( ‖w ( 2 ) ‖22+‖W ( 1 ) ‖2F ) . The norm of f ∈H is the minimum norm required to represent f : ‖f‖= inf θ̂∈Θ C ( θ̂ ) s.t . fθ̂=f . ( 2 ) Savarese et al . ( 2019 ) showed that this norm is equivalent to Equation 1 for univariate networks . Since these complexity measures typically appear in generalization bounds Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) , we expect to improve generalization error by reducing these complexity measures . Minimum complexity reduces with a denoiser . Given Π ( y ) ∈ argminy∗∈V ‖y∗− y‖2 which is projection onto V ( breaking ties arbitrarily ) , we want to compare the norms of fdirect that represents f ? directly and the minimum norm base predictor that represents f ? : f∗base =argmin f∈H { ‖f‖ : Π◦f ( x ) =f ? ( x ) , x∈X } . ( 3 ) Note that ‖f∗base‖≤‖fdirect‖ since fdirect is a feasible solution . Thus composing can not increase the norm . Adjacent intervals measure stability . Our result depends crucially on the number of non-adjacent pairs of intervals in f ? . Suppose the output dimension is k = 1 . We define a pair of interval indices ( i , i+1 ) as adjacent if there is no valid output value y ∈V such that either y∗i < y < y∗i+1 or y∗i+1 < y < y ∗ i hold . The number of non-adjacent interval pairs characterizes the instability of f ? . Let |J | be the number of non-adjacent pairs and |I| be the number of adjacent pairs , where |I|+|J |=N−1 . Our bound also depends onL=mini|y∗i −y∗i+1| andU=maxi|y∗i −y∗i+1| , the min and max separation between valid points . For higher output dimensions ( k > 1 ) , let y∗i , j be the j-th output coordinate of the i-th valid point and let |Jj | , |Ij | , Lj , Uj be the analogous quantities for each output coordinate j∈ [ k ] . Theorem 1 . Let the valid output space V⊂Rk be a set overN multivariate output values { y∗1 , ... , y∗N } in V . Let f ? : R→Rk be a piecewise constant function defined onX = ] Ni=1 [ xli , xui ] where f ? ( x ) =y∗i if x∈ [ xli , xui ] . Let ∆x be the length of the smallest interval inX . Then ‖fdirect‖ ‖f∗base‖ =Ω NmaxjLj∑k j=1Uj ( |Jj |+δ |Ij |∆x ) ( 4 ) See Appendix A for a proof . If |Jj | are sublinear in N and valid points are evenly spaced , then the gap is Ω ( 1/δ ) which can be arbitrarily large for a fixed output dimension as δ→ 0 and N→∞ . If any |Jj | is linear in N ( many non-adjacent intervals ) , then there is only a constant factor gap in the worst case . Overall , if f ? is stable with respect to its discrete output space , we can learn a simpler base predictor that still represents f ? when composed with the denoiser . Note that in practice , we need to regularize the base predictor to find this low complexity solution .

Summary: This paper proposes a framework for problems where the output has some validity constraints, for e.g. the output must be a valid python program that must compile. These kind of problems arise naturally in settings such as pseudocode to program, and moreover there are many more unlabelled valid programs that are easily available (e.g. on Github) than there are labelled examples - i.e. paired pseudo-code, code examples. In this case, the authors propose the following framework of predict and de-noise: 1) train a de-noiser that learns to map synthetically noised versions of the un-labelled valid examples and 2) compose a predictor on the labelled examples with this de-noiser so that end predictions belong to the space of valid programs. The idea proposed in the paper is simple and intuitive, and the authors show that this approach leads to an improvement of 3-5% on the SPOC pseudo-code to code data-set. The authors also provide some theoretical justification why such a composition is the right thing to do.