Simplifying Models with Unlabeled Output Data

1 INTRODUCTION . We study problems whose outputs have validity constraints . For example , in pseudocode-to-code translation , the output code must compile . Other examples include natural language translation and molecule generation , where outputs should be grammatically correct or chemically valid , respectively . State-of-the-art models typically learn the input-output mapping from expensively-obtained labeled data Kulal et al . ( 2019 ) ; Vaswani et al . ( 2017 ) ; Méndez-Lucio et al . ( 2020 ) ; Senior et al . ( 2020 ) , which may not contain enough examples to learn a complex validity structure on high-dimensional output spaces . However , there are often lots of “ unlabeled ” outputs—outputs without a corresponding input ( e.g. , GitHub has over 40 million public code repositories ) . How do we leverage these with a much smaller amount of labeled input-output pairs to improve accuracy and validity ? In this paper , we present predict-and-denoise , a framework in which we compose a base predictor , which maps an input to a possibly invalid output , with a denoiser , which maps the possibly invalid output to a valid output . We first train the denoiser on synthetic perturbations of unlabeled outputs . Second , we train the base predictor composed with the fixed denoiser on the labeled data ( Figure 1 left ) . By factorizing into two modules , base predictor and denoiser , the framework allows the base predictor to be simpler by offloading the complexity of modeling the output validity structure to the denoiser , which has the benefit of being trained on much more data . We aim to lay down a principled framework for using unlabeled outputs with theoretical justification for improving sample efficiency by reducing the complexity of the learned base predictor . Figure 1 ( middle , right ) shows a pictorial example of a staircase function where valid outputs are integers and requires a complex spline to represent . When composed with a denoiser ( which rounds to the nearest integer ) , a simple linear base predictor can represent the staircase function . We theoretically show that our framework reduces the complexity of a 2-layer ReLU network needed to represent a family of functions on a discrete valid output set in high-dimensions . This complexity gap can be arbitrarily large depending on the stability of the target function being learned . We expect such a lower complexity function to be learnable with fewer samples , improving generalization . Empirically , we show on image generation and two pseudocode-to-code datasets ( synthetic and SPOC Kulal et al . ( 2019 ) ) that predict-and-denoise improves test performance across continuous and discrete output data modalities . In image generation , our framework improves the clarity and styling of font images by learning a low-complexity base predictor to generate an abstract image while the denoiser sharpens the image . For pseudocode-to-code , we consider the more difficult full-program translation task rather than line-by-line translation ( with compiler side information ) studied by previous work Kulal et al . ( 2019 ) ; Yasunaga and Liang ( 2020 ) . We first study a synthetic pseudocode-to-code dataset where the denoiser simplifies the base predictor by helping with global type inference . On SPOC , a recent pseudocode-to-code dataset on programming competition problems , we improve the proportion of correct programs by 3-5 % points over a baseline Transformer . 2 SETUP . We consider prediction problems from an input spaceX ( e.g. , pseudocode ) to an output space Y ( e.g. , code ) where there is an unknown subset of valid outputs V⊆Y ( e.g. , code that compiles ) , where the true output is always valid ( in V ) . We have a labeled dataset ( x1 , y1 ) , ... , ( xn , yn ) where xi ∈X and yi∈V and access to many unlabeled outputs ( ỹ1 , ... , ỹm ) from V . We do not assume access to any black box function for testing validity ( whether y∈V or not ) , allowing for general problems ( e.g . language generation ) where output validity is imprecisely characterized . A predictor f : X →Y from a chosen hypothesis class H maps from inputs to the ambient output space . Our goal is to improve the predictor by leveraging information about the valid space V from the unlabeled examples { ỹi } mi=1 . We leverage a denoiser Π : Y→V , which projects a possibly invalid output inY and to the valid set V . We can use unlabeled outputs to learn an approximate denoiser . Base , composed , and direct predictors . Let ‖·‖ be a norm onH . Let Π◦fbase be a composed predictor that is supposed to represent the target function f ? ( that is , Π◦fbase =f ? onX ) . In the context of a composed predictor , we call fbase the base predictor . We compare against fdirect∈argminf∈H { ‖f‖ : f ( x ) =f ? ( x ) , x∈X } , a minimum norm direct predictor which represents f ? . 3 DENOISERS CAN REDUCE MODEL COMPLEXITY . In this section , we study direct and composed predictors from an approximation standpoint and use complexity measures on predictors as surrogates for sample complexity . We aim to represent a target function f ? : X →V . We assume access to a denoiser Π : Y→V which projects to the nearest valid output for an appropriate metric on the output space ( breaking ties arbitrarily ) . In Section 3.1 , we give a simple example for when composing with a denoiser ( Π ◦ fbase ) can drastically reduce the complexity of the learned predictor . Since fbase becomes easier to approximate , we may expect better generalization Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) ; Wei and Ma ( 2020 ; 2019 ) . In Section 3.2 , we theoretically show for two-layer ReLU networks that the complexity required to directly represent f ? can be arbitrarily larger than representing with a composed predictor depending on the stability of f ? . 3.1 MOTIVATING EXAMPLE . Figure 1 shows a staircase function f ? that requires a complex direct predictor fdirect but the minimum norm base predictor f∗base has low complexity . For 0 < δ < 1 , let the input space X = ] Ni=1 [ i− ( 1−δ ) /2 , i+ ( 1−δ ) /2 ] be a union ofN disjoint intervals and the valid outputs V =Z be the integers , a subset of the ambient output space Y =R . The staircase function is f ? ( x ) = bxe defined onX , which rounds a linear function onto the integers . Following Savarese et al . ( 2019 ) , we define the norm of a univariate function f : R→R as ‖f‖= 1 2 max ( ∫ ∞ −∞ |f ′′ ( x ) |2dx , |f ′ ( −∞ ) +f ′ ( +∞ ) | ) . ( 1 ) This norm measures the ( lack of ) stability of f . Complex functions will have a higher norm . Consider representing f ? with linear splines , a family of piecewise linear functions . In linear splines , the norm in Equation ( 1 ) becomes roughly the sum of absolute changes in slope between piecewise segments . If we represent f ? directly with a linear spline fdirect , the norm of fdirect has to be large due to the large number of slope changes : ‖fdirect‖= ( N−1 ) /δ ( Figure 1 left ) . Suppose we have access to a denoiser Π ( y ) =bye , which projects onto V=Z . Then a linear function f∗base composed with Π can represent the staircase onX , reducing the norm to 1 ( Figure 1 right ) . By not requiring f∗base to represent the local complexity and discreteness in f ? , the base predictor f∗base better captures the underlying globally linear structure of f ? . 3.2 ANALYSIS FOR 2-LAYER RELU NETWORKS . We extend to more general hypothesis classes and high dimensional outputs . Our setting is motivated by the task of generating images of font characters from attributes , which we study empirically in Section 5.1 . In font image generation , there is a discrete set of valid font images in the continuous ambient output space . Formally , we take the valid set V= { y∗1 , ... , y∗N } to be a discrete set overN output values in Rk and f ? is a piecewise constant function defined onN disjoint intervalsX = ] Ni=1 [ xli , xui ] ( in ascending order ) , where there is a δ > 0 gap between each interval and the next . The target function f ? is defined such that if x∈ [ xli , xui ] , then f ? ( x ) =y∗i . We study 2-layer ReLU networks , often studied as a first step towards understanding the expressivity of neural networks Neyshabur et al . ( 2014 ) ; Savarese et al . ( 2019 ) ; Eldan and Shamir ( 2016 ) . Following Savarese et al . ( 2019 ) , we define fθ∈H as fθ ( x ) = h∑ l=1 w ( 2 ) l [ 〈w ( 1 ) l , x〉+b ( 1 ) l ] + +b ( 2 ) l on x∈Rd , where we will take d= 1 throughout . Here , [ x ] + = max ( x,0 ) is the element-wise ReLU nonlinearity . The parameters θ contain the hidden unit size h∈N and all weights and biases . We let W ( 1 ) ∈Rh×d denote the matrix with w ( 1 ) l ∈Rd as rows and let b ( 1 ) , b ( 2 ) , w ( 2 ) ∈Rh be vectors with b ( 1 ) l , b ( 2 ) l , w ( 2 ) l ∈R as elements respectively . We let Θ denote this parameter space . Measure of complexity . Following Savarese et al . ( 2019 ) , the complexity of a network is associated with the squared Euclidean norm of the weights C ( θ ) = 1 2 ( ‖w ( 2 ) ‖22+‖W ( 1 ) ‖2F ) . The norm of f ∈H is the minimum norm required to represent f : ‖f‖= inf θ̂∈Θ C ( θ̂ ) s.t . fθ̂=f . ( 2 ) Savarese et al . ( 2019 ) showed that this norm is equivalent to Equation 1 for univariate networks . Since these complexity measures typically appear in generalization bounds Bartlett et al . ( 2017 ) ; Neyshabur et al . ( 2017 ) , we expect to improve generalization error by reducing these complexity measures . Minimum complexity reduces with a denoiser . Given Π ( y ) ∈ argminy∗∈V ‖y∗− y‖2 which is projection onto V ( breaking ties arbitrarily ) , we want to compare the norms of fdirect that represents f ? directly and the minimum norm base predictor that represents f ? : f∗base =argmin f∈H { ‖f‖ : Π◦f ( x ) =f ? ( x ) , x∈X } . ( 3 ) Note that ‖f∗base‖≤‖fdirect‖ since fdirect is a feasible solution . Thus composing can not increase the norm . Adjacent intervals measure stability . Our result depends crucially on the number of non-adjacent pairs of intervals in f ? . Suppose the output dimension is k = 1 . We define a pair of interval indices ( i , i+1 ) as adjacent if there is no valid output value y ∈V such that either y∗i < y < y∗i+1 or y∗i+1 < y < y ∗ i hold . The number of non-adjacent interval pairs characterizes the instability of f ? . Let |J | be the number of non-adjacent pairs and |I| be the number of adjacent pairs , where |I|+|J |=N−1 . Our bound also depends onL=mini|y∗i −y∗i+1| andU=maxi|y∗i −y∗i+1| , the min and max separation between valid points . For higher output dimensions ( k > 1 ) , let y∗i , j be the j-th output coordinate of the i-th valid point and let |Jj | , |Ij | , Lj , Uj be the analogous quantities for each output coordinate j∈ [ k ] . Theorem 1 . Let the valid output space V⊂Rk be a set overN multivariate output values { y∗1 , ... , y∗N } in V . Let f ? : R→Rk be a piecewise constant function defined onX = ] Ni=1 [ xli , xui ] where f ? ( x ) =y∗i if x∈ [ xli , xui ] . Let ∆x be the length of the smallest interval inX . Then ‖fdirect‖ ‖f∗base‖ =Ω NmaxjLj∑k j=1Uj ( |Jj |+δ |Ij |∆x ) ( 4 ) See Appendix A for a proof . If |Jj | are sublinear in N and valid points are evenly spaced , then the gap is Ω ( 1/δ ) which can be arbitrarily large for a fixed output dimension as δ→ 0 and N→∞ . If any |Jj | is linear in N ( many non-adjacent intervals ) , then there is only a constant factor gap in the worst case . Overall , if f ? is stable with respect to its discrete output space , we can learn a simpler base predictor that still represents f ? when composed with the denoiser . Note that in practice , we need to regularize the base predictor to find this low complexity solution .

Summary: The authors propose a more data-efficient way to train generative models with constraints on the output; specifically they evaluate on image generation and pseudocode-to-code (SPoC) tasks. They train two separate models, a “predictor” and a “denoiser”, which they then compose: the output from the “predictor” is further processed by the “denoiser”. For the SPoC task they show an improvement of 3-5% over a simple transformer baseline.