Weakly Supervised Label Learning Flows

1 INTRODUCTION . Machine learning has achieved great success in many supervised learning tasks . However , in practice , data labeling is usually human intensive and costly . To address this problem , practitioners are turning to weakly supervised learning ( Zhou , 2018 ) , which trains machine learning models with only noisy labels that are generated by human specified rules or pretrained models for related tasks . Recent research shows that these models trained with weak supervisions can also perform well . Many existing weakly supervised learning methods ( Ratner et al. , 2016 ; 2017 ; Bach et al. , 2019 ; Arachie & Huang , 2021b ) learn a deterministic function that estimates the unknown labels y given input data x and weak signals Q . Since the observed information is incomplete , the predictions based on it can be varied . However , these methods ignore this uncertainty between x and y . In this paper , we develop label learning flows ( LLF ) , a general framework for weakly supervised learning problems . LLF is a flow-based generative model ( Dinh et al. , 2014 ; Rezende & Mohamed , 2015 ; Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) . The main idea behind LLF is that we define the relationship between x and y with a probability distribution p ( y|x ) , which is modeled by a conditional flow . In training , we use the weak signals Q to define a constrained space for y and then optimize the likelihood of all possible y that are within this constrained space . Therefore , this model captures all possible relationships between the input x and output y . Learning LLF can be defined as a constrained optimization problem . We develop a learning method for LLF that trains the conditional flow inversely and avoids estimating the labels . For prediction , we use sample-based method ( Lu & Huang , 2020 ) to estimate the labels . We apply LLF to three weakly supervised learning problems : weakly supervised classification ( Arachie & Huang , 2021b ; Mazzetto et al. , 2021b ) , weakly supervised regression , and unpaired point cloud completion ( Chen et al. , 2019 ; Wu et al. , 2020 ) . These three problems have very different label types and weak signals . Our method outperforms all other state-of-the-art methods on weakly supervised classification and regression , and it can perform comparably to other recent methods on unpaired point cloud completion . These experiments show that LLF is versatile and powerful . 2 BACKGROUND . In this section , we introduce weakly supervised learning and conditional normalizing flows . Weakly Supervised Learning . Given a dataset D = { x1 , ... , xN } , and weak signals Q , weakly supervised learning finds a model that can predict the unknown label yi for each input data xi . Some methods , e.g. , ALL ( Arachie & Huang , 2021b ) , define a set of constrained functions based on Q and x . These functions form a space of possible y . The methods then look for one possible y within this constrained space . In this work , we follow this idea and use constrained functions to restrict the predicted y . Conditional Normalizing Flows . A normalizing flow ( Rezende & Mohamed , 2015 ) is a series of invertible functions f = f1 ◦ f2 ◦ · · · ◦ fK that transform the probability density of output variables y to the density of latent variables z . In conditional flows ( Trippe & Turner , 2018 ) , a flow layer function fi is also parameterized by the input variables x , i.e. , fi = fx , φi , where φi is the parameters of fi . With the change-of-variable formula , the log conditional distribution log p ( y|x ) can be exactly and tractably computed as log p ( y|x ) = log pZ ( fx , φ ( y ) ) + K∑ i=1 log ∣∣∣∣det ( ∂fx , φi∂ri−1 ) ∣∣∣∣ , ( 1 ) where pZ ( z ) is a tractable base distribution , e.g . Gaussian distribution . The ∂fx , φi ∂ri−1 is the Jacobian matrix of fx , φi . The ri = fx , φi ( ri−1 ) , r0 = y , and rK = z . To use normalizing flows , we need to develop flow layers that are invertible and have tractable Jacobian determinant . In this paper , we use affine coupling layer ( Dinh et al. , 2014 ; 2016 ) to form normalizing flows . It splits the input to two parts , and force the first part only relate to the second part , so that the Jacobian is a triangular matrix . For conditional flows , we can define conditional affine coupling layer as ya , yb = split ( y ) , zb = s ( x , ya ) yb + b ( x , ya ) , z = concat ( ya , zb ) , where s and b are two neural networks . The split ( ) and the concat ( ) functions split and concatenate the variables . 3 PROPOSED METHOD . In this section , we introduce the label learning flows ( LLF ) framework for weakly supervised learning . Given Q and x , we define a set of constraints to restrict the predicted label y . These constraints can be inequalities , formatting like f ( x , y , Q ) ≤ b , or equations , formatting like f ( x , y , Q ) = b . For simplicity , we represent this set of constraints as C ( x , y , Q ) . Let Ω be the constrained space of all possible y defined by C ( x , y , Q ) . Previous methods only look for one possible y within Ω . In contrast , LLF optimizes the conditional log-likelihood of all possible y within Ω , resulting in the following objective max φ Epdata ( x ) Ey∼U ( Ω ) [ log p ( y|x , φ ) ] , ( 2 ) where U ( Ω ) is a uniform distribution within Ω , and p ( y|x ) is a continuous density model of y . Let ŷ be the true label . We assume that each data point xi only has one unique label ŷi , so that pdata ( x , ŷ ) = pdata ( x ) . Let q ( ŷ|x ) be a certain model of ŷ . Traditional supervised learning learns a q ( ŷ|x ) that maximizes the cross entropy Epdata ( x , ŷ ) [ log q ( ŷ|x , φ ) ] . Following theorem indicates that maximizing log p ( y|x ) can be interpreted as maximizing a lower bound of log q ( ŷ|x ) . Theorem 1 Let Ω∗ ⊆ Ω is a tight enough space satisfying that ŷ ∈ Ω∗ and for two different ŷi and ŷj , the Ω∗i and Ω ∗ j are non-overlapped . The volume of Ω ∗ is bounded such that 1|Ω∗| ≤ M . The relationship between p ( y|x ) and q ( ŷ|x ) can be defined as : q ( ŷ|x ) = ∫ y∈Ω∗ p ( y|x ) dy . Then maximizing log p ( y|x ) can be interpreted as maximizing the lower bound of log q ( ŷ|x ) . That is , Epdata ( x ) Ey∼U ( Ω∗ ) [ log p ( y|x , φ ) ] ≤MEpdata ( x , ŷ ) [ log q ( ŷ|x , φ ) ] ( 3 ) The complete proof is in appendix . Theorem 1 indicates that , when Ω is well-defined , learning LLF is analogous to dequantization ( Theis et al. , 2015 ; Ho et al. , 2019 ) . That is , the method optimizes the likelihood of dequantized true labels . In practice , the real constrained space , i.e. , Ω , may be loose and does not fulfill the conditions in Theorem 1 , which will result in inevitable errors that come from the weakly supervised setting . Moreover , for some regression problems , the ideal Ω∗ only contains a single point : the ground truth label , i.e. , Ω∗ = { ŷ } . 3.1 LEARNING AND PREDICTION . Since y is unobserved , directly optimizing the conditional likelihood , i.e. , Equation 2 , is impossible . Using the invertibility of normalizing flows , we can rewrite log p ( y|x ) as log p ( y|x ) = log pZ ( fx , φ ( y ) ) + K∑ i=1 log ∣∣∣∣det ( ∂fx , φi∂ri−1 ) ∣∣∣∣ = log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ , ( 4 ) where gx , φi = f −1 x , φi is the inverse flow . With the inverse flow , Equation 2 can be interpreted as a constrained optimization problem max φ Epdata ( x ) EpZ ( z ) [ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ ] , ( 5 ) s.t . C ( x , gx , φ ( z ) , Q ) . Note that in Equation 5 , the original constraint for z is gx , φ ( z ) ∈ Ω , and this constraint can be replaced with C ( x , gx , φ ( z ) , Q ) . For efficient training , this constrained optimization problem can be approximated with the penalty method , resulting in the objective max φ Epdata ( x ) EpZ ( z ) [ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣+ λCr ( x , gx , φ ( z ) , Q ) ] , ( 6 ) where λ is the penalty coefficient , and Cr ( ) means we reformulate the constraints to be penalty losses . For example , an inequality constraint will be redefined as a hinge loss . In practice , the expectation with respect to pZ ( z ) can be approximated with Monte Carlo estimate with L samples . Since we only need to obtain stochastic gradients , we follow previous works ( Kingma & Welling , 2013 ) and set L = 1 . Given a trained model and a data point xi , prediction requires outputting a label yi for xi . We follow ( Lu & Huang , 2020 ) and use a sample average , i.e. , yi = ∑L j=1 gxi , φ ( zj ) as the prediction . In our experiments , we found that L = 10 is enough for generating high-quality labels . 4 CASE STUDY . In this section , we illustrate three scenarios of using LLF to address weakly supervised learning problems . 4.1 WEAKLY SUPERVISED CLASSIFICATION . We follow previous works ( Arachie & Huang , 2021b ; Mazzetto et al. , 2021b ) and consider binary classification . For each example , the label y is a two-dimensional vector within a one-simplex . That is , the y ∈ Y = { y ∈ [ 0 , 1 ] 2 : ∑ i y [ i ] = 1 } , where y [ i ] is the ith dimension of y . Each ground truth label ŷ ∈ { 0 , 1 } 2 is a two-dimensional one-hot vector . We have M weak labelers , which will generate M weak signals for each data point xi , i.e. , qi = [ qi,1 , ... , qi , M ] . Each weak signal qi , m ∈ Q = { q ∈ [ 0 , 1 ] 2 : ∑ i q [ i ] = 1 } is a soft labeling of the data . In practice , if a weak labeler m fails to label a data point xi , the qi , m can be null , i.e. , qi , m = ∅ ( Arachie & Huang , 2021a ) . Following Arachie & Huang ( 2021b ) , we assume we have access to expected error rate bounds of these weak signals b = [ b1 , .. , bM ] . Therefore , the weak signals imply constraints N∑ i=1 , qi , m 6=∅ ( 1− yi ) qi , m + yi ( 1− qi , m ) ≤ Nmbm ∀m ∈ { 1 , ... , M } , ( 7 ) where Nm is the number of data points that are labeled by weak labeler m. This problem can be solved with LLF , i.e. , Equation 5 , by defining C ( x , gx , φ ( z , Q ) to be a combination of weak signal constraints , i.e. , Equation 7 , and simplex constraints , i.e. , y ∈ Y . The objective function of LLF for weakly supervised classification is max φ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ +λ1 [ gx , φ ( z ) ] 2 + + λ2 [ 1− gx , φ ( z ) ] 2 + + λ3 ( ∑ i gx , φ ( z ) [ i ] − 1 ) 2 +λ4 M∑ m=1 N∑ i=0 qi , m 6=∅ ( 1− gx , φ ( z ) i ) qi , m + gx , φ ( z ) i ( 1− qi , m ) −Nmbm 2 + , ( 8 ) where the second row describes the simplex constraints , and the last row is the weak signal constraints . The [ . ] + is a hinge function that returns its input if positive and zero otherwise . We omit the expectation terms for simplicity .

Summary: This work focuses on the problem of learning predictive models where no ground truth labels are provided.  Instead a set of weak constraints over instances that restrict the space of possible target values are provided.  The proposed method, LLF, is based on the core idea that instead of determining a single true weak label for each instance, a model can be learned by considering the expectation over all possible labels in the feasible set defined by the given constraints.  The authors utilize a conditional normalizing flow model as a generative process for labels and constrain it's output by imposing regularization terms on the training objective that represent different constraints imposed by the weak supervision.  The authors show how their framing of weakly supervised learning can be applied go specific settings in classification, regression, and unpaired point cloud completion.  In their evaluation, the authors compare LLF to a number of weakly supervised baselines and show superior performance on classification and regression tasks, while largely being outperformed by one method in point cloud completion.