Weakly Supervised Label Learning Flows

1 INTRODUCTION . Machine learning has achieved great success in many supervised learning tasks . However , in practice , data labeling is usually human intensive and costly . To address this problem , practitioners are turning to weakly supervised learning ( Zhou , 2018 ) , which trains machine learning models with only noisy labels that are generated by human specified rules or pretrained models for related tasks . Recent research shows that these models trained with weak supervisions can also perform well . Many existing weakly supervised learning methods ( Ratner et al. , 2016 ; 2017 ; Bach et al. , 2019 ; Arachie & Huang , 2021b ) learn a deterministic function that estimates the unknown labels y given input data x and weak signals Q . Since the observed information is incomplete , the predictions based on it can be varied . However , these methods ignore this uncertainty between x and y . In this paper , we develop label learning flows ( LLF ) , a general framework for weakly supervised learning problems . LLF is a flow-based generative model ( Dinh et al. , 2014 ; Rezende & Mohamed , 2015 ; Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) . The main idea behind LLF is that we define the relationship between x and y with a probability distribution p ( y|x ) , which is modeled by a conditional flow . In training , we use the weak signals Q to define a constrained space for y and then optimize the likelihood of all possible y that are within this constrained space . Therefore , this model captures all possible relationships between the input x and output y . Learning LLF can be defined as a constrained optimization problem . We develop a learning method for LLF that trains the conditional flow inversely and avoids estimating the labels . For prediction , we use sample-based method ( Lu & Huang , 2020 ) to estimate the labels . We apply LLF to three weakly supervised learning problems : weakly supervised classification ( Arachie & Huang , 2021b ; Mazzetto et al. , 2021b ) , weakly supervised regression , and unpaired point cloud completion ( Chen et al. , 2019 ; Wu et al. , 2020 ) . These three problems have very different label types and weak signals . Our method outperforms all other state-of-the-art methods on weakly supervised classification and regression , and it can perform comparably to other recent methods on unpaired point cloud completion . These experiments show that LLF is versatile and powerful . 2 BACKGROUND . In this section , we introduce weakly supervised learning and conditional normalizing flows . Weakly Supervised Learning . Given a dataset D = { x1 , ... , xN } , and weak signals Q , weakly supervised learning finds a model that can predict the unknown label yi for each input data xi . Some methods , e.g. , ALL ( Arachie & Huang , 2021b ) , define a set of constrained functions based on Q and x . These functions form a space of possible y . The methods then look for one possible y within this constrained space . In this work , we follow this idea and use constrained functions to restrict the predicted y . Conditional Normalizing Flows . A normalizing flow ( Rezende & Mohamed , 2015 ) is a series of invertible functions f = f1 ◦ f2 ◦ · · · ◦ fK that transform the probability density of output variables y to the density of latent variables z . In conditional flows ( Trippe & Turner , 2018 ) , a flow layer function fi is also parameterized by the input variables x , i.e. , fi = fx , φi , where φi is the parameters of fi . With the change-of-variable formula , the log conditional distribution log p ( y|x ) can be exactly and tractably computed as log p ( y|x ) = log pZ ( fx , φ ( y ) ) + K∑ i=1 log ∣∣∣∣det ( ∂fx , φi∂ri−1 ) ∣∣∣∣ , ( 1 ) where pZ ( z ) is a tractable base distribution , e.g . Gaussian distribution . The ∂fx , φi ∂ri−1 is the Jacobian matrix of fx , φi . The ri = fx , φi ( ri−1 ) , r0 = y , and rK = z . To use normalizing flows , we need to develop flow layers that are invertible and have tractable Jacobian determinant . In this paper , we use affine coupling layer ( Dinh et al. , 2014 ; 2016 ) to form normalizing flows . It splits the input to two parts , and force the first part only relate to the second part , so that the Jacobian is a triangular matrix . For conditional flows , we can define conditional affine coupling layer as ya , yb = split ( y ) , zb = s ( x , ya ) yb + b ( x , ya ) , z = concat ( ya , zb ) , where s and b are two neural networks . The split ( ) and the concat ( ) functions split and concatenate the variables . 3 PROPOSED METHOD . In this section , we introduce the label learning flows ( LLF ) framework for weakly supervised learning . Given Q and x , we define a set of constraints to restrict the predicted label y . These constraints can be inequalities , formatting like f ( x , y , Q ) ≤ b , or equations , formatting like f ( x , y , Q ) = b . For simplicity , we represent this set of constraints as C ( x , y , Q ) . Let Ω be the constrained space of all possible y defined by C ( x , y , Q ) . Previous methods only look for one possible y within Ω . In contrast , LLF optimizes the conditional log-likelihood of all possible y within Ω , resulting in the following objective max φ Epdata ( x ) Ey∼U ( Ω ) [ log p ( y|x , φ ) ] , ( 2 ) where U ( Ω ) is a uniform distribution within Ω , and p ( y|x ) is a continuous density model of y . Let ŷ be the true label . We assume that each data point xi only has one unique label ŷi , so that pdata ( x , ŷ ) = pdata ( x ) . Let q ( ŷ|x ) be a certain model of ŷ . Traditional supervised learning learns a q ( ŷ|x ) that maximizes the cross entropy Epdata ( x , ŷ ) [ log q ( ŷ|x , φ ) ] . Following theorem indicates that maximizing log p ( y|x ) can be interpreted as maximizing a lower bound of log q ( ŷ|x ) . Theorem 1 Let Ω∗ ⊆ Ω is a tight enough space satisfying that ŷ ∈ Ω∗ and for two different ŷi and ŷj , the Ω∗i and Ω ∗ j are non-overlapped . The volume of Ω ∗ is bounded such that 1|Ω∗| ≤ M . The relationship between p ( y|x ) and q ( ŷ|x ) can be defined as : q ( ŷ|x ) = ∫ y∈Ω∗ p ( y|x ) dy . Then maximizing log p ( y|x ) can be interpreted as maximizing the lower bound of log q ( ŷ|x ) . That is , Epdata ( x ) Ey∼U ( Ω∗ ) [ log p ( y|x , φ ) ] ≤MEpdata ( x , ŷ ) [ log q ( ŷ|x , φ ) ] ( 3 ) The complete proof is in appendix . Theorem 1 indicates that , when Ω is well-defined , learning LLF is analogous to dequantization ( Theis et al. , 2015 ; Ho et al. , 2019 ) . That is , the method optimizes the likelihood of dequantized true labels . In practice , the real constrained space , i.e. , Ω , may be loose and does not fulfill the conditions in Theorem 1 , which will result in inevitable errors that come from the weakly supervised setting . Moreover , for some regression problems , the ideal Ω∗ only contains a single point : the ground truth label , i.e. , Ω∗ = { ŷ } . 3.1 LEARNING AND PREDICTION . Since y is unobserved , directly optimizing the conditional likelihood , i.e. , Equation 2 , is impossible . Using the invertibility of normalizing flows , we can rewrite log p ( y|x ) as log p ( y|x ) = log pZ ( fx , φ ( y ) ) + K∑ i=1 log ∣∣∣∣det ( ∂fx , φi∂ri−1 ) ∣∣∣∣ = log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ , ( 4 ) where gx , φi = f −1 x , φi is the inverse flow . With the inverse flow , Equation 2 can be interpreted as a constrained optimization problem max φ Epdata ( x ) EpZ ( z ) [ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ ] , ( 5 ) s.t . C ( x , gx , φ ( z ) , Q ) . Note that in Equation 5 , the original constraint for z is gx , φ ( z ) ∈ Ω , and this constraint can be replaced with C ( x , gx , φ ( z ) , Q ) . For efficient training , this constrained optimization problem can be approximated with the penalty method , resulting in the objective max φ Epdata ( x ) EpZ ( z ) [ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣+ λCr ( x , gx , φ ( z ) , Q ) ] , ( 6 ) where λ is the penalty coefficient , and Cr ( ) means we reformulate the constraints to be penalty losses . For example , an inequality constraint will be redefined as a hinge loss . In practice , the expectation with respect to pZ ( z ) can be approximated with Monte Carlo estimate with L samples . Since we only need to obtain stochastic gradients , we follow previous works ( Kingma & Welling , 2013 ) and set L = 1 . Given a trained model and a data point xi , prediction requires outputting a label yi for xi . We follow ( Lu & Huang , 2020 ) and use a sample average , i.e. , yi = ∑L j=1 gxi , φ ( zj ) as the prediction . In our experiments , we found that L = 10 is enough for generating high-quality labels . 4 CASE STUDY . In this section , we illustrate three scenarios of using LLF to address weakly supervised learning problems . 4.1 WEAKLY SUPERVISED CLASSIFICATION . We follow previous works ( Arachie & Huang , 2021b ; Mazzetto et al. , 2021b ) and consider binary classification . For each example , the label y is a two-dimensional vector within a one-simplex . That is , the y ∈ Y = { y ∈ [ 0 , 1 ] 2 : ∑ i y [ i ] = 1 } , where y [ i ] is the ith dimension of y . Each ground truth label ŷ ∈ { 0 , 1 } 2 is a two-dimensional one-hot vector . We have M weak labelers , which will generate M weak signals for each data point xi , i.e. , qi = [ qi,1 , ... , qi , M ] . Each weak signal qi , m ∈ Q = { q ∈ [ 0 , 1 ] 2 : ∑ i q [ i ] = 1 } is a soft labeling of the data . In practice , if a weak labeler m fails to label a data point xi , the qi , m can be null , i.e. , qi , m = ∅ ( Arachie & Huang , 2021a ) . Following Arachie & Huang ( 2021b ) , we assume we have access to expected error rate bounds of these weak signals b = [ b1 , .. , bM ] . Therefore , the weak signals imply constraints N∑ i=1 , qi , m 6=∅ ( 1− yi ) qi , m + yi ( 1− qi , m ) ≤ Nmbm ∀m ∈ { 1 , ... , M } , ( 7 ) where Nm is the number of data points that are labeled by weak labeler m. This problem can be solved with LLF , i.e. , Equation 5 , by defining C ( x , gx , φ ( z , Q ) to be a combination of weak signal constraints , i.e. , Equation 7 , and simplex constraints , i.e. , y ∈ Y . The objective function of LLF for weakly supervised classification is max φ log pZ ( z ) − K∑ i=1 log ∣∣∣∣det ( ∂gx , φi∂ri ) ∣∣∣∣ +λ1 [ gx , φ ( z ) ] 2 + + λ2 [ 1− gx , φ ( z ) ] 2 + + λ3 ( ∑ i gx , φ ( z ) [ i ] − 1 ) 2 +λ4 M∑ m=1 N∑ i=0 qi , m 6=∅ ( 1− gx , φ ( z ) i ) qi , m + gx , φ ( z ) i ( 1− qi , m ) −Nmbm 2 + , ( 8 ) where the second row describes the simplex constraints , and the last row is the weak signal constraints . The [ . ] + is a hinge function that returns its input if positive and zero otherwise . We omit the expectation terms for simplicity .

Summary: This article proposes **label learning flows (LLF)**, a general framework for weakly supervised learning tasks. The modelling framework is primarily driven by 1) conditional normalizing flows (Trippe & Turner, 2018), which is used as a flexible conditional likelihood model, with an affine coupling flow layer, and 2) constrained optimization formulation of maximum likelihood, where weak signals/error rate bounds enter as constraints (Arachie & Huang, 2021). The main practical formulation uses Lagrange multiplier to define an unconstrained stochastic loss function that requires Monte-Carlo samples during training (Equation 6). Ultimately, they obtain a trained inverse flow, which conditional on a covariate, can produce prediction samples.