WEAKLY SUPERVISED LABEL LEARNING FLOWS
Year: 2021

Abstract:
Supervised learning usually requires a large amount of labelled data. However, attaining ground-truth labels is costly for many tasks. Alternatively, weakly supervised methods learn with cheap weak signals that only approximately label some data. Many existing weakly supervised learning methods learn a deterministic function that estimates labels given the input data and weak signals. In this paper, we develop label learning flows (LLF), a general framework for weakly supervised learning problems. Our method is a generative model based on normalizing flows. The main idea of LLF is to optimize the conditional likelihoods of all possible labelings of the data within a constrained space defined by weak signals. We develop a training method for LLF that trains the conditional flow inversely and avoids estimating the labels. Once a model is trained, we can make predictions with a sampling algorithm. We apply LLF to three weakly supervised learning problems. Experiment results show that our method outperforms many state-of-the-art alternatives.

heading: 1 INTRODUCTION
text: Machine learning has achieved great success in many supervised learning tasks. However, in practice, data labeling is usually human intensive and costly. To address this problem, practitioners are turning to weakly supervised learning (Zhou, 2018), which trains machine learning models with only noisy labels that are generated by human specified rules or pretrained models for related tasks. Recent research shows that these models trained with weak supervisions can also perform well.
Many existing weakly supervised learning methods (Ratner et al., 2016; 2017; Bach et al., 2019; Arachie & Huang, 2021b) learn a deterministic function that estimates the unknown labels y given input data x and weak signals Q. Since the observed information is incomplete, the predictions based on it can be varied. However, these methods ignore this uncertainty between x and y. In this paper, we develop label learning flows (LLF), a general framework for weakly supervised learning problems. LLF is a flow-based generative model (Dinh et al., 2014; Rezende & Mohamed, 2015; Dinh et al., 2016; Kingma & Dhariwal, 2018). The main idea behind LLF is that we define the relationship between x and y with a probability distribution p(y|x), which is modeled by a conditional flow. In training, we use the weak signals Q to define a constrained space for y and then optimize the likelihood of all possible y that are within this constrained space. Therefore, this model captures all possible relationships between the input x and output y. Learning LLF can be defined as a constrained optimization problem. We develop a learning method for LLF that trains the conditional flow inversely and avoids estimating the labels. For prediction, we use sample-based method (Lu & Huang, 2020) to estimate the labels.
We apply LLF to three weakly supervised learning problems: weakly supervised classification (Arachie & Huang, 2021b; Mazzetto et al., 2021b), weakly supervised regression, and unpaired point cloud completion (Chen et al., 2019; Wu et al., 2020). These three problems have very different label types and weak signals. Our method outperforms all other state-of-the-art methods on weakly supervised classification and regression, and it can perform comparably to other recent methods on unpaired point cloud completion. These experiments show that LLF is versatile and powerful.

heading: 2 BACKGROUND
text: In this section, we introduce weakly supervised learning and conditional normalizing flows.
Weakly Supervised Learning. Given a dataset D = {x1, ...,xN}, and weak signals Q, weakly supervised learning finds a model that can predict the unknown label yi for each input data xi. Some methods, e.g., ALL (Arachie & Huang, 2021b), define a set of constrained functions based on Q and x. These functions form a space of possible y. The methods then look for one possible y within this constrained space. In this work, we follow this idea and use constrained functions to restrict the predicted y.
Conditional Normalizing Flows. A normalizing flow (Rezende & Mohamed, 2015) is a series of invertible functions f = f1 ◦ f2 ◦ · · · ◦ fK that transform the probability density of output variables y to the density of latent variables z. In conditional flows (Trippe & Turner, 2018), a flow layer function fi is also parameterized by the input variables x, i.e., fi = fx,φi , where φi is the parameters of fi. With the change-of-variable formula, the log conditional distribution log p(y|x) can be exactly and tractably computed as
log p(y|x) = log pZ(fx,φ(y)) + K∑ i=1 log ∣∣∣∣det(∂fx,φi∂ri−1 )∣∣∣∣ , (1)
where pZ(z) is a tractable base distribution, e.g. Gaussian distribution. The ∂fx,φi ∂ri−1
is the Jacobian matrix of fx,φi . The ri = fx,φi(ri−1), r0 = y, and rK = z.
To use normalizing flows, we need to develop flow layers that are invertible and have tractable Jacobian determinant. In this paper, we use affine coupling layer (Dinh et al., 2014; 2016) to form normalizing flows. It splits the input to two parts, and force the first part only relate to the second part, so that the Jacobian is a triangular matrix. For conditional flows, we can define conditional affine coupling layer as
ya,yb = split(y), zb = s(x,ya) yb + b(x,ya), z = concat(ya, zb),
where s and b are two neural networks. The split() and the concat() functions split and concatenate the variables.

heading: 3 PROPOSED METHOD
text: In this section, we introduce the label learning flows (LLF) framework for weakly supervised learning. Given Q and x, we define a set of constraints to restrict the predicted label y. These constraints can be inequalities, formatting like f(x,y,Q) ≤ b, or equations, formatting like f(x,y,Q) = b. For simplicity, we represent this set of constraints as C(x,y,Q). Let Ω be the constrained space of all possible y defined by C(x,y,Q). Previous methods only look for one possible y within Ω. In contrast, LLF optimizes the conditional log-likelihood of all possible y within Ω, resulting in the following objective
max φ Epdata(x)Ey∼U(Ω) [log p(y|x, φ)] , (2)
where U(Ω) is a uniform distribution within Ω, and p(y|x) is a continuous density model of y. Let ŷ be the true label. We assume that each data point xi only has one unique label ŷi, so that pdata(x, ŷ) = pdata(x). Let q(ŷ|x) be a certain model of ŷ. Traditional supervised learning learns a q(ŷ|x) that maximizes the cross entropy Epdata(x,ŷ) [log q(ŷ|x, φ)]. Following theorem indicates that maximizing log p(y|x) can be interpreted as maximizing a lower bound of log q(ŷ|x).
Theorem 1 Let Ω∗ ⊆ Ω is a tight enough space satisfying that ŷ ∈ Ω∗ and for two different ŷi and ŷj , the Ω∗i and Ω ∗ j are non-overlapped. The volume of Ω
∗ is bounded such that 1|Ω∗| ≤ M . The relationship between p(y|x) and q(ŷ|x) can be defined as: q(ŷ|x) = ∫ y∈Ω∗ p(y|x)dy. Then maximizing log p(y|x) can be interpreted as maximizing the lower bound of log q(ŷ|x). That is, Epdata(x)Ey∼U(Ω∗) [log p(y|x, φ)] ≤MEpdata(x,ŷ) [log q(ŷ|x, φ)] (3)
The complete proof is in appendix. Theorem 1 indicates that, when Ω is well-defined, learning LLF is analogous to dequantization (Theis et al., 2015; Ho et al., 2019). That is, the method optimizes the
likelihood of dequantized true labels. In practice, the real constrained space, i.e., Ω, may be loose and does not fulfill the conditions in Theorem 1, which will result in inevitable errors that come from the weakly supervised setting. Moreover, for some regression problems, the ideal Ω∗ only contains a single point: the ground truth label, i.e., Ω∗ = {ŷ}.

heading: 3.1 LEARNING AND PREDICTION
text: Since y is unobserved, directly optimizing the conditional likelihood, i.e., Equation 2, is impossible. Using the invertibility of normalizing flows, we can rewrite log p(y|x) as
log p(y|x) = log pZ(fx,φ(y)) + K∑ i=1 log ∣∣∣∣det(∂fx,φi∂ri−1 )∣∣∣∣
= log pZ(z)− K∑ i=1 log ∣∣∣∣det(∂gx,φi∂ri )∣∣∣∣ , (4)
where gx,φi = f −1 x,φi is the inverse flow.
With the inverse flow, Equation 2 can be interpreted as a constrained optimization problem
max φ Epdata(x)EpZ(z)
[ log pZ(z)−
K∑ i=1 log ∣∣∣∣det(∂gx,φi∂ri )∣∣∣∣ ] , (5)
s.t. C(x,gx,φ(z),Q).
Note that in Equation 5, the original constraint for z is gx,φ(z) ∈ Ω, and this constraint can be replaced with C(x,gx,φ(z),Q). For efficient training, this constrained optimization problem can be approximated with the penalty method, resulting in the objective
max φ Epdata(x)EpZ(z)
[ log pZ(z)−
K∑ i=1 log ∣∣∣∣det(∂gx,φi∂ri )∣∣∣∣+ λCr(x,gx,φ(z),Q) ] , (6)
where λ is the penalty coefficient, and Cr() means we reformulate the constraints to be penalty losses. For example, an inequality constraint will be redefined as a hinge loss.
In practice, the expectation with respect to pZ(z) can be approximated with Monte Carlo estimate with L samples. Since we only need to obtain stochastic gradients, we follow previous works (Kingma & Welling, 2013) and set L = 1.
Given a trained model and a data point xi, prediction requires outputting a label yi for xi. We follow (Lu & Huang, 2020) and use a sample average, i.e., yi = ∑L j=1 gxi,φ(zj) as the prediction. In our experiments, we found that L = 10 is enough for generating high-quality labels.

heading: 4 CASE STUDY
text: In this section, we illustrate three scenarios of using LLF to address weakly supervised learning problems.

heading: 4.1 WEAKLY SUPERVISED CLASSIFICATION
text: We follow previous works (Arachie & Huang, 2021b; Mazzetto et al., 2021b) and consider binary classification. For each example, the label y is a two-dimensional vector within a one-simplex. That is, the y ∈ Y = {y ∈ [0, 1]2 : ∑ i y[i] = 1}, where y[i] is the ith dimension of y. Each ground truth label ŷ ∈ {0, 1}2 is a two-dimensional one-hot vector. We have M weak labelers, which will generate M weak signals for each data point xi, i.e., qi = [qi,1, ...,qi,M ]. Each weak signal qi,m ∈ Q = {q ∈ [0, 1]2 : ∑ i q[i] = 1} is a soft labeling of the data. In practice, if a weak labeler m fails to label a data point xi, the qi,m can be null, i.e., qi,m = ∅ (Arachie & Huang, 2021a). Following Arachie & Huang (2021b), we assume we have access to expected error rate bounds of
these weak signals b = [b1, ..,bM ]. Therefore, the weak signals imply constraints
N∑ i=1,qi,m 6=∅ (1− yi) qi,m + yi (1− qi,m) ≤ Nmbm ∀m ∈ {1, ...,M}, (7)
where Nm is the number of data points that are labeled by weak labeler m.
This problem can be solved with LLF, i.e., Equation 5, by defining C(x,gx,φ(z,Q) to be a combination of weak signal constraints, i.e., Equation 7, and simplex constraints, i.e., y ∈ Y . The objective function of LLF for weakly supervised classification is
max φ
log pZ(z)− K∑ i=1 log ∣∣∣∣det(∂gx,φi∂ri )∣∣∣∣
+λ1 [gx,φ(z)] 2 + + λ2 [1− gx,φ(z)] 2 + + λ3 (∑ i gx,φ(z)[i]− 1 )2
+λ4 M∑ m=1  N∑ i=0
qi,m 6=∅
(1− gx,φ(z)i) qi,m + gx,φ(z)i (1− qi,m)−Nmbm  2
+
, (8)
where the second row describes the simplex constraints, and the last row is the weak signal constraints. The [.]+ is a hinge function that returns its input if positive and zero otherwise. We omit the expectation terms for simplicity.

heading: 4.2 WEAKLY SUPERVISED REGRESSION
text: For weakly supervised regression, we predict one-dimensional continuous labels y ∈ [0, 1] given input dataset D = {x1, ...,xN} and weak signals Q. We define the weak signals as follows. For the mth feature of input data, we have access to a threshold m, which splitsD to two parts, i.e.,D1,D2, such that for each xi ∈ D1, the xi,m ≥ m, and for each xj ∈ D2, the xj,m < m. We also have access to estimated values of labels for subsets D1 and D2, i.e., bm,1 and bm,2. This design of weak signals tries to mimic that in practical scenarios, human experts can design rule-based methods for predicting labels for given data. For example, in disease prediction, a medical doctor can predict the disease rates for patients based on their ages. For a group of people whose age is greater than a threshold, an experienced physician would know an estimate of their average disease rate. Assuming that we have M rule-based weak signals, the constraints can be mathematically defined as follows:
1 |Dm,1| ∑
i∈Dm,1
yi = bm,1, 1 |Dm,2| ∑
j∈Dm,2
yj = bm,2, m ∈ 1, ...,M. (9)
Plugging in Equation 9 to Equation 6, we have
max φ
log pZ(z)− K∑ i=1 log ∣∣∣∣det(∂gx,φi∂ri )∣∣∣∣+ λ1 [gx,φ(z)]2+ + λ2 [1− gx,φ(z)]2+
+λ3 M∑ m=1  1 |Dm,1| ∑ i∈Dm,1 gx,φ(z)i − bm,1 2 +  1 |Dm,2| ∑ j∈Dm,2 gx,φ(z)j − bm,2 2 (10)

heading: 4.3 UNPAIRED POINT CLOUD COMPLETION
text: Unpaired point cloud completion (Chen et al., 2019; Wu et al., 2020) is a practical problem in 3D scanning. Given a set of partial point clouds Xp = {x(p)1 , ...,x (p) N }, and a set of complete point clouds Xc = {x(c)1 , ...,x (c) N }, we want to restore each x (p) i ∈ Xp. Each point cloud is a set of points,
i.e., xi = {xi,1, ...,xi,T }, where each xi,t ∈ R3 is a 3D point, and the counts T represent the number of points in a point cloud.
Note that the point clouds in Xp and Xc are unpaired, so directly modeling the relationship between x(c) and x(p) is impossible. This problem can be interpreted as a weakly supervised learning problem, in which the weak signals Q are derived from the referred complete point cloudsXc. We predict complete point clouds y ∈ Y for partial point clouds in Xp. The conditional distribution p(y|xp) is an exchangable distribution. We follow previous works (Yang et al., 2019; Klokov et al., 2020) and use De Finetti’s representation theorem and variational inference to compute its lower bound as the objective.
log p(y|xp) ≥ Eq(u|xp) [ Tc∑ i=1 log p(yi|u,xp) ] − KL(q(u|xp)||p(u)), (11)
where q(u|xp) is a variational distribution of latent variable u. In practice, it can be represented by an encoder, and uses the reparameterization trick (Kingma & Welling, 2013) to sample u. The p(u) is a standard Gaussian prior. The p(yi|u,xp) is defined by a conditional flow. We follow Chen et al. (2019); Wu et al. (2020) and use the adversarial loss and Hausdorff distance loss as constraints. The final objective function is
max φ
Eq(u|xp) [ Tc∑ t=1 log pZ(zt)− K∑ i=1 log ∣∣∣∣det(∂gu,xp,φi∂rt,i )∣∣∣∣ ] − KL(q(u|xp)||p(u))
+Eq(u|xp) [ λ1(D(gu,xp,φ(z))− 1)2 + λ2dH(gu,xp,φ(z),xp) ] , (12)
where D() represents the discriminator of a least square GAN (Mao et al., 2017), and dH() represents the Haudorsff distance, which measures the distance between a generated complete point cloud its corresponding input partial point cloud. For clarity, we use zt and rt to represent variables of the tth point in a point cloud, and gu,xp,φ(z) to represent a generated point cloud. Detailed derivations of Equation 12 are in appendix.
Training with Equation 12 is different from the previous settings, because we also need to train the discriminator of the GAN. The objective for D() is
min D
Epdata(xc) [ (D(xc)− 1)2 ] + Epdata(xp),pZ(z),q(u|xp) [ D(gxp,u,φ(z)) 2 ] . (13)
The training process is similar to traditional GAN training. The inverse flow gu,xp,φ can be roughly seen as the generator. In training, we train the flow to optimize Equation 12 and the discriminator to optimize Equation 13, alternatively.

heading: 5 RELATED WORK
text: In this section, we introduce the research that most related to our work.
Weakly Supervised Learning. For weakly supervised classification, we use the same strategy as adversarial label learning (ALL) (Arachie & Huang, 2021b) to define constraint functions based on weak signals. ALL then uses a min-max optimization to learn the model parameters and estimate y alternatively. In contrast to ALL, LLF is a generative model, so it can learn the model parameters and output y simultaneously, and it does not need a min-max optimization. Moreover, LLF optimizes the likelihoods of all possible ys within Ω, while ALL only estimates one possible y. Other methods also constrain the label space of the predicted labels using weak supervision (Arachie & Huang, 2021a;b; Mazzetto et al., 2021a;b). These methods are deterministic and developed for classification tasks. However, LLF can be used for regression and uses sampling during inference.
Non-constraint based weak supervision methods typically assume a joint distribution for the weak signals and the true labels of the data. These methods use a latent variable model to estimate the labels of the data while accounting for the dependency among the weak signals (Ratner et al., 2016; 2019; Fu et al., 2020). Like these methods, we assume a family of distributions for the label space of the data. This space is defined by the constraints of the weak supervision and the data. Unlike these methods, we use a flow network rather than a graphical model to solve for the label of the
data. Additionally, we do not solve for the dependence amongst the weak signals thereby avoiding the need for making extra assumptions.
Normalizing Flows. Normalizing flows (Dinh et al., 2014; Rezende & Mohamed, 2015; Dinh et al., 2016; Kingma & Dhariwal, 2018) have gained recent attention because of their advantages of exact latent variable inference and log-likelihood evaluation. Specifically, conditional normalizing flows have been widely applied to many supervised learning problems (Trippe & Turner, 2018; Lu & Huang, 2020; Lugmayr et al., 2020; Pumarola et al., 2020) and semi-supervised classification (Atanov et al., 2019; Izmailov et al., 2020). However, normalizing flows have not previously been applied to weakly supervised learning problems.
Our inverse training method for LLF is similar to that of injective flows (Kumar et al., 2020). Injective flows are used to model unconditional datasets. They use an encoder network to map the input data x to latent code z, and then they use an inverse flow to map z back to x, resulting in an autoencoder architecture. Different from injective flow, LLF directly samples z from a prior distribution and uses a conditional flow to map z back to y conditioned on x. We use constraint functions to restrict y to be valid, so that does not need an encoder network.
Point Cloud Modeling. Recently, Yang et al. (2019) and Tran et al. (2019) combine normalizing flows with variational autoencoders (Kingma & Welling, 2013) and developed continuous and discrete normalizing flows for point clouds. The basic idea of point normalizing flows is to use a conditional flow to model each point in a point cloud. The conditional flow is conditioned on a latent variable generated by an encoder. To guarantee exchangeability, the encoder uses a PointNet (Qi et al., 2017) to extract features from input point clouds.
The unpaired point cloud completion problem is first defined by Chen et al. (2019). They develop pcl2pcl—a GAN (Goodfellow et al., 2014) based model—to solve it. Their method is two-staged. In the first stage, it trains autoencoders to map partial and complete point clouds to their latent space. In the second stage, a GAN is used to transform the latent features of partial point clouds to latent features of complete point clouds. In their follow-up paper (Wu et al., 2020), they develop a variant of pcl2pcl, called multi-modal pcl2pcl (mm-pcl2pcl), which incorporates random noise to the generative process, so that can capture the uncertainty in reasoning.
In contrast to pcl2pcl, LLF can be trained end-to-end. When applying LLF to this problem, LLF has a similar framework to VAE-GAN (Larsen et al., 2016). The main differences are that LLF models a conditional distribution of points, and its encoder is a point normalizing flow.

heading: 6 EMPIRICAL STUDY
text: In this section, we evaluate LLF on the three weakly supervised learning problems.
Model architecture. For weakly supervised classification and unpaired point cloud completion, the labels y are multi-dimensional variables. We follow Klokov et al. (2020) and use flows with only conditional coupling layers. We use the same method as Klokov et al. (2020) to define the conditional affine layer. Each flow model contains 8 flow steps. For unpaired point cloud completion, each flow step has 3 coupling layers. For weakly supervised classification, each flow step has 2 coupling layers. For weakly supervised regression, since y is a scalar, we use simple conditional affine transformation as flow layer, which is defined as:y = s(x) ∗ z + b(x), where s and b are two neural networks that take x as input and output parameters for y. The flow for this problem contains 8 conditional affine transformations.
For the unpaired point cloud completion task, we need to also use an encoder network, i.e., q(u|xp) and a discriminator D(). We follow Klokov et al. (2020); Wu et al. (2020) and use PointNet (Qi et al., 2017) in these two networks to extract features for point clouds.
Experiment setup. In weakly supervised classification and regression experiments, we assume that the ground truth labels are inaccessible, so tuning hyper-parameters for models are impossible. We use default settings for all hyper-parameters of LLF, e.g., λs and learning rates. We fix λ = 10 and use Adam (Kingma & Ba, 2014) with default settings, i.e., η = 0.001, β1 = 0.9 and β2 = 0.999. We use an exponential learning rate scheduler with a decreasing rate of 0.996 to guarantee convergence. We track the decrease of loss and when the decrease is small enough, the training stops. Following previous works (Arachie & Huang, 2021b;a), we use full gradient optimization
to train the models. For fair comparison, we run each experiment 5 times with different random seeds {0, 10, 100, 123, 1234}. We split each dataset to training, simulation, and test sets. We follow Arachie & Huang (2021b;a) and create weak signals with randomly chosen features, and estimate error bounds and thresholds on simulation set.
For experiments with unpaired point cloud completion, the datasets contain validation sets, so we tune the hyper-parameters using these. We use Adam with an initial learning rate η = 0.0001 and default βs. The best coefficients for the constraints in Equation 12 are λ1 = 10, λ2 = 100. We use stochastic optimization to train the models, and the batch size is 32. Each model is trained for at most 2000 epochs.

heading: 6.1 WEAKLY SUPERVISED CLASSIFICATION
text: Datasets. We follow Arachie & Huang (2021a;b) and conduct experiments on 12 datasets. Specifically, the Breast Cancer, OBS Network, Cardiotocography, Clave Direction, Credit Card, Statlog Satellite, Phishing Websites, Wine Quality are tabular datasets from UCI repository (Dua & Graff, 2017). The Fashion-MNIST (Xiao et al., 2017) is an image set with 10 classes of clothing types. We choose 3 pairs of classes, i.e., dresses/sneakers (DvK), sandals/ankle boots (SvA), and coats/bags (CvB), to conduct binary classification. We follow Arachie & Huang (2021b) and create 3 synthetic weak signals for each dataset. The IMDB (Maas et al., 2011), SST (Socher et al., 2013) and YELP are real text datasets. We follow Arachie & Huang (2021a) and use keyword-based weak supervision. Each dataset has more than 10 weak signals. For the experiments on tabular datasets, we set the maximum epochs to 2000. Since the experiments on real text datasets are larger, we set the maximum epochs to 500.
Baselines. We compare our method with state-of-the-art methods for weakly supervised classification. For the experiments on tabular datasets and image sets, we use ALL (Arachie & Huang, 2021b), generalized expectation (GE) (Druck et al., 2008; Mann & McCallum, 2010) and averaging of weak signals (AVG). For experiments on text datasets, we use CLL (Arachie & Huang, 2021a), Snorkel MeTaL (Ratner et al., 2019), Data Programming (DP) (Ratner et al., 2016), regularized minimax conditional entropy for crowdsourcing (MMCE) (Zhou et al., 2015), and majority-vote. We also show supervised learning (SL) results for reference.
Results. We report the mean and standard deviation of accuracy on test sets in Table 1 and Table 2. LLF outperforms all baseline methods on almost all datasets. On some datasets, LLF can perform as well as supervised learning methods. These results prove that LLF is powerful and effective. In our experiments, we also found that the performance of LLF will also be impacted by different initialization of weights. This is why LLF has relatively larger variance on some datasets.
Ablation Study. We can directly train the model using only the constraints as the objective function. In our experiments, we found that training LLF without likelihood (LLF-w/o-nll) will still work. However, the model performs worse than training with likelihood. We believe that this is because the likelihood helps accumulate more probability mass to the constrained space Ω, so the model will more likely generate y samples within Ω, and the predictions are more accurate.

heading: 6.2 WEAKLY SUPERVISED REGRESSION
text: Datasets. We use 3 tabular datasets from the UCI repository Dua & Graff (2017): Air Quality, Temperature Forecast, and Bike Sharing dataset. For each dataset, we randomly choose 5 features to develop the rule based weak signals. We split each dataset to training, simulation, and test sets. The simulation set is then used to compute the threshold s, and the estimated label values bs. Since we do not have human experts to estimate these values, we use the mean value of a feature as its threshold, i.e., m = 1|Dvalid| ∑ i∈Dvalid xi[m]. We then compute the estimated label values bm,1 and bm,2 based on labels in the valid set. Note that the labels in the simulation set are only used for generating weak signals, simulating human expertise. In training, we still assume that we do not have access to labels. The original label is within an interval [ly, uy]. We normalize the original label to within [0, 1] by computing y = (y − ly)/(uy − ly). In prediction, we recover the predicted label to original value by computing y = y(uy − ly) + ly . Baselines. To the best of our knowledge, there are no methods specifically designed for weakly supervised regression of this form. We use average of weak signals (AVG) and LFF-w/o-nll as baselines. We also report the supervised learning results for reference.
Results. We use root square mean error (RSME) as metric. The results of test set are in Table 3. In general, LLF can predict reasonable labels. Its results are much better than AVG or any of the weak signals alone. Similar to the classification results, training LLF without using likelihood will reduce its performance.

heading: 6.3 UNPAIRED POINT CLOUD COMPLETION
text: Datasets. We use the Partnet (Mo et al., 2019) dataset in our experiments. We follow Wu et al. (2020) and conduct experiments on the 3 largest classes of PartNet: Table, Chair, and Lamp. We treat each class as a dataset, which is split to training, validation, and test sets based on official splits of PartNet. For each point cloud, we remove points of randomly selected parts to create a partial point cloud. We follow Chen et al. (2019); Wu et al. (2020) and let the partial point clouds have 1024 points, and the complete point clouds have 2048 points. We let the latent variable u of the VAE to be a 128-dimensional vector.
Metrics. We follow Wu et al. (2020) and use minimal matching distance (MMD) (Achlioptas et al., 2018), total mutual difference (TMD), and unidirectional Hausdorff distance (UHD) as metrics. MMD measures the quality of generated. A lower MMD is better. TMD measures the diversity of samples. A higher TMD is better. UHD measures the fidelity of samples. A lower UHD is better.
Baselines. We compare our method with pcl2pcl (Chen et al., 2019), mm-pcl2pcl (Wu et al., 2020), and LLF-w/o-nll. We use two variants of mm-pcl2pcl. Another variant is called mm-pcl2pcl-im, which is different from the original model in that it jointly trains the encoder of modeling multimodality and the GAN.
Results. We list the test set results in Table 4. In general, pcl2pcl has the best fidelity, i.e., lowest UHD. This is because pcl2pcl is a discriminative model, and it will only predict one certain sample for each input. This is also why pcl2pcl has the worse diversity as measured by TMD. Mm-pcl2pcl has the best diversity. However, we found in our experiments that some samples generated by mmpcl2pcl are invalid, i.e., they are totally different from the input partial point clouds. Therefore, mm-pcl2pcl has the worse fidelity. LLF scores between pcl2pcl and mm-pcl2pcl. It has better UHD than mm-pcl2pcl, and better TMD and MMD than pcl2pcl. The LLF-w/0-nll has a slightly better UHD than LLF. We believe this is because, without using the likelihood, LLF-w/o-nll is trained directly by optimizing the Hausdorff distance. However, the sample diversity and quality, i.e., TMD and MMD, are worse than LLF. As argued by Yang et al. (2019), the current metrics for evaluating point cloud samples all have flaws, so these scores cannot be treated as hard metrics for evaluating model performance. We therefore visualize some samples in appendix, which show that LLF can generate samples that comparable to mm-pcl2pcl.

heading: 7 CONCLUSION
text: In this paper, we propose label learning flows, which represent a general framework for weakly supervised learning. LLF uses a conditional flow to define the conditional distribution p(y|x), so that can model the uncertainty between input x and all possible y. Learning LLF is a constrained optimization problem that optimizes the likelihood of all possible y within the constrained space defined by weak signals. We develop a specific training method to train LLF inversely, avoiding the need of estimating y. We apply LLF to three weakly supervised learning problems, and the results show that our method outperforms many state-of-the-art baselines on the weakly supervised classification and regression problems, and performs comparably to other new methods for unpaired point cloud completion. These results indicate that LLF is a powerful and effective tool for weakly supervised learning problems.

heading: A PROOF OF THEOREM 1
text: The proof of Theorem 1 is similar to the proof of dequantization (Theis et al., 2015; Ho et al., 2019). The complete proof is as follows.

heading: Proof.
text: Epdata(x)Ey∼U(Ω∗) [log p(y|x, φ)] = ∑ x pdata(x) ∫ y∈Ω∗ 1 |Ω∗| log p(y|x)dy
≤ M ∑ x pdata(x) log ∫ y∈Ω∗ p(y|x)dy
= M ∑ x,ŷ pdata(x, ŷ) log q(ŷ|x)
= MEpdata(x,ŷ) [log q(ŷ|x)] (14)
In the second row, we use the properties that 1|Ω∗| ≤M , and the Jensen’s inequality. In the third row, we use the assumption that pdata(x) = pdata(x, ŷ), and the relationship between p(y|x) and q(ŷ|x).

heading: B LABEL LEARNING FLOW FOR UNPAIRED POINT CLOUD COMPLETION
text: In this section, we provide complete derivations of LLF for unpaired point cloud completion. The conditional likelihood log p(y|xp) is an exchangeable distribution. We use De Finetti’s representation theorem and variational inference to derive a tractable lower bound for it.
log p(y|xp) = ∫ p(y,u|xp)du
= ∫ p(y|u,xp)p(u)du
≥ Eq(u|xp) [log p(y|u,xp)]− KL(q(u|xp)||p(u))
≥ Eq(u|xp) [ Tc∑ i=1 log p(yi|u,xp) ] − KL(q(u|xp)||p(u)), (15)
where in the third row, we use Jensen’s inequality to compute the lower bound, and in the last row, we use De Finetti’s theorem to factorize p(y|u,xp) to the distributions of points. The least square GAN discriminator and the Hausdorff distance for generated complete point clouds can be treated as two equality constraints
D(y) = 1
dH(y,xp) = 0.
Note that the dH() is non-negative. Convert these two constraints to penalty functions, we have
max φ
Eq(u|xp) [ Tc∑ t=1 log pZ(zt)− K∑ i=1 log ∣∣∣∣det(∂gu,xp,φi∂rt,i )∣∣∣∣ ] − KL(q(u|xp)||p(u))
+Eq(u|xp) [ λ1(D(gu,xp,φ(z))− 1)2 + λ2dHL(gu,xp,φ(z),xp) ] . (16)

heading: C EXPERIMENT DETAILS
text: In this section, we provide more details on our experiments to help readers reproduce our results.

heading: C.1 MODEL ARCHITECTURE
text: For experiments of weakly supervised classification, and unpaired point cloud completion, we use normalizing flows with only conditional affine coupling layers Klokov et al. (2020). Each layer is defined as
ya,yb = split(y) s = ms(wy(ya) wx(x) + wb(x)) b = mb(cy(ya) cx(x) + cb(x)) zb = s yb + b z = concat(ya, zb), (17)
where m,w, c are all small neural networks.
For LLF, we only need the inverse flow, i.e., gx,φ, for training and prediction, so in our experiments, we actually define gx,φ as the forward transformation, and let y = s z + b. We do this because multiplication and addition are more stable than division and subtraction.

heading: C.1.1 WEAKLY SUPERVISED CLASSIFICATION
text: In this problem, we use a flow with 8 flow steps, and each step has 2 conditional affine coupling layers. These two layers will transform different dimensions. Each w and c are small MLPs with two linear layers. Each m has one linear layer. The hidden dimension of linear layers is fixed to 64.

heading: C.1.2 WEAKLY SUPERVISED REGRESSION
text: In this problem, since the label y is a scalar, we use conditional affine transformation introduced in Section 6, as a flow layer. A flow has 8 flow layers. The s and b in a flow layer are three layer MLPs. The hidden dimension of linear layers is 64.

heading: C.1.3 UNPAIRED POINT CLOUD COMPLETION
text: The model architecture used LLF used for this problem is illustrated in Figure 2. We use the same architecture as DPF (Klokov et al., 2020) for point flow. Specifically, the flow has 8 flow steps, and each step has 3 conditional affine coupling layers, i.e., Equation 17. Slightly different from the original DPF, the conditioning networks wx, cx, wb, and cb will take the latent variable u and the features of partial point cloud xp as input. The ws and cs are MLPs with two linear layers, whose hidden dimension is 64. The ms are one layer MLPs.
We use a PointNet (Qi et al., 2017) to extract features from partial point cloud xp. Following Klokov et al. (2020), the hidden dimensions of this PointNet is set as 64−128−256−512. Given the features of xp, the encoder E then uses the reparameterization trick (Kingma & Welling, 2013) to generate latent variable u. The encoder has two linear layers, whose hidden dimension is 512.
The GAN discriminator uses another PointNet to extract features from (generated) complete point clouds. We follow Wu et al. (2020) and set the hidden dimensions of this PointNet as 64−128−128− 256−128. The discriminatorD is a three layer MLP, whose hidden dimensions are 128−256−512.

heading: C.2 MORE EXPERIMENT DETAILS

heading: C.2.1 WEAKLY SUPERVISED CLASSIFICATION
text: We use the same way as Arachie & Huang (2021b;a) to split each dataset to training, simulation, and test sets. We use the data and labels in simulation sets to create weak signals, and estimated bounds. We train models on training sets and test model on test sets. We assume that the models do not have access to any labels. The labels in simulation sets are only used to generated weak signals and estimate bounds.
For experiments on Fashion-MNIST and the tabular datasets, we follow Arachie & Huang (2021b) and choose 3 features to create weak signals. We train a logistic regression with each feature on the simulation set, and use the label probabilities predicted by this logistic regression as weak signals. We compute the error of this trained logistic regression on simulation set as estimated error bound.
For experiments on real text datasets, we use the same keyword-based method as Arachie & Huang (2021a) to create weak supervision. Specifically, we choose key words that can weakly indicate positive and negative sentiments. Documents containing possitive words will be labeled as positive, and vice versa.
We list some main features of these datasets in Table 5. We refer to their original papers for more details. For those datasets without official splits, we randomly split them with a ratio of 4 : 3 : 3.

heading: C.2.2 WEAKLY SUPERVISED REGRESSION
text: We use three datasets from the UCI repository. For each dataset, we randomly split it to training, simulation, and test sets with a ratio of 4 : 3 : 3. We use the simulation set to create weak signals and estimated label values. We choose 5 features to create weak signals for each dataset. The detailed introduction of these datasets are as follows. Table 6 summarize the statistical results of them.
Air Quality. In this task, we predict the absolute humidity in air, based on other air quality features such as hourly averaged temperature, hourly averaged NO2 concentration etc. The raw dataset has 9, 358 instances. We remove those instances with Nan values, resulting in a dataset with 8, 991 instances. We use hourly averaged concentration CO, hourly averaged Benzene concentration, hourly averaged NOx concentration, tungsten oxide hourly averaged sensor response, and relative humidity as features for creating weak signals.
Temperature Forecast. In this task, we predict the next day maximum air temperature based on current day information. The raw dataset has 7, 750 instances, and we remove those instances with Nan values, resulting in 7, 588 instances. We use present max temperature, forecasting next day wind speed, forecasting next day cloud cover, forecasting next day precipitation, solar radiation as features for creating weak signals.
Bike Sharing. In this task, we predict the count of total rental bikes given weather and date information. The raw dataset has 17, 389 instances, and we remove those instances with Nan values, resulting in 17, 379 instances. We use season, hour, if is working day, normalized feeling temperature, and wind speed as features for creating weak signals.

heading: C.2.3 UNPAIRED POINT CLOUD COMPLETION
text: Datasets. We use the same way as Wu et al. (2020) to process PartNet Mo et al. (2019). PartNet provides point-wise semantic labels for point clouds. The original point clouds are used as complete point clouds. To generate partial point clouds, we randomly removed parts from complete point
clouds, based on the semantic labels. We use Chair, Table, and Lamp categories. The summary of these three subsets are in Table 7.
Metrics. Let Xc be the set of referred complete point clouds, and Xp be the set of input partial point clouds. For each partial point cloud x(p)i , we generate M complete point cloud samples y
(1) i , ...,y (M) i . All these samples form a new set of complete point clouds Y . In our experiments,
we follow Wu et al. (2020) and set M = 10.
The MMD Achlioptas et al. (2018) is defined as
MMD = 1 |Xc| ∑
xi∈Xc
dC(xi,NN(xi)), (18)
where NN(x) is the nearest neighbor of x in Y . The dC represents Chamfer distance. MMD computes the distance between the set of generated samples and the set of target complete shapes, so it measures the quality of generated.
The TMD is defined as
TMD = 1
|Xp| |Xp|∑ i=1  2 M − 1 M∑ j=1 M∑ k=j+1 dC(y (j) i ,y (k) i )  . (19) TMD measures the difference of generated samples given an input partial point cloud, so it measures the diversity of samples.
The UHD is defined as
UHD = 1
|Xp| |Xp|∑ i=1  1 M M∑ j=1 dH(xi,y (j) i )  , (20) where dH represents the unidirectional Hausdorff distance. UHD measures the similarity between generated samples and input partial point clouds, so it measures the fidelity of samples.
Samples. We compare LLF to mm-pc2pc in Figure 3, and more samples of LLF in Figure 4, Figure 5, and Figure 6. LLF can generate samples that are as good as mm-pc2pc. Mm-pc2pc has a higher diversity in samples, but some generated samples may be unreasonable.

Review summaries:
Different from many existing weakly supervised learning methods, which learn a deterministic function that estimates labels given the input data and weak signals,  this paper proposes label learning flows (LLF), a general framework for weakly supervised learning problems, and it is a generative model based on normalizing flows. The main idea of LLF is to optimize the conditional likelihoods of all possible labeling of the data within a constrained space defined by weak signals.  The proposed method is applied to three weakly supervised learning problems. Experimental results show that this method outperforms many state-of-the-art alternatives.

This paper proposes a new method for weakly supervised learning where the true label of a data sample is unknown but there are known constraints on the label. The proposed method is based on conditional normalising flows, where the inverse flow is used to model the conditional label distribution and optimised with the label constraints. The proposed method is used in three applications including classification, regression, and point cloud completion. The proposed method is shown to have better performance in these applications in the comparison with several baseline methods.  

This article proposes **label learning flows (LLF)**, a general framework for weakly supervised learning tasks. The modelling framework is primarily driven by 1) conditional normalizing flows (Trippe & Turner, 2018), which is used as a flexible conditional likelihood model, with an affine coupling flow layer, and 2) constrained optimization formulation of maximum likelihood, where weak signals/error rate bounds enter as constraints (Arachie & Huang, 2021). The main practical formulation uses Lagrange multiplier to define an unconstrained stochastic loss function that requires Monte-Carlo samples during training (Equation 6). Ultimately, they obtain a trained inverse flow, which conditional on a covariate, can produce prediction samples.

This work focuses on the problem of learning predictive models where no ground truth labels are provided.  Instead a set of weak constraints over instances that restrict the space of possible target values are provided.  The proposed method, LLF, is based on the core idea that instead of determining a single true weak label for each instance, a model can be learned by considering the expectation over all possible labels in the feasible set defined by the given constraints.  The authors utilize a conditional normalizing flow model as a generative process for labels and constrain it's output by imposing regularization terms on the training objective that represent different constraints imposed by the weak supervision.  The authors show how their framing of weakly supervised learning can be applied go specific settings in classification, regression, and unpaired point cloud completion.  In their evaluation, the authors compare LLF to a number of weakly supervised baselines and show superior performance on classification and regression tasks, while largely being outperformed by one method in point cloud completion.