Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Authors: Kosuke Sugiyama
Masato Uchida
Venue: ICML 2025 poster (2025)
Keywords: weak features learning
impute-then-regress
missing value
weak supervised learning
Primary area: theory->learning_theory
TL;DR: This paper proposes a unified framework for learning with low-quality continuous features and provides a theoretical analysis of the interplay between feature quality improvement and downstream predictive performance.

Abstract: This paper addresses weak features learning (WFL), focusing on learning scenarios characterized by low-quality input features (weak features; WFs) that arise due to missingness, measurement errors, or ambiguous observations. We present a theoretical formalization and error analysis of WFL for continuous WFs (continuous WFL), which has been insufficiently explored in existing literature. A previous study established formalization and error analysis for WFL with discrete WFs (discrete WFL); however, this analysis does not extend to continuous WFs due to the inherent constraints of discreteness. To address this, we propose a theoretical framework specifically designed for continuous WFL, systematically capturing the interactions between feature estimation models for WFs and label prediction models for downstream tasks. Furthermore, we derive the theoretical conditions necessary for both sequential and iterative learning methods to achieve consistency. By integrating the findings of this study on continuous WFL with the existing theory of discrete WFL, we demonstrate that the WFL framework is universally applicable, providing a robust theoretical foundation for learning with low-quality features across diverse application domains.

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Kosuke Sugiyama
Masato Uchida
Abstract
and privacy concerns often result in features deviating from their true values. These deviations manifest as missing values, erroneous observations, or ambiguous information. Such low-quality input features are termed Weak Features (WFs) (Sugiyama & Uchida, 2025). Due to the absence of direct observation of true values, WFs destabilize the leam- ing process of predictive models, leading performance degradation. Furthermore, the inaccuracies and incomplete- ness inherent in WF's significantly hinder the explainability of model outputs and decision-makin processes. To ad- dress these challenges, methods such as impute-then-regress (ItR) (Josse et al., 2024; Bertsimas et al., 2021; Le Morvan et al., 2020a;b; 2021), which imputes missing data prior to prediction, and complementary features learning (CFL) (Sugiyama & Uchida, 2024), which utilizes values differing from the true values, have been proposed. Additionally, generalized framework, weak features learning (WFL) (Sugiyama & Uchida, 2025), has been developed to system- atically address a broader range of WFs.
This paper addresses weak features learning (WFL). focusing on learning scenarios charac- terized by low-quality input features (weak fea- tures; WFs) that arise due to missingness, mea- surement errors, or ambiguous observations. We present theoretical formalization and error anal- ysis of WFL for continuous WFs (continuous WFL.,, which has been insufficiently explored in existing literature. A previous study established formalization and error analysis for WFL with dis- crete WFs (discrete WFL.); however, this analysis does not extend to continuous WFs due to the in- herent constraints of discreteness. To address this, we propose theoretical framework specifically designed for continuous WFL., systematically cap- turing the interactions between feature estima- tion models for WFs and label prediction models for downstream tasks. Furthermore, we derive the theoretical conditions necessary for both se- quential and iterative learning methods to achieve consistency. By integrating the findings of this study on continuous WFL with the existing theory of discrete WFL, demonstrate that the WFL framework is universally applicable, providing robust theoretical foundation for learning with low-quality features across diverse application do- mains.
WFL aims to mitigate the impact of WFs and improve the generalization performance and explainability of predictive models (Sugiyama & Uchida. 2025). A common approach involves constructing feature estimation models g to esti- mate the true values of WFs (termed exact values) from observed features (ordinary features; OFs) that are not WFs, and then lcarning a label prediction model f using both the estimated values for the WFs and the OFs to predict the labels of downstream tasks. In practice, ItR and CFL employ sequential learning, where g and f are optimized consecutively, or iterative learning, where they are opti- mized alternateyy (Yoon et al., 2018; Mattei & Frellsen, 2019; Le Morvan et al., 2020a; Ipsen et al., 2021; 2022; Zaffran et al., 2023; Sugiyama & Uchida, 2024).
1. Introduction
The performance and explainability of machine learning models are strongly influenced by the quality of input fea- tures. However, in many real-world applications, constraints such as high observation costs, low observation precision,
However, fundamental questions remain unanswered, such as how the interaction between g and influences learning errors and under what conditions these methods can be guar- anteed to be effective. Specifically, these questions can be framed as: (1) How do g and f interact and influence each other's error bound? and (2) What are the precise conditions under which consistency with the optimal hypothesis is the- oretically guaranteed? For WFL with all WFs restricted to discrete (discrete WFL), unified formulation has been
'Major in Computer Science and Communications Engineering, Waseda University, Tokyo, Japan. Correspondence to: Kosuke Sugiyama <kohsukk032226 @ wasegga.assd....p.. Masato Uchida <m.uchida@ @ waseda.jp>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
proposed, offering systematic answers these questions via generalization error analysis within generalized class of learning algorithms (Sugiyama & Uchida, 2025). Discrete WFs include cases such as missing or noisy discrete features, sets that contain the exact discrete value, and observations consisting solely of incorrect discrete values. Discrete WFL is designed to address the presence of such discrete WFs.
ties investigate the theoretical integration of discrete and continuous WFL..
3. We conduct a theoretical investigation into the influence of on f's learning via the generalization error analysis in step (ii) of LAC-cWFL (Section 4.2). Furthermore, our theoretical framework can be combined with the theories of existing methods for step (i) to establish conditions under which sequential learning, executing (i) and (ii) consecu- tively, achieves consistency.
In contrast, the theoretical framework for continuous WFL that uniformly addresses continuous WFs, such as missing or noisy continuous values and interval observations that contain the exact continuous value, is not yet well estab- lished. This is primarily because the theoretical framework for discrete WFL heavily depends on the mathematical prop- erties of discreteness, making it difficult to directly extend to continuous WFL. Although some theoretical analyses exist for ItR applied to missing data, they lack inite-sample gener- alization error analyses for arbitrary distributions and fail to accommodate diverse types of continuous WFs (Josse et al., 2024; Bertsimas et al., 2021; Le Morvan et al., 2020a;b; 2021). Additionally, while cases involving erroneous obser- vations (Ristovski et al., 2010; Hu et al., 2020; Berikov & Litvinenko, 2021) or interval-based observations contain- ing exact values (Cheng al., 2023) have been analyzed in contexts where they are observed as target labels, no analyses have addressed scenarios where such observations serve as input features affecting a downstream task. There- fore, constructing a more general and systematic theoretical framework for continuous WFL remains a critical challenge. Such framework should not only function equivalently to the discrete WFL. framework and provide consistent guaran- tees across various types of WFs, but also adapt flexibly to arbitrary generative distributions of continuous WFs.
4. We Theoretically analyze the influence of f on g's learn- ing in step (ii) of LAC-cWFL (Section 4.3). By integrating the analyses in Contributions 3 and 4, we derive the con- ditions under which iterative learning, alternating between steps (ii) and (iii), achieves consistency.
5. We extend our analysis to derive analogous theoretical results for ItR in scenarios where all features are continuous and could be missing (Appendix B). This extension estab- lishes a comprehensive foundation for the generalization error analysis of ItR with continuous missing features.
Our results are correspond in parallel with those of discrete WFL (Sugiyama & Uchida, 2025). Consequently, the anal- yses in this paper demonstrate that the WFL framework (Sugiyama & Uchida, 2025) serves as universal frame- work, equally applicable to both discrete and continuous WFs.
2. Related work
2.1. Review of Ordinary Supervised Learning
This paper builds on the foundational principles of empirical risk minimization (ERM) in supervised learning (Shalev- Shwartz & Ben-David, 2014; Mohri et al., 2018). which we briefly revisit here. Let the input space be defined as X E TRd and the label space as y E R. Here, d E N represents an input dimension. Denote the random vari- ables representing instances by X and the random vari- able for labels by Y. Their realizations (ar, y) are assumed to independently follow the true distribution pa y) over X x V. The goal of ERM is to learn label prediction model f X' + y E F that minimizes the expected risk with respect to a loss function L: y x y - R4:
Our main contributions in this paper are the following:
1. We propose a risk-based formulation to unify the treat- ment of arbitrary continuous WFs. By employing a novel proof technique distinct from that used in discrete WFL, we theoretically demonstrate that the proposed objective func- tion is valid for modeling the true input-output relationship (Section 3.1). This establishes the theoretical justification of our formulation. Based on this, we introduce the learn- ing algorithm class for continuous WFL (LAC-cWFL), a counterpart to LAC-dWFL. (Sugiyama & Uchida, 2025). LAC-cWFL offers a consistent theoretical framework for continuous WFs and flexibly incorporates three learning steps: (i) Learning g using WFs as weak supervision, (ii) Learning f with g fixed, and (iii) Learning g with f fixed.
R₂( = Ep. (z. (X))), Y)
==())==pp.cer
(2.1)
where F is the hypothesis set of label prediction models. Since p. (xx y) is unknown, ERM approximates R& with the empirical risk computed as an average with finite samples and learns f by minimizing this empirical risk.
2. We derive new inequalities to analyze the interaction between g and f in continuous WFL (Section 4.1). These inequalities quantitatively characterize how the mutual de- pendence between g and / influences learning errors, form- ing foundation for comprchensively understanding WFL algorithms. In Appendix A, we leverage these inequali-
2.2. Discrete WFL
Discrete WFL was formulated as follows (Sugiyama & Uchida, 2025). In WFL, the random variables represent-
2

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
ing an instance X are partitioned into the exact values of WFs, denoted by X", and OFs, denoted by X such that X = (x", X.). The possible value sets of X and X are denoted by x = I sElF X and X re- spectively, where X x X = X, Fw is the number of WFs, and [F-] = [I....,F"),
signals and estimating the exact values. The former ap- proach is implemented using decision-tree-basee methods (Twala et al., 2008; Chen & Guestrin, 2016) or through im- putation with placeholder values that signify missingness (Josse et al., 2024). However, when the presence or absence of missingness is independent of Y. this approach may not contribute meaningfully to predicting Y. In such cases, em- ploying ItR with imputation through a feature estimation model g could become essential.
Notably, x= is a finite set. The observed values of WFs are denoted by the random variables X". Feature estimation models, which estimate the exact values of WFs X",, are defined as g (g),.... ,FFW) E G Gi X ... x GF- : - A²", where Oj E 9 A -
Several theoretical studies have investigated ItR. Josse et al. demonstrated that in regression or binary classification tasks where only one variable is missing and the missing mechanism is Missing at Random, there exists a predictive function / such that f o g serves as the Bayes rule for any constant-value imputation g (Josse et al., 2024) Bertsimas et al., while restricting their analysis regression tasks, demonstrated that for any almost everywhere continuous function g. there exists an I such that f Tog is the Bayes rule (Bertsimas et al., 2021). Morvan et al. extended the work of Bertsimas et al., providing a comprehensive analysis of the Bayes rule under arbitrary missing mechanisms and any measurable g. where all features could potentially be miss- ing (Le Morvan et al., 2021). Morvan et al. also conducted generalization error analysis in the restricted setting where the true regression model is linear (Le Morvan et al., 2020b).
X for all j E [FW]. Here, 9j represents the hypothesis set for estimating Xy.
N
In discrete WFL. where all WFs are discrete, the goal is to minimize the generalization error Ri.g of f. evaluated using the loss function I, and the estimation error Roi, of Jj, measured by the 0-1 loss (Sugiyama & Uchida, 2025):
(f) == (p. (a Ro == p.y [1 los (X*)+x,, Vj E [F*]. (2.3)
X"), Y),,
(2.2)
Minimizing these risks aims to achieve a highly accurate and improve the explainability of f by mitigating inac- curacies or ambiguities in the input features. The objective function of discrete WFL. is defined as the weighted sum of these risks using a weighting parameter X E R_:
R = R (J) + AIE ( (gy) (2.4) 1
Despite these advances, the aforementioned studies did not investigate learning algorithms capable of asymptotically obtaining the optimal f o g. Moreover, generalization error analysis under finite sample regimes, applicable to arbitrary data distributions, has yet to be unachieved. In this study, we conduct finite-sample generalization error analysis of ItR for any downstream task. This analysis encompasses situations where any continuous feature can be missing, for any measurable g. Through this generalization error analysis, we further investigate the conditions under which sequential or iterative learning methods for f and g achieve consistency. The scenario where only some features may be missing is addressed in Sections 3 and 4, while the case where all features are missing is discussed in Appendix B.
It has been shown that for any / bounded by U²- < 0o, E and g E G, the inequality (f) < R@WFF (f.g) holds (Theorem 3.1 in (Sugiyama & Uchida, 2025)). This result showed that minimizing R contributes to obtaining an f that captures the true input-output relationship, thereby justifying the formulation. This work builds on (Sugiyama & Uchida, 2025) further conducted a theoretical analysis of generalized learning algorithm class for discrete WFL under the formulation.
However, this analysis expresses the estimation errors of g using the 0-1 loss, rendering the same proof strategy inap- plicable to continuous WFL. In this study, we construct a theoretical framework for continuous WFL that accounts for the unique characteristies of continuous WFs. Specifi- cally, we represent the estimation errors of g using the mean squared error (MSE) instead of the 0-1 loss, thereby estab- lishing a theoretical framework for continuous WFL. that parallels that of discrete WFL.
3. Formulation
3.1. Formulation of continuous WFL
In this section, we present the formulation of continuous WFL from the perspective of risk minimization. Most no- tations remain consistent with Section 2.2 but two key differences exist. First, in discrete WFL, 2' is finite set, whereas in continuous WFL, it is defined as X C FR". Next, we define the probability density function (PDF) asso- ciated with the feature estimation models g as gg (x (=²). The PDF dg will later be utilized as a probabilistic model derived from the deterministic model g. Similar discrete
2.3. Theoretical Analysis for ItR
ItR is a framework that imputes missing values in input features and subsequently performs predictions using the imputed data (Le Morvan et al., 2021). Approaches for handling missing values are generally categorized into two strategies: leveraging the absence of values as informative
3

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
WFL (Sugiyama & Uchida, 2025), focus on binary clas- sification the downstream task; but, formulation and analysis can be readily extended to other downstream tasks without losing generality.
Theorem 3.1. Let a R4, and define 4g as the PDF of N (gg(a)), o2 xF") For any measurable I E F. E and I bounded by Ui co, the following holds:
R(( -
The main objectives of WFL are twofold: improving the gen- eralization performance in a downstream task and restoring the explainability lost due to WFs. The loss of explainabil- ity stems from the imprecision of information encoded in WFs, X". A natural approach to address this issue is to accurately estimate the exact values of WF', X". There- fore, in continuous WFL, it is natural to learn f and g by minimizing the following two risks. The first risk evaluates the generalization error of f:
(3.8)
Rf.g()) + U.fCo + 20
1 Eje(pu) R
Here, Co =
(p.(aayllogpp (X"| Y)] +
The definition of Og corresponds to adding Gaussian noise with mean 0 and variance 0² to the output of g. This definition imposes no constraints on the learning of g, thus preserving the generality of our results. According to the definition of Co, when for any j E [F"), finite-preciss precision decimals (in this case, (x Dl p. (X" X Y)] I 0 holds), and dg is defined with 0² = (2x) -1 it follows that C, S O. Thus, under this mild assumption, Co can be disregarded. The LHS of Eq. (3.8) quantifies how well / captures the true input-output relationship. Thus, Theorem 3.1 establishes that minimizing the two components of our objective function ensures that f effectively captures the true input- information.
log / (2
(f) = Ey_ (a w) (-m mr) (X((), -Bp. (m a) xx) Y)
(3.5)
2T" is a set of finite
The second risk measures the estimation errors of g:
RASEE (0)) == Ep. (ax) [ (X"),
["],
(3.6)
where (MS y/) = (y -²²² represents the mean squared error (MSE). The objective function for continuous WFL is defined as a linear combination of these two risks using a weight parameter À E R4:
3.2. Learning Algorithm Class for continuous WFL
In this section, we introduce a unified learning algorithm class capable of handling not only arbitrary continuous WFs but also various methods continuous WFL. Under the proposed formulation in Section 3.1, we define the following learning algorithm class:
ReWFL (f.g) = (f) REX
RASE. (gg)) (3.7) SE[FMM
Σ
The objective function R:WFL, facilitates the unified treat- ment of any continuous WFs. This is because the error of 9j is defined through the risk RMSE. to be minimized irre- spective of the type of WF. For various types of continuous WFs, weakly supervised learning methods that minimize RASE, using Xy as weak supervision have already been established (Wasserman & Lafferty, 2007: Li et al., 2017: Kostopoulos et al., 2018; Ristovski et al., 2010; Hu et al., 2020; Berikov & Litvinenko, 2021; Cheng et al., 2023). Therefore, to minimize REWFF, these WSL methods can be employed to learn Oj based on the type of WF.
Definition 3.2 (LAC-cWFL). The learning algorithm class for continuous WFL (LAC-cWFL) refers to the set of al- gorithms that learn the feature estimation models g and label prediction model f through any combination of the following three steps:
(i) Learning g by using X as weak supervision and min- imizing Ej{(r)) RMSE. either directly or indirectly.
Fixing g and learning f by minimizing Ri
(iii) Fixing f and learning g by minimizing RoWFL
Moreover, HI WFL. can be interpreted as a natural formula- tion, where Rot,j replaced by RMSE. in RdWFI. defined in Eq. (2.4). However, justifying this formulation for con- tinuous WFL is non-trivial. The justification for discrete WFL relies on the inequality derived using the properties of Rot, (Theorem 3.1 in (Sugiyama & Uchida, 2025)), which cannot be directly applied to continuous WFL. In the case of discrete WFL, the key approach involved expressing the upper bound of (f) - Ri.g (J) () in terms of Roi,( (gx). In this work, we successfully derive the upper bound of Rf( - Ri.g (f), expressed in terms of RMSE, (g;). The validity of our formulation is demonstrated by the following theorem. The proof can be found in Appendix C.I.
LAC-eWFL acts as a continuous counterpart to LAC-WWFL (Sugiyama & Uchida, 2025). Similar to LAC-dWFL, LAC- cWFL unifies a broad range of methods, such as sequential learning (comprising steps (i) and (ii) and iterative learning (comprising steps (ii) and (iii) (Yoon al., 2018; Mattei & Frellsen, 2019; Ipsen et al., 2021; Josse et al., 2024: Le Mor- van et al., 2020a; 2021; Ipsen et al., 2022; Sugiyama & Uchida, 2025). Furthermore, various weakly supervised learning methods can be applied to step (i) by utilizing
X w as weak supervision (Wasserman & Lafferty, 2007; Li et al., 2017; Kostopoulos al., 2018; Ristovski et al., 2010;
4

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Hu al., 2020; Berikov & Litvinenko, 2021; Cheng al., 2023). In Section 4, perform a generalization error anal- ysis of this unified class of learning algorithms, revealing the mutual influences between g and f during learning and the conditions under which WFL attains optimal models.
specific case of Lemma 4.1. The proof is provided in Appendix C.3.
Lemma 4.2, Let 9g be the PDF of N(g(2")) I xx=x where 0² E R... For any measurable f E F. E 9 and bounded by Ut < oo, the following inequality holds:
4. Theoretical analysis
[Ri(]) - -u ( (V + v (T)
4.1. Construction of an Analytical Tool
(4.10
x 22ui(Co + 20 EjE(o-) RMSE)( }
The formulation introduced in Section 3 and LAC-cWFL correspond to those of discrete WFL (Sugiyama & Uchida, 2025). However, determining whether similar theoretical analyses can be applied to them, as in the case of discrete WFL, is not straightforwwrd. This difficulty arises because the analytical method for discrete WFL is derived using the properties of Roi.j. and the method cannot be applied directly to continuous WFL. In discrete WFL, deriving an in- g f in terms of (gş) and RJ) enabled discussions on how g influences the learning of f and vice versa (Lemma 4.1 in (Sugiyama& & Uchida, 2025)). Therefore, for con- tinuous WFL, it is essential to identify an inequality that expresses the upper bound of [R( - [f) in terms of RMSE., (g3) and (f). The challenge is distinct from Theorem 3.1, the upper bound must explicitly depend Rf) Equation (3.8) in Theorem 3.1 does not express this dependency.
Equation (4.10) of Lemma 4.2 corresponds to the inequal- ity in discrete WFL. which expresses the upper bound of [f) - Rv. (f) in terms of Rot,; and (f) (E..64.6) in (Sugiyama & Uchida, 2025)). This parallel suggests that Eq. (4.10) provide a sufficient foundation for deriving anal- ogous analyses to those discrete WFI.. Using Lemma 4.2, we theoretically elucidate the mutual influence of g and f in LAC-eWFL. In Section 4.2, we analyze the learning of ʃ under LAC-eWFL's step (ii), whereas in Section 4.3, we focus on the learning of g under the step (iii).
equality that expresses the upper bound of
4.2. Analysis of Learning Label Prediction Model f
This section examines the learning of ʃ in LAC-eWFL's step (ii). As f in WFL utilizes the output of g as input, the learning of f inherently depends on We theoretically investigate how the estimation errors of g influence the learning of ʃ.
We derive an inequality that resolves this issue. First, we establish the following lemma, which is valid within a more general framework that is independent of the definition of 0g !. The proof is provided in Appendix C.2.
For theoretical analysis, we introduce the following defini- tions. Let S == {(x,, y)) denote ordinary dataset and 3 = {aa) a?, yi) weak dataset, where both datasets contain n E N4 samples, and the i-th sample in S and S corresponds to the same instance, for any E [m)]. Let Rú and Ri denote the empirical risks computed by sample average over S and 5, respectively. For any E G, the empirical risk minimizer of LAC-cWFL's step (ii) is defined as follows:
Lemma 4.1. For any measurable I E F, (g(x (x²" and bounded by U1= < oo, the following inequality holds:
[R((f) - [f) S (VRi(f) + (R..o()) x (2U (p X g) (X (X
(4.9)
Here, Du denotes the Hellinger distance.
Although Lemma 4.1 has a broad scope, one of the our goals is analyzing the relationship between the estima- tion errors of g. represented by Ej{(yn] (Mg.. and f. with the goal of constructing a theory that parallels discrete WFL. To this end, we define (g as the PDF of W(g(z"), I Ig=x and derive the following lemma as
fg.s== := minsi (f).
Using Lemma 4.2, the error bound for fg.s learned in LAC- cWFL's step (ii) is established in the following theorem. The proof provided in Appendix C.4.
'Lemma 4.1 is valid for both discrete and continuous WF', and it remains applicable even when the feature estimation models are probabilistic. However, this lemma cannot directly be used to derive the results of discrete WFL presented in (Sugiyama & Uchida, 2025). In contrast, the lemma has the potential to enable unified analysis of discrete WFL and continuous WFL. This direction explored in Appendix
Theorem 4.3. Suppose S and S are ordinary and weak datasets consisting of n samples, respectively. Let qg be the PDF of (gxx))), 0,2 (II==xF)) where o² E RR.. For any measurable g E G. Li-Lipschitz continuous bounded by Ul < 00 and 8 E (0, 1), the following inequality holds with
5

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
probability at least 6:
bination demonstrates that learning g improves the error bound's order from Op(1/n// to (2(1/(m)))//)) where m S is the number of samples available for learning g. Consequently, learning g improves convergence rates.
- 1 (F) - (F + V
log(4/8)
2n
Additionally, this combination enables a consistent analysis of sequential learning as represented by LAC-<WFL's steps (i) and (ii), thereby elucidating conditions for achieving consistency as follows theorem. This theorem shows that sequential learning is asymptotically sufficient under these conditions. The proof is provided in Appendix C.5.
(2(Ril(=) + (T, 2 2n
log(2/6)
(4.11)
(2ui(Co +
1 Eyj RASE $1
20
x {220 Co +
1. EjE(EM) RMSE, (0:)
20
Theorem 4.4. Suppose that X is set of finite- preiti--reeiioon decimals, for any j E [FW]. Assume the existence of true de- terministic functions 93 : x - xx," exist for any j E {,"*,, such that (gi...., -T- E g and f" : :X + +Y such that J F. Furthermore, assume that bounded by Ut- < oo is La-Lipschitz continuous, that R2, and Rq, (F) asymp- totically converge to O as n - 30, and that for all j E 1,*,,, the number of samples available for learning 9j tends in- finity as - co. Then, using consistent methods for learn- ing g and settings o² = =(x) for 9g. sequential learn- ing under continuous WFL is consistent (i.e. as n + oc, Ri (f + +R(fF)).
where fF == arg fer RJ( represents the true risk minimizer in ordinary supervised learning.
Here, (F) and Rg (F) denote the Rademacher com- plexities associated with the distributions p. (2:) and p.(x (2g(2"12²) respectively, and represent the complex- ity of F. For kernel ridge regression and multilayer per- ceptrons, the order of the Rademacher complexity' 's upper bound is (2)n (Mohri et al., 2018; 8; Neyshabur et al., 2015). the subsequent discussions, R,, (F) and ( (F) are assumed to have this order. Additionally, we assume that F is sufficiently expressive and that [fx O. Given that XT is a set of finite decimal numbers, for any j E [FW] and o² (2T) -1 follows that C. S 0. Therefore, under this mild assumption, C, will be ignored in the subsequent discussion.
4.3. Analysis of Learning Feature Estimation Models g
This section examines the learning of g in the context of LAC-cWFL's step (iii). We aim to elucidate how the gen- eralization performance and properties of f influence the learning of g when the objective function 12;WEL is mini- mized with f fixed.
Theorem 4.3 establishes an error bound for learning f un- der continuous WFL, which parallels the result for discrete WFL (Theorem 4.2 in (Sugiyama & Uchida, 2025)). This demonstrates that our framework successfully achieved re- sults for learning f in continuous WFL, analogous to those in discrete WFL. The main contributions of Theorem 4.3 are twofold:
LAC-eWFL's step (iii) focuses on minimizing the two terms on the RHS of Eq. (3.7) with respect to g. This requires a distinct analytical approach compared to the single-risk minimization in Section 4.2. In the analysis of discrete WFL (Sugiyama & Uchida, 2025), structural risk minimization (SRM) was employed to handle the minimization of mul- tiple terms (Mohri et al., 2018). To develop a theoretical framework of continuous WFL that corresponds in parallel with that of discrete WFL, we analyze the learning of using SRM.
The first contribution of Theorem 4.3 lies in elucidating how the estimation errors of fixed g, measured by the MSF, influence the convergence rate of f's generalization error RI,g (f) with respect to the number of training samples n. Specifically, the orders of the first and second terms on the RHS of Eq. (4.11) are (p(1/n///) and Op(1//'/// respectively. Therefore, when ]EE[F>] RMSF,; (95) ) is large, the second term, which decreases more slowly, becomes dominant.
As a preparation, for any j E [FD), we intro- duce the following definitions according to the theory of discrete WFL (Sugiyama & Uchida, 2025). Let L denote the loss function for gj. calculated using Xy. Define the datasets S = {(E,, " Let R1, (9)) == (a.))) (4/(1) (g, Xx))) represent the expected risk of gj computed using Xy and Ry repre- sent the empirical risk, which approximates R, by taking the sample average over 3y. We assume that Ri, satisfies either Roi, (0,) R,, (0,) or Roi, S Ri, (gz) for any gj. or that the optimal solutions of Rr, equal those of Roi..
The second contribution of Theorem 4.3 lies in its ability to combine Eq. 14. with the theoretical bounds for learning g in LAC-cWFL's step (i), thereby enabling an analysis of the effect of g learning on f's learning and the asymptotic properties of sequential learning. For example, in the case of WFs with missing data, if 03 is obtained as an empirical risk minimizer for RMSE,J, an error bound of RMSE, (0,) can be derived (c.g., Theorem 11.3 in (Mohri et al., 2018)) and subsequently incorporated into Eq. (4.11). This com-
6

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
For any (ri (ris..,, (pp) Rf" define the following 2018):
hypothesis set:
RMSEJ (g. I 80 MUsse Ry,
(93) + 2U M/SSS V
log(1/8)
2n1
- (7.5))9((11.)) 51) x ... x gF" (rpw, 3pw),
(4.14)
where, (,(r,,,,) = f9ylo; E 9 (1)
where ni denotes the number of training samples used to obtain (ry). Under the assumption that the missing- ness mechanism is MCAR, it follows that n'; -> oc as n + 00. Details are provided in Appendix C.7. Accord- ingly, if (T 5z) is defined as the set of ERM solutions obtained by such method achieving consistency, and if the order of R; (93) is (p(n/n1/²), then the order of the (rz)) is also Op(1/n2/2). More- 0 as n oo, then it follows that R (2)) + 0 as n + co. In the following discussion, (03))
[F*]. In this section, (f) is expressed as (g)
95.5"
R to emphasize the optimization with respect to g.
The empirical risk of Rg)(gg) is described as (g)
(f)). Using these definitions, the learning of g in LAC-
cWFL's step (iii) is expressed as follows:
(r) f.3
arg min (g) gE)(r,5)
(4.12)
upper bound of RMSE. (9..)
over, if R (G))
By applying Lemma 4.2, we derive the following error bound for (r) The proof is provided in Appendix C.6.
we assume that RMSE. (r5)) can be upper bounded by a probability inequality of the same form as 14.
f,5
5.
Theorem 4.5. Suppose S and S are ordinary and weak datasets consisting n samples, respectively. Let qg be the PDF of (g(x)), o² Ip-xx)),, where 0² E R4. For any measurable label prediction model J E F, I bounded by Ui= < co and d E (0, 1), the following inequality holds with probability at least 1- --:; 8:
Similar to the analysis of f's learning in Section 4.2, Theo- rem 4.5 yields results that closely align with those of discrete WFL (Theorem 4.4 in (Sugiyama & Uchida, 2025)). This confirms that the analysis of g learning in LAC-cWFL's step (iii) aligns with its counterpart in discrete WFL.
R() S Re,lo 535
(1)) - (
Theorem 4.5 elucidates how the generalization performance and properties of f affect the learning of g in continuous WFL. Equation (4.13) expresses the effect of f's general- ization error (f) on the convergence rate of Ru, 's error bound with n training samples. Specifically, the orders of the first and second terms on the RHS of Eq. (4.13) are (1(2////)) and Op (1/n2//)), respectively. Therefore, as Rf( increases, the slower-decreasing second term domi- nates the reduction rate. Although (f) affects the learning of g and cannot be minimized directly. Theorem 3.1 demon- strates that the two terms in Rg as defined in Eq. (3.7), contribute to the minimization (f). Consequently, a re- duction in (f) can be expected as a result of the iterative learning process.
(49R) (G. 3)) + 20
log(2/8) 2n
+ +(2 Ri(f)
RMSE [055 (r2))) $1 }
+ (2ti(Co+ + 202 Eyelon]
x x2Ui(Co
1
Σ
RMSE, ( 1)}
202
jE[FM]
(4.13)
Here, Sp(r = arg min, (r,,(r,,33) (9;) and (r, = {(x",)) + I(f(g(x)), (,0()))))))))))))))))))))))))))) x,),)) : g E ((r,
Furthermore, Theorems 4.3 and 4.5 reveal the conditions un- der which the iterative learning composed of LAC-cWFL's steps (ii) and (iii) achieves consistency. The proof is pro- vided in Appendix C.S.
The term RMSE,, (o in Eq. (4.13) can be further upper- bounded by defining (r(r,. 3,) as the set of empirical risk minimizers obtained through learning method us- ing Sy- Consider, for instance, the case where Xy de- notes features with missing values, and the missingness mechanism is missing completely at random (MCAR). As- sume further that Gj is sufficiently expressive such that mingres, RASSE,(9)) = O, and that y is a bounded inter- val such that IMSE < ULMEE- Suppose that Gy(r,, 3,) is defined as the set of empirical risk minimizers obtained us- ing only the samples for which Xy is observed, where these samples are independently and identically distributed. Then, (r5)) can be upper bounded
Theorem 4.6. In addition to the conditions stated in Theo- rem 4.4, assume that I obtained in LAC-cWFL'ss step (ii) is Lipschitz continuous. Furthermore, for any E [FT define Gy (r,)55)) as the set of empirical risk minimizers obtained by methods that use 5, and are guaranteed to achieve con- sistency. Moreover, suppose the Rademacher complexities about g converge to O as n + xc. Under these conditions, the iterative learning composed of LAC-cWFL's steps (i) and (ii)) achieves consistency.
for any 5 E (0, 1). RM $.j with probability at least 8, as follows (Mohri et al.,
In the preceding discussion, for the sake of simplicity, the
7

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
variance o² of 4g was defined as a constant independent of x. In contrast, it is also possible to define this variance function (x*)). In this case, for Theorem 3.1, Lemma 4.2, Theorem 4.3, and Theorem 4.5, the inequalities presented in each result continue to hold when o² in the upper bounds is replaced by max.. (x)).
R+ using the following sampling method to generate the
predicted value sy for an exact value my. where j [F*]:
B Bern(112), where B (0.1), 27 ~ x N(a + BV (1 (1-3)1V,,000 a/aa,
(5.15)
where l E R., In this study, k is set to 2. This approach allows precise control over the estimation errors of g. en- abling an investigation of f's learning results across varying levels of g s errors.
5. Experiments
In Section 4, it was theoretically demonstrated that sequen- tial learning alone is sufficient for continuous WFL.. In this section, we investigate whether the error bound for f. de- rived in Theorem 4.3, can adequately explain the relation- ship between the number of training samples n and the generalization error (g,و in practical tasks. We adopt the experimental methodology of discrete WFL (Sugiyama & Uchida, 2025) to examine the relationship between them across various estimation errors for g.
Figure 5.1 illustrates the relationship between the number of training samples n and the generalization error Rt,g (f) when f trained with g of varying estimation errors. Here, the MSE of is controlled within the range [0.001, 0.5). The results indicate that lower estimation error for result in a faster decrease in Bi as n increases.
Additionally, Figure 5.2 compares the observed general- ization error Ri.g( (f)) with the error bound in Theorem 4.3. From Figure 5.2, the decrease in g and the error bound shows similar behavior, with greater reductions when the MSE of g is smaller. This indicates that Theorem 4.3 can well explain the relationship between g's MSE and [ Discrepancies between Ri, s) and the er- ror bound can be caused by the fact that the error bound does not account for the importance of WFs in predicting Y. as in the case of discrete WFL (Sugiyama & Uchida, 2025). While our derived bound cannot account for the feature importance of WFs, it captures fundamental charac- teristics common across diverse scenarios. Additionally, our results are considered to provide an important foundation for deriving error bounds that reflect the feature importance of WFs.
5.1. Experimental Settings
For real-world datasets, we utilize four datasets from OpenML (Vanschoren et al., 2013): als4mi_Jhc.jett Jilf (Pierini et al., 2020). electricity (Harries et al., 2014), my (Luis, 2014), and Run_or_wald informmaoootnnonn (Viktor, 2017). We refer to them as Jets, Electricity, Mv, and Run-or- Waal,,,Walk, respectively. Details of these datasets are summarized in Appendix F.1. Half of the samples in each dataset were used as test data to estimate the expected risk. This experiment focused on scenarios involving OFs alongside continuous features with missing values as WFs. Here, the missing mechanism was assumed to be Missing Completely At Ran- dom, with a missing rate of 50% for any WFs. For and f. we used two-layer perceptrons with hidden layers of width 500 and ReLU as an activation function, following the exper- imental setup of discrete WFL (Sugiyama & Uchida, 2025). Logistic loss is employed as l. Details of the experimental settings are summarized in Appendix F.1. The experimental results shown below are the average of 5 trials. The experi- mental seripts used in this paper are available at the follow- ing URL: http://ggguuuboooo/KSSSEPMPoo///
g.
6. Conclusion
In this paper, we presented a unified framework for the formulation and thcoretical analysis of continuous WFL. First, we proposed a formulation of continuous WFL that is capable of handling arbitrary continuous WFs. We val- idated this formulation by demonstrating that minimizing the proposed objective function helps to the label predic- tion model f to capture the true onput-output relationships. Under this formulation, we conducted generalization error analyses for LAC-cWFL, which encompasses both the fea- ture estimation models g and f. Through this analysis, we provided a detailed understanding of how g and f affect each other's error bound in finite-sample settings. Further- more, we identified the conditions under which sequential and iterative learning strategies within LAC-CWFL. achieve consistency, Finally, through numerical experiments using real-world datasets, we confirmed that our theoretical find- ings align well with the actual learning behavior. The results of this study are shown to correspond parallelly with the
WTL
5.2. Impact of g on f's learning
We examines whether Theorem 4.3 can explain the learning behavior of f with g fixed in real-world data. To verify this, g must be created with several estimation errors. However, precisely controlling estimation emrors of g via learning al- gorithms is challenging. In discrete WFL, since WFs are discrete, this issue was addressed by creating randomly in- correct models with certain error rates (Sugiyama& & Uchida, 2025). In contrast, continuous WFL. requires controlling the MSE of g, making the same approach infeasible.
To address this limitation, we construct 9, with MSE a e
8

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
x100
Jeta
X10
Electricity
X1C
X10-
Ron-or-Walk
6,5
6
Ramn, (6) -0,00 Y) [F*]
(8) -0000 [+
6.6-
Rass, -0.01 4j (M
5-
/a, -0.6 vj [
Nost, -1.1 ky [F"]
Rask (5,) =0.3 kg [F1]
8
Muss, (a)) -0.3/3 6 [F*]
6.0
Ranz, -0.4%) [F*]
5-
Muse, (9,) -0. by [F*]
0.8
01
03
0.5
0.7
0.0
111
1.3 1.5 x10
0.1
03
0.5
0.7
0.0
L1
133
1.5 X10
0.1
0.3
0.5
0.7
0.0
1.1
1.5
1.5
0.1
0.3
0.5
0.7
0.9
1.5
X10
Figure 5.1.
The relationship between R and the training data size for various MSEs of g.
Jets
Electricity
Mv
Run-or- Walk
10
0 mondu Sondunnnnnoooomm DO0D
Re Barifall (Ruse -0.005 We (F7))
7 Round (Razn.))) (o,) -0.00 vj [PM])
Burifga aa (Rase (0y) -0.00
(F7))
(Bern, (6)) snion to (871)
11p
Batifg (Rass /oy) -i1. by (17D)
Boand (hose, (0)) m0.1 Vy (177))
10
107
Bound (Ruse, (ot) -0.5 Vj [P*T)
Rasifo a) (Rassale)) -0.3 e (F71)
000000
0000000
00200 88
Rouad (Rusa, (o,) -0.5 Vj [P*])
Ratifo () (Rassuloy -0.5 V/ (571)
0.5 0.5
0.7
0.5
0.1
<10
X30*
Figure 5.2. A comparison between (.g,5) for various MSEs of and the error bound derived in Theorem 4.3.
findings of discrete WFL. which addresses discrete WFs. Thus, our findings demonstrate that the WFL framework operates universally for both discrete and continuous WFs, offering theoretical support for wide range of application domains where low-quality features are prevalent.
A Acknovledgements
This work was supported in part by the Japan Society for the Promotion of Science through Grants-in-Aid for Scientific Research (C) (23K11111).
References
Impact Statement
Berikov, V. and Litvinenko, A. Weakly supervised regres- sion using manifold regularization and low-rank matrix representation. In Mathematical Optimization Theory and Operations Research: 20th International Conference, MOTOR 2021, Irkutsk, Russia, July 5-10, 2021, Proceed- ings 20, pp. 447-461. Springer, 2021.
Our paper presents a formalization and theoretical analysis of continuous WFL. that accommodates arbitrary continu- ous WFs. The relationship revealed in this study between low-quality continuous features and the predictive perfor- mance in downstream tasks is also relevant to discussions on the safety of machine learning. For example, our find- ings may inform questions such as: what kinds of WFs tend to induce models with socially undesirable bias, or which WFs contribute to model vulnerabilities. The theoretical re- sults presented in this work considered to offer critical foundation for such discussions.
Bertsimas, D., Delarue, A., and Pauphilet, J. Prediction with missing data. stat, 1050:7, 2021.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boost- ing system. In Proceedings of the 22nd ACM SIGKDD

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
International Conference on Knowledge Discovery and Data Mining, KDD '6, pp. 785-794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1 00.1149999967222999975
Twenty Third International Conference on Artificial In- telligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 3165-3174, PMLR, 26- 28 Aug 2020b.
Cheng, X., Cao, Y., Li, X., An, B., and Feng, L. Weakly supervised regression with interval targets. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 5428-5448. PMLR, 23-29 Jul 2023.
Le Morvan, M., Josse, J., Scornet, E.,, and Varoquaux, G. What's good imputation to predict with missing values? In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Informa- tion Processing Systems, volume 34, pp. 1 11530- 11540. Curran Associates, Inc., 2021.
Li, Y.-F. Zha, H.-W., and Zhou, Z.-H. Learning safe predic- tion for semi-supervised regression. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 1..09/aai..3lil.10885..
Harries, M., Gama, J., and Bifet, A. electricity. OpenMI., 2014.
Hu, W., Li, Z., and Yu, D. Simple and effective regular- ization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations, 2020.
Luis, T. mv. OpenMI., 2014.
Mattei, P.-A. and Frellsen, MIWAE: Deep generative modelling and imputation of incomplete data sets. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 41144441444444 4423. PMLR, 09-15 Jun 2019.
Ipsen, N., Mattei, P.-A., and Frellsen, J. How to deal with missing data in supervised deep learning? In Pro- ceedings of 2022 International Conference on Learn- ing Representations, 2022. URL https: / /iclr..... /c/ Con Conferences/ 2022. 10th International Conference on Learning Representations, ICLR 2022: Conference date: 25-04-2022 Through 29-04-2022.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of Machine Learning. The MIT Press, 2nd edition, 2018. ISBN 0262039400.
Ipsen, N. B., Mattei, P.-A., and Frellsen, J. not-(miwae): Deep generative modelling with missing not at random data. In International Conference on Learning Represen- tations, 2021.
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based ca- pacity control in neural networks. In Grünwald, P., Hazan, E., and Kale, S. (eds.), Proceedings of The 28th Confer- ence on Learning Theory. volume 40 of Proceedings of Machine Learning Research, pp. 1306-1401, Paris, France, 03-06 Jul 2015. PMLR.
Josse, J., Chen, J. M., Prost, N., Varoquaux, G.,, and Scornet, E. On the consistency of supervised learning with missing values. Statistical Pupers, 65(9):5447-5479) 2024.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Rep- resentations, 12 2014.
Pierini, M., Duarte, J. M., Tran, N., and Freytsis, M. HLS4ML LHC Jet dataset (150 particles). OpenML, 2020. DOI: https://doi.org/10.55 1ttp://00///2emooo36666 /22oodd.30000..
Kostopoulos. G., Karlos, S., Kotsiantis, S., and Ragos, O. Semi-supervised regression: A recent review. Journal of Intelligent & Fuzzy Systems, 35(2):1483-1555 2018
Ristovski. K., Das, D., Ouzienko, V., Guo, Y., and
Obradovic, Z. Regression learning with multiple noisy oracles. In ECAI 2010, pp. 445-450. IOS Press, 2010.
Le Morvan, M., Josse, J., Moreau, T., Scornet, E., and Varoquaux, G. Neumiss networks: differentiable pro- gramming for supervised learning with missing values. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. pppppp..9..... 5980-5990. Curran As- sociates, Inc., 2020a.
Shalev-Shwartz, S. and Ben-David, S. Understanding mu- chine learning: From theory to algorithms. Cambridge university press, 2014.
Sugiyama, K. and Uchida, M. Learning from complemen- tary features. arXiv preprint arXiv:2408. 14788, 2024.
Le Morvan, M., Prost, N., Josse. J.. Scornet, E., and Varo- data
Sugiyama, K. and Uchida, M. A unified framework for generalization error analysis of learning with arbitrary discrete weak features. In Proceedings of the 42nd Inter- national Conference on Machine Learning, 2025.
quaux, G. Linear predictor on with missing values: non consistency and solutions. In Chiappa, and Calandra, R. (eds.), Proceedings of the
10

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Twala, B., Jones, M., and Hand, D. Good methods for cop- ing with missing data in decision trees. Pattern Recogni- tion Letters, 29(7):50--55, 2008. ISSN 0167-8655. doi:
https://oii.rr// https://bb.orr//00000////pppeeee 00t1:00000.0......00000000000000
Vanschoren, van Rijn, N., Bischl, B., and Torgo, L. Openml: networked science in machine learn- ing. SIGKDD Explorations, 15(2):99-00, 2013. doi:
Viktor, M. run-or-walk. OpenML, 2017.
Wasserman, L. and Lafferty, J. Statistical analysis of semi- supervised regression. In Platt, J., Koller, D., Singer, Y., and Roweis, S. (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Yoon, J., Jordon, J., and van der Schaar, M. GAIN: Missing data imputation using generative adversarial nets. Dy, and Krause, A. (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5689- 5698. PMLR, 10-15 Jul 2018.
Zaffran, M., Dieuleveut, A., Josse, J., and Romano, Y. Conformal prediction with missing values. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 40578- 40604. PMLR, 23-29 Jul 2023.

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
A. Considerations for the theoretical integration of discrete WFL and continuos WFL
The results presented in the main body of this paper provide a unified theoretical foundation for WFL in a wide range of scenarios where some features are continuous WFs. Combined with the analysis results from discrete WFL (Sugiyama & Uchida, 2025), our findings clarify numerous theoretical properties of WFL across various problem settings. However, these frameworks do not address situations where discrete and continuous WFs coexist. This limitation arises because discrete WFL represents the estimation errors of the feature estimation models g using the 0-1 loss, whereas continuous WFL expresses them using the mean squared error (MSE).
On the other hand, Lemma 1, which is fundamental to the results in this paper, is applicable regardless of the nature of the WFs (discrete or continuous). Hence, this section, we explore a methodology for extending Lemma 4. to enable an analysis of an integrated WFL framework that unifies discrete and continuous WFL. We aim to explore how the theoretical framework established for discrete WFL and continuous WFL can be extended the broader case of integrated WFL, which encompasses arbitrary WFs regardless of whether they discrete continuous.
Most of the notations used here are consistent with those in the main body of this paper. By contrast, since Lemma 4.1 assumes that the feature estimation model is probabilistic model, we define the feature estimation model as probabilistic model (z") in this section. Thus, the discussion in this section can also be interpreted as a generalization of the analysis of discrete WFL and continuous WFL with respect to feature estimation models, which were previously restricted to deterministic models.
A.1. Formulation of Integrated WFL
In this section, we formulate integrated WFL. based on risk minimization. integrated WFL., the objective is to improve the generalization performance of downstream tasks and restore the explainability lost due to the presence of WFs by accurately estimating the exact values of WFs using probabilistic model q. To achieve this, we aim to learn a label prediction model f and feature estimation model q that minimize the following two objectives. The first objective is the generalization error of f, evaluated by the following risk:
R1 E... (wo v)((=> (e>) ((S((),) Y)]-
(A.16)
The second objective is the estimation error of q, evaluated by the following Hellinger divergence:
Ex (m a) (p. (X X Y), 0).4(2 (x" [
(A.17)
Finally, the objective function for integrated WFL is defined as a linear combination of these two terms:
RIWEL (f.g) == (f) + (Ep [p X Y). ()) X*)], X E
(A.18)
The objective function RL is justified by the following theorem, the proof of which can be found in Appendix D.1. Theorem A.1. For any measurable J E F, probubilistic model q(=="(/r and bounded by Ub the following inequality holds:
B(()) < (f) + (DH X), (x))).
(A.19)
Theorem A. demonstrates that minimizing the two terms defining R,WFL contributes to construction of f that captures the true input-outpur relationship. This result validates the formulation of integrated WFL in manner consistent with the discrete and continuous WFL frameworks, affirming the justification of the objective function under this formulation.
Next, we define a generalized class of learning algorithms for integrated WFL, analogous to LAC-dWFL (Sugiyama & Uchida, 2025) and LAC-cWFL,, as follows:
Definition A.2 (LAC-iWFL). The learning algorithm class for integrated WFL (LAC-IWFL) refers to the class of learning algorithms that learn the feature estimation model q and the label prediction model f in integrated WFL through any combination of the following three steps: (i) Learning by directly or indirectly minimizing the second term on the RHS of Eq. 18). (ii) Learning f with a fixed using Rt,g as the objective function. (ii) Learning with a fixed f using R as the objective function.
The next section analyze the generalization error of LAC-IWFL.
12

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
A.2. Theoretical Analysis for Learning / with fixed !
In this section, we analyze the generalization error when learning f using a fixed q as part of LAC-iWFL's step (ii). For theoretical analysis, we introduce some definitions. Similar to Section 4.2, we define the ordinary dataset S where the i-th sample in S and S correspond to the
{ ,=...n...., and the weak dataset S = {(o"" , (4f, same instance for any E [n]. The empirical risks Rg and R1.g are computed by the sample average over S and S, respectively. For any 9, the empirical risk minimizer obtained from LAC-iWEL's step (ii) is defined follows:
a :- arg minser R(o(s)-
Based on Lemma 4,1, the generalization error bound for fq.3, learned through LAC-WWFL's step (ii), is derived in the following theorem. The proof is provided in Appendix D.2.
Theorem A.3. Suppose S and S are ordinary and weak datasets consisting of n samples, respectively. For any measurable E G, L--ipschitz continuous I bounded by Ut < oo and ó E (0, 1). the following inequality holds with probability at least 1 8:
R1.g RL.((5g.5) (g,5) -R((fr)
log log(4/))
$ 4 LiR (F) + (F) +
2n
2 R((f=)+ + 2 log(4/0)
$
B2U (w) XX
x
2n
(Di ( X Y), ,)xx(x(x))))
(A.20)
Here, fy == argg Rf( is the true risk minimizer ordinary supervised learning. Also, R7 and Rg (F) denote the Rademacher complexities calculated using p. p.(ar,)) v) and p. (a",y)g(a") (x²), respectively.
f=
Theorem A.3 allows for the analysis of generalization error in learning f while accounting for the estimation error of the probabilistic feature estimation model in scenarios involving mixed discrete and continuous WFs. This result elucidates the effect of q's estimation error on the error bound for f. We assume that F is sufficiently expressive and that R((fs) = O. Additionally, we assume that the orders of (F) and Rg (F) are Op 2/2). Then, the order of the first term on the RHS of Eq. (A.20) is (p(n while the order of the second term is Op (1/n1/4), Therefore, Eq. (A.20) implies that larger the estimation error in g, as measured by the Hellinger distance, result in the slower-converging second term dominating the error bound, thereby slowing the rate of error reduction.
While Theorem A.3 clarifies the impact of q's estimation error on learning f. constructing a parallel discussion analogous to Section 4.2 remains challenging. This difficulty arises because the Hellinger distance, used to measure g's estimation error, evaluates the divergence between the conditional joint distributions X"I",, , Y and X I without assessing estimation errors for individual WFs. Moreover, to the best of ours' knowledge, there are no theoretically guaranteed methods for learning q from X w as weak supervision that their theories can be combined with Theorem A.3, as in Theorem 4.3. For the reasons outlined above, it is currently challenging to conduct further theoretical analysis of integrated WFL. This limitation also applies to the learning of q with fixed in LAC-iWFL's step (iii).
In contrast, the discussions above can also be extended to scenarios where q's estimation error is measured using the Kullback-Leibler (KL) divergence (-I.). Specifically, integrated WFL can be formulated with the following objective function, allowing for similar discussions:
WFL R.X.LL (S.4) := (f) + (D X Y))
RER
(A.21)
This is because, by utilizing the following inequality, we can derive equivalent inequalities corresponding to Theorem A.I and Theorem A.3, which evaluates the estimation error of q in terms of the KL divergence:
[
(x X Y) log (= X YY
Epp
(we (wr
w)
() & Ep_ (2
xx2" (x))
da
(A.22)
DKI (x X
(x))..
13

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
This suggests that the integrated WFL framework is compatible with methods that learn q as a maximum likelihood estimation problem. However, it remains challenging to express the error bound of in form that depends on divergences individually evaluating the estimation enror for each WF. Addressing these challenges in the theoretical analysis of integrated WFL will be an important direction for future research.
B. Impute-then-Regree
the Case All Features Are Continuous and Can Be Missing
The results presented in the main body of this paper provide a unified theoretical framework for WFL that handles datasets where some features are continuous WFs, clarifying several theoretical properties. However, the analysis in Section 4 has limitation: it cannot accommodate cases where all features are WFs. This limitation arises because the analysis in Section 4 assumes that the exact values of WFs are estimated using OFs. This issue is particularly significant in the context of ItR problems, In real-world scenarios, it is not uncommon for all features to be subject to missingness. Moreover, the latest theoretical studies on ItR assume scenarios where all features are continuous and may be missing (Le Morvan et al., 2021). Therefore, to conduct a comprehensive generalization error analysis for ItR from the perspective of continuous WFL, necessary to extend the analysis Section 4 to include cases where "all features are continuous and may be missing."
In this section, we extend the analysis in Section 4 to conduct finite-sample generalization error analysis for the case where "all features are continuous and may be missing." These results correspond in parallel with those presented in Section 4. The findings of this section, combined with the results of Section 4 and the theoretical analysis of discrete WFL (Sugiyama & Uchida, 2025), establish a unified generalization error analysis framework applicable to a broad range of ItR problem settings.
B.1. Preliminary
To address cases where all features may be missing, we introduce definitions partially different from those in the Main Body, adhering to the definitions of the latest theoretical study on ItR (Le Morvan et al., 2021). Let X = (X1..., denote the sequence of random variables representing the complete input features without missingness, and let Y denote the random variable representing the downstream task label. Here, d represents the dimensionality of the input features. The sets of possible values for X and Y are denoted by X' = II== X2 S Rd and Y, respectively. Define X = (X....., xd)) as the sequence of random variables representing the incomplete input features that may contain missing values. The set of possible values for X is denoted by X I (X, U (NA)). where NA indicates a missing value. Let M = (M, Ma) represent the sequence of random variables indicating the missingness of features. The set of possible values for M {0, 13dd For any j E [d]. M, indicates that X, is observed (i.e., X, X2). while M, (i.e., Xj= NA). By definition, X is uniquely determined by X and M. Additionally, M can be uniquely determined from X. From this fact, omit M when both X and M are arguments. the realizations of X, Y, M,, and X are represented as ar y. m and æ. respectively. The true joint probability distribution of X, Y and M is denoted as p. (x. y, m.). All samples (z, y, m) are assumed to be independently drawn from p. (x, y, m). For a given m. let obs(mn) S (1,..., d} denote the set of indices corresponding to observed features, and mis(m) C {1,.... d} denote the set of indices corresponding to missing features.
indicates that X, is missing
Define the label prediction model as f X + Y,, where F is the hypothesis class for label prediction models. Define the feature impulation models g (g),-.-, :9d) G1 Ga X x (0, 1)dd + X follows:
X; (10) Xobe(M))
if M₁ = 0
9j
(B.23)
if M₁ = 1
where ( Xobe(M) + X2 and Gj represents the hypothesis class of feature imputation models for the j-th feature. The PDF associated with g is denoted by 9g (Fimis(m) (obssmm) m.)
B.2. Formulation
we formalize ItR in the scenario where all features are continuous and potentially missing so that it corresponds in parallel to the discussion in Section 3.1. ItR aims to achieve two main goals: improving the generalization performance of downstream tasks and restoring the interpretability lost due to missing values. Consequently, we aim to learn f and g that minimize the
14

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
following two risks. The first risk evaluates the generalization error of the label prediction model f:
R (f) := (m)))) (tt.aimmmmmlm(((.).ma [(((())) Y)] = (.y.
(B.24)
The second risk evaluates the estimation error of the feature imputation models g under specific missing patterns:
R (gg) = Ep mm) (M, By [X)] Ep. (a)) m) IMSE (X, Pj (m)) (Xohst obs (m)) )) Vj [d].
(B.25)
where (y, 3// = (y 3²²² represents the mean squared error (MSE). Finally, the objective function for ItR with only continuous and potentially missing features is defined as linear combination of these risks:
R (U.9) = ( A.p (m)
:
Σ
RMSE, (g1) À E IR...
(B.26)
(E
Theorem 3.1, which corresponds to the formalization involving OFs and continuous WFs, can generalized to ItR framework. Consequently, the validity of the formalization with the objective function RIR in the current setting is demonstrated as the following theorem. The proof is provided in Appendix E.
Theorem B.1. Let 4g be the PDF of (g mm) ( (ooba(m)): : o² mis(m)) [mmmsmm)))) where o E R.,. For any measurable I E F. g G, and bounded by Uj 20, the following inequality holds:
[1 Ri(f) S R(g(f)) (f
))
C.
202 3 pp
Σ
(MM (g:)
(B.27)
jEmis(M)
where C² = E.. (P= (mmis(M Xobs(Md) Y, M)] + pp. ( log(2 (unis (M) ()mis(
According to the definition of Cy., when for any j E [d]. X, is set of finite-precision decimals (in this case, (a.y.. [P. (X. mimm(M) Xobe((M) M), Y, M)] < holds), and Qg is defined with o² = (2m)-4 it follows that C² < 0. Thus, under this mild assumption, C² can be disregarded.
Next, we define a general class of learning algorithms for ItR, analogous to LAC-cWFL, follows:
Definition B.2 (LAC-ItR). The learning algorithm class for Impute then Regress (LAC-IIR) refers to the set of algorithms for learning the feature imputation model and the label prediction model f in ItR using one or a combination of the following three steps:
Learning g by directly indirectly minimizing the second term the RHS of Eq. (B.26).
iii) Learning f using Rigg as the objective function while fixing g.
(iii) Learning g using L,A as the objective function while fixing f.
Analogous LAC-cWFL, LAC-IIR encompasses wide range of methods, including sequential learning, which involves steps (i) and (ii), and iterative learning. which alternates between steps (ii) and (iii) (Yoon et al., 2018; Mattei & Frellsen, 2019; Ipsen et al., 2021; Josse et al., 2024; Le Morvan et al., 2020a; 2021; Ipsen et al., 2022). In the following sections, we provide a generalization error analysis for this class of learning algorithms.
B.3. Deriving Analysis Inequalities
To conduct a theoretical analysis for ItR involving continuous and potentially missing features, analogous to Section 4, it is necessary to derive results corresponding to Lemmas 4.1 and 4.2. In fact, the proofs of these lemmas can be extended to the problem setting of this section, leading to the following two lemmas. The first lemma assumes the use of an arbitrary probabilistic model for missing value imputation and corresponds to Lemma 4.1 in parallel. The proof is provided in Appendix E.2.
15

---

where E R4, Let S and 5 represent Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Lemma B.3. For any measurable probabilistic model q (m)) ( (obe(m); m)), )) S E F and i bounded by Uj oc, the following inequality holds:
(R((T) Rigg(D
(R(()) + R(ta())
(B.28)
x { 22 (m (mobe(m) wlm)
(Du (miss(Mr) (o) Y,M, 9 (iis((M) My Xabo( (X M))]
When the probabilistic model N (I (m))) o² (mmis(m)) [m]ss(m)]) where is a deterministic feature imputation model and E R+. the second lemma holds. The proof is provided in Appendix E.3.
q (m)) (m)) m)) is defined as the PDF of
I [|mis(m))) where o E Rt. For any measurable Lemma B.4. Let 9g be the PDF of ((gmis(m) (oohs(mm)) [mnis(()) FE E 9 and l bounded by U1 <. co, the following inequality holds:
RaISE (0:))) }
3
[Ri(s) (f) (VR((f) + V (i.o(() 2U1
C² 20
1
(m.)
(m)
Σ
(B.29)
jEmis M)
where, C% = =p. -Fp.. [p- (X mis(M)) XX obs( 7,M + (m))m G log (2tr) (ile( (M) g (1mmls(MM))
Lemma B.4 correspond to Lemma 4.2. While Lemma B.3 is more broadly applicable than Lemma B.4, our analysis, similar to Section 4, focuses on the relationship between the estimation errors of represented by MSE and f. Hence, proceed by using Lemma B with lg set as the PDF of (gmiss(m) (oohs(m)) o I x [mis(m))) This setting does not restrict the learning method for 9, preserving the generality of our analysis for deterministic models g.
B.4. Theoretical Analysis for Learning f with fixed
In this section, we analyze the learning of f by minimizing Ri.g in LAC-ItR's s step (ii) with g fixed. We first assume that samples (ax Vi, m=)))= are independently drawn from (xr m). The ordinary dataset and weak dataset are defined as S = {(,,, y:) ))) and S == {(a,, yr, m. )2 respectively. Here, the i-th samples in and 5 correspond to the same instance, for any i E [n]. Hereafter, Ri and Ri.g denote the empirical risks that approximate R₁ and Ri.g by sample averages based on and 5, respectively. For any E G, the empirical risk minimizer obtained from LAC-ItR's R's step (ii) is defined as follows:
fg. :- arg minger (
From Lemma 4.2, the error bound for fg.3 obtained using LAC-ItR's step (ii) is derived in the following theorem. The proof is provided in Appendix E.4.
Theorem B.S. Let 4g be the PDF of N (Forss(m)))))))))) o² ( the ordinary dataset and weak dataset, respectively, consisting of n samples. For any measurable g E G, La-Lipschitz continuous I bounded by U1 < oo and 8 E (0, 1), the following inequality holds with probability at least 8:
R1.g RL.g gg,zz) -R((sr)
1og(4/6)
(F) + (F) + V.y
2n
}/
+ {2 R((fz) + (1) + 201
(4)))
+
20:
C² +
E.p (m)
Σ
RASE, (0:))
2n
20
jEmis(MM
{ 20 (c=- 1
(0:E, }}
E,
(m))
Σ
R
20"
JE
(B.30)
16

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Here, JF arg mi R(( is the Irue risk minimizer in ordinary supervised learning, and Cy (p.(a..m) [p. mii) obe(M)) Y, M)] + (m))2 log(2x) a log(2))) (M))
According to the definition of Cg when for any j E [d]. is set of finite-precisie decimals (In this case, (a..m) (p. (X mis(M) X obs(MM) Y, M)] I O holds), and 9g is defined with x² = (2x)=², it follows that C₁ I 0. Thus, under this mild assumption, C can be disregarded in discussions concerning error bounds.
Theorem B.5 parallels the result in Theorem 4.3 Therefore, it provides theoretical insights equivalent to those discussed in Section 4.2 for ItR involving only continuous missing features, elucidating the influence of g on the learning efficiency of f. This result further reveals conditions under which sequential learning the current setting becomes consistent, follows. The proof is shown in Appendix E.5.
Theorem B.6. Suppose N2 is set of finite-precision decimals for j E [d]. Assume the existence of true deterministic functions 93 X x (0, 1) + X3.Vj Vj E [d], such that (gi,...,97 92) E G and f* X y such that f E F. Additionally, assume that l is Li-Lipschitz continuous, that Rū (J) and (F) asymptotically converge to 0 as n + cc, and that for all j E [d], the number of samples available for learning 9j tends to infinity as n - co. Under these conditions, using consistent methods for learning g and setting = (2T) in 9g. sequential learning in ItR becomes consistent, i.e., as R((f)).
+ 0o, RR.g(g.
B.5. Theoretical Analysis for Learning g with fixed f
This section analyzes the learning g by minimizing (J in LAC-ItR's step (iii) while f is fixed. To conduct a similar analysis as in Section 4.3, we introduce the following definitions.
Definition B.7. Let p(m) denote the empirical distribution of M computed from finite number samples. For any (Fi,...,ra)) E TRQ and any S, defined the following hypothesis set:
(ri, G(r x... Ga(ra, Sa);
where, (y(r,, (9119) E gj Epony)
(o))
S STz).
5, [ = 0, E [d]. vj E [d].
M,, denotes the empirical risk approximating MMSE Here, \/ represents the index set excluding j. and M calculating the sample average over 5,,.
(M,)) by
The learning of g under LAC-ItR's s step (iii) is then expressed as follows:
//.3 105 ..... ......5..
(r) (ru) (ra)
95.
(r,)
(.))). (oo). vj [d].
where I33 = arg
min
E Ep(m))))
(B.31)
(ry.3)
Under these definitions, the following theorem holds for the case where the missing mechanism is Missing Completely At Random (MCAR). The proof is provided in Appendix E.6.
Let 4g be the PDF of ((gmis(m)) (m)))(m)) o [mmis(m) m)]x (mis(m))). where o E R., Suppose S and represent the ordinary dataset and weak dataset, respectively, consisting of n samples, and the missing mechanism is MCAR. For any measurable' f E F, bounded by Ut d0 and 8 E (0, 1), the following inequality holds with probability at least
Theorem B.8.
17

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
1- 6:
(r) Ri( (o, 1.5
log((d= + 1)/6)
2
Hy (T,),,,,, + UiV
2n
2V R + 2
C² + 20
Bp(mm
Σ
RMSE (B0(r.sh.))
jEEmis(M)
log((d + 1)/5)
+ 499 499 (3, + 2
2n.
x
201
C²2 + 202
Bp(m))
Σ
HMSEE, (B6ir.911)
+ 1 8,)) + 201V
log( + 1)/8)
2n
(B.32)
Here, Sg) {(rm))
91 E (m. (ni()))
IMSE (25,(22999 m))
g,(r,, S,)} C. and Ig(r,3),5
(a..m)) [p. (X mis X (ob((M) M)) Y, M)]
arg min (r, (pmm) E EjEmis(M) R (9,))-
Theorem B.8 parallels Theorem 4.5. Hence, it provides theoretical insights equivalent to Section 4.3 regarding the influence of f on the learning under MCAR assumptions for ItR settings with only continuous missing features. Additionally, Theorems B.5 and B.8 elucidate the conditions under which iterative learning achieves consistent when all features are continuous and potentially subject missingness, as follows: The proof provided in Appendix E.7.
Theorem B.9. In addition to the conditions stated in Theorem B.6, suppose f obtained in LAC-ItR's step (ii) be Lipschitz contimuous, and the Rademacher complexities about converge to 0 asymptotically as n - co. Under these conditions, the iterative learning in LAC-ItR's ' steps (ïi) and (iii) achieves consistency.
As in Section 4, for the sake of simplicity, the variance o² of 9g is defined in this section as a constant independent of x", but it is also possible to define this variance as a function o²²")) that depends on æ". In this case, Theorem B. 1, Lemma B.4, Theorem B.5, and Theorem B.8 still hold, with each occurrence of the constant the corresponding upper bounds replaced by max o² (x:").
C. Proofs for Main Body
C.1. Proof of Theorem 3.1
In the corresponding theorem for the formulation of discrete WFL (Theorem 3.1 in (Sugiyama & Uchida, 2025)) derives an upper bound on the 0-1 loss with respect to using the fact that the WFs are discrete and that qg is defined as deterministic probability distribution. In contrast, for continuous WFL, taking into account the continuity WFs, we define 9g as Gaussian distribution. By leveraging the definition, we derive the term including the MSE with respect to gj as being upper-bounded by Rf) (f)
18

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Proof of Theorem 3.1. For any measurable f G, and / bounded by Uj oc, the following inequality holds:
Rf(f) Ri,g(f) = (m.w) (() -Ep, Ep. 2000 (20) ((( X)]
/ Xm
= P. =.p (0) 2)
1[f(x" ((())",))), {) (x:" X V) - 4g (x=" 1 xx) )dk
S (1= ,) - (x x)) }dd
< U/ v))
1
Pa (x: Xo; Y)
U. (2),9) s))
Pa Pa X | Y) log
dx
9g (x" X
log Pa 1x(X"
U1
E (a.))
gg lg(X*|X))
Ui {
( (TI.p.
(p. 9g 4g x xx)))
(C.33)
Here, Lhn, (m.y) (log p- (X (X" Y)). In the first inequality, the condition ((y, y') S Uz, Vy. y is used. The second inequality employs Pinsker's 's inequality, while the third uses Jensen's 's inequality.
From the definition of gg. it follows that - -Bp. (x..w (log 9g [x| [x)] can be reformulated as follows:
1
EEp (I=) (log 2g (x)] = log v (2x) 22F
+
E
pm)
(Xx g( (x²")) g(X)))]
20
1
= log V (2T) g2F
+
E (m) - (x(2))
20
FE
1
log (2x) 02
R
20
JE
Substituting this result into Eq. (C.33), we obtain:
(1 (1/6)
1 Σ
Ra( Ri, (
SU
Cr +
RMSE (gs)
(C.34)
202
jE[Fw]
Hence, Theorem 3.1 is established.
C.2. Proof of Lemma 4.1
In the corresponding lemma for the theoretical analysis of discrete WFL (Lemma 4.1 in (Sugiyama & Uchida, 2025))
derives an upper bound on the 0-1 loss with respect to 9, using the fact that the WFs are discrete and that g, is defined as a deterministic probability distribution. In contrast, the analysis continuous WFL cannot rely on these properties. To address this, we first derive Lemma 4.1, and by defining as a Gaussian distribution, we subsequently derive Lemma 4.2, which plays a central role the theoretical analysis of continuous WFL.
19

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Proof of Lemma 4.1. The LHS of Eq. (4.9) can be upper-bounded follows:
[f) R (f)
(a.,)) ((f(()() (1) ((X))
1xx X=
(erp(armw) (w) w)
1[f("", (p, x xx,Y7 Y) Og (x:" )
f 1([(x", ,x),, (20 x V) Og (ar x \10)a
By, (ee w)
EE,, (w m)
1[f(x", (1(x",x)),1)
Vp X / // (x (xo)
New
Va X0, V2 Xo)
dx=
EE, (wr w) [/ 1(f(x" X²), ()),)) Xo,
/ (xx Xo) dxx"
Pa 1x Xo, Y)
Ep. (0 w) (/ee ((s("" X"), Y) 4g (20 X
1 X0
V (oww xo)
dxe"
pm (((F(X),))3)) (x)))) 1
X
go (2 (Xo)
dx
(a))
+
E (ww..))), (= (e) ( (m...)
(x Xo,Y)
V 9g (x
/2w
(X))
dx
(C.35)
x
(a3)
(a2)
Here, in the second inequality, Cauchy-Schwarz inequality is used since I, p. and dg are measurable functions. The terms (al) and (a3) in Eq. (C.35) can be bounded as follows:
(al) = Ep, (a. (()(((()))))))))))))))))) Ep. (e
s P. (XX)), 2U)
(C.36)
(a3) Ex, (w" v0%, (- ( (() Ep, (wo wa (w (00) (ee (1)() 00]
< 2U E.. (w" wi,, (e (-)) ( [()()X))))))))))))) Y)] = 2U, Rg (f).
(C.37)
Here in terms (al) and (a3), the fact that the function ~ x² is - continuous on the interval [0, Ui] utilized. Applying the above inequalities to Eq. (C.35), Lemma 4.1 can be showed:
(R(()) - (f)) (VR((f) + (iill))
2 (ar ( Diil,. Y). 9g (X (x1X))]
(C.38)
Here, Dh represents the Hellinger distance.
C.3. Proof of Lemma 4.2
Proof of Lemma 4.2. The term Ep.( (p. Xo, (X xx)))) in Eq. (4.9) can be upper-bounded as fol- lows:
E (m) ( < Ep, (1) w) f P= log p= X Y) 9g (x X dx
X
LLp. log Xx)
(C.39)
20

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Here, LL,, (x.))
(x. logp- X", Y)J. Defining 4g by the PDF of (g((())) I xF where o² R4, the term
(x)) [log 4g X" [ can rewritten follows:
p. log 0g (X" log V (2th) FM 02F"
Ey, (a) ((xx)) (X" (X""))"XXXXXXXX""""""" ((x)))
20²
log V (2T)) FW 02Fm
E (p. (x))}
20
JE[FM]
log v (27) 62r" 622
R
20
JE(F=]
Substituting this result into Eq. (C.39), we obtain:
E (x0 (IDi x (Xx S LLp. + +((0))
Σ
(C.40)
20 JE[PM]
Here, Z(o2)== = :olog v /2 Fm o2F"
Applying the above result to Eq. (4.9), we establish Eq. (4.10) in Lemma 4.2:
[Ri(]) - R1) (VRi(f) + v /Bia()) 22
)
20: [LL, + Z(00) + 20
Σ
RMSE
(C.41)
jE(FM)
C.4. Proof of Theorem 4.3
Using Lemma 4.2, we establish the following lemma:
Lemma C.1. Let 9g be the PDF of N (g(xx), o² where o² E R4. For any measurable E F, g 9 and l bounded by Ut < co, the following inequality holds:
(f(f)) - Rf)(f))
2 Bi(f)+ +
12 Co
Σ
RMSE,;(9))
}
20 Co +
1
Σ
RMSES (o)) )
(C.42)
20
20²
jE(Fw)
jE(F-)
Here, C2 = (w.,) (e. (X Xo, Y)] + log
fw g2F
Proof of Lemma C.I. From Eq. (C.41), for any measurable F.g G and / bounded by Uj oo, the following holds:
1/2
(R(() R < 2U, Lbp. Z(00)
Σ
RMSE.S(95))
(C.43)
20
jE(
Thus, /Ri (f) can be upper-bounded as follows:
(f) RR(() + ((()) V RR.
$ (B()) + 20 (LLp + =(002++ + 20 JE(FM)
Σ
RMSE (9))
(C.44)
21

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Applying Eq. (C.44) to Eq. (4.10) in Lemma 4.2, conclude that Lemma C.I holds:
[ Rf)
1 2 R + 2U LLp. + z((0²)
1
E
RMSE, (g5)
20²
jE[FM]
x 201 201(LLg LLp. + =Z(0²) +
1
Σ
RMSE.J(9))
(C.45)
20
jE(FM)
Using Lemma C.I, Theorem 4.3 is demonstrated follows:
Proof of Theorem 4.3. In the ordinary supervised learning setting described in Section 2.1, we define the empirical risk minimizer as follows:
fs = arg min R((
The LHS of Eq. (4.11) can be decomposed as follows:
(fg5() RR(fs) RR. Ri.g( + R5.g( RI. (fs)
(al)
(n2)
(C.46)
R4 (Js) (f()s) + Ri(fs) R((FF).
(a3)
(a4)
The terms (al) and (a2) in Eq. (C.46) can be upper-bounded as follows:
fEF (al) < max Rt.g (f) (r)l
(a2) < (fs) - Ri,g(fs) I max Rf (f)).
fef
The term (a3) in Eq. (C.46) can be upper-bounded using Lemma C.I as follows:
(a3)= 2/ 2/Ri(fs))
2 (2ui(c, Co +
1
Σ
RMSE,
RASSE)(9))) (01)) {{
201 (C. Σ
RMSE, (O)EE.(())) (1) (C.47)
202 JE(F")
20
FE(F=]
Here, R((fs) can be further bounded follows:
R((ss) = R((Ss) RG(Ss) + Ra(S) -
S R((s) Ri(fs) + +R((s) +
< R((f)) + max [R(f) - -R(())-
(C.48)
JEF
Applying the above inequality to Eq. (C.47), the term (a3) in Eq. (C.46) can bounded as follows:
(207 Co +
(a3) < 2 R((fs) + mmxx RR(f) - -R((F)) + 20
Σ
RMSE, (g:)
fEF
202 JE[F#]
(C.49)
(
2U (Cs + 202
Σ RMSE.)(3)))
x
jE[Fw]
22

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Similarly, the term (a4) in Eq. (C.46) can be bounded follows:
Rs(Js) R((fx) S 2 max
Ra( - R(()
SEF
Combining the results above, the LHS of Eq. (C.46) can be upper-bounded as follows:
(gg R((Fz)
s 2 max (Rig(s)- Rr.g(f) + max [R((S) - R((S) FEF
/EF
+ 2 Ri(FF) + 2 max Jef
[R(()) - + +(200/ 20/ Qui(Cor+ +
Σ [Fw]
RMSE (g1)
(C.50)
20
}}
x
20
Co +
Σ
RMSE,500)
20²
JE[Fn]
Using the uniform law of large numbers (Mohri et al., 2018), for any E (0, 1), the following inequality holds with probability least 1 8/2:
log (4g(44))
a (1) (F1 + U
max fer
2n
1og(4/0)
[R(() - Ri(O) < (Fi) + Vip
max feF
2n
Here, Fi = ((r,y) - I((()),)) : FJ. Additionally, from the assumption that is Li-Lipschitz continuous and Lemma 26.9 in (Shalev-Shwartz & Ben-David, 2014), it follows that (Fi) < L,R; and (Fg s (F). Thus,
for any 5E E (0, 1), the following inequality holds with probability at least - 5:
Rg. g. R((F))
(og(4/))
4 (F) + (F) + VIV
2n
{ 2 Ri(FF) + R((f))+ (L, + 201
log(4/8)
+ 2U1
Co
Σ sE(F
RMSE (g:)
(C.51)
2n
20
{222( 20l (0 + 202 jE[Fw]
Σ
RMSE,1(9))
C.5. Proof of Theorem 4.4
Proof of Theorem 4.4. Firstly, since XW is set of finite-precision decimals, for any j E [FW] and o² = (2T) follows that C, S
The remainder this proof follows a structure similar that of the proof for discrete WFL (Sugiyama & Uchida, 2025).
By assumption, there exist true deterministic functions 0; 0 X for any [F"], and (gi,... fFw) G. Therefore, when 9s- :Us.::....5.F.. is obtained by the methods that achieve consistency (Mohri et al., 2018; 88; Cheng et al., 2023). the following holds:
DC, RMSE, (95.)) + 0, Vj E [F"".
(C.52)
23

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
By assumption, there exists deterministic function f" X + y for label prediction, and f F. Hence, the following holds:
R+(f=)= =
(C.53)
Thus, (F) and R (F) are monotonically decreasing with respect to and converge to the error bound established in Theorem 4.3 converges to s0.
C.6. Proof of Theorem 4.5
For weak dataset S and positive real-valued vector r, define the following feature estimation models:
(r)
(95,1
(r1)
-5.
(r (
(C.54)
5
(ry) arg min (Ms. Vj [P"].
where 93. 9, (0(,,
(C.55)
Using Lemma C.I, we establish Theorem 4.5 as follows.
Proof. Proof of Theorem 4.5
The LHS of Eq. (4.13) be reformulated as follows:
Ri
(r) f - R(( R f.3)
(r) Rg
f 1.3 - Ro
+
(g
(r))
Ri, (g
(r)
R(()).
(C.56)
(al)
(a2)
(23)
The terms (al) and (a2) in Eq. (C.56) can be upper-bounded as follows:
(al) v max (Rir(g) RR,s(g)),
(C.57)
gEg(r.3)
(a2) < (9
(r) R (r) < max
R (
(C.58)
gEG(r.S)
The term (a3) in (C.56) can be upper-bounded using Lemma C.I as follows:
(a3) < (9))) (r))
Ro(f)l
< 2//R(ff++ + 20 Co
Σ
(r)) RMSE,
(3)) {{
20 Co + 202
Σ
RMSE, (r))
202 JE[F"]
(955
JE[F"]
(C.59)
Therefore, by applying Eqs. (C.57), (C.58), and (C.59) to Eq. (C.56), we obtain:
R1 (r) Ri(f)
22 max [g) Rg)) (E
(C.60)
2 Ra(f) + 2 Co +
1
RMSE
(g
(953)
(1s)
20
Ca +
RMSE. ( )
20
20
JE(F=)
jE(Fm)
From the uniform law of large numbers (Mohri et al., 2018), for any 8 E (0, 1), the following holds with probability of at least 1 8:
log(2/5)
max gEg(r.
(R,r(g) Rg) < (r., + U.
(C.61)
2n
24

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Furthermore, by applying Eq. (C.61) to Eq. (C.60), we obtain that, for any 8 (0, 1), with probability least 6, Eq. (4.13) holds:
Ra, ( Rf)
(2/(2/8)
x 49,, (r,,, + 2
2n
(C.62)
2v R + (201
22 Co (2ti(Cot
1
Σ
R (1y)
))
{{
201
Co
1
Σ
RMSE. (155
(1)) }
2r²
2x²
jE(FM)
je(PM]
C.7. Proof of Eq. 4.14
Proof of Eq. 4.14. Since, by the assumption about Gy(r,, S,) and the fact that I5. E (g(r,, Sy), the following holds:
RMSE (955
(1:) min RMSE, (g3).
(C.63)
91E9;
Moreover, by the assumption about 9j, the following holds:
AMSE,1(9G)) = O,
(C.64)
where 00, == arg min RMSE.J (Mj) In this case, the I.HS of Eq. 4. can be upper bounded as follows: EBj
RMSE. (05} ( RASE. (05,) RMSE. (0 (15)) + RMSE, (11)) RuMEE.(99;)
- RuSEE, (1z)) - RASE (1y)) + RMSE, (g9,) RMSE,,(99,)
(9 (g
2 max [RMsF, RMSE. (g;)l- 9,E0;
(C.65)
By applying Theorem 11.3 in (Mohri al., 2018) the RHS of the last inequality above, we obtain that, for any 5 E (0, 1),
with probability at least 1- 8, the following holds:
R (15)) S MUMBE Riy (9;) + 20 Usis V
(
log(1/0)
(C.66)
2n's
C.8. Proof of Theorem 4.6
Proof of Theorem 4.6. Firstly, since X. is set of finite-precision decimals, for any j E [FW], and o² = (2T) follows that C, S.. O.
The remainder of this proof follows strueture similar to that of the proof for discrete WFL (Sugiyama Uchida, 2025). From assumption, there exist true deterministic functions 9) d + X²" for any j E [FW], and (gf...., gF-) G. In this
case, for any and 3, it holds that g" E G(r,3) 3). Hence, the following holds:
RMSE (Mss,.(90(r.5..)) 0, Vj [F')..
(C.67)
For any j [F*),, define Grfr, Sy) the of hypotheses that satisfy the following two conditions: (i) each element is solution obtained by methods that are guaranteed to achieve consistency (Mohri et al., 2018; Cheng et al., 2023), and (ii) its empirical risk is at most r, As n increases and r; > 0, the assumptions on RG, and the theoretical guarantees of these leaminng methods for 9; imply the following:
RMSE,;(9)) 0 Vgj (g Vj E |
(C.68)
25

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Additionally, by assumption, there exists true deterministic function f" X - y for label prediction, and f F. Therefore, the following holds:
R((fr) 0.
(C.69)
Thus, under the conditions of Theorem 4.4, the following holds for fg.3 obtained through LAC-dWFL's step (ii):
+ 00, Ri.g(fg.5) + 0.
(C.70)
Furthermore, using Theorem 3.1 with Co S and 0² (2x))1, and additionally letting ry 0 as increases for any j E [F*], the following holds:
+ 00, R.(fg33) 0 Ag ((r
(C.71)
Since l is L,- Li-isshhhtz continuous and g.5 -Lipschit continuous, the following holds (Shalev-Shward & Ben-David, 2014):
R (9 S)) LL S)).
(C.72)
Consequently, if Ri, (G(r, S)) and R (T, 8,)) for any [F]] monotonically decreasing with respect to and converge to 0, the error bound established in Theorem 4.5 converges to 0 as + cc.
D. Proofs for Appendix A
D.1. Proof of Theorem A.1
Proof of Theorem A.I. For any J F, and bounded by U, RJ( Rr, can be upper-bounded as follows:
R(( Rig(f)
-Ep (Z..)) (e.vi (I()) (ap(aa))g(a (e (1X)))
[ xrw
s Epp (20 w)
1[f("" ,x)), (),, X),Y q a )
- [/ 1(f(=" (o((a"X)),)) (x X Y) g(20
(VP_ 11(001100 dow
1
Ep. (()_( Ep. w) / (VDr Va(l" Xo) [xo)
2
do
Ep (20 20)9(10000000000)))))))))))))) (19 F))2] x Ep.(ar Srn (V (VDa(20 Xo.V) (V Vo(20
2
dx
(VRi(f+ + / (e, 2U p. (ar) D 1o, g(xx
$ 2U% (D (p. X Y), q. (x)))
(D.73)
Since p. f and are all measurable functions, the second inequality uses the Cauchy-Schwar incquality. The third inequality uses the fact that I upper bounded by Uj. Hence, Theorem A.I is showed.
D.2. Proof of Theorem A.3
Using Lemma 4.1, we establish the following lemma:
26

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Lemma D.I. For any measurable FF q and I bounded by Us the following inequality holds:
[Ri(f) R.,( 2 ((()
(2 (P X 18,)),9 (
(D.74)
x (X
Proof of Lemma D.1. Using Lemma 4.1, for any measurable f E F, q and bounded by U4 < co, the following inequality holds:
[R((f) - Ra < QU (ar X
From this inequality, (f) can be upper-bounded as follows:
(R.g(f) S v Ri((f) + V Ri(f)) v (f)
(f) 2U:Ep, (m wn
xx
(D.75)
Applying the above result Eq. (4.9) in Lemma 4.1 establishes Lemma (D.1))
[R((f) (1) 2 Ri(f)
(D.76)
(D.77)
Using Lemma D.I, we now prove Theorem A.3 as follows.
Proofs of Theorem A.3. The empirical risk minimizer in the ordinary supervised learning setting defined as follows:
fs = arg minseF Ri(f).
The LHS Eq. (A.20) can be decomposed as follows:
R - RR(fs)) Ru. (F.g(F + (fs)
(al)
(02)
(D.78)
+ (fs( RR(fs) + Ri(fs) - R((fF).
(a3)
(n4)
The terms (al) and (a2) in Eq. (D.78) can be upper-bounded as follows:
(al) < max RI (f) - (1)b FEJ
(a2) RSs( - RSg S max (i, Re.
fEF
The term (a3) in Eq. (D.78) can be upper-bounded using Lemma D.I follows:
(a3) < 2 R((Js) + (2UtEp. (w0 [X " xxx(xx
(D.79)
(Xi X². (1
27

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Here, Rs(ss) can be upper-bounded follows:
R((fs) Ri(fs) Ri(fs) + R((fs) B((f))+
S Ri(fs) Ri(fs) + -R(fF) +
I R((fs) + max Ri(f) - -R((F))
(D.80)
JEF
Applying the above inequality to Eq. (D.79), the term (a3) in Eq. (D.78) can be bounded as follows:
R((II) +
(a3) - R((fr) max JEF
[R(())
(X,, (x))
(D.81)
(2 .v)
D (p. (X X g(x ,) X(1"x)
Similarly, the term (a4) in Eq. (D.78) can be upper-bounded as follows:
Rf(fs) - R((f=)= < 2 max R2(f) R((f
JEF
Combining the above results, the LHS of Eq. (D.78) can be aoper-boundee as follows:
(f,.() R((ff)
$ max FEF
Rf) R(.g(f) 2 maax [R(() RI(III
FEF
2 R((fr) + 2 max [f(()) - -R((P)))) + (2UrE), (2o (2 (Di( X( }}
(D.82)
FEF
2U D (p. I Y), x
By the uniform law of large numbers (Mohri et al., 2018), for any 5 E (0, 1), the following holds with probability at least
1 1- 8/2:
log log(4/8)
(1) Ri, 29% (F) Up
SEF
2n
log(4/8)
R((f) - R((()) & 29,, ( V
max FEF
2n
Here, F. = (x.,v) + ([((), v) f E F). From the assumption of and Lemma 26.9 in (Shalev-Shwartz & Ben-David, 2014), it follows that 9t, (Fi) L (F) and Ri (Fi) Li9g (F). Thus, for any d E (0, 1), the following inequality holds with probability at least - 6:
Rt.g (fg,8) -R((F)
log (4/5)
< 4 (F) + (F) +F) Vi
2n
+ 2 R((fz) + 44, + 20 (og(4/0)
(2 [D 1), F), g)x(xxxx)))
2n
(X X ,)) (111x)))) 7
(D.83)
This concludes the proof Theorem A.3.
28

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
E. Proofs for Appendix B
E.1. Proof Theorem B.1
Proof of Theorem B.1. The LHS of Eq. (B.27) can be upper-bounded as follows:
R(() (f pp (wr,)) (m (())) Ep, mo (m)l. [([((),))))
E. (m.u/m)) [V Y] Ep, (y.re)yg ((Fmin())) (alba(m)) m.p, (cruts(m)) m.y) (X(()); Y)]
= (m))
[ (..))))) (X)). M) - Pp (Bobo(( M (#0mi (e) Reab M)) (x), Y)]]
< ppp (m²)
B (p
(ffXXobst Fobs(
Xmin(ha)
]]
x
pa (= ( Xo) Y, M) - 9g (Emin(Md) (Xbss M) dx (M))
Ep, (ostara M) I XmiutSa)
< U.Ep (m))
P- (2 Mobe( M) - 9g (Mi)ss Xobu(ar), M) de (M))
S (m)
(y. w|) M))
Po (F X (b)) Y, M,M Xaishha)
P. (m M)
X ooba MY Y,M)
x log
dxiiis (M)
9g (m M))
X (bs((M) M)
P. X mia(M)) X Mb) Y, M)
$
$ Up Ep. (m)
log
2
9g X Mis((
X Xobs(M)) (M), M)
log 9g (Xiis(M) X Xohs(M M) 1)}
Ui
LLP. + (a.
(E.84)
2
Here, LLg, = Ep. (m.y.)) [p= (X (M)ssMM) Xobs(M)) Y, M)]. The second inequality uses (yy. s UI, Vy, E V. The third inequality uses Pinsker's inequality, and the fourth inequality uses Jensen's 's inequality. Furthermore, the following holds:
(z.g.m) [log 9g (M)is(M) (obo(M)) M)]
log log(2r))lmixt (MM) M|)
2
Ep. (m.
X mis((M) Imiss(M) (Xobs(Mr))) (Xmis
( 9mis(M) (X (Xoto(aa))))
20
(m))
log( 2Tr) |mmi (MT)) ((mlis((M
(m))
(p.(a -8)(8))
2
20
jEmis(M)
1
(m))
log (Trr
[M)) 2|mla((M)
+ Pp))
MSE. (g3)
(E.85)
2
20
jEmis(M)
Substituting this result into Eq. (E.84) establishes Eq. (B.27) in Theorem B.I:
Σ R (41)
})}
Bi(f - RI, s Up 1 C² + (m)) jEmiss(M)
(E.86)
202
Here, C² (e..).)) 12- (0 (iaaeMM)\obbssii (obs(M)) V.M) + pmp log(2 [mia(M) ([m
29

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
E.2. Proof of Lemma B.3
Proof of Lemma B.3. The LHS of Eq. (B.28) can be bounded from above as follows:
[R((f) - R1. (f)l
Pp. (o).w) [([(X,, Y) - R.. (v. (aly. (X)(), Ki]l
Pp. (m)p- (w.ylmm (x),, Y] Ep. (v. go (IEmin(e)) (obaa( pp (1oobs(m)) m. m..b))((((x)) Y)
Ey (m)
(m. (x), (X).)) Bp- (W M/ Mg (25mle we (aate way [((((X), Y)]
pp.(m)) (m)
E pp. (m
(ff(Xohs( (M) Dohs(
Xmiaaie::
x
P. (2 Xobo(at), Y, M) - Og (mmis(() X Xoba(M)) M) d
Ep. (m)
E p. ((
(b))) Fobo))))))))))))))))))))))))))))
Xmiat
x
Pa (3F Xobe(Ma)s Y, M) + V
(g (amite X Xbb((M) M)
x
Pr (TF (d)) Y, M) V (2 (M) X M) M) dæ dx
[1
(m))
Ep. (o
[(f(Xobe(A1)) (obe Y) ( Miis) Xobe(Na), Y, M)
Xmini
x
P. (PFiint (M)) X Xobs(M M,M)
V -9 (F M) X (ob) M) ma
// Xiutscae
+ (m))
(m)
Ep. (essor
(d)) H)
M Uo (mis(M X (obe(M)) M)
V Po (M Xobo(M Y, M) V Ag Miiiss X Xobs(MM) M)
Ep, (ep
(a1)
Ep. (m)). at w//r)
(i (Xmis(M) Xohs(MM) M Y, Y,MM,, 9g (Xmis(M) (X Xobes( M) M))
(a2)
+ Ep. (M))p-((totooc)))) 9p (2 (m)))
aabs m2) ((((())))))
(a3)
X E Ep. (m)), (M m))
(X.
mis M) X MY, ( (Xmis(M) X Xobs( MY) M))
(a2)
(E.87)
Here, the second inequality uses the Sauchy-Schwarr inequality, since I, f. Pe. and 9g are all measurable functions. The terms (al) and (a3) in Eq. (E.87) can be bounded as follows:
(al) = (m.( [(f((),,)) Y)² 0]
< 2U, (m.)) [(((X), Y) 2U, ((),
(E.88)
30

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
(a3) Ep, (m)p. (Bota(mm)) gg (mii))))) (otes(m)) mm ())
< 20 Ep, (m)p- (Robn(mm) 3/ (m2)9pp (miss()) (Bbbs(mm) ma) [ YA] Ru
(F.89)
In the above, it is used that the function + x² is 20 -ip-sshti continuous when its domain is restricted to [0, Applying these results to Eq. (E.87), we have demonstrated Lemma B.3:
Ui].
(R((f) (1)
$ ((V Ro(f)+ + v (ooo(f))
(2 (me)p. (#eorarm) ma) (p (Xiis (obe((N) Y, M), 9g (X (Mias X (obs( M))
E.3. Proof of Lemma B.4
Proof of Lemma B.4. The term P., (m)), (Xobbx(m) drl m)) Eq. (B.28) can be bounded as follows:
DR (pa Xmis(N))Xoba(MM)) Xoba(M)) Y, M), 2g ( miis (obe( MM)bbt)n)))))))))) M))
p. (m)), (Fobs(m)-)//mm)
D (Mioo("Xiscaos(Xooooaaa M,, 4g (iin (obu
S Ep (m)p, (Moba(m))
S Jannnae Pa (a my M). Y, M)) log
P+ (. Xobot M) Y, M)
dxmmis
9g X (is) X X M) M)
T.L, Po. [lo Mg X Mis(M) X oba M)) M)).
(F.90)
Here, L.I.
p.. (2. (D_ (X mis( X (M) Y, M)).
Defining 9g by the PDF of
A (ggnis(m) (Foomm)))Cosoom)) (Fobs(m)) o² ( [mmssmm where a E R4, The term (... [log dg (X (ii)s(M)) Xobs(M)) M)] can be rewritten as follows:
Ep, (m. [og Ug (X Xmis(M)) X Xobs(M); M)]
log(2r) (tn [ MM)i
+ Ep. (m.
202 (X (Xiis(M) Mmis(Mr) X (oha(Mr)))
(XXi)) - 9mis (M) (X (Xone(Mni))
= [m)
log(2rr) (iii ( ( 22mmi((MM)
+ Ep.
Σ
(p, - -x
2
202
jEmis(M)
(m))
log(2
(miss [MI) (llmss(M
Bp.(m)
Σ
M (g,)
(E.91)
2
20 jEmis(M)
Substituting the above into Eq. (E.90), we obtain:
E (mm)p, (//
{
(H(pp(Xmisd (Xmis(M) (obs(M)) Y,M 9g (mmis(M))
(M) (X (obs))XbbssMM))-o. Mi))
< LL', +Z' + E (m))
M (9,)
(E.92)
202
JE
Here, 7'(02) : Ep.(m))
31

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Thus, applying the above results to Eq. (B.28) completes the proof of Lemma B.4 stated in Equation (B.29).
[f( (f)
$ (R((f)+ + v (lo( { 22 LL/2 L11. + z'(22) + -Bp.(m)
(E.93)
20
Σ
R (0))
20
(Emis(M)
E.4. Proof of Theorem B.5
First, we use Lemma B.4 to prove the following lemma:
Lemma E.I. Let 9g be the PDF of (F ) 0² T [mis(m)) (mmis(m))) where RR4- For any measurable f F. 9 and bounded above by Ui < the following inequality holds:
[R(()) Rf)l 2/ Ro(f)+++
1 20 C₂+
Bm.(m)
Σ
RSE (MSE,(00))) (0)))
(M)
202
jEmis(Mi)
(E.94)
11)}
x 2U:
C² +
1
p. (mr)
Σ jemis(M)
R (gs) )]
(M)
20
where C² (m..m))) (P Mis X M)) Y, M)] + pm)) ) log(2T) (1123rr)(mms(MM) (Mi((M)) 2
Proof of Lemma E.I. From Lemma B.4, for any measurable f E F.g. g E 9 and / bounded above by U1 < oo, the following holds:
1))
(R((f) /Ri & 20 LI,, + Z' + Ep (mx)
Σ
RIM RASE, (03)
20
jEmia(M)
Using the above result, v RRig(f) can be bounded follows:
v (f) (R((f ( (R((F) ( -Boool))
1)} 7
(R((f) 20 L.I. z'(02) E (m)
MASE (0))
(M)
(E.95)
20
jEmis( M)
Substituting this inequality into Eq. (B.29) completes the proof of Lemma E.1 as given in Eq. (E.94).
[R( (J) { (2/R(())
+ 20 [. + (2(03) +
20 (m))
1
Σ
((M)
RSE (9:)
(E
20 LL'. + Z'2222 + 1
Ep. (m)
Σ
R
R [9;1]
}}
(E.96)
202
jEmis M)
Using Lemma E.I, we now prove Theorem B.5.
Proof of Theorem B.5. Define the empirical risk minimizer under the ordinary supervised learning setting described in Section 2.1, as follows:
fs == arg min Ra(f). Jef
(E.97)
32

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
The LHS of Eq. (B.30) can be rewritten follows:
(f R((fF) R1gg(g.))) Ri.g(f (Fg.5) (f,5) R
(al)
(n2)
(E.98)
R1.g (fs) - R((ss) Ro(fs) Ra(fr)-
(a3)
(o4)
The terms (al) and (a2) in Eq. (F.98) can be upper-bounded as:
(a1) v max (f) - (f) JEF
(a2) S (Ss) Rig S max Big (1). SEF
(E.99)
The term (a3) in Eq. (E.98) can be upper-bounded using Lemma E.1 follows:
(a3) < 2 2 Ri(ss) + 20/ C. +
20 202 (m)
1
Σ
(M) (g:)]
(E.100)
M)
x { 20 C. + (m))
Σ
( (oss,/(,))]]
(E.101)
202
MEmis((M
E F}. C% Ep. (m. pp- (oinst Y M,) + Ep.(m) 12g(2 ( ( MI. Using these, (fs) can be upper- bounded follows:
Here, F ((x,y) y)
((((x),))
R((fs) = Ri(fs) Ro(fs) + Ri(fs) - R R((Ss) R((Ss) + R((FF) RG(F)) Re(FF)
S R((FF) + max Ri(f) -R((I)-
(E.102)
JEF
Hence, the term (a3) in Eq. (E.98) can also be upper- upper-bounded as:
1))² (E.103)
(a3) < 2 ( + max (Ri()) - R((S))) 1/2 + 2U/ C² 20 Bp_(m)
1
Σ
(M)
(0,)]]
7/2
x
C² + 20
Bp
Σ
MASE (لاه))
(E.104)
202
ME)is(M)
Similarly, the term (a4) in Eq. (E.98) can be upper-bounded as:
R((fs) - Ri(fr) S maax RR( -R())-((()).
feF
From the above, the LHS of Eq. (B.30) can be upper-bounded as:
RI. (f..5) Ri(fs)
< max Rf) ()) + 2 max [R(()) R(INI
FEF
JEF
+{ 2 R(((f)) max [R(f) - R((())) leF
/20/
)) #1
+
+ 20 C² +
202 Bp)mm)
1 Σ
R (g:))
(E.105)
(Emis(M)
(
1
x 20 C² +
20z (ma)
Σ
RAASE, (0:))
JEmis(M)
33

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
By the uniform law of large numbers (Mohri al., 2018), for any ő E (0, 1). the following holds with probability least
1 6/2:
(4g(4/))
max FEJ
Ru (f) - (f)) (F)) + ViV
2n
log(4/6)
R((f) - R 29 V
max FEF
2n
Here, := = [2, + ((f()) FF.
From the assumption that and Lemma 26.9 in (Shalev-Shwart, & Ben-David, 2014), it follows that R8 (Fi) - (F)
and R2 (Fi) L19 Thus, for any 8 (0, 1). the following holds with probability least 1 ō:
R1.g (g,z) - R((fr)
log(4 (4g(
(F) + (F) + ViV
2n
3/2
+ $ R((fF) + (L)R,(FF 201
log( (4/0)
+
2U
C² +
pp.(m) (m)
Σ
RASE (0,)
2n
20
jEmis(M)
(E.106)
200 (02 C% + 20 jEmis(M)
M (0:)) }}
(m)
Σ
E.5. Proof of Theorem B.6
Proof of Theorem B.6. Firstly, since X₂ is set of finite-precision decimals, for any j [d]. and 0² = (2m)-1 it follows that C² < 0. The remainder of this proof follows structure similar to that of the proof for discrete WFL (Sugiyama & Uchida, 2025). By assumption, there exist true deterministic functions 9e X x (0, 1] - X3 for any j E [d]. and .gi...., E G. Therefore, when Is = (9.. :9s...) is obtained by the methods that achieve consistency (Mohri et al., 2018), the following holds:
DC, (m))
Σ
R
(05 + 0, Vj [d].
(E.107)
(Emiss(M)
By assumption, there exists a true deterministic function f + y for label prediction, and f F. Hence, the following holds:
R(ff)) 0.
(E.108)
Thus, if (F) and Big (F) monotonically decreasing with respect and converge the error bound established in Theorem B.5 converges 0 as
E.6. Proof of Theorem B.8
Using Lemma E. first prove the following Theorem E.2. Theorem E.2 holds for any missing mechanism.
Theorem E.2. Let be the PDF of N (gmis(m)) (Xons(m)) o² [mmis(m]] (miss(m) ). Let S and 3 represent ordinary and weak datasets with n samples, respectively. For any measurable G, bounded by Uj < oc and E (0, 1), the following
34

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
inequality holds with probability least ö:
R1,5 (1)) - Rf(
1.3
(0)(1/0)
2
H (9 + V.y
2n
2 2xU1 R((f)
U1
max Ø5 (m.,5)
(m)) (m²)
Σ
RASSE
(33,
(r:)
Bp(m)
Σ
RISSE
93
(r:)
jEmiss(M) M)
jEmini M)
(E.109)
+ Σ ri jE(d]
X
E (m))
(MM HMSE,
(1)))
Bp(m))
Σ
RMSE.
(5)
0,E0g(r).35)
JEmis(M)
-JEmis(M)
Σ rj jE(d]
Here, Sg) 1(x0, 2.)) + IMSE
9i
Gy(r,,
Vj
[Fw]. C²
(x....m (p. (a X Xobe(M)) Y, M)] + Epp, (m))2 log(2 (M)) [M
Proof of Theorem E.2. The LHS of Eq. (E.109) can be rewritten as:
Ru (r) RR(f) (9 R (r) + (9 Re,5
(r)
(r)
(1))
(9 (r) R(()).
(E.110)
(al)
(a2)
(a3)
The terms (al) and (a2) in Eq. (E.110) can be upper-bounded as follows:
(al) < max (Ri,(g) RRi,((9).
(E.111)
gEG(r.9)
(a2) < Re, (5
(r)) ( (
< max (g) Ri,r(g))
(E.112)
gEG(r.5)
The term (a3) in Eq. (E.110) can be upper-bounded using Lemma E. as follows:
(a3) < (9 (r)) Ri(f)l
$ 2 2/R((f)+ + 20 C. +
ppa(m)
Σ
R
202
95.5
MM)
(E.113)
20/
C + 20
1
Ep, (ma)
Σ
R
93
(r))
jEmis(M)
35

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
By definition, the following holds:
(m))
Σ
RISSE
(9... )
(r5))
jEmis( M)
(M)
(r:))
Σ
(rs)
Eg (m)
(m)
RMSEJ
Bp(m))
RMSE (95.)
jEmis(M)
(Emiss(M)
+ ())m)
Σ
MSE. |535
By(m)
Σ
R (0(6) )
jEmis((M)
jEmis(()
+ Ep(mi
Σ
BASSE
(B6(m.59,)
jEmis(M) M))
S Ep. (m²)
Σ
RAMSEA (1))))
Ep(m)
Σ
RMSSR (ry)
(95) )
jEmis(()
"jEuis((M)
+ Ep(m)
Σ
AIM
(Botr.51s)
Ep(m)
Σ
BiM)
(96(r.5).)
jEmis(M) M)
jEmis(()
pp(m)
Σ
MSE. (Bo(r.5.3)
jEmis(M)
2 max 96 EC, (ry)S,)
E (m)
Σ
RISEE (953)
(ry)
E (p(m)
Σ
R (95r))
(M)
MEmis((M
Σ
MSE. (Bo(r.51.)
+ Bp(m)
jEmia(M)
(E,114)
Thus, the term (a3) in Eq. 100) can be further upper-bounded as:
(a3) <
2 (v/T( + 2U:
Ch + 20²
Ep(mm)
Σ
RoASE & (B61r.81s)
jEmis((M) M)
+ max
E (m))
Σ
RiSE, (1:)))
Ex(m)
Σ
R
(555
8: (r,,3,)
jEmiss M)
jEmis(M)
x 20
C² +
1
Ep(mm)
Σ
RMSRR (go(r.5hs)
20
ME
+2 max .9 (rg,5(ry,55)
[pp.(m)
Σ
RASE
(55))
(r)) E.pm)))
Σ
(95.)
(5)
jEmis(M)
jEmiss (M)
(E.115)
36

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Applying the above results to Eq. (E.110), have:
Ro.r( 1.5 R2(
2 max gES((,.)
[g), R
+ 2V R((f)+ + 20 C² + 202
1
Bp(m)
Σ
R ( (90().3.5)
(Mmls(M)
+
max 9, 9, (r))
Ep Ep- (m))
Σ
BASEE.
[gs.j
(19))
Bp(m)
Σ
R
(r))
jEmis M))
jEmis( (M)
20 C + 20
By(mma)
Σ
RASE (B5(r.51.)
jEmis(M) M)
max (,E9)(r5.83)
(m))
Σ
MSÉ.
(5.5
(rx) Ep(m)
Σ
((M) MSE.
(r)) (55.)
SEmis(M)
jEmis((M
(E.116)
Using the uniform law of large numbers (Mohri et al., 2018), for any (0, the following inequality holds with probability least 1- d:
1og(2/8)
max geg(r.S)
(g) Rg)) s 2 + U. 2n
(E.117)
Hence, for any å E (0, 1), Eq. (E.109) holds with probability at least 6:
Ri
(r) 13
RF(f)
log(1/0)
2
Hy (Gr(r S,)) + U
2n
2v R((F)
20 C² 202
1
By(m))
Σ
MSE. (90tr.5h.) }
jEmiss(M)
+2 max (r5,
Ep (m)
Σ
(95.)) ((())))
E (((m)
Σ
A
(5)
ME)
JEmls(M)
20
C² + 20
Ep(m)
Σ
MMSE
(De(r.51.))
jEmis( M)
+ 2
max
Ep.
Σ
MSE.
(5.5
(ry)
)
Ep(m)
Σ
(
(1))
(r,,,(ry.3,)
jEmis(M)
(Emis(M)
(E.118)
By adding the MCAR constraint Theorem E.2, we prove Theorem B.8 follows:
37

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Proof of Theorem B.8. First, we have:
Ep, (m²) (m²)
Σ
RMSE..
Ep(m)
RMSE.
(r:)
(M)
jEmis M)
(m))
RMSEE
(355)
((1)))
Bp(m)
(ry) 535
(E.119)
JE
Here, define (r,; 5,) ((. + (a Oj ())) 9j E (r)),,,)))) Under the MCAR assumption, Theorem 11.3 of (Mohri et al., 2018) implies that for any [d] and any å E (0, 1), the following holds with probability at least 1 (/ 1):
Ep. (m))
RiAssE [3.5
(1:) Ep(m)
RASSE $3.
(
< 29R, (9, (r,) S2)) + UiV
log( (d + 1)/8)
(E.120)
2n
Thus, for any E (0, 1). Eq. (B.32) holds with probability at least 1 - 8:
R 1.8 - Rf)
log((d + 1)/8)
2 AM (T,(,(()):,))) + V
2n
+ 22 Ri(f)( + 20/ C² +
1
Ep(m)
Σ
BASE (Sc(r.59s)
20
jEmia((M
log((d (d+ 1)/8)
(T,;,,,)))) 201V
499%
2n
x
C² + 20
1
Po(mm)
Σ
R(M)
(B6(r.59.) + 49R; (G,(,, + 201
log( + 1)/8)
20
2n
MM)
(E.121)
E.7. Proof of Theorem B.9
Proof of Theorem B.9. Firstly, since X, is set of finite-pree mnite-precision decimals, for any j [d]. and 0² = (2m)-1, that C² < 0. The remainder of this proof follows a structure similar to that of the proof for discrete WFL (Sugiyama & Uchida, 2025). From assumption, there exist true deterministic functions 9; A x 10, + X2 for any j E [d]. and
-1 it follows
(gf ................................ ..) E G. In this case, for any and S, holds that g" E Grr, Hence, the following holds:
RASS, (Bo(r.5).) O, Vj E [d]. vm E {0,
(E.
For any j E [d], define Gy(r,, S;) as the set of hypotheses that satisfy the following two conditions: (i) each element is a solution obtained by methods that are guaranteed to achieve consistency (Mohri al., 2018), and (ii) its empirical risk is most ry- As increases and r, O, the theoretical guarantees of these learning methods for gj imply the following:
((m) (g,) + 0, Vg, E (r Vj e [d] vm E {0, 1]d.
(E. 123)
Additionally, by assumption, there exists true deterministic function f X - y for label prediction, and fr E F. Therefore, the following holds:
Rí(fz) 0.
(E.124)
38

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Thus, under the conditions of Theorem B.6, the following holds for fg.5 obtained through LAC-dWFL's step (ii):
+ 00, Ri,g(gg5) 0.
(E.125)
Furthermore, using Theorem B.1 with C² I 0 and 0² = 12 and additionally letting rj - 0 increases for any j E [d], the following holds:
+ 00, Ri Vg (r,
(E.126)
Since I is Ly-Lipschit continuous and I 9.3 3 is h.=Liipschtz.commimuuus, continuous, the following holds (Shalev-Shwartz & Ben-David, 2014):
( S)) (G
(E.127)
Consequently, if (r,, S)) and R2 (r, for any E [d] are monotonically decreasing with respect to n and converge to O, the error bound established Theorem B.8 converges to 0 as n + co.
F. Detail Information of Experiments
E1. Details of Datasets
For real-world datasets, we utilize four datasets from OpenMI. (Vanschoren et al., 2013): hls4ml Ihe jets hif (Pierini et al., 2020), electricity (Harries et al., 2014). my (Luis, 2014), and wwwlorrallliiooooeeee as Jets, Electricity, Mv, and Run-or- Walk, respectively. Table F.1 summarizes these datasets. All binary features were constrained to take values of either or 1. One-hot encoding was applied to all categorical features, and continuous features were scaled to ensure their values fall within the range [0, 1].
(Viktor, 2017). We refer to them
Table F.I. Outline of datasets. binary, categorical, and numerical represent the number of features of each type, respectively.
dataset
Jets
Electricity
Mv
Run-or- Walk
data size
830000
45312
40768
88588
binary
0
0
categorical
0
0
numerical
7
6
target
binary
binary
binary
binary
F.2. Details of Experimental setup
The experimental settings, except for the dataset, were almost identical to those used in the discrete WFL experi- ments (Sugiyama & Uchida, 2025). The differences lie in the settings for o² and (p.((..)) [logp, (X" [x Y)]. In this experiment, o² was set to (2r)-1 For continuous features, all were assumed to be umite-precision decimals, and (x.w) [logp. (a..( (X (X"Ix X Y)]
assumed. Consequently, the bound was computed by setting C. 0.
All other settings follow those in the discrete WFL experiment (Sugiyama & Uchida, 2025), as follows. The feature estimation models g and the label prediction model f were represented as two-layer perceptrons with hidden layers of width 500, using ReLUU as the activation function. All models were trained using Adam (Kingma & Ba, 2014) and the following hyperparameters: learning rate of 0.0005, batch size of 512, 100 epochs, and a weight decay 0.0002. Logistic loss is employed for training f.
We summarize the method used calculate the error bound presented in Theorem 4.3 for this experiment. First, the numerical values related to the error bound were set as 8 = 0.0001, Rf(fs) 0, and Uj 2.0. R((f>) was set under the assumption that F is sufficiently complex. In this experiment, the predicted label is determined by the label corresponding to the largest output value of f. Since scaling the outputs of f does not affect the results, it was assumed that the maximum value of each element in f's output is 1, and U, was set to 2.0. The logistic loss I, used in this experiment, is 1-Lipschitz continuous. Therefore, it follows that Ri, (Fi) = R (F) and Dig = RR(FF) (Lemma 26.9 in (Shalev-Shward Devid, 2014)).
39

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
Similar to the experiment in discrete WFL (Sugiyama & Uchida, 2025), Re (F) and Ree (F) were calculated using the upper bound on the Rademacher complexity of multilayer perceptron derived by Neyshabur et al. (Theorem in (Neyshabur et al., 2015)). For the parameters in this bound, we set μ 1 and p q q= 2. The parameter μ represents the upper bound on the 1-- nn-mo of all parameters in the model, and p specifies the type of 1p norm. Since all parameters of f can be scaled without affecting the inference results for predicting a single label, /s was set to 1.
G. Additional Experiments
In Section 5, we experimentally validated our theoretical results under the setting where all WFs are continuous variables that may contain missing values. In this setting, since the exact values of WFs are not always missing, some instances have observed exact values for certain WFs. In contrast, this section investigates scenarios in which all WFs for all instances are observed either with substantial noise or as intervals that contain the exact values.
Figure G.3 illustrates the relationship between the generalization error of fg.3 and the number of training samples n, for several g with varying levels of estimation error, Figure G.4 compares these generalization errors with the theoretical error bound presented in Theorem 4.3, Consistent with the results in Section 5, these findings confirm that our derived error bound effectively captures the relationship between the estimation error of g and the rate which the generalization error of gg decreases increases.
Jels
Electricity
My
-100- Run or Wilk
6,8
Raux/aa) -0.00 Vi [F*)
0.6
Mast -0.01 Vi 6 [P]]
Rasi, =0.55 Yj e (PM)
6.4-
Muss, (a.) -0.3 k5 [F*]
0.4
Rast (s.) -0.2 by 6 [F"]
6.2
(8)) -0.3
6.2
6.0-
Past, (a) -0,4 [F"]
Ranx,aa) -0.5 [F*]
1.1
1.5
00.
13
0.3
C.T
111
1.5
0.9
Figure G.3. The relationship between (
g. s) and the training data size n for various MSEs of g. This figure shows the cases in which all WFs for all instances are observed either with substantial noise or as intervals that contain the exact values.
Jets
Electricity
Mv
Run-or-Was
Peng GELDSSEUS
#
10
100
10"
Barifga (Rase (9.) -0,000 by (M7))
(Rorn, (0i) m0o0 (P7)
4 Barifg a)) (Rass -0,00 by (P*))
D. Bound (Rosn, -0ot by e (179)
Rasifo a) (Rassalo) -0.3 MJ (171)
Round (Rusa,, (o)) miL1 V0 e (PT))
Batifo a) (Rassuloy -0.3 Wy (171))
Bound (Ressu (n) -0.5 Vj e (171)
00000 8888888880
Barifus (Rareale)) -0.5 (177)
Bound (Ruan, (oy) -0.5 by (171):
0.1 0.8 0.5
0.7
0.
1.3 1.5
001 0.3
0.0
1.5 1.5
0.1 0.3
0.5
07
0.0
2.1
1.3
1.5
0.3 0.8
0.6
0.7
09
1. 1.5
<10
Figure G.4. A comparison between Ri, (f,,)) s) for various MSEs of g and the error bound derived in Theorem 4.3.
40

---

Unified Analysis of Continuous Weak Features Learning with Applications to Learning from Missing Data
H. Limitation
In this paper, we provided a unified theoretical analysis of continuous WFL, in parallel with the discrete WFL framework proposed by (Sugiyama & Uchida, 2025). At the same time, our analysis inherits several limitations similar those in the discrete WFL setting (Sugiyama & Uchida, 2025).
The first limitation is that the derived emror bounds do not take into account the feature importance of each WF with respect to the downstream prediction task. Extending the theoretical framework to incorporate feature importance remains an important direction for future research. The second limitation is that our analysis does not cover joint learning approaches where f and g are represented by a single model, nor does it handle settings in which the estimated outputs for some WFs are used as input features for estimating other WFs. Such more complex formulations are often important in practical applications, and developing methods and theoretical guarantees for them promising avenue for future work. It is considered that the theoretical results presented in this paper provide a solid foundation for addressing both of these future directions.


Reviews:
[{"review_id": "qHb7Gpu84b", "reviewer": "Reviewer_wW1H", "summary": "This paper aims to provide a systemical theoretical framework for continuous weak feature learning (WFL). Previous studies focus on discrete WFL while neglecting the continuous weak features. Moreover, they have not addressed the fundamental questions such as the influence of feature estimation and label prediction models on each other, and the precise conditions for theoretical guarantees. Built upon these motivations, this paper proposes to construct a general and systematical theoretical framework for continuous WFL. Furthermore, the authors extend the framework with discrete WFL to construct a more general framework.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and the detailed feedback, which will greatly help us enhance the quality of our paper.\n\n**The importance of continuous WFL:**\n\nExisting research on specific cases of WFL, such as ItR and CFL, has established them as a recognized research area. The discrete WFL framework, detailed in our supplemental material, provides a unified approach for handling various discrete WFs. However, numerous real-world scenarios involve continuous WFs, such as missing or noisy values, observational errors, or interval-based observations containing exact values. For instance, continuous WFs are caused by measurement errors or insufficient precision in sensor readings. Additionally, anonymized personal data where numerical attributes like age, blood pressure, or purchase frequency are provided as intervals. These cases frequently arise in medical and industrial applications, highlighting the significance of continuous WFL. Since discrete WFL alone cannot theoretically accommodate these scenarios, developing a framework for continuous WFL is crucial.\n\n**The theoretical difference between discrete WFL and continuous WFL:**\n\nAs stated in the introduction, discrete WFL relies on the discrete nature of WFs and cannot be trivially extended to continuous cases. This is evident when comparing the proofs of Theorem 3.1 and Lemma 4.2 in continuous WFL with the proofs of Theorem 3.1 and Lemma 4.1 in discrete WFL, which employ fundamentally different approaches. Moreover, while previous studies have addressed individual cases of continuous WFs, no unified theoretical framework exists. Our work formulates and analyzes such a framework, making a significant contribution to the field. To better convey the importance of our research, we will expand the introduction in the Camera Ready version. Additionally, we will clarify the differences in proof techniques by adding further details in the Appendix.\n\nThe theoretical significance of continuous WFL and its distinction from discrete WFL are discussed in our response to Reviewer Ck1h (Reply Section: Clarification of the Theoretical Contribution of continuous WFL). We kindly refer you to that section for further details.\n\n**Experiments Conducted on Other Types of Data:**\n\nWhile our experiments focus on missing data, the proposed framework can be applied to scenarios involving measurement errors and ambiguous observations as well. We are currently conducting additional experiments to evaluate the performance under these conditions. The preliminary results indicate that the experimental results conducted on other types of WFs remain qualitatively similar to the missing data case. We will include these experimental results in the Appendix of the Camera Ready version.\n\n**The hypothesis of random weak features:**\n\nWe acknowledge that restricting the probabilistic model $q_g$​ for modeling the randomness of weak features (WFs) to a Gaussian distribution is a limitation of our approach. However, this theoretical hypothesis allowed us to derive rich analytical insights, such as the relationship between the risk of $f$ and the MSE-based risk of $g$. Extending beyond this assumption is an important direction for future work.\n\nRegarding the assumption that the variance $\\sigma^2$ is constant, we clarify that it can be generalized to depend on observed ordinary features $X^\\mathrm{o}$, i.e., $\\sigma^2(x^{\\mathrm{o}})$. In this case, the current error bound involving $\\sigma^2$ remains valid by replacing $\\sigma^2$ with $\\max_{x^{\\mathrm{o}}}\\sigma^2(x^{\\mathrm{o}})$. We did not explicitly state this in the paper and will clarify it in the Camera Ready version. \n\n**About Citation “Anonymous. A unified …”:**\n\nThe cited paper is our previous work on discrete WFL, which is currently under review. Following ICML submission guidelines, we have anonymized it and included it in the supplementary material. Regarding the citation after acceptance, if the discrete WFL paper is accepted by then, we will update the reference accordingly. Otherwise, we will upload the discrete WFL paper to arXiv and update the citation with the corresponding information.", "claims_and_evidence": "Weaknesses:\n- The importance of continuous WFL is underexplored since the value of discrete WFL has not yet been acknowledged.\n- Moreover, there is no adequate explanation of continuous WFL, as well as its difference from discrete WFL. It is better to provide some examples and their applications, otherwise this paper will be hard to follow.\n- There is an over-claim regarding the weak features. The claim of weak features includes missingness, measurement errors, or ambiguous observations, while the experiments are only conducted on missing data.", "methods_and_evaluation_criteria": "Strengths:\n- The experimental settings and evaluation criteria are properly provided.", "theoretical_claims": "Strengths:\n- The proposed framework has appropriately analyzed the error bounds with feature estimation and label prediction models. The theoretical analysis makes sense.\n\nWeaknesses:\n- The hypothesis of random weak features is a bit weak. In practice, the weak features may be relative to the exact features or the observed features.", "experimental_designs_or_analyses": "Strengths:\n- The experimental results can support the proposed idea.\n\nWeaknesses:\n- The experiments are only conducted on learning with missing data.", "relation_to_broader_scientific_literature": "Strengths:\n- This paper is related to learning from missing data. The authors have adequately discussed the works in such areas.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n- The reference for \"Anonymous. A unified framework for generalization error analysis of learning with arbitrary discrete weak features\" is confusing. This paper cannot be searched on the web now. How will you refer to this citation after your paper is accepted?", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4hBVcmWEd2", "reviewer": "Reviewer_Ck1h", "summary": "This paper proposes a unified analysis framework of weak features learning, where part of the features are inaccurate. The paper analyzes generalization performance of a class of learning algorithms, in which a feature predictor $g_j$ is learned for all dimension of \"weak feature\", and a classifier $f$ is learned to make the final prediction. The authors analyze the generalization error of $f$ when $g$ is fixed, as well as that of $g$ when $f$ is fixed, demonstrating that weak features learning is feasible under some conditions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your careful examination of our paper and for their helpful suggestions to improve the clarity and depth of our research.\n\n**Clarification of the Theoretical Contribution of Continuous WFL:**\n\nThank you for your suggestion. Our main contribution is a unified framework for continuous WFL with theoretical analysis. We acknowledge its limitations and summarize them below:\n\n(1) the main challenge of weak features learning:\n\nThe main challenge in WFL is that simply combining existing learning theories for $f$ and $g$ does not guarantee consistency and convergence rates in WFL. Specifically, the learning of $f$ depends on $g$, and vice versa, but supervised learning theory does not account for this interdependence. Consequently, prior theoretical frameworks could not explicitly analyze how errors in $g$ (or $f$) affect the learning efficiency of $f$ (or $g$).\n\n(2) how the current analysis resolve the challenge:\n\nTo address this issue, our analysis explicitly models the interaction between $f$ and $g$. We first develop mathematical tools to capture this mutual dependence (Lemmas 4.1 and 4.2). Using these tools, we derive error bounds that illustrate how the risk of $g$ (or $f$) influences the convergence rate of the risk of $f$ (or $g$) (Theorems 4.3 and 4.5). These error bounds provide precise insights into how the consistency and convergence rates of WFL evolve based on the risk of $g$ (or $f$), as detailed in Sections 4.2 and 4.3. For distinctions between discrete WFL and our setting, please refer to our response to Reviewer wW1H.\n\n(3) the limitation of current analysis due to the development of the supervised learning theory:\n\nAs you pointed out, our analysis builds upon existing supervised learning theory. Consequently, our framework does not account for the feature importance of WFs in the downstream task, which remains a limitation. \n\n(4) the limitation of current framework due to the mathematical model itself:\n\nMoreover, since our mathematical model explicitly separates $f$ and $g$, it does not theoretically address approaches that jointly learn them as a single model. Addressing these challenges is an important direction for future work.\n\nFor (1) and (2), we will clarify the significance and positioning of our theoretical results in the discussions following each analysis. For (3) and (4), we will explicitly state these limitations as future work in the Conclusion.\n\n**Question (1a) The reason for adopting MSE:**\n\nWe adopt MSE for both analytical necessity and simplicity. It allows us to directly apply existing error bounds for learning $g$, which typically use MSE, to Theorem 4.3, where the error bound of $f$ is also expressed in MSE-based risk. Additionally, MSE enables key mathematical tools, such as Lemma 4.2, which captures the interaction between $f$ and $g$ and facilitates our theoretical analysis of WFL. While similar analyses might be possible with other metrics like mean absolute error, we have not yet explored this direction in depth.\n\n**Question (1b) The Reason for the Assumption of Feature Estimation Models:**\n\nThe Gaussian assumption for feature estimation models is made for both analytical necessity and simplicity. This assumption enables us to relate the risk of $f$ to the MSE-based risk of $g$. However this assumption is not necessarily unrealistic, as Gaussian noise is often added to deterministic outputs of $g$ to represent prediction uncertainty, and Bayesian models frequently yield Gaussian predictive distributions. Thus, while it simplifies analysis, it remains relevant in practical settings.\n\n**Question (2) Non-triviality of Our Generalization Bounds and the Condition:**\n\nAlthough the formulas are complex, Theorem 4.3 provides a non-trivial bound, as the RHS of Eq. (4.11) does not include second-order terms of $U_l$​. The first term is purely first-order, while the second involves products of ${U_l}^{1/2}$, leading to at most a first-order dependence.\n\nRegarding Theorem 4.5, as you noted, it includes a quadratic term in $U_l$​. However, this term is scaled by $\\sqrt{\\log(1/\\delta) / 2n}$​, which becomes small when $n$ is large. Thus, the quadratic term’s overall impact remains limited, ensuring that Eq. (4.13) remains non-trivial in large-sample regimes.\n\n**Question (3) Applicability of Our Theoretical Bounds to Alternative or Simultaneous Update:**\n\nTheorems 4.3 and 4.5 are applicable to an alternative update. Theorem 4.3 provides a generalization bound for $f$ given any $g$, while Theorem 4.5 does so for $g$ given any $f$. During training, Theorem 4.3 applies when updating $f$ with a fixed $g’$, and Theorem 4.5 applies when updating $g$ with a fixed $f’$. This allows generalization error analysis at each step of an alternative update process.\n\nHowever, our current framework does not provide theoretical guarantees for simultaneous update, which remains an important direction for future work.\n\n\n**About Typos:**\n\nThank you for pointing that out. We will correct it.", "claims_and_evidence": "Generally yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The theoretical claims are generally correct. However, some of the error bounds are a bit too loose, making them unable to provide theoretical insights, which is particularly important as the paper is mainly about \"unified analysis\". Please see weaknesses for the detailed comment.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: The problem formulation is clear and general, the symbols are standard, and the proofs are relatively easy to follow.\n\nWeaknesses: \n\nMy main concern is that the results seems a bit too straightforward, thus lack of insight more than supervised learning. Specifically, Theorem 4.3 and 4.4 demonstrate that a good generalization performance can be obtained when the feature predictors $g$ and the classifier $f$ can both be learned well. I think the problem of weak features learning is not merely a process of learning the value of weak features via ordinary features plus a process of classical supervised learning. \n\nDue to this concern, I wonder if the authors could conduct a clearer discussion that separates the following topics: (1) the main challenge of weak features learning, (2) how the current analysis resolve the challenge, (3) the limitation of current analysis due to the development of the supervised learning theory, and (4) the limitation of current framework due to the mathematical model itself. \n\nI think a convincing discussion of above topics in the paper will change my evaluation, even if the result of discussion is rather negative (weak feature learning is feasible only if the features are actually not weak).", "other_comments_or_suggestions": "Typos: Line 237~238, lack a colon \":\"; Line 239, \"for any...\" → \"For any...\"", "questions_for_authors": "The authors made several assumptions throughout the paper. While each of the assumptions has been justified, I still have some questions regarding the necessity and generality of some assumptions:\n\n(1a) In line 172-173, the performance of feature learner is measured via MSE. I wonder if this is necessary for the subsequent analysis, or just assumed for simplicity. \n\n(1b) In line 209-210, the randomized feature estimation model is assumed to be Gaussian. Is Gaussian a common model in real-world scenarios? Is Gaussian a necessary assumption, or just assumed for simplicity?\n\nBesides, I also have the following questions:\n\n(2) In Theorems 4.3 and 4.5, the upper bounds include a quadratic term of $U_l$, thus I wonder when would this bound be non-trivial (except the cases that either $f$ or $g$ has learned very well).\n\n(3) Moreover, Theorems 4.3 and 4.5 present the theoretical guarantee when either $f$ or $g$ is \\emph{fixed} and the other is optimized. I'm curious if there is any overall guarantee when $f$ and $g$ is both optimizable (i.e., alternatively update, or simultaneously update).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "caZVOq0BhN", "reviewer": "Reviewer_dvad", "summary": "This paper proposes a unified theoretical framework for Continuous Weak Feature Learning (or continuous WFL), addressing scenarios where input features are low-quality due to missing data, measurement errors, or ambiguous observations. The authors introduce a novel risk-based formulation specifically tailored for continuous weak features, distinguishing it clearly from previous work that focused only on discrete weak features. The paper derives new theoretical bounds to characterize the interaction between feature estimation models and downstream prediction models. Furthermore, the paper provides conditions under which sequential and iterative learning algorithms achieve theoretical consistency. Experimental results on real-world datasets empirically validate the theoretical claims.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you dedicated to evaluating our work and are grateful for your insightful feedback and constructive suggestions.\n\n\n**Practical Implications and Robustness of Mathematical Assumptions:**\n\nWe think that our assumptions, such as Lipschitz continuity and bounded loss functions, are standard in theoretical studies and not particularly restrictive. Many commonly used loss functions naturally satisfy these properties, ensuring the validity of our theoretical results in practical settings.\n\nWhile some loss functions, like MSE and 0-1 loss, do not strictly meet these assumptions, this is not a practical concern. MSE satisfies them when target data values are bounded, which is always the case in computational settings. The 0-1 loss is rarely used directly due to optimization challenges, and surrogate losses like hinge loss, which are Lipschitz continuous, are typically employed.\n\nEven if a loss function does not fully satisfy these assumptions, our framework remains practically relevant. Since it integrates existing learning methods for $f$ and $g$, an effective learning algorithm in standard supervised learning is likely to be effective in WFL as well.\n\n**Evaluation on More Complex and Realistic Scenarios:**\n\nWe appreciate your suggestion. As you pointed out, evaluating the framework on high-dimensional or more complex data distributions would further strengthen its practical relevance and validation. A similar concern was raised by Reviewer wW1H (Experiments Conducted on Other Types of Data). In the response, we have conducted additional experiments using different types of WFs to demonstrate the framework’s effectiveness in a broader range of scenarios. These experiments have further enhanced its practical relevance.\n\nTheoretically, our framework is not constrained by data complexity. However, empirical validation on high-dimensional datasets remains essential. We acknowledge that such evaluations would provide deeper insights and help identify future research directions. In this study, we focused on establishing the effectiveness of our framework in fundamental settings. Nevertheless, we plan to explore evaluations with more complex datasets in future work.\n\n\n**Potential Applicability to High-Impact Domains (e.g. Diffusion Model):**\n\n\nOur theoretical analysis establishes properties that are independent of the specific choices of learning methods for $f$ and $g$, meaning our framework is flexible and can incorporate models such as Diffusion Models. This flexibility is a key strength, suggesting that our approach is well-suited for contemporary high-impact domains. The precise performance when integrating Diffusion Models and other domains remains an open and intriguing question. In this study, we focused on developing and analyzing a general framework, and exploring its application to such advanced models is an important direction for future work.\n\n\n**Explicit Impact Statement of our paper:**\n\nWe have addressed a similar question from Reviewer wW1H. Please refer to our response (top 2 response for reviewer wW1H) there for details.\n\n\n**Applicability to Modern AI Models and High-Dimensional Feature Spaces:**\n\nAs mentioned earlier, our framework can be applied to various modern AI models. This is because it focuses on integrating different methods for learning $f$ and $g$. Moreover, the derived error bounds provide a theoretical analysis of how such $f$ and $g$ influence each other when using state-of-the-art learning methods. For instance, our framework enables a detailed understanding of how improvements in $g$'s error affect the learning efficiency of $f$. \nAdditionally, our theoretical results hold regardless of the data distribution as long as the data values are bounded. Therefore, our theory remains valid even in high-dimensional feature spaces, offering valuable insights.\n\n**Trade-offs Between Sequential and Iterative Learning:**\n\nIntuitively, iterative learning is expected to be more effective than sequential learning, albeit at the cost of increased training time. This advantage arises because iterative learning can enhance the accuracy of both $g$ and $f$.\n\nFirst, in iterative learning, the learning of $g$ benefits not only from the observed values of WFs but also from the information of $Y$ through $f$, potentially leading to lower error compared to sequential learning. Moreover, according to Theorem 4.2, such an improved $g$ contributes to a more accurate $f$, further reinforcing the benefits of iterative learning. However, the quantitative evaluation of this trade-off depends on the specific methods and datasets used, making it difficult to determine a generalizable conclusion without large-scale empirical validation.", "claims_and_evidence": "The main theoretical claims regarding the generalization error bounds and conditions for consistency appear rigorously presented, but their practical significance heavily relies on certain mathematical assumptions (e.g., Lipschitz continuity and bounded loss functions). While the empirical experiments generally support the theoretical findings, it remains unclear how sensitive these theoretical conditions are to deviations in practical scenarios. It would be helpful if authors could clarify these assumptions' practical implications or robustness.", "methods_and_evaluation_criteria": "The proposed theoretical framework and methods for evaluating the error bounds seem appropriate and logically sound. However, the evaluation heavily relies on relatively simple datasets from OpenML. Considering the complexity of recent machine learning applications, additional evaluation using more complex or realistic scenarios (such as datasets involving generative models or high-dimensional data) would enhance the practical relevance and validation of the proposed framework.", "theoretical_claims": "Due to limited familiarity with the detailed mathematical underpinnings of the theorems presented, I could not thoroughly verify the correctness of all proofs provided. However, the logical structure and steps outlined seem consistent with standard practices in theoretical machine learning. A thorough peer check from a reviewer deeply familiar with continuous week feature learning theory would be beneficial.", "experimental_designs_or_analyses": "I examined the experimental section, particularly Section 5, focusing on the generalization error analysis on real-world datasets. The experimental design effectively illustrates the proposed theoretical results about error convergence. However, the datasets used (Jets, Electricity, etc.) represent relatively simple, low-dimensional scenarios. Extending these evaluations to scenarios involving modern generative models, such as LLMs or diffusion models, would provide stronger evidence for the applicability and utility of the theory in contemporary, high-impact research domains.", "relation_to_broader_scientific_literature": "The paper effectively relates its contributions to previous theoretical work on discrete weak features learning, clearly highlighting the gap addressed by this work. Additionally, it integrates related work effectively, such as imputation methods (e.g., impute-then-regress) and complementary features learning, positioning its contributions within an established body of literature.", "essential_references_not_discussed": "At present, no glaring omissions of essential references are apparent. However, further scrutiny by reviewers more familiar with continuous weak features or semi-supervised learning might identify additional relevant studies.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Presents a unified theoretical framework clearly extending discrete WFL theory to continuous cases.\n2. Provides rigorous theoretical conditions for sequential and iterative learning methods ensuring consistency.\n\n**Weaknesses**\n\n1. The practical implications of theoretical assumptions are not fully discussed, potentially limiting the practical applicability.\n2. Experimental validations are restricted to relatively simple and low-dimensional datasets, lacking evaluation on more complex, high-dimensional tasks. Applicability to contemporary high-impact domains such as Large Language Models (LLMs) or Diffusion models is not explored, which could limit the perceived relevance of the proposed method in current research contexts.", "other_comments_or_suggestions": "- The paper is missing an explicit impact statement. Given the theoretical nature of the work, it would be helpful if the authors provided a discussion on the broader implications of their findings, including potential societal or industrial impacts. Clarifying how Continuous WFL could influence real-world machine learning applications would strengthen the paper.", "questions_for_authors": "- Given that many modern AI models deal with weak or missing features (e.g., masked tokens in LLMs, noise in generative models), it would be interesting to explore how your framework could be extended or adapted to these domains. Would the theoretical results hold in high-dimensional feature spaces commonly seen in LLMs?\n- What are the practical trade-offs between sequential learning (impute-then-predict) and iterative learning (joint training of feature estimation and prediction)? While the theoretical results suggest conditions for consistency in both learning paradigms, it is unclear in which real-world scenarios one approach may be preferable over the other.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides a unified theoretical framework for continuous weak feature learning, which is distinguished from previous work focusing on discrete weak feature learning. It also conducts experiments on real-world datasets to validate the effectiveness. The reviewers have raised concerns related to the practicality of theoretical assumptions, the intuitive theoretical results, and the relatively simple validation datasets. During the rebuttal phase, the authors have addressed most of the concerns of the reviewers. Although some concerns regarding the theoretical assumptions have not been completely addressed, the theoretical contribution of this paper is somewhat meaningful. The authors need to carefully revise their paper according to the comments, and include necessary evaluation experiments under more scenarios (According to their rebuttal to Reviewer wW1H, they have conducted additional experiments under measurement errors and ambiguous observations). Consequently, I recommend a weak acceptance of this paper.
Decision: Accept (poster)