S$^2$MAM: Semi-supervised Meta Additive Model for Robust Estimation and Variable Selection

Authors: Xuelin Zhang, Hong Chen, Yingjie Wang, Zeyu Zhang, Tieliang Gong, Bin Gu, Feng Zheng
Primary area: learning theory
Keywords: manifold regularization, bilevel optimization, sparse additive model, robustness, learning theory

Abstract:
Semi-supervised learning with manifold regularization is a classical family for learning from the labeled and unlabeled data jointly, where the key requirement is the support of unknown marginal distribution enjoys the geometric structure of a Riemannian manifold. Usually, the Laplace-Beltrami operator-based manifold regularization can be approximated empirically by the Laplacian regularization associated with the whole training data and its graph Laplacian matrix. However, the graph Laplacian matrix depends heavily on the pre-specifying similarity metric and may result in inappropriate penalties when facing redundant and noisy input variables. In order to address the above issues, this paper proposes a new semi-supervised meta additive model (S$^2$MAM) under a bilevel optimization scheme to automatically identify the informative variables, update the similarity matrix, and achieve the interpretable prediction simultaneously. Theoretical guarantees are provided for S$^2$MAM including the computing convergence and the statistical generalization bound. Experimental assessments on synthetic and real-world datasets validate the robustness and interpretability of the proposed approach.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729514982827

thread_type:
official_review

<thread_object>
creator: reviewer_5g1j

<content>
summary:
This paper proposes a new semi-supervised meta additive model (S2MAM) under a bilevel optimization scheme to automatically identify the informative variables, update the similarity matrix, and achieve the interpretable prediction simultaneously. Theoretical guarantees are provided for S2MAM including the computing convergence and the statistical generalization bound. Experimental assessments on synthetic and real-world datasets validate the robustness and interpretability of the proposed approach.

strengths:
This paper inject the meta learning strategy and sparse additive models into manifold regularized SSL framework, and formulate a new semi-supervised meta additive model (S2MAM) to realize automatic variable masking and sparse approximation for high-dimensional inputs even with noisy variables.

There are no technical errors, and the presentation and writing are clear.

An efficient implementation is employed here to solve the bilevel optimization problem, which avoids the heavy computing burden on the implicit hypergradient calculation.

weaknesses:
I am absolutely not in this field and the comments from me are not relatively professional. The comments in the following are just raised from the presentation or organization.                       

The authors propose the manifold regularized semi-supervised additive model. However, the authors do not analyze the important part "manifold regularized" in deepth for the presentation of the whole paper, making the paper inevitably lacks the novelty, i.e., the formulation of the manifold regularized sparse additive model is built by the combination of the existing works and the rationality behind this combination and the related theoretical analysis are not clearly given.

I wonder why the authors give many parts in presenting the semi-supervised additive model, i.e., Manifold Regularized Sparse Additive Model, Discrete Bilevel Framework for S2MAM, and Probabilistic Bilevel Framework for S2MAM. These parts seem to be all needed to be presented. However, the limited space in the paper seem to be the combination of these parts and the novelty of each part is dispersed to some degree. The definition of upper level and the lower level of discrete bilevel framework lack clear definition and the realted analysis in Eq. (3) and Eq. (4). The rationality of just using Bernoulli random variable to denote mi should be given, i.e., why it it simple and wheter the other types of random variables can be better in probalisitic bilevel framework.

The authors should add more recent methods for comparison in Table 3 and Table 5 to better show the effectiveness of the proposed method.

The authors just perform evaluations of S2MAM on eight real-world datasets from UCI repository. I think the authors can add other datasets for comparison in the experiment, which is not limited to UCI repository.

questions:
1. Can the authors analysis the extreme cases when the fixed label percentages are high, i.e., r=50% and the noisy variables are high in the experiment, i.e., 100 noisy variables.

2. The authors can further analyze why the average accuracy on Breast Cancer achieve such high performance on Breast Cancer, i.e., the proposed method on unlabeled case is about 88 and the second best performance is just 77.197.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729843752105

thread_type:
official_review

<thread_object>
creator: reviewer_iykb

<content>
summary:
This paper presents a model named S2MAM, which aims to improve classical graph-based semi-supervised learning (GSSL) by combining bilevel optimization with manifold regularization to select relevant features and adapt similarity matrices in noisy and redundant data settings. S2MAM addresses issues in traditional manifold regularization, where fixed similarity metrics can perform poorly with uninformative features. The authors propose a probabilistic bilevel optimization framework to apply adaptive variable masking and provide theoretical proofs of convergence and generalization bounds. Empirical evaluations on synthetic and small real-world data suggest that S2MAM performs well in both regression and classification tasks, particularly under noisy conditions.

strengths:
* The method partially solves the issue of poor similarity matrix in the graph-based SSL methods
* The authors provided convergence and generalization bounds, which enhances the model's credibility
* The proposed method outperforms the other GSSL methods of the same kind

weaknesses:
The method still has many limitations of the graph-based GSSL methods. Therefore the impact of the method is very limited.
* It is expensive. The W or L matrix is infeasible when l is large. The method involves multiple kernel matrices, which is even more expensive.
* The assumption of a single bandwidth mu is unrealistic
* The construction of W is problematic for many data types, e.g., images, text documents, and biology sequences.
* The experiments are not convincing because all tested datasets are very small.

questions:
* The computational complexity is not ananlyzed.
* The prediction function requires all training samples. This is rather infeasible in practice. If changed to other prediction functions, does the theoretical guarantee still hold?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729993997941

thread_type:
official_review

<thread_object>
creator: reviewer_hk4x

<content>
summary:
This paper proposes a novel semi-supervised meta additive model where a bilevel optimization scheme is employed to identify the informative features. The authors analyze the properties of the model including the convergence and statistical generalization   bound. Experimental results on toy and real-world data sets demonstrate promising improvements over previous methods.

strengths:
1. The paper is well-organized.
2. This paper proposes a bilevel optimization scheme to select informative features.
3. The authors provided theoretical guarantees for the proposed model.

weaknesses:
1. The authors employed a sampling strategy from Bernoulli distributions to select informative features. Do selected features contain noisy features by using this strategy? If this occurs, how to remove noisy features?
2. Hyperparameter C affects the selection of  informative features. It would be much better to discuss the effect of this hyperparameter or give the specific values in the experiments.
3. The authors used the batch in each iteration. In practical implementations,  what is the proportion of the labeled samples in the batch since the unlabeled samples dominate the training set? 
4. The authors assume  that $\Phi(s)$ is L smooth. For the model in this paper,  how to compute this constant L?
5.  The authors employed dimension reduction techniques to reduce the dimensions of high-dimensional data.  Is the method effective for directly selecting features of high-dimensional data such as gene expression data?

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731018467

thread_type:
official_review

<thread_object>
creator: reviewer_gvds

<content>
summary:
The paper introduces a new Semi-Supervised Meta Additive Model (S2MAM) that aims to improve the robustness and interpretability of semi-supervised learning through bilevel optimization. The proposed model combines manifold regularization, meta-learning, and sparse additive modeling to automatically identify informative features while simultaneously performing semi-supervised learning tasks. Unlike existing methods that rely on pre-specified similarity matrices, S2MAM leverages probabilistic bilevel optimization to adaptively update both the similarity matrix and decision functions. The authors provide theoretical guarantees regarding convergence and generalization, and they validate the proposed model using synthetic and real-world datasets, demonstrating its robustness and interpretability against noisy and redundant variables.

strengths:
(1) The paper introduces an innovative bilevel optimization framework for manifold regularization that addresses key limitations of existing semi-supervised learning models, particularly those involving noisy and redundant input variables. 
(2) The paper offers a rigorous theoretical foundation, including proofs for optimization convergence and bounds for generalization error. These theoretical results provide assurance about the robustness and applicability of S2MAM in different learning scenarios.
(3) Extensive experiments on both synthetic and real-world datasets show that S2MAM outperforms other state-of-the-art semi-supervised learning models, particularly in scenarios with noisy and redundant features. The robustness and interpretability claims are supported by competitive performance results.

weaknesses:
(1) While the paper provides a novel framework, the bilevel optimization procedure can be computationally expensive, especially for large-scale datasets. Despite the probabilistic bilevel framework simplifying the optimization process, the overall complexity might still be a limitation in practical applications.
(2) The proposed method involves both Laplacian matrix computation and optimization over probabilistic masks, which may not scale efficiently to very high-dimensional datasets or very large sample sizes. Although theoretical guarantees are provided, the lack of a discussion on practical scalability leaves questions about how S2MAM would handle real-world big data scenarios.
(3) While the paper compares S2MAM against a wide variety of baselines, some comparisons (especially against modern deep learning-based semi-supervised models) are limited. This is particularly important since deep learning approaches have become dominant in semi-supervised learning tasks.

questions:
(1) Could the authors clarify the learning dynamics of the probabilistic mask variables? Specifically, how does the model ensure convergence of these masks to truly informative features, and is there a risk of instability in mask selection during the optimization?
(2) How does S2MAM's computational complexity compare to models like LapSVM or AWSSL? Specifically, can the authors clarify its scalability for high-dimensional datasets and large sample sizes?
(3) The paper compares S2MAM with several established baselines, including LapSVM, AWSSL. However, the comparisons with recent deep learning-based semi-supervised learning methods are limited. Could the authors provide additional experimental comparisons with state-of-the-art deep learning-based SSL methods to better evaluate S2MAM's competitive performance?
(4) The datasets used in the experiments appear relatively small. Could the authors evaluate S2MAM on larger, more complex datasets to demonstrate its scalability and robustness in real-world scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732190243765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common concerns from reviewers

comment:
We thank all reviewers for their valuable and constructive comments. We address the common
concerns here and post a point-to-point response to each reviewer as well. Notice that some new results are reported in the resubmitted PDF, highlighted in red color.

### **Common Concerns (1): Experiments on Complex Data**

Indeed, this paper conducts experiments on 4 synthetic datasets, 8 UCI datasets and **one image-type COIL dataset**. And the following complex data scenarios have already been empirically considered:

1. **Large-size Data.** Both Buzz-Regression and Buzz-Classification datasets contain 38393 samples with 77 features. Please refer to Appendix B.1 for detailed descriptions of the selected datasets.  
2. **High-dimensional Data.** The used images are selected from the COIL20 image library with 16384 original features (128x128). Corresponding results are shown in Tables 11\&12.
3. **More Labeled Training Samples.** In Table 10 (Appendix B.5, Page 25), the training set contains 5% or 50% labeled samples.
4. **Various Noisy Corruption.** The irrelevant and extremely noisy features are drawn from the distributions like Chi-square distribution and Gaussian distribution. The number of noisy dimensions varies in [0,10,100].



### **Common Concerns (2): Extensions to high-dimensional or non-numeric data**

1) **Existing high-dimensional applications**

As stated in Line 530 (Page 10), results of high-dimensional applications (e.g., images) are left in Appendix B.7 (Tables 11\&12). 


2) **Methodology for processing high-dimensional or non-numeric data**

Inspired by some supervised [1] and semi-supervised works [2], an interesting approach for dealing with high-dimensional data like images and texts is to extract the feature vectors first. Following [2] and AWSSL, we first use a CNN to learn the vectors with a fixed number of features, which realizes rough dimensional reduction. However, this step may not remove those irrelevant or even noisy features.

Similar preprocessing methods for feature extraction and dimensional reduction also apply to larger or non-numeric datasets (image, text or biological data). 

> [1] Stab-GKnock: Controlled variable selection for partially linear models using generalized knockoffs. 
>
> [2] Robust embedding regression for semi-supervised learning.
>
> [3] Predicting effects of noncoding variants with deep learning–based sequence model. 

### **Common Concerns (3): More deep learning-based competitors**

Following your constructive suggestions, we've added discussions and empirical comparisons on some recent deep learning baselines [1,2]. 

1) New results are reported in the revised pdf highlighted in red color. 

Unsurprisingly, the supervised neural network [1] fails to select the informative features or predict with few labeled samples. 
S2MAM still shows better robustness against noisy dimensions than those semi-supervised networks including [2], VAE and SSDKL, especially under noisy circumstances.

2) Model descriptions for [1,2] are added in Appendix B.2.

> [1] Consistent feature selection for analytic deep neural networks. 
>
> [2] NP-Match: When Neural Processes meet Semi-Supervised Learning. 



### **Common Concerns (4): Computation cost analysis**

**Indeed, the computational complexity has been theoretically analyzed in Appendix E.5.** The computation complexity of S2MAM reaches $\mathcal{O}\left(\frac{p(l+u)}{\epsilon_1^2 \epsilon_2} \right)$, where $\epsilon_1,\epsilon_2$ are the stationary parameters, $p,l,u$ stand for the sample dimensions, labeled and unlabeled sizes.

And the **empirical comparisons in training time cost on complex image data are left in Appendix B.7.**

In the following, we further restate the reasons at Appendix F for the higher computation costs as well as some techniques for acceleration.

**1) Why higher cost?**
As pointed out in Appendix B.7 and Appendix F, we admit the computation limitations of S2MAM. 
This is mainly caused by:

1) The bilevel optimization requires more iterations to learn the additional masks.
2) The additive scheme expands the data dimensions to provide interpretable feature-wise contributions.

**2)  How to accelerate?**

Indeed, the probabilistic formulation and policy gradient estimation have been used in this work to reduce the computation burden of bilevel optimization.

To further accelerate the computation process, the random Fourier acceleration technique [1,2] and some dimensional reduction strategies [3] can be exploited to approximate the additive Gram matrix, which has been previously validated to be effective for additive models [4]. 

> [1] Random features for large-scale kernel machines.
>
> [2] Towards a unified analysis of random Fourier features.
>
> [3] Robust embedding regression for semi-supervised learning.
>
> [4] Tilted Sparse Additive Models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733051915991

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dear Reviewers,

comment:
Once again, we sincerely appreciate your comments, which has been extremely helpful for our work. We hope this message does not cause any inconvenience, but as we only have a few days left for discussions, we would like to ask whether our responses have sufficiently addressed your concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a Semi-Supervised Meta Additive Model (S2MAM) that aims to improve the robustness and interpretability of semi-supervised learning through bilevel optimization. The proposed model combines manifold regularization, meta-learning, and sparse additive modeling to automatically identify informative features while simultaneously performing semi-supervised learning tasks. Theoretical guarantees are presented along with numerical experiments.

While reviewers appreciate the novelty of the method and the theoretical analysis, they also have major concerns about its high computational cost, especially for large data sets.