Weakly-Supervised Disentangled Representation Learning via Filter-Based Adaptive Swapping
Authors: Zhenyu Zong
Qidi Wang
Simon Yu
Hongpeng Cao
Yanbing Mao
Han Zhao
Lui Sha
Huajie Shao
Venue: Accepted by TMLR ()

Abstract: Disentangled representation learning (DRL) aims to uncover semantically meaningful latent factors from observed data, thereby improving both interpretability and generalization of machine learning (ML) models. Despite remarkable progress, unsupervised DRL cannot achieve complete disentanglement without inductive biases or supervision. To address this challenge, existing approaches either rely on full supervision, which demands extensive manual labeling, or weak supervision, which involves complex training strategies that often result in unstable training. To address these limitations, we propose Filter-VAE, a weakly supervised variational autoencoder (VAE) that introduces a filter-based adaptive swapping strategy to learn stable and meaningful disentangled representations. Specifically, a relevance filter removes semantically meaningless latent factors, while an adaptive swapping filter exchanges those latent factors that have reached stability. With these two filters, Filter-VAE adaptively swaps only stable and semantically aligned latent factors, leading to robust and meaningful representations. We evaluate Filter-VAE on three standard benchmarks and our created traffic sign dataset in two downstream tasks: disentanglement and adversarial robustness. Experimental results demonstrate that Filter-VAE achieves strong disentanglement performance with reduced supervision and delivers remarkable robustness against diverse adversarial attacks and corruptions. The code is released at https://github.com/ZY-Zong/Filter-VAE.git.

Published in Transactions on Machine Learning Research (01/2026)
Weakly-Supervised Disentangled Representation Learning via Filter-Based Adaptive Swapping
Zhenyu Zong
zzong@uum.co edu
Deportment of Computer Science William & Mary
Qidi Wang
qwang25@amm @vomm edu
Department Computer Science William & Mory
Simon Yu
jundayu2@llinoiss edu
Department of Electrical and Computer Engineering University of Mlinois Urbana- Champoign
Hongpeng Cao
cao aao.hongpeng@tum.dde
School of Engineering and Design Technical Universily of Munich
Yanbing Mao
hmm99662hhaamme..d
Engineering Technology
Wayne State University
Han Zhao
bunchao@ilknnosss edu
Department of Computer Science
University of Blinois Urbona- Chanmipgn
Lui Sha
Ins@iliinois. edu
Department of Computer Science University of Blinois Urbena- Chaappiin
Huajie Shao
hshao Erooo edu
Department of Computer Science Willimm & Mary
Reviewed on OpenReview: htt tp //op vi .n et /f or K6 KK 2U
Abstract
Disentangled representation learning (DRL) aims to uncover semantically meaningful la- tent factors from observed data, thereby improving both interpretability and generalization of machine learning (ML) models. Despite remarkable progress, unsupervised DRL can- not achieve complete disentanglement without inductive biases or supervision. To address this challenge, existing approaches either rely on full supervision, which demands exten- sive manual labeling, weak supervision, which involves complex training strategies that often result in unstable training. To address these limitations, propose Filter-VAr VAE, weakly supervised variational autocncoder (VAE) that introduces filter-based adaptive swapping strategy to learn stable and meaningful disentangled representations. Specifi- cally, a relevance filter removes semantically meaningless latent factors, while an adap- tive swapping filter exchanges those latent factors that have reached stability. With these two filters, Filter VAE adaptively swaps only stable and semantically aligned latent fac- tors, leading to robust and meaningful representations. We evaluate Filter- three

---

Published in Transactions on Machine Learning Research (01/2026)
standard benchmarks and our created traffic sign dataset in two downstream tasks: disen- tanglement and adversarial robustness. Experimental results demonstrate that Filter-VAR achieves strong disentanglement performance with reduced supervision and delivers remark- able robustness against diverse adversarial attacks and corruptions. The code is relcased at httpp//ttuu..cm//2222o//e//eeee/
1 Introduction
Disentangled representation learning (DRL) aims uncover semantically meaningful latent representations that independently capture the underlying generative sources of variation in observed data. Such represen- tations contain high-level concepts that are understandable to humans- fonnans-foraa example, the shape and color of different traffic signs. This offers several benefits, including improved interpretability, enhanced sample efficiency in downstream tasks, and better generalization capability across domains. DRL has demonstrated success in several domains such as image generation, natural language processing, and recommendation systems (Wang et al.] 2024).
Over the past few years, a wide range of DRL methods have been proposed, broadly categorized into three groups: unsupervised, fully supervised, and weakly supervised approaches. Unsupervised methods such as 3-VAE (Higgins et al. 2017) and FactorVAE (Kim & Mnih] 2018) aim to discover disentangled factors without any labeled supervision. Despite the success of unsupervised DRL on synthetic datasets, [ocatello ett al. (019 have pointed out that unsupervised methods are not effective without. inductive biases or auxiliary information. To address this issue, some works have introduced fully-supervised DRLs, such as Concept Bottleneck Models (Koh et al.] (2200) and ML-VAE (Bouchacourt ct al., 2018), which use explicit annotations that are aligned with latent variables to guide the learning process toward meaningful latent factors. However, existing methods are limited by the need for extensive supervision, which may be costly and impractical in many world applications.
Weakly-supervieed approaches offer a compromise by leveraging limited or indirect supervision, such as partial labels, pairwise similarity, causal priors, and labeling functions (Feng et al. 2018 Locatello et al., 2020at Shen et al., 2022 Tonolini et al.] 2023), combined with specifically designed training strategies. For example, a very recent method, SW-VAE (Zhu et al. 2023), utilizes annotations indicating the maximum number of varying factors and introduces an optimized latent swapping strategy to improve disentanglement. However, SW-VAE still faces two major limitations. First, it relies
warm-up stage that swaps latent factors without confirming whether semantically meaningful features have been learned. This can lead to unstable training when the latent factors are not yet fully disentangled. Second, SW-VAE requires annotations specifying multiple distinct generative factors for all pairwise samples, which still demands substantial labeling effort. More recent weakly-supervised methods have shifted their focus toward other downstream tasks such as binary classification (Tonolini et al. 2023), image restoration (Zheng et al. 2024 and 3D face modeling (Li et al. 2024), rather than disentanglement.
Motivated by the limitations of SW-VAE, we propose Filter-VAL VAE, weakly-supervised variational autoen- coder that employs a filter-based adaptive swapping strategy to learn stable and meaningful disentangled representations. However, developing Filter-VAL poses two key challenges: (i) not all latent dimensions encode semantically meaningful features, and (ii) frequently swapping latent factors that are not yet fully disentangled can result. unstable feature learning.
To address the first challenge, we propose a relevance filter that removes meaningless (noisy) latent factors by applying threshold to identify meaningful latent factors. To tackle the second challenge, we introduce adaptive swapping filter that identifies stable latent factors by computing the KL divergence between pairs of latent variables and then adaptively swaps them based on this divergence. Compared to previous works, the combination of these two filters can locale positions of semantically meaningful latent factors and prevent the swapping of unstable latent features. Additionally, our method requires less prior knowledge, needing only single distinct generative factor for small amount of the pairwise data. We evaluate Filter- Vilter-VAE on multiple benchmark datasets and a created traffic sign dataset in two downstream tasks: disentanglement and adversarial robustness. Extensive experimental results demonstrate that our method
2

---

Published in Transactions on Machine Learning Research (01/2026)
outperforms both unsupervised and weakly-supervised basclines in the disentanglement task. Moreover, it achieves superior adversarial robustness compared to existing defense baselines under various attacks and corruptions. These findings underscore the effectiveness of the proposed method in learning robust and stable latent representations from observed data.
In summary, our contributions are three-fold as follows:
Novel disentanglement learning model. We propose Filter-VAE, novel weakly supervised VAE that incorporates two filters to achieve stable disentanglement using small amount of labeled pairwise data.
Synthetic traffic sign dataset. We create new synthetic traffic sign dataset to assess both the disentanglement performance and adversarial robustness of our method.
Superior performance in two downstream tasks. Evaluation results demonstrate that the proposed Filter- VAE achieves better disentanglement performance than the baselines on multiple datasets with less supervision. Moreover, it can significantly enhance the adversarial robustness of ML predictions under 17 adversarial attacks and corruptions.
2
Related Work
2.1 Unsupervised DRL
One representative type of unsupervised DRL is the Variational Auto-encoder (VAE) (Kingma & Welling 2014), which encompasses an encoder and a decoder. The core idea is to model data distributions by maximizing their variational inference in an unsupervised manner (Wang et. al. 2024). In the past few years, many studies have proposed unsupervised VAE and its variants (Burgess et al. 2018 Chen et al.
2018), such as 3-VAE (Higgins et al., 2017), FactorVAE (Kim Mnih 2018), and ControlVAL
(Shao et al.
2020b 2022p. learn the disentangled representations from observed data without the requirement of labels. These methods introduce the trade-off between reconstruction quality and disentanglement by adding new regularizer in the objective function, such mutual information total correlation (Chen et al. 2018). While they have shown good performance in synthetic datasets, Locatello et al. (2019) pointed out that it is theoretically impossible to fully disentangle latent factors using unsupervised learning without inductive bias
2.2 Fully-supervised DRL
Fully-superviedd DRL approaches address the limitation of unsupervised methods by leveraging full supervi- sion, where ground-truth generative factors concept annotations are assumed available. Models like Label Supervised VAE (Bouchacourt al.) 2018), directly conditioned labeled factors, achieving disentangle- ment by aligning latent variables with supervised attributes. Concept Bottleneck Models (CBMs) (Koh et al. 2020) and ProtoPNet (Chen et al. 2019). focused on interpretable classification by forcing the model to base decisions on human-detined concepts. CausalVAE (Yang et al., 2021) further leveraged supervision signals to learn structural causal model as prior. While supervised DRL can achieve high disentanglement quality, it relies heavily on labeled data, which often costly or infeasible obtain in real-world scenarios.
2.3 Weakly-supervised DRL
To avoid massive labeling requirements while still achicving disentanglement, researchers have explored weakly-suerriseed VAEs that leverage partial or indirect supervision. Early approaches leverage limited labels on input subsets (Locatello et al.) 2020b) or pairwise annotations (Locatello et al. 2020a). Subse- quent works extend this ides using different forms of supervision such as similarity measurements (Chen) & Batmanghelich 2020), the number of different generative factors (Locatello al. 2020a). and the max-
imum number of different generative factors (Zhu al. 2023 More recently, weakly-supervised VAFs
3

---

Published in Transactions on Machine Learning Research (01/2026)
have been adapted for other downstream tasks, including binary classification (Tooolin et al. 2023), im- age restoration (Zheng et al. 2024), and 3D face modeling [Li et al. 2024). which lie outside the scope of disentanglement itself.
A parallel line of work builds on the idea of latent swapping, which is first introduced by DNA-GAN (Xiao eet al. 2018) in the supervised setting. This strategy has been adopted both self-supervised and weakly- supervised VAEs. For example, Swap-VALE (Liu al. 2021) proposed block-wise swapping self-supervised framework, DSD (Feng et al. 2018) extended it with dual swapping under weakly supervision, and SW-
VAE Zhu et al. 2023) further optimized it by progressively increasing the number of swapping factors, However, these methods often perform swaps without verifying whether the swapped factors are scmantically meaningful, leading to unstable training if the latent factors are not yet fully disentangled. In this work, we address these shortcomings by introducing an adaptive swapping strategy that delays swapping until semantically meaningful latent. factors have stabilized. Moreover, our method achieves disentanglement with less supervision, requiring only 10% of pairwise inputs annotated with single distinct generative factor.
3 Preliminaries
3.1 Problem Statement
The goal of this work to learn semantically meaningful latent. representations from observations with limited supervised signals. The primary challenge lies in effectively disentangling independent latent factors that are aligned with the ground truth generative factors. Given batch of N observed image {æ11 2., ZN ) DRL aims to find mapping f((x) - where E Rd denotes the learned factors in latent space with the dimension size of d (Eastwood & Williams 2018). Each latent factor is expected to represent one and only one explanatory factor of æ so that modifying one latent factor doesn't affect the other factors of variation. This guarantees each latent factor is independent of the others. However, since some factors are not semantically meaningful, the number explanatory factors always less than or equal the number of latent factors; namely, [o| < d.
3.2 Variational Autoencoder
Variational autoencoder (VAE) KKnggma & Welling 2014 a popular unsupervised deep generative model.
It consists of two main components: (1) The encoder go (z maps input data æ into a probabilistic distribu- tion in the latent space, which is often assumed to be Unit Gaussian. (2) The decoder pe (x) 2)) reconstructs the input based on the representations sampled from the latent space. In general, rescarchers often optimize the following evidence lower bound (ELBO):
Lyae (=) [log po (x) x)] DKI (KL(o((/=))llp())
(1)
To better learn disentangled representations, variants like 3-VAE HHggins et al. 2017 are proposed, which adds 3 (B 1) in front of the KI-divergence in the objective. However, large and fixed / could lead to poor reconstruction quality (Sao et al. 2020a).
3.3 ControlVAE
To effectively trade off the reconstruction quality and disentanglement performance, ControlVAF (Shao et al.) 2020b) incorporates controller from automatic control into the basic VAE model to dynamically adjust the hyperparameter 3 during training. The objective of ControlVAE is given by
Leantral (x,z) = (t (o (m) (x|1)
(())
(2)
where B(t) is the output of non-linear PI controller as follows:
Kp K; Σ e(() Bmins
B(t) 1 + exx(e(())
(3)
1=0
4

---

Published in Transactions on Machine Learning Research (01/2026)
adaptive swapping filter
F2
Po EobDa
7ap
Zia
Map
Xa
Encoder
colevance fitter
Zib 22n
23.0
Zia
2 -. Eub
X
pair-wise input
relevant
inelevant
Fi
Zp
Au
Z
Decoder
Decoder
data flow
weight sharing
swap
original
swapped
latent factor
latent factor
recons.
8
8a
%
Figure The overall framework of weakly-supervised Filter-VAE. pair of images with one distinet gen- erative factor is fed into the encoder de generate latent factors. Then two filters are implemented guarantee stable swapping: (1) relevance filter Pi filters out irrelevant (meaningless) latent factors z for only swapping relevant (mcaningful) factors x* in the latent space; (2) Adaptive swapping filter F2 sets upper bound for KT- KT--divergence to prevent swapping undesired latent factors (=ī) and unstable latent factor during training. It then swaps all stable and relevant latent factors (z 2x)) except for the most different one (zf)). The decoder reconstructs the input data based on both original (Za: 2b) and swapped (z/, zf,) latent factors.
where Kp and K, are hyperparameters the algorithm, and Amin is an application-specii constant. e(t) is the error between the designed KT--divergence and the real output.
Like other unsupervised VAB-based methods, Cont still struggles achieve perfect disentangle- ment from complex data. To address this problem, will extend ControlVAE to develop a novel weakly- supervised model for disentangled representation learning.
4 Proposed Method
As mentioned above, current supely-s DRL methods, such as SW-VAE (Zhu et al. 2023 perform latent factor swapping without accounting for the disentanglement status of the latent space, often resulting in training instability. To solve the limitation, we propose new weakly-supervised Filter-VAr in this work. As shown in Figure 1, Filter-VAE consists of four main components: encoder, relevance filter, adaptive swapping filter, and decoder, with lwo fillers as cores. The main idea is that it first encodes the input pairwise data into latent representations. The relevance filter then filters out meaningless latent factors. After that, the adaptive swapping filter finds disentangled stable factors and implements swapping. Finally, the swapped latent features are decoded to reconstruct the input data. We elaborate on each component in the following section.
4.1 Encoder for latent representation learning
First, we adopt encoder to learn latent representations of pairwise input. Let (æa, 2b) denote a pair of samples with one distinct factor. As shown in Figure [ the encoder maps (æa, xb) into latent variables (2,, 23), respectively. Latent variables are expressed as probabilistic Gaussian distributions, which are approximated by a convolutional neural network. The encoded representations can be formulated as:
(z a.) ~ V(uz,
where {a, b}. Uz, and 02, represents the mean and variance of latent variable 2, Latent variables are obtained with the reparameterization trick allow tractable computation (Kingma & Welling 2014). 2; contains d latent factors {21,1 22,1 -.. 2dz), where not all of them semantically meaningful.
5

---

Published in Transactions on Machine Learning Research (01/2026)
4.2 Relevance filter
We then use the relevance filter to remove irrelevant latent factors, thereby overcoming the current methods' limitation that often swap some meaningless latent factors during model training. To achieve this, we propose to separate all latent factors into relevant and irrelevant parts. Inspired by empirical experiments in prior work (Shao et al. 2020b), we can that meaningful factors exhibit higher dim-wise KT. divergence than those meaningless ones. Based on this observation, develop filter Fi with threshold to select meaningful factors below:
= { 2/ DKl(z) 2a Fi 2;) = 2i (KL(z)) <a
(5)
Here, suppose the KT. divergence between the i-th latent factor of pair of data. samples exceeds the threshold In this case, consider this latent factor to encompass semantically meaningful concepts, such shapes and colors. These latent factors are relevant for expressing meaningful high-level features of the data and are denoted with + symbol.
4.3 Adaptive swapping filter
After identifying relevant latent factors, the third step is to use the second filter to find stable latent factors for implementing swapping. Since latent factors in early training iterations may not be disentangled well, swapping these factors may result. in training instability. To overcome this issue, we design the second filter F/ based on wise KL-divergence between zn" and 2; as follows:
/ (0.0(1))) log 2oia(z;) dzgt
DEL
90. (2tt)
0. ofa + (Mi. - m2,0)
--2 log
+
(6)
o.b
where (0.a(2;) and (0,6(2;) denote the posterior probability of the i-th latent factor learned by the encoder
9s- Both of them follow Gaussian distribution (u4,03) (Burgess et al. 2018 For simplification, we use D(2tt) to represent DKL (1o,a(1;)) [lg),(=;+t) in the following.
Next, we need figure out which latent factor should be swapped basod on Eq. equation ? straightforward method to swap all meaningful factors except for the one with the largest dim-wise KL-divergence, denoted by j argmax, D(=+t. This seems effective because there exists only input sample. However, in practice, we find that sometines the largest D(z,t) doesn't represent the most distinct factor between a pair of samples during model training if the latent distribution is not learned well, especially in early training iterations. As a result, swapping could occur unexpectedly in the non-target factors, lcading to training instability. To resolve this problem, we further set an upper bound y for D(2;t) avoid unexpected swapping. Thus, we have the second filter F2 determine which latent factor should be swapped:
distinct factor for cach pairwise
swap if max D(=,) >g swap if max D(=+) S and i == argmax, D(ztt) swap+ if max D(2+t) S and + argmax, D(2;))
=2(),2)==
(7)
where y is a threshold, swap+ means swapping will be implemented and swap means not. The first condition indicates that we will not implement unexpected swapping when D(z²)) greater than 7. This can avoid swapping entangled latent factors. For the second condition that D(2;") is lower than y, if the i-th latent factor corresponds the largest dim-wise KL divergence, we do not implement swapping since it is more likely to be distinct factor. If the first and second conditions do not hold, we will swap the latent factors.

---

Published in Transactions on Machine Learning Research (01/2026)
4.4 Decoder for reconstruction
After swapping latent factors, the final slep is reconstructing the imput data. We input both the original latent factors &as and swapped factors Za's zy into the decoder to reconstruct input data &as Lb. & and E,, respectively. Both reconstructions should remain identical to their original imputs because the swapping occurs between latent factors with the same semantic meaning. This motivates the design of a reconstruction objective that encourages latent factors to align with the desired distributions..
4.5 Objective of DRL
The overall objective of DRL comprises two loss functions: the VAE loss and the reconstruction loss between the original and reconstructed samples. Specifically, we combinc the ControlVAE introduced in Sec. B with the proposed swapping lechnique learn disentangled representations. The loss of Con for pairwise input samples can be written
Spair Coontrot (da; Za) + Leontrol (2.: Zo).
After swapping, we ensure that the reconstructed image close to the original input. Thus, we have the following reconstruction loss
Sawap || 21113 ||, 16.
(9)
Combining the ControlVAF loss and reconstruction loss above, the overall objective is given by
Loveralt Lpair + whswap:
(10)
where w a hyperparameter to balance the second term.
We summarize the proposed Filter- VAE in Algorithm ! Lines 5-12 aim extract meaningful latent factors from the latent representations. In lines 17-21, swap the identical latent factors between pair of samples with one distinct factor.
5 Experiments
We first evaluate the disentannleement. performance of Filter-VAF. three benchmark datasets and traffic sign dataset that created. Then we verify its robustness against adversarial attacks in traffic sign detection. Lastly, we explore the impact of important hyperparameters and components model performance. The detailed model configurations and hyperparameter settings are presented in Appendix A
5.1 Datasets
5.1.1
Benchmark datasets
We first evaluate the disentanglement performance of our method using three benchmark datasets: dSprites (Matthey et al. 2017]. 3dShapes (Burgess & Kim 2018), and 3dChairs (Aubry et al., 2014
Each benchmark contains different ground-truth factor labels and corresponding numbers of factors:
1. dSprites Matthey et al. 2017) contains 737, 280 binary 64 x 64 images of 2D shapes generated by 6
ground truth factors (number of factors): color (1), shape (3), scale (6), orientation (40), x- position (32), y-position (32).
2. 3dShapes (Burgess & Kim 2018 contains 480, 000 RGB 64 x 64 x 3 images with 6 ground truth factors (number of factors): floor huc (10), wall huc (10), object huc (10), scale (8), shape (4), orientation (15).
3dChairs (ubry 2014) contains 86, 366 synthesized RGB 64 64 x 3 images with ground factors (number of factors): style (1,393), horizontal orientation (31), vertical orientation (2).

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1: The Filter-VAB Algorithm
Input:
Pairwise data samples {(0,, i 1,2,...N N. latent dimension d, threshold a for KL, threshold y for adaptive swapping, iterations T
Output: ΦT, OT
Initialize Φ1 81
for t= to I do
Sample a pair of images (æa, Lbb
Za: 2b + qá. (In), 4o, (2%)
for i to do
6
(xI - (Z) + (xe(z),))
if (x) 2 then
Obtain relevant. factors at Pib
9
else
10
Get meaningless factors T. 2ib end
11
12
end
for i= to 2+1 do D(2+t) t DKL (z, (=)),o,)))))
14
15
end
16
2/21 z/ f Za; 2h for i to 1=|| do
17
if max D(2,) S y and i * argmax D(x)) then
18
19
Swap z,,0 and z
20
end
end
22
Ra 0bb <- Po, (=a), Po, (26)
23
G. 2/ + pe, (z()) (,,,,0 Po, (z()
24
Update d,, 0, through gradient decent of Loverall
25
end
5.1.2
TrafficSign dataset
We create a traffic sign dataset for further assessing the disentanglement and robustness of the proposed Pilter-VAE, Adversarial attacks and defenses traffic sign recognition (TSR) have been widcly investigated, they are critical to autonomous driving. However, current benchmark datasets like German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp et al.) 2011 and UsinghuaTTencen 100K (TT100K) (Zhu) ct al. 2016 lack factor labels like traffic sign shape, thus they cannot be directly applied to discntangled representation learning. Inspired by Sim2Real (Kadian et al. 2020), we build our traffic sign dataset in a simulation enviromment. by controlling different latent factors, such as shape, color, and orientation. Note that we adopt different colors for each traffic sign to facilitate disentangled representation learning. also contribution our work that leverages simulated data for real-world prediction.
As shown in Figure 2 it contains 8 different traffic sign models with different. shapes: stop (octagon), warn- ing (triangle), speed limit (tall rectangle), oneway (long rectangle), deer crossing (rhombus), handicapped crossing (rhombus), left curve (rhombus), workers ahead (rhombus). Each sign's texture has 10 different colors. Each traffic sign model was rendered on a blue-white checkerboard with a plain background. We orient each model around its bottom in the 2D plane with [-0.2, 0.2] radians and the step size of 0.004. The camera takes an image at a fixed viewpoint. The resulting dataset contains x x 100 = 8, 000 RGB 128 128 images. This dataset will be publicly available upon publication of the paper.
5.1.3 Data Preparation

---

Published in Transactions on Machine Learning Research (01/2020)
SPEED LIMIT 25
STOP
STOP
ONE WAY
(a) Fight traffic signs in our dataset.
(b) Generated images.
Figure 2: Our synthetic traffic sign dataset. (a) Textures of eight traffic signs cover five different shapes (octagon, triangle, tall rectangle, long rectangle, rhombus). (b) Rendered images in the PyBullet (Greff
et al. 2022 simulator.
Table 1: Performance of different methods the TrafficSign dataset using DCT (Disentanglementt) SAP, and MIG, averaged over 5 random seeds. Bold: the best method. Underline: the second best method.
Dataset
TrafficSign
Metrics
DCI1
SAPT
MIG1
Unsupervised methods
B-VAEB
0.239
0.042
0.177
B- B-VAEB
0.079
0.005
0.116
FactorVAE
0.343
0.036
0.276
3-TCVAE
0.261
0.048
0.310
ControlVAB
0.460
0.133
0.442
Weakly-supervel
methods
Ada-GVAE
0.394
0.049
0.398
Ada-ML-VAE
0.397
0.051
0.387
SW-VAE
0.422
0.102
0.394
Filter-VAE (Ours)
0.482
0.205
0.581
In our experiment, we label only small amount of data for weakly-supervised disentanglement learn- ing. Specifieally, we label 10% of the sample pairs (æa, 2s), where each pair shares identical generative factors except for one distinct factor, as illustrated in Figure This selective labeling facilitates effec- tive latent factor swapping during model training with minimal supervision.
[0, 1,4]
[0, 0,41
[0, 1, 98]
Figure 3: Data samples with three factor labels ([shape, color, orientation).. Samples within each pair only vary in one factor, highlighted with different col- ors. The dashed box represents pair of samples.
5.2 Evaluation Disentanglement
We evaluate the disentanglement performance of Filter-VAE four datasets mentioned above, in- cluding 3 benchmarks and the TrafficSign dataset.
We compare it with 5 unsupervised basclines: 8-VAE, (Higgins et al. 2017), A-VAEB Burgess et al. 2018), PactorVAB (Kim & Mnih 2018), 3-TCVAE (Chen et al. 2018), Control VAE
Shao et al. 2020b),
and 3 weakly-supervisee methods: Ada-GVAF, Ada-MI-VAE [Locatello et al. 2020b),
SW-VAE Zhu et al.
2023
For disentanglement performance evaluation, we use commonly used metrics, DCI-Disentanglement East-
wood & Williams 2018), Separated Attribute Predictability (SAP) (Kumar al. 2018 and Mutual Infor-

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Performance different methods on three benchmarks using DCI (Disentanglement), SAP, and MIG, averaged over random scods. Bold: the best method. Underline: the second best method.
Dataset
3dChairs SAP1
3dShapes
dSprites
Metrics
DCM
MIG1
DCM
SAPt
MIG
DCF
SAPt
MIG
Unsupervised methods
B-VAEH
0.174
0.011
0.080
0.733
0.086
0.322
0.205
0.042
0.134
3-VAEB
0.117
0.010
0.063
0.439
0.052
0.349
0.389
0.081
0.360
FactorVAE
0.121
0.011
0.057
0.685
0.066
0.249
0.199
0.036
0.123
3-TCVAE
0.181
0.016
0.084
0.893
0.107
0.481
0.399
0.076
0.255
ControlVAE
0.162
0.090
0.138
0.815
0.143
0.513
0.531
0.067
0.494
Weekkyysuperrvee
methods
Ada-GVAE
0.090
0.007
0.040
0.542
0.049
0.519
0.410
0.067
0.344
Ada-ML-VVA
0.094
0.006
0.039
0.496
0.040
0.465
0.438
0.070
0.376
SW-VAE
0.170
0.073
0.092
0.772
0.068
0.328
0.560
0.090
0.472
Filter- VAE (Ours)
0.230
0.133
0.175
0.836
0.171
0.523
0.514
0.104
0.455
Filter-VAE
ControlVAE
shappe arient ocale
shape
Figure 4: Traverse visualization of top 2 methods dSprites and TranficSign datasets. Each represents one latent feature. dSprites: traverse in range of [-3, 3] with step= 2/3. Filter-VAB and ControlVAE successfully discntangle all five factors; however, the orientation range of Filter-VAE. is small. TrafficSign: traverse in range of [- with step= 4/9. Filter- VAE can effectively disentangle color and orientation, whereas Con does not perform well.
mation Gap (MIG) (Chen et al. 2018 CCI-Disentannlement measures the weighted average entropy of the probability of each latent. factor to predict each generative factor. SAP attributes a prediction score each pair of latent and generative factors, and measures the average difference between the two highest scores for all generative factors. MIG computes the average difference between the top two latent variables with the highest mutual information over all latent factors.
We present the comparison of disentanglement for different methods in Table 1 and Table It can be observed that the proposed Filter-VAB outperforms baselines using 3dChairs and TrafficSign datasets in all three metrics. Additionally, our method consistently ranks among the top 2 three metrics in 3dShapes experiments. The improvement is attributed to our method's ability to learn stable and relevant latent factors. For the simple dSprites dataset, the proposed method is slightly worse than ControlVAE and SW-VAE. The main reason is that it does not perform well in disentangling the orientation factor, which
10

---

Published in Transactions on Machine Learning Research (01/2026)
is illustrated in the following Figure 4 Overall, we conclude that the proposed Filter-VAB exhibits superior performance over baselines in three of these four datasets, except for the simplest one.
Additionally, we illustrate the qualitative results for our Filter-VALE and the second-best baseline in Figure 1 For Trafiicign,, we can see that the proposed Filter-VAr can better disentangle two latent factors, shape and orientation. For dSprites, it can be observed that both our method and ControlVAF can successfully discover all five latent factors. However, the orientation (in the 4th row) disentangled by Filter-VAL has small ranges, leading to low MIG in Table This is caused by (1) low explicitness of latent representations and (2) overlap between shape and orient features in the dSprites dataset.
Table 3: Classification accuracy over 17 attacks/corruptionn on the TrafficSign datasct. Bold: The best method.
Classification Accuracy
attacks//cormupiioss
Filter- VAE
KEMLP(AdvTrain)
KEMLP(DOA)
FSR(Resnet15)
SPN (Ours)
Impulse Noise (s=5)
0.6056
0.6225
0.7595
0.6153
Gaussian Noise (s=5)
0.4906
0.6206
0.7310
0.7689
Shot Noise (s=5)
0.5381
0.6252
0.7755
0.8119
Defocus Blur (s=5)
0.2525
0.2894
0.6000
0.9123
Glass Blur (s=5)
0.5531
0.5719
0.5815
0.7866
Motion Blur (s=5)
0.6131
0.6256
0.6020
0.8449
Zoom Blur (s=5)
0.4956
0.6288
0.4195
0.5006
Brightness (s=8)
0.6175
0.6119
0.1890
0.4478
Snow (s=5)
0.6100
0.5813
0.6050
0.7044
Frost (s=5)
0.3475
0.4600
0.4345
0.5630
Fog (s=5)
0.1500
0.2394
0.1350
0.3963
Contrast (s=5)
0.3544
0.5763
0.1385
0.1885
Elastic (s=5)
0.6063
0.6119
0.5975
0.7145
Pixelate (s=7)
0.4400
0.6213
0.6205
0.8595
JPEG (s=5)
0.6031
0.6225
0.6240
0.8798
Sticker (5x x 5)
0.0467
0.5100
0.1245
0.5700
L.x (e 8/255)
0.6056
0.1381
0.3960
0.8069
5.3 Evaluation on Adversarial Robustness
Next, we compare the performance of our method against adversarial attacks with two basclines: KEMLP (Gürel el al. 2021 and FSR (Kim et al. 2023).
In the experiments, our method is evaluated to classify traffic signs under 17 adversarial attacks that contain: (1) Common corruptions (Hendrycks & Dietterich 2019): It contains 13 types of corruptions caused by noise, blur, weather, and digital. We follow the same severity settings 8 as KEMLP. (2) L. PGD attack (Madry et al., 2018); we set e 8/255 with 40 iterations to construct perturbations that are unnoticeable to humans while still causing incorrect predictions. (3) Sticker attack: This attack attaches a sticker to traffic signs. We use the same adversarial sticker patch generated by (Eykholt et al. 2018 with the size of x 5. An exhaustive search conducted to find the best attack location. Additionally, we use classification accuracy assess model performance.
The classification is carried out using Sum-Produee Product Network (SPN) Poon & Domingos 2011
probabilis- tic graphical model structured as a rooted acyclic directed graph (DAG). Compared to standard classification models, such as multilayer perceptrons (MLPs), SPNs offer probabilistic interpretations, enabling reasoning over dependencies among latent factors with a small sacrifice to accuracy. For completeness, we also report classification results using MLPs and linear classifiers the Appendix C
Table 3 illustrates the accuracy comparison of different methods. It can be observed that the proposed Filter- VAE remarkably outperforms baselines in most attack scenarios. This is because method adopts
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Ablation study of two filters on the TrafficSign dataset. Bold: the best method.
F
F2
DCI 1
SAP f
MIG f
X
0.4532
0.1307
0.4203
V
0.4587
0.1551
0.4873
X
0.4697
0.1895
0.5514
Ours with both
0.4873
0.2013
0.5831
generative high-level factors instead of low-level features to detect traffic signs, improving its robustness against adversarial examples. However, it exhibits limited robustness to strong attacks such as contrast and brightness, as these attacks significantly degrade image quality and interfere with high-level factors such shape and color. Moreover, leverage tractable SPN to provide logical reasoning for predictions. Please refer to the graphical structure in Appendix B In summary, disentangled latent factors generated by
Filter- VAE are robust to most adversarial attacks.
We also investigate the adversarial robustness of Filter-VAL on real-world traffic sign images in Appendix The results indicate the potential of the proposed method for real-world traffic sign applications.
5.4 Ablation Studies
Lastly, we explore the impact of two key components, hyperparameteess and input labels on model perfor- mance. We first study the influence of two filters: the relevance filter F1 and the adaptive swapping filter F2. Then we igves the effect of three hyperparameters: w in the objective function and two threshold values of two filters. Lastly, we conduct experiments with imperfect supervisions to evaluate the sensitivity to input labels.
5.4.1 Effect of two filters in the smooth swapping.
Table ! shows the experimental results with or without using two filters in the smooth swapping. We can observe that the disentanglement performance will drop one or two filters are removed. These empirical results suggest the importance of two filters to achieve stable swapping.
5.4.2 Effect of Hyperparameters
Effect of in objective. Figure Appendix shows the impact w on disentanglement performance. We increase from to to balance the swapped reconstructions loss and the original VAE objective. It be observed that
0.4 exhibits the best result. In addition, there are no significant changes for three metrics with w > 0, which indicates Filter-VAF is not sensitive with > 0. Since 0.4 gives the best performance, we set w = 0.4 for our experiments.
Effect of a in Filter 1. We also study the effect of a that separates meaningful latent factors in the latent space. Setting a = 0 implies all latent factors are considered irrelevant, thereby disabling the swap- ping mechanism. As shown in Figure 7 of Appendix D using a larger than improves disentanglement performance, highlighting the effectiveness of the swapping strategy. Additionally, small a values can better identify relevant latent factors in the early training iterations. This is attributed to the magnitude gap between relevant and irrelevant latent factors. We set = 0.2 in our experiments.
Effeet of b in Filter 2. Lastly, we investigate the effect of by varying its value from 0 to 1. As illustrated in Figure S of Appendix D Filter-VAL achieves the highest disentanglement performance at 7 = 0.4. Setting y = disables swapping, while 7 allows unconstrained swapping. Both extreme conditions result in suboptimal performance. Based on these observations, we choose ? 0.4 in our experiments.
5.4.3 Performance under imperfect supervision
We further assess Filter- VAE under imperfect supervision to examine its sensitivity to the selection of labeled pairwise data. Here, imperfect supervision refers cases where pair of inputs may differ in more than
12

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Performance of Filter-VAB on the TrafficSign dataset with one and random distinct generative factors.
Metrics
DCI1
SAPT
MIG1
Filter- VAE with one distinct factor
0.487
0.201
0.583
Filter- VAE with random distinct factors
0.476
0.185
0.557
one generative factor. shown in Table the performance of Filter-VAL remains stable even when we gencrate image pairs with random number distinct factors, indicating that our method is sensitive to imperfect supervision. We attribute this robustness to the adaptive swapping filter F2: when multiple latent factors differ, F2 flexibly treats them either the first the second condition in Equation 7 thereby preventing any distinct latent factors from being incorrectly swapped.
6 Conclusion
In this paper, proposed Filter- VAE, weakly supervised variational autoencoder that employs filter- based adaptive swapping technique to learn disentangled representations using limited supervision. Specifi- cally, we designed two filters, a relevance filter and an adaptive swapping filter, to identify meaningful latent factors and prevent undesirable swapping of unstable, entangled factors. Filter-VAL can achieve effective disentanglement using only a small subset pairwise data that contains a single distinct generative factor. Extensive experiments on four datasets demonstrated the superiority of Filter-VAN VAE over existing basclines in both disentanglement and adversarial robustness tasks.
7 Limitation
While our proposed method achieves strong performance in both disentanglement and adversarial robustness, it relies heuristic assumption that high dim-wise KI. divergence corresponds to semantically relevant latent factors. However, this assumption is inspired by prior work (Shao et al. 2020b) and currently lacks theoretical foundation. As part of future work, we plan to develop formal justification to better support this design choice.
Acknowledgments
Rescarch reported in this paper was sponsored in part by NSF CPS 2311086, NSF CIRC 716152, NSF RITEL 2506890, and Faculty Research Grant at William & Mary 141446.
13

---

Published in Transactions on Machine Learning Research (01/2026)
References
Mathicu Aubry, Daniel Maturana, Alexei Efros, Bryan Russell, and Josef Sivic. Sceing 3d chairs: exemplar part-based 2d-3d alignment using large dataset of cad models. In CVPR, 2014.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI, 2018.
Chris Burgess and Hyunjik Kim. 3d shapes dataset.
2018. 2018.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 8-vae, 2018.
Chaofan Chen, Oscar Li, and Alina et al. Tao. This looks like that: Deep learning for interpretable image recognition. In NeurIPS, 2019.
Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise similarities, 2020.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and
R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL 6a25a3587799722322-2P8 pdf
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. URL https://opent
flev.net/forum?ld-By)
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust phyrical-world attacks on deep learning visual classification. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018. doi: C0.1199/CVPR.2016
Zunlei Feng, Xinchao Wang, Chenglong Ke, An-Xiang Zeng, Dacheng Tao, and Mingli Song. Dual swap disentangling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Robert Gens and Domingos Pedro. Learning the structure of sum-product networks. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 873-880, Atlanta, Georgia, USA, 17-19 Jun www.ennggmmm.....mmmmm......e.em mlr. wttpp//e//seeeeeeeeeeeeee html
2013. PMLR. URL https:
Klaus Greff, Francois Belletti, Lueas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Tagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Hemning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mchdi S. M. Sajjadi, Matan Sela, Vineent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3749 3761, June 2022.
Nezihe Merve Gürel, Xiangyu Qi, Luka Rimanic, Ce Zhang, and Bo Li. Knowledge enhanced machine learning pipeline against diverse adversarial attacks. In International Conference on Machine Learning, pp. 3976 3987. PMLR, 2021.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- Learning basic visual concepts with a constrained vari- ational framework. In International Conference on Learning Representations, 2017. URL https:
//openreview http://wwww..etettoouoodiiddaaae htt///ooruuumddddyy222zggg
Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters, 5(4):6670-661) 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.); Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 2649-265X PMLR, 10-15 Jul 2018. URL https::/proceeddng.. mllr. prrss
2/v80/kim18b.. himl
Woo Jae Kim, Yoonki Cho, Junsik Jung, and Sung-Ein Yoon. Feature separation and recalibration for adversarial robustness. In CVPR, 2023.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Confer- ence on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Pang Wei Koh, Shiori Sagawa, and Hampus et al. Marklund. Concept bottleneck models. In ICML, 2020.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations, 2018. URL httts:///axix: wwwp:/oor/bbb....ooooo
Guohao Li, Hongyu Yang, Di Huang, and Yunhong Wang. 3d face modeling via wealviys dis- entanglement network joint identity-consistentn prior. Tn 2024 IEEE 18th International Conference Automatic Face and Gesture Recognition (FG), pp. 1-10. IEEE, 2024.
Ran Liu, Mchdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith B. Hengen, Michal Valko, and Eva L. Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. CoRR, abs/2111.02332 2021. URL http:://aaxx...rrr//a////11111 abtp///ss//111/1 02338
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschan- nen. Weakly-supervised disentanglement without compromises, 2020a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020b. URL https:://ppeereeriee... wetprrwwwwwwwweeeeeeeeeeeeeeeeee wwtp://weeee///m/ieiiaae /tt.///rrnnddyygapp......
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
Hoifung Poon and Pedro Domingos. Sum-produet networks: A new deep architecture. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2011. doi:
10.1109/ICCVW. www.CCC.....1..6.100000 1.6130310.
Huajic Shao, Haohong Lin, Qinmin Yang, Shuochao Yao, and Tarck F. Abdckaaer. Challenging 8-vac with 8<1 for disentanglement via dynamic learning. CoRR, abs/2009.06795, 2020a. URL https: sttpss//arxiv.or fabbs/000900799)
15

---

Published in Transactions on Machine Learning Research (01/2026)
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. ControlVAF: Controllable variational autoencoder. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings ml
of Machine Learning Research, pp. 8653-8664. PMLR, 13-18 Jul 2020b. URL
r. F.press/v119/// mttss//v19//hho2b...oo
Huajie Shao, Yifei Yang, Haohong Lin, Longzhong Lin, Yizhuo Chen, Qinmin Yang, and Han Zhao. Re- thinking controllable variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 19250-19259, 2022.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241); 1-55, 2022. URL
html
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: A multi-class classification competition. In The 2011 International Joint Conference Newral Networks, pp. 1453 1460, 2011. doi: 10.1109/LJCNN, 2011....30..0033305
Francesco Tonolini, Nikolaos Aletras, Yunlong Jiao, and Gabriella Kazai. Robust weak supervision with variational auto-encoders. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 34394-34408, PMLR, 23 29 Jul 2023. URL https::/pproceeddngg mlr.
wtwp://nnwws......rr.eeeee.e.eee www.ppressssssr wttps////////ooo1o2222aaa html
Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. DNA-GAN: Learning disentangled reprosentations from multi-attribule images, 2018. URL
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvac: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9593-9600 9602, June 2021.
Zhuolin Yang, Zhikuan Zhao, Boxin Wang, Jiawei Zhang, Linyi Li, Hengzhi Pei, Bojan Karlaš, Ji Liu, Heng Guo, Ce Zhang, and Bo Li. Improving certified robustness via statistical learning with logical reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL httpt:://opporeeiieeeee///// http:///o/oormrrmdd/
Dihan Zheng, Yihang Zou, Xiaowen Zhang, and Chenglong Bao. Senm-vae: Semi-supervised noise modeling with hierarchical variational autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25880-25899, 2024.
Jiageng Zhu, Hanchen Xie, and Wael Abd-Almageed. Sw-vae: Weakly supervised learn disentangled rep- resentation via latent factor swapping. In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino (eds.), Computer Vision ECCV 2022 Workshops, pp. 73-87 87, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-25063-7.
Zhe Zhu, Dun Liang, Songhai Zhang, Xiaolei Huang, Baoli Li, and Shimin Hu. Traffic-sign detection and classification in the wild. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2110 2118, 2016. doi: 2..1109/CVP..2016.222
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Model Configurations and Hyperparameter Settings
A.1 Filter-VAE
In this experiment, our Filter- VAE uses the same encoder and decoder ControlVAE (Shao et al. 2020b): The model configurations and hyperparameter settings presented in Table We adopt the Adam optimizer with 31 0.90, 32 0.99 and a learning rate of 10-4. In addition, following the same settings of ControlVAE, we choose the desired KL-divergence as 19, 18, 10 for dSprites, 3D Chairs, and 3dShapes datasets. For TraffieSign dataset, the desired KT. value is set to 11. The dimension of the latent space Zdim is set to 10 for all experiments.
For a, y, and w, we tuned these parameters in the TraffieSign dataset and applied values to other datasets. is selected based on the observation that meaningful representation z has higher dim-wise KT. values than others. is determined by the highest Dxz (gs.aaaa2z) (lga.,))))) )) among at in the training process. w is the weight to adjust the swap reconstruction loss. Based on the ablation study in Section 5.4 we choose w = 0.4, 0.2, = 0.4 in all experiments.
A.2 SPN
SPN is trained with three Gaussian and one categorical distributions representing class labels with clustering algorithms (Gens & Pedro 2013 Yang el al., 2022). The result SPN a valid (complete and decomposable) 7-layer rooted DAG with 240 nodes (13 sum nodes, 77 product nodes, and 150 leaves) and 239 edges. In the trained SPN, we found that splitting color and shape contributes more to the classification because it gives 8 and 9 branches respectively, while orientation only gives 2 branches. It is reasonable because the traffic sign class is mainly determined by its color and shape.
Table 6: Model architectures of Filter- VAE
Encoder
Decoder
Input: 64 64 image
Input: RR_dimm
x 4 conv, stride 2, pad 1. 32 ReLU.
FC 256. 256 ReLU.
4 conv, stride 2, pad 1. 32 ReLU,
4 x transconv. 256 ReLU.
X 4 conv, stride 2, pad 64 ReLU,
4 x transconv, stride 2, pad 1. 64 ReLU,
4 x 4 conv, stride 2, pad 1. 64 ReLU,
4 x 4 transconv, stride 2, pad 1. 64 ReLU,
4 conv, stride 1, pad 1. 256 ReLU.
4 transconv, stride 2, pad 1. 32 ReLU.
FC 256.
4 x 4 transcony, stride 2, pad 1. 32 ReLUU Output: ex 64 64
FC z dim 2.
B SPN Visualization
Following the idea. of the prior work YYang et al. 2022). we draw knowledge graph to showcase the logical inference process of SPN for classification in Figure 5 As an instance of an adversary scenario, if the texture of "workers_aaeee ahead' covered with stickers, SPN can still distinguish it from other signs through its color and shape.
C Evaluation with Different Classifier
We conduct traffic sign classification using three different backbone models as the classifier: a single-layer linear classifier, multilayer perception (MLP), and sum-product network (SPN). Each model is provided
17

---

Published in Transactions on Machine Learning Research (01/2020)
SPE S
ON
Independent Feature
Feature Decision
SPEED LIMIT 25
STOP
ONE WAY
Shape
Triangle
Rectangle
Rhombus
Octagon
Rectangle
Color
Yellow
Green
Orange
Texture
Orientation
DeerCrossing
LeftCurve
ori 0
ori 15
ori 30
Knowledge graph of the TraffieSign dataset. The red arrow means independent features between data samples. The blue arrow represents the logical feature decision process.
Figure 5:
with ground-truth traffic sign labels along with 3 relevant latent factor values corresponding to shape, orientation, and color.
The results are shown in Table 7 Both the MLP and SPN outperform the linear classifier, as they can capture complex interactions among latent factors. Although the SPN achieves slightly lower accuracy than the MLP, provides tractable decision-making and allows explicit reasoning about how each latent factor contributes to the final prediction. These findings suggest that robustness primarily stems from the disentangled representations rather than the specific choice of classifier.
D Effect of Important Parameters
In this section, illustrate the impact of three hyperparamcters w, a, and y disentanglement performance using the TrafficSign dataset.
D.1 Effect of in the objective function.
is the weight to balance pairwise ControlVAE loss and swapping reconstruction loss in Equation 10 To investigate its effect, we increase w from O to 1. As shown in Figure 0 Filter-VAB achieves the best performance in three matrices when = 0.4.
D.2 Effect of a in relevant latent filter.
a is the threshold value to filter out irrelevant latent codes in the relevant latent filter F,. As shown in Figure 7 we investigate the effect of a in the range from to 1. When a 0, all latent dimensions are treated as irrelevant, and no latent code is swapped. Setting a > 0 yields improved results, indicating that swapping enhances disentanglement performance. The best performance is achieved at 0.2. Bocause meaningless latent codes typically have smaller magnitudes than meaningful ones, a small a value allows the filter to identify relevant latent codes more effectively. In experiments, set 0.2.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Classification accuracy over atfacks/coruptions on TrafficSign dataset with different classifica- tion backbones. Bold: the best method.
Classification Accuracy
attacks/corruptions
Filter VAE + Linear
Filter- VAE + MLP
Filter- VAE SPN
Impulse Noise (8=5)
0.5051
0.6429
0.6153
Gaussian Noise (s=5)
0.6599
0,8310
0.7689
Shot Noise (5=5)
0.6791
0.8606
0.8119
Defocus Blur (s-5)
0.7399
0.9294
0.9123
Glass Blur (8=5)
0.6709
0.8314
0.7866
Motion Blur (s=5)
0.7003
0.8829
0.8449
Zoom Blur (s=5)
0.4375
0.5153
0.5006
Brightness (8=8)
0.3778
0.4573
0.4478
Snow (s=5)
0.5950
0.7748
0.7044
Frost (s=5)
0.4579
0.6145
0.5630
Fog (s=5)
0.3405
0.4109
0.3963
Contrast (s=5)
0.1250
0.1955
0. 0.1885
Elastic (s=5)
0.6183
0.7655
0.7145
Pixelate (s=7)
0.7115
0.9008
0.8595
JPEG (s=5)
0.7119
0.9046
0.8798
Sticker (5
0.3650
0.5400
0.5700
C. 8/255)
0.6250
0.8175
0.8069
D.3 Effect of y in adaptive swapping filter.
is the upper bound used to determine the stability of latent codes in the adaptive swapping filter F2. Its effect is evaluated by varying from to 1, as shown in Figure Filter-VAE achieves the best disentanglement performance when 7 = 0.4. When 7 0, all latent codes are considered unstable, and no swapping is performed. Conversely, when y 1, all relevant codes are unconstrained by the upper bound, which means the training stability is ignored. We
that both extreme conditions lead to degraded performance. Based on these observations, set y 0.4 in experiments.
0.6
0.65
0.6
0.55
0.25
0.55
DCI
0.5
SAP
0.2
MIG
0.5
0.45
0.15
0.45
0.4
0.1
0.4
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
@
Figure 6: Effeec of w on disentanglement performance using TrafficSign dataset. We can see that our method is not sensitive with w > 0.
E Latent Traversals Visualization
We illustrate the latent traversals of our method and the basclines in Figures. D 10 and 11 The basc-
line methods like Ada-GVAE, Ada-ML-VAE, and 3-VAEg fail trade off the reconstruction error and disentanglement, which results in blurry reconstructed images.
19

---

Published in Transactions on Machine Learning Research (01/2026)
0.6
0,3
0.65
0.55
0.25
0.55
DC1
SAP
MIG
0.5
0.5
0.45
0.15
0.45
0.4
0.1
0.4
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure Effect of disentanglement performance using TrafficSign dataset.
0.6
0.3
0.65
0.6
0.55
0.55
DC1
SAP
0.5
0.2
0.5
0.45
0.15
0.45
0.4
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
y
Figure 8: Effect of y on disentanglement performance using TrafficSign datasct.
E.1 TrafficSign
Filter- VAE can effectively disentangle color and orientation. Other baselines, except for 3-TCVAE, cannot learn these two factors well. For instance, ControlVAE entangles all features together in the 2nd row. Though 3-TCVAE can also learn both color and orientation, it suffers from poor reconstruction in the dSprites dataset. In addition, separating color from shape remains challenging for all methods when data contains complex texture information.
E.2 3dShapes
Fil can effectively disentangle all factors except for object_ hue (the 3-rd row), which is entangled with slight shape changes. Though 3-TCVAE achieves good disentanglement result in 3dShapes indicated in Table 2 it sacrifices reconstruction quality to get better disentanglement, especially in Figure 11
E.3 dSprites
Filter-VAE achieves better disentanglement performance than most basclines, except that the orientation is slightly worse compared to ControlVAE. However, ControlVAE does not perform well in disentangling orientation the TrafficSign dataset. Moreover, it can only disentangle shape and wall hue in the 3dShapes dataset.
F Real-world Experiments
The effectiveness of the proposed method on real-world data crucial for practical deployment. However, a significant limitation lies in the absence of real-world traffic sign benchmarks with controlled latent factors. To address this issue, we leverage In- Context Learning (ICL) with the state-of-the-ar vision-language model Gemini Nano Banana to construct dataset. of 1,960 real-world traffic sign images. Specifically,
20

---

Published in Transactions on Machine Learning Research (01/2026)
shape orient
FilterVAAE
Ada-GVAE
Ada-ML-VAR
shape
SWVAE
ControlVAE
B-TCVAE
shape orient
FactorVAE
B-VAEB
B-VAEH
Figure 9: Latent traversals on the TrafficSign dataset. Each row represents one latent factor. Traverse in the range of 2- with step Filter- VAE and 3-TCVAE can disentangle color and orientation successfully. However, the reconstruction quality of 3-TCVAE is worse than Filter-VAE.
publicly available traffic sign images as ICL examples and design prompts to generate images across eight traffic sign categories under 35 distinct real-world driving scenarios. Each scenario includes seven randomized 8 1, 960 images. The latent factors
rotations and eight color variations, resulting in total of 35 explicitly controlled via prompt engincering. Notably, due Gemini's limitation in
(color, orientation) executing precise rotation degrees from prompt specifications, each traffic sign is randomly rotated seven times to approximate orientation diversity. Details of the dataset and the generation pipeline are provided in the supplementary material.
We present classification results in Table S Filter- VAE achieves either the best or second-best performance under most attacks and corruptions, with exceptions of Frost and Gaussian Noise. Nonetheless, its perfor- mance under these two corruptions remains competitive, exhibiting only minor drops of 0.0750 and 0.0276 respectively, compared to the best results. These findings demonstrate the strong potential of our approach for real-world traffic sign applications.
21

---

Published in Transactions on Machine Learning Research (01/2026)
huee
hue
oboject huse
scale
shape
orient
Filter-VAS
Ada-GVAE
Ada-ML-VAE
floor hue
wall huee
object
scale
shape
oriant
SWVAE
ControlVAF
B-TCVAE
floor
hue
wall
Tue
object
scale
FactorVAE
B-VAEg
B-VAEg
Figure 10: Latent traversals on the 3dShapes dataset. Each row represents one latent factor. Traverse the range [-1,1] 1] with step 1/2. Filter-VAL can successfully disentangle all features except for slight shape change in object hue.
22

---

Published in Transactions on Machine Learning Research (01/2026)
x
sc
orient
shape
Filte-VAB
Ada-GVAE
Ada-ML-VAE
ortenk
shape
SWVAE
ControlVAE
B-TCVAE
orient scale
Fac
B-VAEs
B-VADy
Figure 11: Latent traversals on the dSprites datasct. Each row represents one latent factor. Traverse the range of [-3,3] 3] with step 3/2. We can observe that Filter- VAE and ControlVAB can successfully disentangle five latent factors, while the other baselines cannot.
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 8: Classification accuracy over 17 attacks/corruption on the real-world traffic sign datasct. Bold:
The best method. Underline: The second best method.
Classification Accuracy
attacks/corruptions
Filter-VAl
HEMMP(AdvTrain
KEMLP(DOA)
FSR(Resnet18)
SPN (Ours)
Impulse Noise (8
0.9245
0.8020
0.8102
0.8704
Gaussian Noise (s=5)
0.9245
0.8265
0.9490
0.9214
Shot Noise (x=5)
0.9245
0.8265
0.9347
0.9281
Defocus Blur (s=5)
0.9143
0.8184
0.8878
0.9352
Glass Blur (s=5)
0.8939
0.8020
0.8939
0.9306
Motion Blur (s=5)
0.8816
0.7918
0.8286
0.9332
Zoom Blur (s=5)
0.8224
0.7735
0.8388
0.8643
Brightness (s=8)
0.2163
0.1755
0.2204
0,2189
Snow (s=5)
0.5082
0.5755
0.3776
0.5724
Frost (x=5)
0.4551
0.4959
0.4776
0.4209
Fog (s=5)
0.3265
0.3000
0.1776
0.3133
Contrast (s=5)
0.2714
0.2816
0.2122
0.2862
Elastic (s=5)
0.8959
0.7980
0.8857
0.9112
Pixelate (s=7)
0.9122
0.8163
0.9020
0.9357
JPEG (s=5)
0.9245
0.8224
0.9327
0.9378
Sticker (5 x 5)
0.6694
0.9082
0.2633
0.8952
Lx (e = 8/255)
0.9265
0.6122
0.9265
0.9071
24


Reviews:
[{"review_id": "K6ef3JOAfh", "reviewer": "Reviewer_3nbZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper proposes Filter-VAE, a weakly supervised vae designed to learn disentangled representations using limited pairwise supervision. Building on prior work in latent factor swapping (e.g., SW-VAE), the key contribution is a filter-based adaptive swapping strategy that aims to improve training stability and semantic alignment of latent factors. Specifically, the method introduces: (1) a relevance filter, which removes latent dimensions that is semantically meaningfulless based on the KL divergence. (2) an adaptive swapping filter, which swaps only latent factors that are both relevant and sufficiently stable, as measured by pairwise KL between latent posteriors. The method is evaluated on three standard disentanglement dataset and a newly introduced synthetic dataset. The results indicate that it generally outperforms unsupervised and weakly supervised baselines, particularly on more complex datasets, while using relatively weak supervision.\n\nStrengths:\n\nClear and Intuitive High-Level Idea: The paper addresses a well-known issue in weakly supervised disentanglement: unstable or premature latent swapping, by introducing a principled gating mechanism. The use of KL divergence both to identify meaningful latent dimensions and to assess their stability is conceptually simple and well-motivated.\nIncremental but Solid Methodological Contribution: While the approach builds directly on prior latent-swapping VAEs, the introduction of two explicit filters (relevance and stability) represents a meaningful refinement over SW-VAE. The method reduces reliance on strong assumptions (e.g., warm-up stages or precise annotations of varying factors), which is aligned with the goals of weak supervision.\n\nWeaknesses:\n\n1. Although sensitivity analyses are provided, the method still depends on manually chosen thresholds that are tuned on a specific dataset and then transferred to others. This raises direct questions about robustness across domains with very different latent distributions.\n2. On dSprites, Filter-VAE does not consistently outperform strong baselines such as ControlVAE, particularly on orientation factors.\n3. The robustness gains are demonstrated using an SPN classifier trained on disentangled representations. While the results are strong, it is not entirely clear how much of the improvement is attributable to disentanglement itself versus the choice of classifier or the synthetic nature of the dataset.\n4. Both the relevance filter and the adaptive swapping filter rely on dim-wise KL divergence as a proxy for semantic meaningfulness and stability. While empirically motivated, this choice is heuristic and lacks a strong theoretical justification. In particular, high KL divergence does not necessarily imply semantic alignment with a single generative factor, especially in entangled or partially disentangled cases.\n\nOverall: \n\nThis paper presents a competent and well-executed incremental contribution to weakly supervised disentangled representation learning. The high-level idea is clear, the method is carefully designed, and the experimental evaluation is largely sufficient to support the main claims, particularly regarding stability and reduced supervision. While the novelty is moderate and the reliance on heuristic thresholds is a limitation, the work is solid and relevant to the TMLR audience, especially for readers interested in practical improvements to disentanglement methods rather than fundamentally new theory. Hence I recommend for acceptance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission are supported by the designed experiments.", "audience_yn": "Yes", "audience_explanation": "The topic is relevant and potentially interest the community.", "requested_changes": "See Weaknesses Above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k6HSSdzk0j", "reviewer": "Reviewer_f7K6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the paper, the author presents Filter-VAE, a weakly supervised VAE using a filter-based adaptive swapping strategy for stable, meaningful disentanglement. The paper introduces two key filters: a \"Relevance filter\" that removes semantically meaningless latent factors, and an \"Adaptive swapping filter\" that exchanges latent factors that have reached stability during training. This dual-filter mechanism enables Filter-VAE to adaptively swap only stable and semantically aligned latent factors, resulting in robust and meaningful representations.\nThe author also presents comparable results with other weakly supervised methods on the traffic sign dataset. And the results look promising.\nAlthough the method appears promising compared to others, the dataset seems somewhat limited as it’s synthetic, allowing the authors significant control over the attributes of the traffic dataset, including the shapes and colors of traffic signs.\n\nThe paper states that Filter-VAE targets disentangled representation learning for traffic sign recognition (TSR), noting that existing benchmarks, such as GTSRB and TT100K, lack factor labels, including sign shape. However, the paper doesn’t clearly explain how this approach could be applied to real-world traffic scenes, where such controlled conditions don’t exist.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the paper shows a comparison of results with other relevant algorithms.", "audience_yn": "Yes", "audience_explanation": "Yes, Developers working on diffusion models, representation learning, and robust perception for autonomous driving (not limited to) would benefit most.", "requested_changes": "The paper would be even stronger if it included benchmarks on real-world data, where conditions are less controlled. Even if the results differ from those on synthetic datasets, such experiments would provide valuable insights into real-world performance and help future developers understand practical challenges and key takeaways.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x7PfUhUjbB", "reviewer": "Reviewer_fqeo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Filter-VAE, a weakly supervised model that improves disentangled representation learning. It uses two filters: one to remove meaningless latent factors and another to align stable ones,  allowing more reliable disentanglement with minimal labeled data. The authors also build a synthetic traffic sign dataset to evaluate both disentanglement quality and adversarial robustness. Experiments show that Filter-VAE outperforms existing methods on several benchmarks while using less supervision, and it also notably improves model robustness against a wide range of adversarial attacks.\n\nStrength:\n1. Proposed a new algorithm, VAE with two filters.\n2. Developed a synthetic traffic sign dataset.\n\nWeakness:\n1. The overall presentation feels somewhat rushed and underdeveloped.\n2. Did not justify reproducibility.\n3. Experiment results (Table 1&2) are based on only 3 replications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "If the authors could justify reproducibility, and explain why there are only three replications, the results should be convincing.", "audience_yn": "Yes", "audience_explanation": "Yes, the topic of weakly supervised disentanglement and model robustness is relevant to TMLR readers, although the current version may need stronger presentation and empirical support to attract wider attention.", "requested_changes": "See weaknesses.\n1. Presentation. The authors should carefully revise their manuscript, including but not limited to:\n\n1.a. Overall Organization: The main text often feels loosely structured: key algorithmic components are deferred to the supplementary material, while large figures and tables take excessive space without conveying much additional insight.\n\n1.b. Problem Statement: I'm quite confused. Did you omit the dimension of each data point x_i, or in your mapping, f(x) -> z, x indeed contains N data points and each data point is a scalar? If x_i (i=1,2,...,N) is a vector, you should use bold x_i, and \"Given a batch of N observed data\" (sample-level) and mapping (model-level) should not be in same sentence. If x in \"f(x) -> z\" indeed contains N data points  and each data point is a scalar, what does it mean that using N observations to learn one latent factor?\n\n1.c. Math notations: some text in math looks not nice, e.g., ConVAE in equation (2), Diff in equation (7); in the last paragraph of Page 5, Gaussian distribution N(mu, sigma^2), why does here (mu, sigma^2) not contain any subscript or superscript; in equation (4), q should follow (using \"~\" in math) instead of equals to (\"=\") normal distribution.\n\n1.d. Wording: In section 5.1.2, \"stop (octagon), warning (triangle), speed_limit (rectangle_tall), oneway (rectangle_long), deer_crossing (rhombus), handicapped_crossing (rhombus), left_curve (rhombus), workers_ahead (rhombus).\", the underscore is not the scientific presentation stype.\n\n1.e. Citation: almost all of the parenthesis in citations are missing, e.g., \"DRL has demonstrated success in several domains such as image generation, natural language processing, and recommendation systems Wang et al. (2024).\"\n\n2. Reproducibility: please (anonymously) provide (at least part of) the codes to justify reproducibility, especially for the applied work.\n\n3. Results justifiication: Please explain why there are only three replications/seeds.", "broader_impact_concerns": "I did not find any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)