Two-Layer Convolutional Autoencoders Trained on Normal Data Provably Detect Unseen Anomalies
Authors: Yanbo Chen
Weiwei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Learning Theory
Primary area: learning theory

Abstract: Anomaly detection refers to the techniques that identify (probably unseen) rare or suspicious data that deviate significantly from the pre-defined normal data (Chalapathy & Chawla, 2019; Ruff et al., 2021). Empirical studies have observed that generative models trained on normal data tend to produce larger reconstruction errors when reconstructing anomalies. Based on this observation, researchers have developed various anomaly detection methods, referred to as reconstruction-based anomaly detection (RBAD) (Lv et al., 2024; Li et al., 2024) in the literature.

Despite the empirical success of RBAD, the theoretical understanding of RBAD is still limited. This paper provides a theoretical analysis of RBAD. We analyze the training dynamics of a 2-layer convolutional autoencoder and introduce the cone set of the features. We prove that the cone sets of the normal features would absorb the (convolutional) kernels of the autoencoder during training and use these absorbed kernels to reconstruct the inputs. The absorbed kernels are more aligned with the normal features, which explains the cause of the reconstruction error gap between the normal data and the anomalies. Synthesized experiments are provided to validate our theoretical findings. We also visualize the training dynamics of the autoencoder on real-world data, demonstrating our proposed cone set intuition.

Published as a conference paper at ICLR 2026
TWO-LAYER CONVOLUTIONAL AUTOENCODERS TRAINED ON NORMAL DATA PROVABLY DETECT UN- SEEN ANOMALIES
Yanbo Chen, Weiwei Liu"
School of Computer Science, Wuhan University, Wuhan, China {yanbo. acad, Liuweiwei163]@mmaill
ABSTRACT
Anomaly detection refers to the techniques that identify (probably unseen) rare or suspicious data that deviate significantly from the pre-defined normal data (Cha- lapathy Chawla, 2019; Ruff et al., 2021). Empirical studies have observed that generative models trained normal data tend to produce larger reconstruction er- rors when reconstructing anomalies. Based on this observation, researchers have developed various anomaly detection methods, referred as reconstruction-batt based anomaly detection (RBAD) (Lvv et al., 2024; Li et al., 2024) in the literature.
Despite the empirical success of RBAD, the theoretical understanding of RBAD is still limited. This paper provides a theoretical analysis of RBAD. We analyze the training dynamics of 2-layer convolutional autoencoder and introduce the cone set of the features. We prove that the cone sets of the normal features would ab- sorb the (convolutional) kernels of the autoencoder during training and use these absorbed kernels to reconstruct the inputs. The absorbed kernels are more aligned to the normal features, which explains the cause of the reconstruction error gap be- tween the normal data and the anomalies. Synthesized experiments are provided to validate theoretical findings. We also visualize the training dynamics of the autoencoder on real-world data, demonstrating our proposed cone set intuition.
INTRODUCTION
Anomalies are rare or suspicious observations that deviate significantly from the pre-defined nor- mal data (Chalapathy & Chawla, 2019; Ruff al., 2021). Examples of anomalies include credit card fraud (Bhattacharyya et al., 2011), defects in manufacturing products (Bergmann et al., 2021), and outbreaks of disease (Díaz-Cao et al., 2023). Anomaly detection (AD) is challenging task since negative samples (i.e., anomalies) are hard to obtain at the training stage. Nevertheless, re- searchers have developed various AD methods that require only normal training data. For example, empirical studies (Salehi et al., 2021; Mousakhan et al., 2023) have observed an intriguing phe- nomenon that generative models trained on normal data tend to produce larger reconstruction errors when reconstructing anomalies than normal data. Accordingly, anomalies can be detected by setting up thresholds for reconstruction errors. Such AD methods are referred to as reconstruction- anomaly detection (RBAD) (Livernoche et al., 2023; Lv et al., 2024; Li al., 2024) in the literature.
Despite the empirical success of RBAD, the theoretical understanding of RBAD is still limited. A detailed discussion the related works is postponed Appendix A due limited space. The main goal of our paper is to establish a theoretical framework for analyzing RBAD and find a plausible explanation for RBAD. More specifically, we raise the following three research questions.
(Q1) How does the model learn the features from normal training data in RBAD?
(Q2) What is the role of the learned Seanurreature features the reconstruction phase?
(Qa) Why are the anomalies harder to reconstruct compared normal data?
This paper comprehensively discusses Q Q3 and provide a brief answer in Section 5.
"Correspondence Weiwei Liu {1 twei we 1..0 com).

---

Published as a conference paper at ICLR 2026
Setup. We analyze the training and reconstructing processes of RBADs using tools from feature learning (Allen-Zhh & Li. 2021; 2023); we refer the readers to Appendix A for more related works. Following Allen-Zhu & Li (2023), we characterize the inputs images an array of P patches. As illustrated Figure 1, patches are small parts of an image that might contain various features and non-feature noise. In this paper, we adopt the signal-noise data model that characterizes the patches as the sum of feature and a random noise. For normal data, the patches contain different features according to some given distribution (ef. Section 2.1). We characterize the anomalies as normal data with some of the normal patches replaced by abnormal patches.
We consider a hwo-layer convolutional autoencoder that includes a convolutional hidden layer with max pooling operation. The network is parameterized by the convolutional kernels w and is trained by minimizing the empirical reconstruction error using gradient descent with random initialization. We analyze the training dynamics of the kernels by setting up probability bounds for the growth rates and the magnitude of the kernels.
Theoretical Analysis. We quantitatively analyze the training and reconstruction phases of RBADs in Section 3. Given feature and iteration step, Definition 3.1 introduces the cone set of the feature at that iteration step. By definition, the "cones" would get sharper as iteration steps increase (ef. Fig- ure 3), which ensures that the kernels in the cone sets would get more aligned to the corresponding feature. We prove that the cone sets of the features would absorb the kernels during training. As a result, the absorbed kernels would eventually approximates the features' directions, and by this means, the model extract and learn the features from the training data. The above analysis provides a plausible answer to question Q1.
We answer the questions Q2 and Q3 by further analyzing the reconstruction phase of RBAD. Our paper characterizes the anomalies as normal data with some of the normal patches replaced by the abnormal patches. In Section 3.2, we study how the learned features process the signal from the input image. Theorem 3 proves that the processed signal of the abnormal patches would be significantly weaker than those of the normal patches, explaining why the anomalies are hard to reconstruct.
Empirical Validation. We conduct experiments on synthesized and real-world datasets to validate the cone set intuition. The setting of the synthesized experiment is identical to our theoretical setting- As reported in Figure 4, there is clear gap between the reconstruction errors of the normal data and the anomalies, which is in consistent with both our theoretical claims and the existing empirical results on RBAD. We also train networks growth of the kernels in these networks. The results exhibit clear feature learning processes, which further validate our proposed cone set intuition.
worldwrrdd anomaly detection datasets and visualize the
Contribution Statement. The main contributions of our paper are summarized as follows.
1. We establish a theoretical framework for analyzing RBAD. The main results and intuitions are summarized in the theoretical roadmap (Figure 2). Our proposed framework introduces the cone set intuition (Remark 3.2 and Figure 3). We characterize how neural networks extract features (Q1) and reconstruct normal data and anomalies (Q2). In particular, we prove that a 2-layer convolutional autoencoder trained on normal data cannot accurately reconstruct the anomalies with high probability (Qx).
2. We conduct experiments on synthesized and real-world datasets to validate our theoretical findings. With an identical setting our theory, the results of the synthesized experiment are consistent with both our theoretical findings and the existing empirical results. We also visualize the feature learning process on real-world datasets, validating our proposed cone set intuition. Remark: It is worth noting that the main goal of this paper is to complement, not compete with, existing RBAD methods. The experiments are mainly conducted to support and validate our theoretical findings.
The remainder of this paper is organized as follows. Section introduces the problem setup and the terms and notations. A detailed discussion on the related works is postponed to Appendix A. We present the main results and intuitions in Section 3, and postpone the detailed proofs to Appendix E. Experimental results and discussions be found in Section 4. We conclude this paper in Section 5.
2

---

Published as a conference paper at ICLR 2026
2 PROBLEM SETUP
Throughout this paper, we denote the vectors by bold letters (e.g.. a; n, and w). The i-th entry of x: is noted by x()) For any N E Øt, we write [A]] := f1,2. ,,..,,N3 and [æx] = { Eí e [N] s.t. = 2 for simplicity. The 2-norm of æ is denoted by ||=||| A summarized list of terms
and notations is included in Appendix C.
P-Patch Data. Following Allen-Zhu & (2023), we consider the following P-patch data (P E Z+) model. As demonstrated in Figure patch refers to small part of an image that might contain various features (e.g., edges and shapes). Previous works of feature learning (Allen-Zhu & Li, 2023; Kou al., 2023) characterize the patches as vectors. In this paper, let H be d- dimensional Hilbert space equipped with the inner product (01, == Eield Ø1 x 2,, 22 E H. In the rest of this paper, we will call æ a patch-like vector if x E H.
for all
be split into patches. Motivated by this, our paper uses disjoint patch-like vectors to represent the images. More specifically, we say that is P-patch data (or z has P-patch structure) if can be represented by an array of patch-like vectors, i.e., (x) 22, 22. pp H² For all i E [P], we call æ; a d-dimensional patch of z.
Real-world images
Our paper assumes that both the normal data and the anomalies are P-patch data with fixed P and d. In the context of convolutional neural networks, the (convolutional) kernels w and the patches have identical sizes and shapes. Therefore, assume that the kernels w also patch-like like vectors. For all v, w E H. we call (æ, w) the convolution of a and w by convention.
Features. We characterize the features as d-dimensional orthogonal patch-like vectors. Specifi- cally, assume that the set of all features H. Based on whether it is related to the semantic information of the normal data, we further cat- egorize the features into normal and auxiliary features. As illustrated in Figure 1, if hand-written digit "I" is considered as the normal data, then the patches Ta,1 L, Ib and T.,1 might contain normal features that capture part the semantic information of the normal data. In Figure 1.c, 2x,2 contains an auxiliary feature that captures the semantic information of a semantic anomaly of "7" instead of the normal data "1". However, we argue that these auxiliary features can be regarded as either acceptable errors or sample diversity, and thus might appear in normal data with a small yet non-negligible probability.
(#1, " Ua)) form orthonormal basis of
In this paper, we formalize the collection of normal and auxiliary features of the normal data as disjoint subsets V. noted by Vner and Vaua. respectively. Without loss of generality, we assume that Vnor {v1, 222 and Vanz {V3, -.. , vaJ. Notice that our results can be easily extended to V Yoor and Vauw taking arbitrary complementary subsets of V at the cost of tedious computation.
Ta.1
Tb,1
Fe..1
aaa
Lb,2
Le,2
(a) Normal data
(b) Non-semantic anomaly
(c) Semantic anomaly
Figure 1: Patches, features, and different types of anomalies. As an illustrative example, consider the handwritten digits with only "1" being normal data. Patches small parts of the images, e.g., those outlined with red boxes. Ea,1. 2a,2, Vt, and C.,1 contain normal feature that captures the semantic information of "I". Figure 1b is called the non-semantic (or sensory) anomaly in the literature since ø6,2 contains non-feature noise. Ie,2 contains an auxiliary feature that does not capture the semantic information of "1", making Figure semantic anomaly.

---

Published as a conference paper at ICLR 2026
2.1 DATA DISTRIBUTION
Our paper assumes that the normal data z is drawn independently from an implicit distribution D Dnar- Given normal data 2, we assume that each of the patches of contains one feature. Here, we say that a patch 2; contains feature Uy if x = + p. in which is noise term that satisfies (p 2y S E 0.
for some given constant
For any k E [d] and z ~ Pnor. we let the probability of the event {3e, E such that æ; contains v be a given constant 3% E [0. 1]. Intuitively, normal data would contain more normal features than auxiliary features. For simplicity, we assume that 3x = for all & E {1,2] 2). i.e., normal data surely contain normal features. As for the auxiliary features, we require 3x < for any E [3,4,-.. d] and Σ By = P to ensure there is no empty patches. In summary, we adopt the following assumption for the normal data.
Assumption 2.1 (Normal data model). We assume that normal data (0); 22, ; ; is drawn independently from an implicit distribution Dnor that satisfies
1. for all i E [P], 3s E [d] such that æ; contains Uki
2. for all E [d]. Dnor Ei E [P] s.t. æ contains UR] = :44
3. 31 = 82 = 1, 3x < for all & E {3, 4, ... d]. and E 3k P.
Given N E Z+, let [zN] be the training data that drawn independently from Dnor. in which 2n E [=N] is noted by Σn := (I 2n. Fn.a) in the remainder of this paper. Notably, RBAD does not require anomalies during training, which constitutes one of the core advantages of this method. We will introduce the data model for the anomalies later in Section 3.2.
2.2 NETWORK STRUCTURE
Let Φ HP - HP be a function that reconstructs the inputs. It is not hard to show that simple models (e.g., principal component analysis) would fail to reconstruct the normal data due to the slightly more sophisticated model, i.e.,
unspecified ordering rule of the patches. Therefore, the 2-layer convolutional autoencoder.
An autoencoder is composed of encoder &e and decoder da. i.e., x0z) = (z() for
all z E HP. Denote the number of channels by C = c2d for some given constant 02 and let w (w), w2. wc) be the weight of d.. in which w, (i [C] are kernels described earlier. Given input z E Z, the output of &e is noted as 6e(2; w) = (0x(2; wi), 6x(2; wc)), in which
(e(2; w)) = (a(ww, m1)), ... (((0),
Here, a()) is the smoothed ReLU function defined as
0
if x < 0; if < x S Co:
(x)
(2c0)-12² x
(2c0)
(2)
x if x Co:
for given constant cp- Unlike the original ReLU function, a is derivable near 0. Note that &e (z; wyy) also P-patch data, but the dimension of the patch reduced from to
The decoder reconstructs z according to the features learned by de. To illustrate how the features are used in construction, consider the following ideal case in which perfectly learns the features, i.e., let ((d) = d and w = v for all j [d]. In this case, we can write
(x, w.) w, = Zi.
(3)
JEld
which provides a trivial yet perfect reconstruction of the patches in z.
We modify the reconstruction in Equation (3) by considering over parapearmeterize setting with max pooling operation. We make these modifications for the following reasons. First of all, the proof of Theorem implies that the reconstruction in Equation (3) would easily converge to local
4

---

Published as a conference paper at ICLR 2026
minima; refer the readers to Remark E. for detailed discussion. To avoid reaching local minima, we consider an over- parametaraaeeed setting with C2 >
In practice, the max pooling operation is placed after the convolution layer to select the most repre- sentative patch, i.., the patch with the maximum convolution value. For case of computation, our work also adds pooling operation in the decoder. For all [C]. let jit be binary-value function that takes when
xE(PI arg max o ((w,, (x))
and O otherwise. Finally, we have the over-parameterized version of Equation (3) with max pooling
(d)(z))
Σ jE(C]
Intuitively, for all v; E [xp]. (0) (oe (z)) reconstructs æ, by adding together those wj such that the "most related" patch to wj. Combining Equations (4) and (5), our decoder Φa is defined as
Φa (o)())) = (o') (ox(z)), ... . (o)(
(6)
In the rest of this paper, we will always omit the parameter w it does not cause confusion.
2.3 TRAINING PARADIGM
We train the autoencoder & under standard empirical risk minimization paradigm (Shalev-Shwartz & Ben-David, 2014). As mentioned in Section 2.1, given N E Z=, the training data [zx] are drawn independently from distribution D To measure how well ø reconstructs the input data, let the reconstruction loss be
((2; w) (1; m) 2|
The corresponding (empirical) reconstruction error is defined Rv(w) = N Ene(Ny {(zn; w)). We minimize (x(w) using gradient descent with random initialization. For all j [C] and t 0, we iterate w; by
w/tt = = - nI Vw, (N
for all / E N. The learning rate Tıt specified in Appendix D, is parameter that gradually decreases to O as the itcration step grows. Note that is independent of wy. For all j E [C]. the random initialization of w, given by w ~O,0 T) in which N(0, T) is the d-dimensional Gaussian distribution and oo is given constant.
3 THEORETICAL ANALYSIS
Our analyses can be roughly divided into two phases: the training phase (Section 3.1) and the reconstruction phase (Section 3.2). The main results and intuitions are summarized in the following theoretical roadmap. We include list of summarized terms and notations in Appendix C. The proofs are postponed Appendix
3.1 TRAINING PHASE
We start with analyzing the growth of the kernels. Given n E [N] and E [P],
An.i In,i 2 (zx))
(9)
to denote the difference between the input patch In,i and its reconstruction of (o (z,)). Recall that the growth of the kernel is given by w - w = -n Ry(w") in Equation (8), which V w RN(w's can be calculated by the following proposition.
Proposition 3.1.
For any j E [C), the gradient of the empirical reconstruction error is
2 Vw, Ry(w)= - Σd (). (o((1) Dn,))) An. + (Anis (),)));))) o ((wj, In,i)) æn.i). (10)
5

---

Published as a conference paper at ICLR 2026
Theoretical Roadmap
Definitions and Assumptions
Results Initialization
Training Phase Results
Assumption 3.1
Theurem
Theorem
Thas assumption characterize normal model.
This theorem gives probubility
This theoeum shows abworbed by a cone, a kernel would never gets out
During training, the cone sets alanch the kernels the network
bounds for initaalzztionn
Delinition 4,1
Slown of Use cvew wels.
Figure and by this mean, network estracts the festures from training daca.
This definition specifies concept of the cope sets (of the features).
Technical Results
Corollary 4,1
This corollary discusses the aharpnes the comes.
Proposition
This proprcetion calcubates the gradient the reconstuction L2700
Legend
Main Intuition?
Reconstruction Phase
Result R
Explanation
The kernels ('()) grow faster along to other features, just like being absorbed in sharpening come.
Theorem
Lemmas 4.1 and 4:2
(iit high prolubiiitty the signal the anomaly putches would (be too small to reconstruct.
B.2 depends on
These are technical lemmas
explains
the following
Figure 2:
A theoretical roadmap of our analysis. We visualize the dependency of the results and attach a brief explanation for each. We also summarize two major intuitions.
Proposition 3.1 implies positive correlation between the growth rate of w, and the size of wy (i.e., [|wy))) considering that the sizes of ((w,, ((W.F.nn))) In,s)), (An,i, wz)) and a d'(wu, In,i)) in Equation (10) all directly related ||,b|-- The following lemma further controls the initial value and the growth rate of ||wy Il-
Lemma 3.1. Let po = pold) and Sn = fi(l) be parameters. For all j E [C1, we have
|ee||| I aoSh
(11)
with probability at least 1 - Po with regard to the random initialization of w. Moreover, for all t e (T), the size of w) is controlled by
|w/|l| S toofn.
(12)
with probability at least 1 Po. Here, [tt, d) is a parameter depending and d.
Remark 3.1 (Parameter Analysis). Some parameters in this paper, e.g., Po, fh. and T, have clear geometrical or probabilistic intuition. We will explain them later parts of this paper. A summary for the parameters can be found in Appendix C. For simplicity, we will omit the sup- or sub-scripts t and d if it does not cause confusion. A set of recommended parameters given in Appendix D.
Note that the probability lower bound for Equations (11) and (12) to hold (i.e., at least 1 - po) is the same because, after random initialization, the kernels would grow in a deterministic manner; the iteration rule given by Equation (8) introduces no randomness. The above discussion implics that those kernels with larger initial values will grow faster. A simple deduction is that after mul- tiple iterations, the paster-growing kernels would take larger values, thus significantly affecting the reconstruction phase of RBAD.
The following definition introduces the main intuition of our analysis.
Definition 3.1 (Cone set). Let 01 = C1 (d) 10 and f. f-(d,t) S ea (dd. be parameters with fr- = Jn o(). For any / E [T' and & [d], let the cone set of feature at iteration 4, denoted by St (k), be the collection of wy [wc] such that
<w/i vk) 2 to and * for any k' + k. (w,", uw)) < too.r-
We say that kernel wy is absorbed by the cone sets of feature U2 if 3tg E [T'] such that w/ E (*(k) for all 2 to, i.e., after the iteration step fo, the kernel w, is always the cone set of feature Uk.
Remark 3.2 (Cone Set Intuition). For any & E [d] and E [T]. k. together with the parameters do- C1. Cn fn and fr specify a d-dimensional hypercone with height topci fn. and radius to According to Definition 3.1. those w/ E 5*)k are bounded by the surface of the cone. Since we let f. = fu o(t), the cone would get sharper during training (cf. Figure 3). As result, the kernels in the cone would get more aligned to feature vy- In particular, the kernels absorbed by the cone set of a feature will eventually approximate to the direction of that feature.

---

Published as a conference paper at ICLR 2026
We can obtain the following lemma directly from Definition 3.1.
Lemma 3.2. For all t 2 1, if (k) and x; contain Dk, then we have o(i,j) = 1.
Lemma 3.2 implies that the kernels in the cones of a feature Us will contribute to the reconstruction of the patches that contain Uy at all iterations. Lemma 3.2 also simplifies the iteration analysis for the kernels in the cones. We can prove the following theorem with the help of Lemmas 3.1 and 3.2.
Theorem 1. For all & E [d], if (k)) 2 1, then (Ik) S S*+1(A) (k) all t E (T].
Remark 3.3 (Interpretation of Theorem 1). Theorem characterizes how the kernels of the autoen- coder extract features from the training data, providing an answer for question Q1. For all &o E [d] and E [T]. Theorem implies that w E (k) will keep growing faster along uy than along Ue for all K' + k. The geometrical intuition is that the kernel will be absorbed by the cone set of feature if it comes into the cone sel once. After multiple iterations, (uwy, vwe) would become almost negligible compared to (ur), 2y) since f. fn. o(t). The direction of wy gradually approximates to the direction of Ut as increases, i.e., wj learns the direction of uk.-
For any i E [d], the following theorem proves that $(K)) is not empty with high probability. Theorem 2. Let p1 (i()) and p2 p2(d) be parameters. For all k E [d]. the size of (k) can be upper-bounded by
(R))
I Cp2 + X
(13)
with probability at least 1 exp
3x²
over random initialization. The lower-bound
62x
|8a
Cp1-A
(14)
hold with probability least 1 exp
X²
20P²
Theorems 1 and together characterize the growth of the kernels wy in the cones of the features.
Figure 3 demonstrates the geometric intuition of our results.
17
V3
173
//
W fi
W
W j1
i0 jz
Wii
U2
U2
U2
V1
V1
V1
(a) At initialization (t 0)
(b) After multiple steps
(c) At convergence
Figure 3: The geometry intuition for Theorems and 2. Consider d 3 and Wj, Wi2 E 8(3) The kernels are absorbed in the cone represented by the light blue surface. Definition 3.1 implies that the cone would get sharper as increases. Theorem proves that w,, Wj, will be absorbed Wj.: Wj,- E $(3) Theorem 2 implies that the cone set non-empty with high probability.
It is worth mentioning that the above results hold for all feature ty with 3x + 0, regardless of whether 2y is a normal feature or an auxiliary feature. Recall that we use auxiliary features to characterize patterns not directly related to the semantics of the normal data. As a minor implication, we prove that neural networks would also learn the auxiliary features. The following corollary discusses the tightness of the bounds for different features.
Corollary 3.1 (Informal). Recall that Definition 3.1 implies a growth rate gap between the kernels growth along the "cone feature's 's direction" " and features direction. For those kernels absorbed the cones of normal features, the growth rate gap is larger than those in auxiliary features.
The formal version of this corollary is postponed to Appendix E due to limited space.
In summary, this section analyzes the training phase of RBADs. Notice that the question Q1 is answered in this section. It remains to discuss how the features are used in the reconstruction phase and why the anomalies are hard to reconstruct.
7

---

Published as a conference paper at ICLR 2026
3.2 RECONSTRUCTION PHASE
Since the learning rate 7) decreases to O as grows, the kernels would gradually converge. Denote the converged kernels by w= (wi, w wip) and the corresponding network by o- Section 3.2 discusses how 6" reconstruct the input images.
We start by introducing the data model for anomalies. Let
Dror be normal data (cf. Assumption 2.1) and aa E H be patch. Given ig E [P], the anomaly is defined as
z(io fa) = (21, Fig Ligt,," IP);
(15)
iee, one of the patches such that Da contains 2g and non-semantic anomaly if otherwise. For non-semantic anomalies, further assume that aa is drawn from the uniform distribution supported on {æ ||||| = 1}. Note that our analysis
replaced by Va We call z(1o; aa) semantic anomaly if 3k E [d]
be trivially extended to the where multiple patches are replaced.
In order to answer Q2, we go on to analyze the behavior of the kernels in the reconstruction phase. Taking 211, æ.) as the input of Φ², by definition, have
t (= Ra); wn)) (((n, m.)) (2;
(16)
for all E [P] such that i io. Since V is orthonormal basis of H. we can decompose aa as
Fo. (Da Uk} Uk.
(17)
kE(d]
By taking Equation (17) into Equation (1), the replaced patch is encoded as follows.
ob (z(io, aa);ww))))))) a(ww),
Σ
(Ca, Uk) vn)) o E
(aa, Un) (woj, va)). (18)
kE(d
kEd
For all [C]. by definition, the max pooling operation only reserves maX a ((w,, ar,)) We have proved in Section 3.1 that the direction of the kernels would gradually get closer to the direc- tions of the features. For non-semantic anomalies, we can prove the following theorem.
Theorem 3. For all 0 E (0, 1), Vj E [C], and vk E [d], if (m), vu) > (1 - 0) -wy then
(w,, wa) ||,|1| 220,
(19)
0vd-1
hold with probability at least 1-20 20 expeexp( (xp(-1
Theorem 3 implies that the convolution between non-semantic anomalies va and the kernels in the cones will take small values with high probability. Therefore, the signal in the replaced patch will be filtered by the max pooling operation, making the replaced patches in the anomalies hard to reconstruct with high probability. The above discussion answers the questions Q2 and Q3-
Remark 3.4 (Semantic Anomalies are hard detect). As reported in the previous survey (Ruff et al., 2021), RBADs are unsuitable for detecting semantic anomalies. As discussed in Allen-Zhu Li (2023), data with different semantic information could contain similar features. In our model, denote the feature contained in the replaced patch of semantic anomaly by Uk. Lemma 3.2, Theo- rem 1, and Theorem together imply that the autoencoder would also learn the semantic information of Uk, if BE, 0. As result, the kernels in the cone of Uk, would align larger value (depending on 3k.) ) to (wj, Ta) for semantic anomalies than non-semantic anomalies. The reconstruction error the semantic anomalies would be smaller than the non-semantic anomalies, making the semantic anomalies harder to detect.
In summary, this section analyzes the reconstruction phase of the obtained autoencoder and explains why the non-semantic anomalies are hard to reconstruct. We also discuss why RBADs unsuitable for detecting semantic anomalies. All the questions raised in Section are answered.
4 NUMERICAL VALIDATION
This section presents synthesized and real-world experiments that validate our theoretical findings. Due to limited space, the implementation details and auxiliary results reported in Appendix G.
8

---

Published as a conference paper at ICLR 2026
Synthesized Experiments We conduct this experiment synthesis dataset that contains 4000 training data and 1000 test data for each of the normal data, semantic anomalies, and non-semantic anomalies. The training and test data are drawn from the distribution specified by Assumption 2. I. We consider different noise levels from e E (001,0.01, 1, 10) and we let d = (nt P) for cs E (1.2, 1.5, 2). We explore other choices of and C' in Appendix G.2. The anomalies are generated according to Equation (15). We consider aver-parameterized scenario with C' (n2(1. The network is specified in Section 2. We use the mean square error (MSE) for the loss function and SGD for optimization.
0.01, 0.001). ). The number patches P is selected
from (20+ + 4u
2209
201-2
64138
33000
33270
0215
53411
38211
(8) C3 1.2
(b) C3 1.5
(c) C3
Figure 4: Results of the synthesized experiments. In these figures, the x-axis is the number of patches P. The y-axis reports the reconstruction losses. We let d = int(cyP) for C3 E {1.2, 2} and C int (1.2d) in the study. All of the experiments exhibit clear gaps between the reconstruction errors of the normal data and the non-semantic anomalies. Besides, as discussed in Remark 3.4, the gap between the reconstruction errors of the normal data and the semantic anomalies not as significant as that between the normal data and the non-semantic anomalies.
Visualization of the kernels Figure visualize the first-layer features learned from MNIST and CIFAR-10, which validates our proposed cone set intuition. It can be observed that the shapes and colors of the kernels do not significantly change during training because, as claimed in Section 3, the kernels are absorbed by the cones of the features. Notice that some of the kemelss remain noisy even after 50 epochs. We interpret these kernels those w, such that j (
(a) Mnist, Epoch 0
(b) Mnist, Epoch 10
(c) Maist, Epoch 50
(d) CIFAR-10, Epoch 0
(c) CIFAR- 10, Epoch
(f) CIFAR-10, Epoch 50
Figure 5: Visualization of the kernels in the first layer the autoencoder.
4.1 DISCUSSIONS
We discuss the significance of the experiments as follows.
Synthetic experiments (Figure 4) The synthetic experiments designed to verify the theo- retical modeling and analysis in Section 3.2. In learning theory community, it is a common prac- tice to conduct synthetic experiments under the same setting the theoretical analysis, serving to

---

Published as a conference paper at ICLR 2026
demonstrate whether the theoretical modeling effectively characterize the practical problems (i.e., RBAD in our paper). the caption of Figure 4, we briefly explain how the experimental re- sults consist with our theory and the existing results. More specifically, the synthetic experiments conducted under our theoretical setting exhibit clear reconstruction error gaps between normal data and the anomalies, which is key rationale underlying RBAD.
Cone set intuition (Figure 5)
We use experiments on real-world datasets illustrate the pro- posed cone set intuition. As a supplement to the existing explanations, the experiments in Figure discusses the manifestations of the cone set intuition networks trained with MNIST and CIFAR- 10, which indirectly verifies the validity of our theory.
Take the results MNIST (Figures 5.(a)-5.(c) an example. The visualized kernels randomly initialized at epoch 0. It can be observed by Figure 5.(a) that the kernels are completely noisy. By comparing Figures 5.(b) and 5.(c), the following two observations are closely related our proposed theory.
1. Some kernels (e.g., (row-2,col-3) and (row-3, col-4)) have exhibit clear contours at the early stage training (epoch 10) and do not undergo significant changes at convergence (epoch 50).
2. Meanwhile, some other kernels (e.g., (row-1, col-1) and (row-3, col-2)) fail to exhibit dis- tinct outlines even after convergence, which aligns with the case when the kernel is not absorbed by any of the cones.
Both of the above points are consistent with the probability- -based training dynamics proposed in our work. As a supplement to the existing explanations, these experiments discusses the manifestations of the cone set intuition on networks trained with MNIST and CIFAR-10, which indirectly verifies the validity of our theory.
CONCLUSION
This paper studies the training dynamics of 2-layer convolutional autoencoder and how it recon- structs different types inputs. Particularly, we answer Q Q3 raised in Section as follows.
(Q1) How does the model learn the features from training data in RBAD? (A1) The kernels absorbed by the cone sets of the features would approximate the features.
(Q2) What is the role of the learned features in the reconstruction phase? (A2) The features process the signal from the inpul patches and use them for reconstruction.
(Qa) Why are the anomalies harder to reconstruct? (Ag) The processed signal from the auxiliary features and the random noise is smaller than that from the normal features.
Our theoretical findings and intuitions are supported by synthesized and real-world experiments.
ETICS STATEMENT
All authors of this submission fully acknowledge and strictly adhere to the ICLR Code of Ethics. Prior to submitting this paper, every author has carefully read and understood the requirements of the Code of Ethics, and commits to complying with all relevant provisions throughout the entire process of paper submission, review, and subsequent discussion.
This research does not involve human subjects, nor does it include the release of any datasets that may pose risks to privacy or security. the design of research methodologies and the discussion of potential applications, the authors have fully considered issues related to fairness, bias, and research integrity, and have taken necessary measures to avoid any potential harm. Regarding conflicts of interest and sponsorship, all authors confirm that there conflicts interest. For matters related to legal compliance and research ethics, the research strictly follows the relevant laws, regulations, and academic norms of the research field.
undisclosed financial
financial
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
The main contribution of this paper is establishing a theoretical framework for analyzing RBAD and providing plausible explanation for RBAD. Complete proof can be found in the Appendix E. Clear explanations and discussions are provided just after every theorem and corollary.
ACKNOWLEDGMENT
This work is supported by Key R&D Program of Hubei Province under Grant 2024BAB038, National Key R&D Program of China under Grant 2023YFC3604702.
REFERENCES
Faruk Ahmed and Aaron C. Courville. Detecting semantic anomalies. In AAAI, 2020.
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In FOCS, 2021.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In ICLR, 2023.
Richard Archibald, Feng Bao, Yanzhao Cao, and Hui Sun. Convergence analysis for training stochastic neural networks via stochastic gradient descent. CoRR, abs/2212.08924, 2022.
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. The mvvec anomaly detection dataset: A comprehensive real-world dataset for unsupervised anomaly detec- tion. Int. J. Comput. Vis., 129(4):0038-1055 2021.
Siddhartha Bhattacharyya, Sanjeev Jha, Kurian Tharakunnel, and J. Christopher Westland. Data mining for credit card fraud: A comparative study. Decision Support Systems, 50(3):602-613, 2011.
Yang Cao, Yanbo Chen, and Weiwei Liu. Prevalence of simplex compression in adversarial deep neural networks. Proceedings of the National Academy of Sciences, 122(17):e2221593122, 2025.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey.
CoRR, abs/1901.03407,, 2019.
Lucas Deecke, Lukas Ruff, Robert A. Vandermeulen, and Hakan Bilen. Transfer-based semantic anomaly detection. ICMI., 2021.
Lutz Dümbben. Bounding standard gaussian tail probabilities. CoRR, abs/1012.20634: 2010.
José Manuel Díaz-Cao, Xin Liu, Jconghoon Kim, Maria Jose Clavijo, and Beatriz Martinez-Lópee Evaluation of the application of sequence data to the identification of outbreaks of disease using anomaly detection methods. Ver Res, 129(4):003--000) 2023.
Mark S. Graham, Walter H. I.. Pinaya, Petru-Daniel Tudosiu, Parashkev Nachev, Sébastien Ourselin, and M. Jorge Cardoso. Denoising diffusion models for out-of-distribution detection. In CVPR, 2023.
Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting for two-layer relu networks. CoRR, abs/2303.04144 004145 2023.
Yuxin Li, Yaoxuan Feng, Bo Chen, Wenchao Chen, Yubiao Wang, Xinyue Hu, Baolin Sun, Chunhui Qu, and Mingyuan Zhou. Vague prototype-oriented diffusion model for multi-class anomaly detection. In ICML, 2024.
Victor Liveroccee, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. CoRR, abs/2305.1 18593, 2023. doi: 10.48550/ARXIV22305.1M 18593.
Wenxi Lv, Qinliang Su, Hai Wan, Hongteng Xu, and Wenchao Xu. Contamination-resllleent anomaly detection via adversarial learning on partially-observed normal and anomalous data. ICML, 2024.
11

---

Published as a conference paper at ICLR 2026
Xinsong Ma, Xin Zou, and Weiwei Liu. A provable decision rule for out-of-distribttio detection. In ICML, 2024.
Xinsong Ma, Jie Wu, and Weiwei Liu. A closer look generalized BH algorithm for out-of- distribution detection. In ICML, 2025a.
Xinsong Ma, Jie Wu, and Weiwei Liu. Sac-bl: A hypothesis testing framework for unsupervised visual anomaly detection and location. Neural Netw. 185(C), 2025b. ISSN 0893-6080.
Xinsong Ma, Jie Wu, Xin Zou, and Weiwei Liu. A unified decision rule for generalized out-of- distribution detection. IEEE Trans. Pattern Anal. Mach. Intell., (8(3):3544-3555 2026.
Arian Mousakhan, Thomas Brox, and Jawad Tayyub. Anomaly detection with conditioned denoising diffusion models. ArXiV preprint, abs/2305. 15956, 2023.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton van den Hengel. Deep learning for anomaly detection: review. ACM Comput. Surv. 54(2), 2022.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. OCGAN: one-class novelty detection using gans with constrained latent representations. In CVPR, 2019.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with ad adversarial autoencoders. In NeurIPS, 2018.
Marco A. Pimentel, David A. Clifton, Lei A. Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Process., 99:215 249, 2014.
Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proc. IEEE, 109(5):756-795, 2021.
Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H. Rohban, and Hamid R. Rabiee. Multiresolution knowledge distillation for anomaly detection. In CVPR, pp. 14902-14912, 2021.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
Vladimir Naumovich Vapnik. The Nature of Statistical Learning Theory, Second Edition. Springer, 2000.
Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptofic Viewpoint. Cambridge Uni- versity Press, 2019.
Hongzhi Wang, Mohamed Jaward Bah, and Mohamed Hammad. Progress in outlier detection tech- niques: A survey. IEEE Access, 7:107964-108000 2019.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In ICML, 2021.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distributiio detection: A survey. CoRR, abs/2110.11334, 2021.
Yibo Zhou. Rethinking reconstruction uutoencoder-baseed out-of-distributian detection. In CVPR, 2022.
A RELATED WORKS
The present paper studies the theoretical properties of RBAD through the lens of feature learning. The related works of RBADs and feature learning discussed follows.
12

---

Published as a conference paper at ICLR 2026
A.1 ANOMALY DETECTION AND RBAD
In contrast to many machine learning topics, anomaly detection (AD) Ma et al. (2025b) adopts the "open-world" assumption in which the training and test data not drawn from the same distribu- tion. According to how the test distribution differs from the training distribution, AD can be further categorized into semantic AD (Ahmed & Courville, 2020; Deecke al.. 2021) and non-semantic AD (Bergmann et al., 2021; Mousakhan et al., 2023). In this work, we characterize the normal data and both types of anomalies using a unified probabilistic model.
The "open- "open-world" setting of AD is also adopted in some closely related topics, e.g., outlier detection (OD) (Wang et al., 2019), novelty detection (ND) (Pimentel et al., 2014), and out-of-distribution (OOD) detection (Graham et al., 2023; Ma et al., 2024; 2026; 2025a). Yang et al. (2021) review the similarities and the subtle differences of these topics. Although AD, OD, ND, and OOD detection different topics, these terms (especially AD, OD, and ND) are used interchangeably in some previous works (Ruff et al., 2021; Pang et al., 2022). is worth mentioning that reconstruction based methods can also be used in ND (Pidhorskyi al., 2018; Perera al., 2019) and OOD detection (Zhou, 2022). The analysis in this paper can be easily extended to the reconstruction -based methods for ND and OOD detection.
A.2 FEATURE LEARNING
In this paper, the term "feature learning" refers to the theoretical paradigm proposed by Allen-Zhu & Li (2021). Compared to other theoretical paradigms, e.g., the uniform convergence paradigm (Wainwright, 2019; Shalev-Shwartz & Ben-David, 2014; Vapnik, 2000) and the convergence analy- sis paradigm (Archibald et al., 2022; Shalev-Shwartz & Ben-David, 2014), feature learning provides a more delicate discussion on how the weights of the neural networks iterates.
As the first work of feature learning, Allen-Zhu & Li (2021) analyzes how a 2-layer ReLU network extracts features from the training data. Following Allen-Zh -Zu & Li (2021), a series of works use the tools from feature learning to study different topics, including knowledge distillation (Allen-Zhu & Li, 2023), contrastive learning (Wen Li, 2021), and neural collapse Cao al. (2025). In this work, we study RBAD under this paradigm by analyzing how 2-layer convolutional autoencoders extract features from training data and reconstruct the samples. To the best of our knowledge, this paper is the first to analyze generative models using feature learning.
B LIMITATIONS
This section further discusses the theoretical assumptions and the limitations of this paper.
In this paper, we introduce the Hilbert space 7 of patch-like vectors and assume the data have P- patch structure. The patches are essentially "sub-matrices" "sub-tensors" of the original image. Previous works have characterized the patches as vectors since the operations (e.g., addition, scalar multiplication, and convolution) of the patches satisfy the axioms of the Hilbert space when the patches are -on-overlapping. However, the patches are not equivalent to vectors in general; to our knowledge, existing structures vector space cannot characterize the overlapping patches.
In practice, the max pooling operation is placed after the convolution layer, calculating the maximum convolution outputs of several overlapping patches close to each other and possibly containing the same feature. In this sense, the patch with the maximum convolution value might contain the most representative version of the feature. Max pooling would greatly reduce the output dimension while not significantly affecting the value since the reserved value acts representative of the others.
In this paper, we simplify the overlapping operation and the relationship between the patches and the features. We use non-overlapping patches to replace the most representative overlapping patches. Besides, the max pooling operation is further simplified in our paper to calculate the maximum value for the convolution of all the patches instead of those close to each other. As result, according to Equation (10), the growth of the kernels can be decomposed to the combination of two directions.
The max pooling operation in our paper significantly simplifies the analysis process. Without max pooling, the growth of w, (j E [C]) would be written as the weighted sum of A,,; and æn,; for all i (P]. However, ignoring the growth of uy along minor directions would cause problems, e.g., the
13

---

Published as a conference paper at ICLR 2026
reconstruction phenomena mentioned in Section 3.2. Nevertheless, the kernel will still grow faster along the direction the most representative patch since max pooling reserves the maximum convolution value. The max pooling operation does not change the intrinsic quality of the problem.
In summary, we argue that the P-patch data structure and the max pooling opeation are coupled with each other and should be adopted entirely. The integrated model could provide simple yet representative characterization of the practical problem.
C TERMS AND NOTATIONS
We include list of terms and notations in this section.
Bold letters (e.g.. æ, v, and w) denote vectors. The i-th entry of is noted by 2(),
[N] 21, NJ denotes the index set from 1 to N.
[xN] = {æ E [N] s.t. a = x;} (= (0), 22 xNn (i ز = a; + x x,)).
H. is task-specific Hilbert space, whose clements called the patch-like vectors. Patches æ, features v, and kernels w are all patch-like vectors.
The images (I1, 22, xp)" T E 7PP are called P-patch data.
the number of patches in an image.
d is the dimension of H. also known as the size of the patches/kernels.
C = e2d is the number of the kernels.
V = {u1, ... , daa
the set of all features.
Vnor and Vauz are the sets of normal and auxiliary features.
Vnor = (w), #2 and Vaur = {w3. "..... , theoretical analysis.
We explore more combinations in the synthesized experiments.
+ for some small random noise A patch æs contains a feature implies that
(ef. Section 2.1). controls the size of p.
E jP]. E [C]. and A: [d] are the exclusive indices for patches (æ;), kernels (wy). and
features (V²), respectively.
ó Qa de is the two-layer convolutional autoencoder defined in Section 2.2.
σ is the smoothed ReLU function defined Equation (2), parameterized by cp-
{(; w) and (x)w) denote the reconstruction loss and error, respectively. controls the size of the (zero-mean Gaussian) random initialization of w.
T' is the maximum number of iterations.
Given t E [2] and & E [d], S*(k) represents the cone set of feature Un at iteration t. fr f-dd, 1), fn [n),,, and constant e1 controls the radius and height of the cone. Pi (d), and p2 p2(d) control the probabilistic bounds. Equation (E.16) specify the expression of po- The upper bounds for |[w]|] in Lemma
po po(d).
3.1 hold with probability at least 1 Po- Pı (specified in Equation (F.28)) and p2 (specified in Equation (F.29)) control the two-side bounds for (K)) (the initial size of the cone sets) in Theorem 2.
D PARAMETERS ANALYSIS
Selecting a of proper parameters can significantly simplify the analyses of feature learning, es- pecially those proofs that involve tight inequalities. However, finding such proper parameters is extremely laborious since every single attempt in a new set requires re-proving most of the results. Besides, we argue that fixing the parameters might confuse the readers and hurt the overall intuitive- ness of the analysis. The choice of parameters is not unique in the analysis of feature learning; a wide range of different choices could lead to similar results. As result, fixing the choice of param- eters could conceal the actual relationship between different variables. For the above reasons, our paper specifies the constraints the parameters instead of fixing them.
14

---

Published as a conference paper at ICLR 2026
D.1 RECOMMENDED PARAMETERS
The patch dimension d, number of patches P. and the probability 8a. (A: E [d] in Assumption 2.1 task-specific constants. It convenient for the readers take P 10, d 20, and f& < 0.5 for all k 2.
The other parameters in our paper can be divided into three groups:
The first group are constants, including co, C1, and c2. It recommended to take co = 0.01, C1 0.5, and 0o 1.2.
2. The second group includes parameters that depend on the dimension d. Recall that we let | 1 for simplicity. To ensure that analysis is without loss of generality, the noise threshold e and the variance the random initialization ap should be re-scaled accordingly. For better understanding,
suggest the readers consider do d-2 and do. e.g.,
= 0 0.001, in this paper.
3. The third group includes parameters that depend on and d, mostly the variables related to the iterations. The constraints are discussed in the rest of this section.
We assume that the learning rate π7 decays the speed 1/. 0.2t as t grow. For completeness, we let 77 = (i.e., the kernels stay unchanged in the first iteration). The proof of the results of our analysis imposes several constraints the parameters. To ensure that Lemma 3.2 holds, assume that
(c1 2v 23) (0.5 22000002) (n) > fd, which weak condition since f+(d, o(t) (d) and s d=2 The proofs of Theorem require C1, and to satisfy
(D.1)
2e > co.
(D.2)
in which co the threshold in the smoothed ReLU function in Equation (2). Denote È 3 +
4s s 0.138, é /2e s 1.045, and ē C1 /2e 0.455. As the main result of our analysis, Theorem require
5c1
N (22 (1 + (e + (laooh))
(D.3)
2
Taking the recommended parameters into Equation (D.3), we have NB& (2 x 0.455 x 1.045 1.045 1.5 10-6 x (tfh)²) 2 1.25,
(D.44
ie., the first constraint on N, Sg. f, and fr. is NB, (0.951 6.27 10 10 (tfh) 2 1.25.
(D.5)
Another constraint from Thcorem is 0.4N oo (fn- (6e + v2= + V2stopfh(e2 - 1)d) + fr. (toofn: - (02-1)d+
+ v2s
(D.6)
+ + (1 1, 1,
Let T = = 100, fn (d) = 0.2vd, and f. fn. (t 1) 00. Equation (D.6) becomes
40,3g (Ji (0.051 0.179 taof)) f.(4tooff + 0.045) < (t+ ()f+(++1.0))
(D.7)
It is not hard to check the validity of the recommended parameters.
E COMPLETE PROOFS
In this section, we go through the proofs in our analysis. We use separate numbering for the equations in this section so that the readers do not need to turn back to the main parts to reference the equations. Denote
A = Dn.i o (che
(E.1)
i..., the difference between the original patch æn,; and the reconstructed patch (00 (0. Then, the empirical reconstruction error can be written as
Ryy Ry(w) ()) = N
Σ E(zn; w)
Σ N
|10(2n; w) 2,
Σ (An,i. An.i) (E.2)
N
N
RE[M]
nE[M] kE(P]
As warm-up, the following proposition computes the gradient of RN (w))
15

---

Published as a conference paper at ICLR 2026
Proposition 3.1. For all [C], the gradient of the empirical reconstruction error is
Vw, Ry(W)) Σ jn(i,)) (o An, + (An,ir wy) o ((w), ande). (E.3)
Proof of Proposition 3.1. By definition, have
( ((zn)
Vw, wy RN(wW = Vw, w,
Σ nE[N]
Ve, ((zn).
(E.4)
N
MEN
Recall
<(z;w) = da (do(2;w)) = (0)) (z)(z;
.88 (P)
(de (oe(z;)))
(E.5)
and
{(;; w) [1 w) - =112.
(E.6)
According to Equations (E.5) and (E.6), for all [N], we have
[(zn) = [l)(zn) =n 2
2 (oe(z,)) Ln.i 28 (pe(2n) In.i)
(Anii An.i).
EEP
SE|PI
(E.7)
and the gradient of ((xx) can be written as
V w Σ E Ana)
E (VV An.i An.))
(E.8)
Here, we abuse the notion of inner product (V An.i: An.i) simplify the calculation of the gradient of l without causing confusion. Consider the right-hand side (RHS) of Equation (E.8). For all i [P]. the term in the sum equal to
o(wwy an,,)) (i. )
(V An.in Ani) = An.i Ve,
In,i
Σ jEC
(E.9)
An.i Vay
Σ
al(myy, am.ii)
J'ES. (6)
() is the natural extension of (i, considering that æ; is replaced by Tn in the proof. (f) is defined as
Here, 6n (i.j)
Sa(1) = (i [C] 8 (i.j) 1}. (t) can be interpreted as the "empirical cone set of the patch x;". From Equation (E.9) we know that (V Www An,i An,i) takes 0 when j & (i). Otherwise, we have
(E.10)
(Vu, An.is An.i) (An,i, Vw, (a((u, )n,)))e;;)) = ((w2) xn,,)) An.i (An,i wr) of ((W1, (n.i)) Fniis
(E.11)
which for all u E I, the derivative of ar at u is
d((u) = {
if u < 0; if < S Co:
(E.12)
if co-
Finally, we can obtain Equation (E.3) by combining Equations (E.4) to (E.12), which completes the proof.
After random initialization, the kernels iterate according to the gradient descent rule, i.e.,
w ++1 w/ 7) Vw, Ry Ry (w").
(E.13)
Let fa (d) be the control function of the growth of the major part of the kernels (i.e., the height of the cone in our cone set intuition. At each iteration step, the size of the kernels can be bounded by the following lemma.
16

---

Published as a conference paper at ICLR 2026
Lemma 3.1. With probability at least 1 Po. the random initialization of w/ satisfies
| aofh-
(E.14)
For all t 2 1, the size of wi is controlled by
||//||
S toufh.
(E.15)
with probability at least Pu- both Equations (E.14) and (E.15), we let pu take
(- (Jn d)²
Po = 2exp
(E.16)
8d
Proof of Lemma 3.1. By definition, we can write
w/
0,(1)
3
0,(2)
w
o0)
(E.17)
and
0,(1)
1
11
Σ
j
Xd
(E.18)
do
in which xa is the chi-squared distribution with d degrees of freedom. Lemma F.3 implies that
P [1|l@]11
afd Aoao|
s 2e 2A-d/s
(E.19)
for all À E (0, 1). We can obtain Equation (E.14) by letting
A fh d
(E.20)
We prove the second result by induction. Equation (E.19) implies that [|w]] || = || S aofn as required. For all t 1, assume that wn-- satisfies Equation (E.15). By definition, we have
|
w 7/--1 w
RN (2x"
Wj
Σ Σ
28m(i,j) (ar (( (n.i)) Ani + (An,i w)) o' ((wjs (n,))) an.i]
(E.21)
N
NE[N] KE(P!
<||w|||+
Σ
Σ 28, (1w, (n.i)) (o An.s (An,s, wy) of ((w3, xn.i)) Dn.il
N
RE(NN iE[P]
Due to limited space, the subscript or superscript t 1 is omitted without causing any confusion. For all E [N] and Vi E [P). it is easy to obtain that
||o ((w,, Fnn))) An,s + (An,i, un,) o' ((w0, (n.)) æn,ill
<2] A, ||wl
(E.22)
<2(1 + +
Since only one the [P] is counted the second summation in Equation (E.21) for all n e [N] due to the property of 6, (i, have
Σ Σ 28, (w., (An.s, w/) o' (lww,, xn.i)) æn.s] <4(1+e)3m) w. ||.
N
RE[N]
which implies that
| $ (1 5))=1))(=;311
(E.23)
We can obtain Equation (E.15) by taking = 0.2t into Equation (E.23), which finishes the proof of this lemma.
17

---

Published as a conference paper at ICLR 2026
Lemma 3.2. For all 2 1, if (kk and æ; (or Fn,z) contains Us. then have 8(2,j) 1 (or 5,(i, 1).
Proof. Recall that ((,,j) is a binary-value function that takes 1 when
i arg max ( (Ww aw))
(B.24)
x'[P
and 0 otherwise. Since a is non-decreasing, we have
arg max o(un, = arg max (wz, mt)
(H.25)
kE(P]
rE(P)
When j (k) and æ, contains uk. (wy, x.) can be bounded by
2 tonci fn. |w/ ||| x Us (w), x;)
(w). va) (w), T; U2
(E.26)
(c1 v 23 fo.
On the other hand, consider and K such that ay contains va. By Assumption 2.1, + l given i' o i. Therefore, we have
(w) = (wj, Uk + (w,, v. 0n) S to + [0, ||
vkll ukl
(E.27)
< (f. + 22 fn fr. 1000
We prove the lemma by combining Equations (F.26) and (F.27).
Denote
P1 =
(cthh)² +1 22T
Ci
e-(ausa)))
|
fr 2T
1 E
f
C-1
(E.28)
and
1 e co(th
fr e
C-1
P2 =
ci
22T
(E.29)
The following lemma proves that the size of at random initialization bounded from both sides. Note that wD w since 7/0 0.
Lemma 2. Let and be parameters depending on d and t. For all k E [d]. the size of (k) can be upper-bounded by
(K)(k) S Cp2 + X,
(E.30)
with probability at least - exp (- 62
3x2
over random initialization. The lower-bound
[})( 2 Cpi - A
(E.31)
hold with probability at least 1 - exp
1cexp
X² 2Cpa
Proof of Theorem 2. According Dümbben (2010), we have
1 l
P g-N((.1)
[g
x|
<
(E.32)
1/2 2
v 2T
for arbitrary x > 0. For all j e [C), we have
(w)) j:
Σ
uk
w
1.(k)
(0. 1).
(E.33)
do
ke
[d]
18

---

Published as a conference paper at ICLR 2026
Take c1 in Equation (E.32),
obtain
a/
1 --(essh
Pug [uw) oa) > onaita]
(B.34)
(cff))+
22
ath 2TT
Similarly, we have
Jr e-
< Pug (20j, 0k) > oufrl < 1 e
(E.35)
f +++v 2m
fr 2T
by letting fr. By definition, we have
Pwo E (c:) Pwg [(w), vk) oafff (1- Puo [ vx) > oofr.) 0-1
(E.36)
Combining Equations (E.34) to (E.36), we obtain that
p1 S Pug (je s p2
(E.37)
By Corollary F.1 we immediately get Equations (13) and (14).
Theorem 1. Assume that (k)) 2 1 for all l [d]. For all & [2] and vk [d], have (*))) (k).
Proof of Theorem I. We prove (*k) (k) by induction. For any j [C]. assume that w (k) at iteration The growth of ws along us given by
(mn w/ vk) =
T/t V w RN (w)) 0%
Σ Σ
2m (i,5) [a ( (ww), xm.,)) (Anss ux) + (An,i, o' ((w0), (n.))) ( 0x7]
iE(P
(E.38)
Given n E [N]. suppose Jan,,, such that a, contains vg. By definition, this condition holds for N many training samples. Then, Lemma 3.2 implics that
(i..)) [o ((1)) .m.)) (An wx) + (An,s. w!) of ((w5) (I m,
VE(PI
(E.39)
=0 ((w), 2n, (Aniin 2k) + (Anin w/} o' ((wf, (Inso ok).
Otherwise, given n [N], let an,i. == when there no an,t such that wnss contains uy with slight abuse of notation. Using these notations, we can write
(w ++1 wjj mx)
21)
Σ ((w)), (Anin va) + + (An,in. w/) of ((w0 annd.)) (o Dk)
RE[N]
(E.40)
It remains to bound the terms in Equation (E.40). We first look at (Aniin Ux))
(((
(Aniin Ux)) = Drndn
Σ
((unty pnii.)) un uk
('E)m((n)
(Dniin ok)
Σ o ((ro) enjr ok
(E.41)
('e),(im)
= (Panin 1h))
E
(or (aas)))))))))))) wx)
(t.)
19

---

Published as a conference paper at ICLR 2026
Apparently, both sides of Equation (E.41) would take zero if 2n.i, O. We only need to consider the case when In,in # O. Since Tníi, contains Uk. for all j E (n()n), we have
a ((wfr. )nijn) I (wf.:In..))
(wyy, Uz) + (wo Dn.inn -vE)
(B.42)
I Wyy Ub + 12
S (1 + (2E
Lemma 3.2 implies that Equation (E.42) also hold for a (Lwf) kn,in)) since j E (k)). Combining Equations (E.41) and (E.42), we have
(An,i, Uy) 2 &
2c)
E
/ 2 1 (1 f1) (E.43)
(I)).
Since we assume (k) 2 1 for all features, we have
(ta( S (02 1)d.
(E.44)
Next, we turn to (An,s wf). Similarly, we have
( Enni Σ
((omi
(Ani, 20,)
((wsy,an.)) w/ voj
J'ES..
= (wnnii w!i)
Σ a ((w5,, (n.i)) wjj, 203
V'ESo,
(E.45)
(an,ir w/i)
Σ ((w',, w/ w0;)
J'ES.,
2 (woj, vx) - 12221112 (1 +1++0 /28)
||00f-11 1 ||110001]
J'ES..
2 looln (a- -2= -11++1 22 (1og
Finally, we derive the lower bound of (un'.
pn.in) by (10; ok) (ur Bn,dn vk)
(wh) Fnn.nn)
2 ww) Un - v 2s wjyl
(E.46)
2 (c1 V2E
which also implies that a ((w)) an.i)))) 2 (cI 22e)looJ and o' ((of, = 1. Combining all above, have
1
(ws+1 - w/. Uk)
21/1
Σ σ (n.)))))))))))) (An,a Ux) + (Aniin w))) o' (uw), andaa) (Inia ok)
xEIN]
(E.47)
2 N, (c1 22e)looSi (1 (1 (1- (22)
+ (1 N) (a1 V2e (1+
(s, ((.)
which implies that
(w ,ux) 2 coosn (1+2c N3
(2ē,E (1+2ci (1 (2))(() ())tof())) - 1)d)
+
(R.48)
20

---

Published as a conference paper at ICLR 2026
As discussed in Appendix D,
2c tmp BR 22 - (1 ( + (t)off (22-))d) _
(B.49)
is not a tight bound. Under mild conditions the choice of the parameters, we can obtain
(w=tt vx> 2 (L+ 1)c/0///
(E.50)
as expected.
Next, for all k * k, from Definition 3.1 we know that
(w). vy S to
(E.51)
For all E [C], the growth of wo along ow is given by
(wf ++1 - w
Σ Σ 2m (1, (nn (Anii Uw + (Am,i 1o)))' (uo), (n,)) (an,,s mw)] (E.52)
nE[N]KE|P]
Note that the evaluation of 8, is independent of & (or k'). Therefore, similar to Equations (E.39) and (E.40), we have
Σ (i,i,) (o ((wf, an.i)) (An,i, vw) + (An,i, w/) o' (Kwo, an.i)) (In,, Meei EE[P]
(E.53)
=a a ((w)), Fmiūn)) (Aninn me) + Anin wz} of ((wo), Rnina)) (Kniin vk)
and
(w) 41 w0j. Uk)) = 21tt Σ a ((n)). (Anin, Uw) + (Amin wgy of ((w0) Fn (ar ow) nE[N]
(E.54)
In Equations (E.53) and (E.54), In,i. refers to the patch that contain AK Ei E [P] such that Øn, contains Ug æn.i,. otherwise (ef. Equation (E.39)). It remains upper-bound the terms in Equation (E.54)
(( Dania
A Aniin Uk. =
Σ a ((nn), wnj
uu
(iniin Ukr)
Σ
Lniba) wy ow wt (nwia) uw)
(E.55)
('EESn
(Eniin Ukd)
Σ
(o
Uni
((En((t))
in which
(In,s., vx) = (Xn,in Ub. ok. + (0k, vk (In,in - Uk vkn) S 22.
(E.56)
By definition, we have (wy,, vwr) 2 0 for all j' E (,n) Therefore, we can bound Equation (E.55) simply by
(Anjin Uw) S V2e. We also need find an upper bound for (Aniin w Similar previous analyses, have
(E.57)
(Aniin un/) = (Knn.i. un/)
Σ
(a ((urt, mntyen5)
j' (i.).
(E.58)
< Stoof 22too toofn (V2etoofa + tooff) -18)
Combine Equations (E.54) to (E.58), finally get
1 ++1 -w0,, vw.) <N BA (2e + + 220)
(w
2ny:
+ 22 Br (toofr + Vetoofn + ttoof (V2stoofa + toof.) (s,( (F.59)
21

---

Published as a conference paper at ICLR 2026
which can be further simplified to (wf+1. v 0.4N Bo (fn- (6e V2E + V2stoofi - 1)d) f. (toofh (o2 1)d + v2s) (E.60)
As discussed in Appendix D, obtain (w]+1 uw S (t + 1)
(E.61)
by setting parameters. Equations (E.48) and (E.60) together prove that wy E (k).
Remark E.1 dever-paramettrrzeed model is necessary). The "perfect reconstruction" given by Equa- tion (3) could easily converge to local minima. According to the proof of Theorem 1, the member of each S(K) is decided by random initialization. It is possible that multiple kernels are included in one Skk). If C = d, then the kernels might not be able to learn all of the features. Therefore, to ensure least with high probability) that every feature with 3x * 0 extracted by least one of the kernels, we let C > d in this paper. For simplicity, we let C = C2d for some constant C2 > 0.
Corollary 3.1. Consider two features 2k, and Uk, with 8k, > Sk.. For all E {1,2}. let w/, be a kernel in the cone of Uk.. Then, the bounds (cf. Equations (F.48) and (E.60) for (Wi,, vk,) (wn, (w) (for all k' + k1) is larger than those for (Wiz, mba) {w]j 1xx) (for all k' * k2).
Theorem 3. For 0 E (0, 1), Vj E [C]. and Ak E [d]. if (wz. 2k) (1 0) ||w0|| then
(w.j v.)
|
m20
(E.62)
0
hold with probability at least 20 exp( (sp(-11(202))
Theorem 3 can be viewed as the direct corollary of the geometrical inequalities in high-dimensional spaces (e.g., Lemma F.4). The proof of Theorem omitted here.
F TECHNICAL LEMMAS
The following two lemmas bound the upper and tail probability of i.idd random variables.
Lemma F.1 (Upper tail). Let X1, X2, ... XL be i.i.d. random variables upper bounded by M.
Denote X Lieg. Then we have
+ exp (- 666 E(X?) + 2MA
312
P 2 E
E
(F.1)
for all X >>0 Lemma F.2 (Lower tail). Let X1, Xx be i.i.d. random variables such that X; - M for all E (C). Denote X = Eiec. Then we have
312
P PIX A exp
(F22
66,, 2MA
for all À
The proof's of Lemmas F.I and F22 can be found in various textbooks, which is omitted here. The following corollary is useful in Appendix E. Corollary F.1. Let X1, X2, ... ,Xo be ii.d. Bernoulli variables such that P = 1] = and
P(X, = = 1 for all E [C]. Let 0 < pr D2 be constants. If p E (pi, P2), then we have
X, < Cp 1 s exp
12
P
(-2Cp2
(F33
2Cp2
SE(C)
and
312
P
Σ
X; 2 Lpp + )
I Sexp
(=6Cp2 + 2A
(F.4
iE|C)
for all
X
22

---

Published as a conference paper at ICLR 2026
Lemma F.3 (Concentration inequality for chi-squared random variables).
P [Ixà -d d 2 Xd] - 2e A2d//
(F.5)
The proof of this lemma can be found in Example 2.11 of Wainwright (2019), which is omitted here.
The following lemma characterizes the concentration of probability d-dimensional sphere.
Lemma F.4. Let X := (X1, X Xaa) be random variable drawn from the uniform distribu- fion on { ||||| 1} C Rd. For all O, the probability
P X
(F.6)
The proof of this lemma can be found the course website of Guruswami (2012).
G EXPERIMENTS DETAILS
This section provides more details on the experiments and some auxiliary results.
G.1 IMPLEMENTATION DETAILS
Both of the experiments conducted Ubuntu 64-bit657 Linux workstation, having 10- core Intel Xeon Silver CPU (2.20 GHz) and 4 Nvidia GeForce RTX658 2080 Ti GPUs with 11GB graphics memory.
G.1.1 SYNTHESIZED EXPERIMENT'
All networks initialized with Gaussian initialization and being trained for 1000 epochs to ensure convergence. We use stochastic gradient descent (SGD) with momentum 0.9 and weight decay 0.001. The model is defined in Section 2. The following algorithm specify the generating process of the normal data in the synthesized experiments.
Algorithm Generate normal data
Require: Sample size N Z+, features V Vnor U Vauaa total patches P, noise size
1: for [N] do
2: Vn,aux randomly sample P 2 features from Vouz without replacement
3: Vn + Vn.urr U Vnor
4: for E do
5:
Uk. < randomly sampled from V,
6:
randomly sampled from (((0 s)
7:
2n <- Uk; +
8:
end for
9:
for UK E V do
10: Bg t (total occurrence of vy in [æN])
11: end for
12:
end for
return [2N], 8
G.1.2 VISUALIZATION OF THE KERNELS
All networks are initialized with Gaussian initialization and trained for 100 epochs to ensure conver- gence. We use stochastic gradient descent (SGD) with momentum 0.9 and weight decay 0.001. The batch size is 32, and the learning rate is 0.001. To obtain images-like like kernels, we visualize 24 kernels from the first layer.
Dataset and network structure To be consistent with the theoretical analysis, Autoen- coders to reconstruct the images from MNIST and CIFAR-10. We visualize the kernels in the first layer of the autoencoder.
23

---

Published as a conference paper at ICLR 2026
G.2 AUXILIARY EXPERIMENTS
This section presents some auxiliary experiments on the synthesized data to further validate our theoretical analysis in Section 3.
G.2.1 HYPER-PARAMETE PARAMETERS STUDY
We mainly discuss the effect of pareaettreeiztioonnn and the number of replaced patches. In Sec- tion 2.2, we require the network to be over-parameterinn In Figure 4, we let C = int 1.2d) and ob- tain desirable results. Here, experiment on the mnder-parameteriiee setting with C int(O.8P) in comparison. The results are presented in Figure G.1. It can be observed that all the reconstruction errors increase significantly. Note that the scales of the y-axis are 0.02 in Figure G.I and 0 to 0.014 in Figure 4. The comparative experiment implies that slight the model better reconstruct all types of inputs.
could help
15/19
3200
342.0
3300
3011
02/15
33817
33215
4813
38811
38013
Testing the under-parameterized setting with C = int(P.8P). By comparing to Figure 4, this set of experimental results implies that slight over-parameterization (i.e., C (nt(1.))))olll help the model better reconstruct all types of inputs. Here, we primarily compare the magnitudes of P and C. Specifically, the scenario where the ratio C/P > I is referred to as over-parameterized, while the converse (i.e., C/P < 1) is defined ander-pprameterized.
Figure G.1:
In Section 3.2, we study the case when only one patch is replaced by the anomaly patch and claim that our analysis can be naturally extended the case when multiple patches are replaced. We con- sider the number of replaced patch NRP = int(c, + 2) with e4 E {0, 0. 0.2}. As demonstrated in G.2, the gaps between the reconstruction errors of the normal data and the anomalies become significantly wider when more patches are replaced, which is consistent with our analysis.
1515
3805
3815
5213
23213
2012
1410
34/00
82113
3200
332/2
445A
25
3800
(a) C4 0
(b) 0.2
(c) C4 0.4
Figure G.2: Testing different numbers of replaced patches. The parameter C2 is used to charac- terize the number of patches replaced in anomalies, where a larger Ca indicates greater number of replaced patches. The gaps between the reconstruction errors of the normal data and the anomalies become significantly wider when more patches are replaced, which is consistent with our analysis.
24

---

Published as a conference paper at ICLR 2026
G.2.2 THE GROWTH THE NORM OF THE KERNELS
This section reports the growth of the norm of the kernels during training. According to Lemma 3.1, the growth rates of the norm of the kernels is upper bounded by Equations (11) and (12). As side product of the synthesized experiments in Appendix G, we record the norm the kernels during training. The results presented in Figures G.3 G.8. In all the experiments, the norm of the kernels increases almost linearly, which is consistent with our analysis in Equations (11) and (12). Additionally, kernels with larger initial values grow faster. It can be observed that as the number of kernels and patches increases (i.c., when the task becomes more complex), the differences between kernels also become more pronounced. The discussion of the results can be found in Remark G.1
G.2.3 INNER PRODUCT BETWEEN THE KERNELS AND THE FEATURES
A key observation underlying our proposed cone set intuition is that kernels tend converge toward features terms of direction, which can be measured by the (normalized) inner product between the kernels and the features. We report the inner products between the kernels and the features during training in с The discussion of the results can be found in Remark G.1
Protaat
20 didh CHOA MMP-13
6411
0111
CAT
CAR
1.0
671
130
0111
4.07
600
4311
CHOE WwPha3 bnadee
2009
20m
CHH
6.00
KITH
3180
45%
JAN
Cam
41411
4141
3009
4101
41111
Figure G.3: Recorded kernels' norm (the below sub-figure) and inner products between the kernels and the features (the above sub-figure) during training. The controling variables are P = 20, d 24, and C 24. In both sub-figures, the æ-axis represents the kernel index j (j e [C]), and the y-axis denotes the training epoch. For better visualization, we have scaled the norm of the kernels by factor of 10 in the below sub-figure.
Remark G.1 (Discussions on the results in Appendices G.2.2 and G.2.3). An intuitive observation is that the color of each column (representing single kernel) becomes progressively warmer from top to bottom in both sub-figures, which implies that both the norm and the inner product increase evidence
with the training epoch. In particular, the growth of the inner product
25

---

Published as a conference paper at ICLR 2026
Figure G.4: The control variables are / = 60, d 72, and C' 72.
for our proposed cone set intuition, i.e. the kernels tend to converge toward features in terms of direction.
The main difference between Figures G.3 G.8 lies in the controling variables P. d, and C, which depict different levels of task complexity. As the dimension increases, randomly initialized kernels become more difficult to be absorbed (reflected in the bound provided by Theorem 2). Figures G.5 G.8 provide more experimental results. The similar captions are omitted for brevity.
Figure G.5: The control variables are P P-28 28, d = 33, and C = 33.
26

---

Published as a conference paper at ICLR 2026
Figure G.6: The control variables are P P336,
-3, d = 43, and C = 43.
Figure G.7: The control variables are P = 44, d 52, and = 52.
Figure G.8: The control variables are P 52, d 62, and 62.
27

---

Published as a conference paper at ICLR 2026
G.3 RAW RESULTS
This section reports the raw results of the synthesized experiments. The control variables (i.e., E, d. and C) noted in the captions of the tables.
Table 0.01, (nt(l. int (0.8P)
(d,
NE + Std.
SE + Std.
NSE + Std.
(22, 20, 16)
0.0120 = 0.0009
0.0169 = 0.0011
0.0197 0.0014
(26, 24, 19)
0.0102 + 0.0008
0.0171 = 0.0012
0.0158 = 0.0011
(30, 22)
0.0132 + 0.0007
0.0157 + 0.0009
0.0172 0.0010
22,
0.0131 + 0.0006
0.0153 + 0.0007
0.0167 + 0.0008
36,
0.0103 = 0.0005
0.0117 = 0.0005
0.0134 = 0.0006
40,
0.0086 + 0.0003
0.0110 0.0005
0.0116 = 0.0005
44,
0.0095 + 0.0003
0.0103 = 0.0004
0.0115 = 0..004
48,
0.0091 + 0.0003
0.0112 = 0.0004
157,52,4 41)
0.0069 = 0.0002
0.0088 = 0.0003
0.0090 = 0.0003
(61, 56,44)
0.0064 + 0.0001
0.0075 + 0.0002
0.0081
0.0002
(66, 48)
0.0069 + 0.0001
0.0066 + 0.0001
0.0079 0.0002
Table 2: 6 0.01, d (1.2P) (0.8P)
(d,
NE + Std.
SE + Std.
NSE + Std.
(24, 20, 16)
0.0135 + 0.0011
0.0176 + 0.0014
0.0192 + 0.0015
(28, 24, 19)
0.0112 0.0008
0.0143 + 0.0011
0.0151 0.0011
(33, 22)
0.0098 + 0.0007
0.0133 = 0.0009
0.0141 = 0.0009
(38,32,25)
0.0093 + 0.0004
0.0114 0.0005
0.0126 = 0.0006
(43,36,28
0.0084 + 0.0004
0.0098 = 0.0004
0.0114 = 0.0005
448)40,32)
0.0082 + 0.0003
= 0 0.0003
0.0105 = 0.0004
(52, 44,35)
0.0088 + 0.0003
0.0108 + 0.0004
0.0114 0.0004
48,38888)
0.0071 + 0.0002
0.0092 + 0.0003
0.0094 + 0.0003
(62, 52)
0.0076 = 0.0002
0.0091 = 0.0002
0.0094 0.0002
(67, 44)
0.0070 = 0.0002
0.0082 = 0.0002
0.0086 = 0.0002
(72, 60,
0.0070 = 0.0002
0.0079 + 0.0002
0.0084 0.0002
Table 3: 5 = 0.01, d imt.5) (0.PP
(d, P,C)
NE Std.
SE + Std.
NSE + Std.
(30, 20, 16)
0.0144 + 0.0009
0.0193 + 0.0013
0.0193 0.0012
(36, 24,119
0.0142 + 0.0007
0.0151 = 0.0008
0.0169 = 0.0009
(42, 28,22
0.0104 + 0.0004
0.0127 + 0.0005
0.0136 + 0.0005
(48,32,25)
0.0093 + 0.0003
0.0110 + 0.0004
0.0121 0.0004
28,
0.0081 + 0.0003
0.0110 + 0.0004
0.0109 + 0.0004
(60,40,32
0.0073 + 0.0002
0.0094 + 0.0003
0.0095
0.0003
34,
0.0078 + 0.0002
0.0084 = 0.0002
0.0092 = 0.0002
(72, 48)
0.0071 = 0.0002
0.0081 = 0.0002
0.0087
0.0002
(78, 52,41)
0.0066 + 0.0002
0.0080 + 0.0002
0.0081 + 0.0002
(84, 56,44
0.0059 + 0.0001
0.0071 + 0.0002
0.0074
0.0002
(90, 60,48)
0.0056 + 0.0001
0.0067 + 0.0001
0.0069 + 0.0001
28

---

Published as a conference paper at ICLR 2026
Table 4: - C C= (1.(
P.C)
NE + Std.
SE + Std.
NSE + Std.
(24, 20, 28)
0.0084 +: 0.0006
0.0128 + 0.0011
0.0140 = 0.0011
(28, 33)
0.0073 = 0.0004
0.0111 0.0006
0.0124 = 0.0008
(33, 28,39)
0.0044 = 0.0003
0.0049 = 0.0003
0.0089 0.0005
(38,32, 45)
0.0040 = 0.0002
0.0058 = 0.0003
0.0077 = 0.0004
(43, 51,
0.0043 + 0.0002
0.0057 + 0.0002
0.0077 0.0003
40,
0.0042 + 0.0001
0.0051 + 0.0002
0.0072 + 0.0003
(52,44,62)
0.0050 + 0.0002
0.0053 = 0.0002
0.0074
0.0003
0.0048 + 0.0002
0.0058 0.0002
0.0070 = 0.0002
(62, 54)
0.0049 = 0.0001
0.0059 0.0002
0.0066
0.0002
(67, 50)
0.0044 = 0.0001
0.0051 = 0.0001
0.0061 = 0.0002
(72, 66,
0.0036 + 0.0001
0.0044 + 0.0001
0.0053
0.0001
Table 5: E = 0.01, int.ttnnn11.)) C == (1.(()
(d, P,C)
NE Std.
SE + Std.
NSE Std.
(30, 20,36) 36)
0.0030 = 0.0002
0.0046 = 0.0004
0.0086 = 0.0006
(36, 43)
0.0074 + 0.0004
0.0083 + 0.0005
0.0107 + 0.0006
(42, 20)
0.0039 + 0.0001
0.0041 + 0.0001
0.0069 + 0.0003
(48,32,57
0.0035 = 0.0001
0.0049 = 0.0002
0.0067 = 0.0003
(54,36,64
0.0025 = 0.0001
0.0032 + 0.0001
0.0052 + 0.0002
70)
0.0029 = 0.0001
0.0036 = 0.0001
0.0052 = 0.0001
(66,44,79)
0.0034 = 0.0001
0.0040 + 0.0001
0.0053 + 0.0001
(72, 46)
0.0030 + 0.0001
0.0039 + 0.0001
0.0049 0.0001
(78, 53,
0.0039 + 0.0001
0.0047 + 0.0001
0.0053 + 0.0001
(84, 56, 100)
0.0041 + 0.0001
0.0053 + 0.0001
0.0057 0.0001
(90, 60, 108)
0.0031 = 0.0001
0.0039 = 0.0001
0.0044 + 0.0001
Table 6: 0.1, int((2), = in.((.2d)
P,C)
NE + Std.
SE + Std.
NSE Std.
(40, 20, 48)
0.0049 + 0.0002
0.0060 0.0002
0.0091 + 0.0004
(48, 24,
0.0046 + 0.0001
0.0061 = 0.0002
0.0077 + 0.0003
(56, 67)
0.0042 = 0.0001
0.0045 0.0001
0.0062 + 0.0002
(64,32,70
0.0033 = 0.0001
0.0046 = 0.0001
0.0056 + 0.0002
86))
0.0027 - + 0.0001
0.0037 + 0.0001
0.0048 + 0.0001
(80,40,96)
0.0042 0.0001
0.0051 + 0.0001
0.0057 + 0.0001
(8,
1.0000000000
0.0033
0.0000
0.0040 0.0001
48, 115)
0.0034 0.0001
0.0045 - 0.0001
0.0048 0.0001
(104, 52, 124)
0.0036 = 0.0000
10044=00000
0.0048 - 0.0001
(112, 56, 134)
0.0033 + 0.0000
0.0041
0.0000
0.0045 + 0.0001
(120, 60, 144)
0.0038 + 0.0000
0.0047 = 0.0001
0.0049 + 0.0001
29

---

Published as a conference paper at ICLR 2026
Table 7: E = 0.01, d int( int((.2d)
P.)
NE Std.
SE = Std.
NSE Std.
(40, 20,48) 48)
0.0047 + 0.0002
0.0057 0.0002
0.0090 + 0.0004
(48, 27)
0.0041 + 0.0001
0.0055 0.0002
0.0072 + 0.0003
(56, 28, 67)
0.0039 + 0.0001
0.0039 = 0.0001
0.0058 + 0.0002
(64,32,
0.0030 = 0.0001
0.0039
0.0001
0.0053 + 0.0001
(72, 36,
0.0026 = 0.0001
0.0036 + 0.0001
0.0047 + 0.0001
(80,40,90
0.0041 = 0.0001
0.0050
0.0001
0.0056 + 0.0001
48,
0.0026 + 0.0000
0.0034 + 0.0000
0.0042 + 0.0001
196,48,115
0.0034 0.0001
0.0044 0.0001
0.0049 0.0001
12,))
0.0035 + 0.0000
0.0043 + 0.0001
0.0047 + 0.0001
(112, 56, 134)
0.0032 = 0.0000
0.0039 + 0.0000
0.0044 0.0001
(120, 60, 144)
0.0037 = 0.0000
0.0046 0.0001
0.0049 + 0.0001
Table 8: E 0.001, d- int(2P)) C in.(1.2d)
P,C)
NE Std.
SE = Std.
NSE Std.
(40, 20, 48)
0.0047 + 0.0002
0.0057 0.0002
0.0090 + 0.0004
(48, 57)
0.0041 = 0.0001
0.0055 = 0.0002
0.0072 + 0.0003
(56, 28, 67)
0.0039 + 0.0001
0.0039
0.0001
0.0058 + 0.0002
(64,32,
0.0030 + 0.0001
0.0039
0.0001
0.0053 + 0.0001
(72, 772,36861
0.0026 + 0.0001
0.0036 - 0.0001
0.0047 + 0.0001
(80,40,96
0.0041 0.0001
0.0050
0.0001
0.0056 + 0.0001
44,
0.0026 + 0.0000
0.0034 + 0.0000
0.0042 + 0.0001
(96,48,111) 115)
0.0034 = 0.0001
0.0044 0.0001
0.0049 + 0.0001
(104, 204,52,12 124)
0.0035 + 0.0000
0.0043 + 0.0001
0.0047 0.0001
(112, 56, 134)
0.0032 + 0.0000
0.0039 + 0.0000
0.0044 4 0.0001
(120, 60, 144)
0.0037 + 0.0000
0.0046 + 0.0001
0.0049 + 0.0001
30


Reviews:
[{"review_id": "I4g0y7Mbnd", "reviewer": "Reviewer_HVHx", "summary": "This paper investigates the behavior of the convolution kernels in reconstruction-based anomaly detection (RBAD). To derive theoretical results, this paper assumes that image data can be decomposed into normal/auxiliary patch features and the model is a two-layer CNN. Under these assumptions, this paper reveals that the kernels are absorbed into the cone set oriented toward the normal feature. The paper claims that it can explain the reconstruction error of anomaly data becomes large and verifies theoretical results through synthetic/real-world data.", "questions": "Could you read Weakness and answer the questions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written except for the section of experiment.\n2. This seems the first theoretical explanation for reconstruction-based anomaly detection.\n3. The absorption into the cone-set appears to be an interesting and reasonable.\n4. Several theoretical results might inspire researchers to propose a new anomaly detection method or to improve existing one.", "weaknesses": "1. Experiments need more explanation and discussions. Section 4 does have the explanation about Figure 4.\n\n2. Empirical evaluations seem insufficient. Which theoretical results do Fig. 4 and Fig. 5 verify?\nI suspect there some theoretical results might be verifiable. For example, since Lemma 3.1 indicates that the norm of kernel increases according to the training step, \nit can be verifiable by plotting the norm vs training steps. I think the cone set is also verifiable by investigating values of the inner product in Def.3.1.\nIf not, why is it difficult and how do Fig. 4 and Fig. 5 support theoretical results?\n\n3. Since fully-connected layers are also commonly used for anomaly detection in AE, a comparison with them might be interesting. \nFor a fully-connected layer, if the input vector contains a normal feature linearly independent of the noise component, intuitively one would expect the weight matrix to have singular vectors in the direction of the normal feature. This would result in singular values of zero for the anomaly feature, leading to large reconstruction errors.\nConsidering that convolution is a special case of a linear layer [a,b], a similar argument holds. Under such assumptions, it seems likely that the frequency components of an image would be formulated as the normal feature. The kernel visualization in Fig. 5 might also support such explanation as each kernel possesses a specific spectrum.\nDoes the proposed theory have any advantages or more reasonable insights over such approach?\n\n[a] Tsuzuku, Y., and Sato, I.  \"On the structural sensitivity of deep convolutional networks to the directions of fourier basis functions\". CVPR2019\n\n[b] Sedghi, H., and et al. \"The singular values of convolutional layers\". ICLR2018\n\n4. The contribution can be a bit weak if theoretical findings can only explalin the reactions to semantic anomalies or the magnitude of reconstruction errors for anomalies.\nThis paper would be strengthened if results can suggest directions for reconstruction-based anomaly detection researches or methods for improvement.\nCan theoretical results suggest such directions or methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ax1gTZzViy", "reviewer": "Reviewer_Po5J", "summary": "This paper gives a theory for why reconstruction-based anomaly detection (RBAD) succeeds on normal data yet struggles on anomalies by analyzing a two-layer convolutional autoencoder with max pooling.\nIt introduces cone sets to show that training aligns kernels to frequent normal features.\nThis alignment yields weak activations and large reconstruction errors for non-semantic anomalies,\nwhile yields small reconstruction errors for semantic anomalies containing learned features.\nThe proposed theory is validated on both synthetic and real datasets.", "questions": "- While this theory analyzes autoencoders, I am curious what role it would play for variational autoencoders (VAEs).\nAs generative models, VAEs assign high likelihood to normal data and low likelihood to anomalies.\nThat is, they reconstruct normal data well and fail to reconstruct anomalies.\nHowever, as noted in [1],\nVAEs can sometimes assign higher likelihood to anomaly data than to normal data.\nSince a VAE can be viewed as a regularized autoencoder,\nI wonder whether this theory applies.\nCould this theory help explain that phenomenon?\n- Could this theory also be effective for more complex architectures, such as ResNet-based autoencoders?\n\n[1] Nalisnick, Eric, et al. \"Do deep generative models know what they don't know?.\" arXiv preprint arXiv:1810.09136 (2018).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Although reconstruction-error–based anomaly detection is widely used, there has been little theoretical analysis of why normal data are well reconstructed while anomaly data are not. This paper provides a theoretical explanation for this.\n- The paper is well written and uses figures effectively to explain complex theory, making it very easy to follow.", "weaknesses": "Please refer to the Questions section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "56S2QFeAH4", "reviewer": "Reviewer_Vbxc", "summary": "This paper proposes a new anomaly detection method, which provides a theoretical analysis of reconstruction-based\nanomaly detection (RBAD). It constructs a two-layer convolutional autoencoder to reconstruct data and proves reconstructing normal data is easier than anomalies.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- This paper proposes an autoencoder model to reconstruct the normal data and anomalies, which trains on normal data. And it shows that the reconstruction error of normal data is smaller than that of anomalies.\n- Theoretical analysis support the observation and conclusion of this paper.\n- Experimental results on synthesis dataset validate the theoretical findings.", "weaknesses": "- How to use the non-semantic anomaly and semantic anomaly during training phase.\n-The organization of section 3 is a bit chaotic. It is unclear to me how this method works.\n- I am confused with the experimental results on real data. I don't know how to observe Figure 5. Please provide more explanations. \n- Why not detect anomalies directly based on the real datasets. Then we can see quantitative results of Acc and F1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3wcFTtCv9J", "reviewer": "Reviewer_2q5h", "summary": "The paper addresses the insufficient theoretical understanding of reconstruction-based anomaly detection (RBAD) and proposes a new theoretical framework introducing the concept of a “cone set” to describe the dynamics of feature learning during the training of convolutional autoencoders (CAEs). Through rigorous derivations, the authors demonstrate how convolution kernel parameters are gradually attracted to the “cone set” and align with corresponding true feature directions. Overall, the theoretical contribution lies in establishing a formal analytical foundation for RBAD, filling a major gap in the literature where theoretical explanations for autoencoder-based anomaly detection were largely absent.", "questions": "1. Could the theoretical assumptions be relaxed, or could the authors discuss whether similar principles hold for other generative models (e.g., diffusion models, GANs)? Additionally, if multiple patches are replaced (non-local anomalies), would the conclusions still hold?\n2. Given the theoretical weakness of RBAD on semantic anomalies, could auxiliary tasks or regularization be introduced to enforce lower reconstruction ability for non-primary features?\n3. The experimental section should be expanded.\n4. In the revised version, clarify the relationship between this paper’s theoretical focus and existing empirical advances—emphasize that the goal is to complement, not compete with, SOTA detection methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "1. The paper’s greatest strength lies in constructing a systematic theoretical framework for RBAD. While previous works achieved strong empirical performance, they lacked a clear explanation for why reconstruction error distinguishes anomalies. Through the cone set and feature absorption mechanism, this paper is the first to explain—via gradient dynamics—why AEs tend to retain only normal features while poorly reconstructing anomalies. The discussion in VPDM [1] of the “identical shortcut” phenomenon indirectly supports this view: if a model relies excessively on raw input rather than stable feature extraction, it may reconstruct anomalies as well, leading to detection failure. The cone set theory precisely explains that models which genuinely learn features, rather than memorizing inputs, naturally exhibit poor reconstruction for anomalies.\n2. The theoretical formulation is highly structured, with well-defined assumptions, lemmas, and proofs. The use of Hilbert space representations for patches, smooth ReLU activations for differentiability, and global max pooling to avoid gradient entanglement all demonstrate careful mathematical design. \n3. The paper’s theoretical results align with long-standing community observations. For instance, it formally explains why RBAD struggles with semantic anomalies (since they may contain normal local features) and why mild overparameterization helps avoid suboptimal reconstructions. These insights transform empirical heuristics into theoretically grounded knowledge, strengthening both credibility and interpretability.\n\n[1] Li Y, Feng Y, Chen B, et al. Vague Prototype-Oriented Diffusion Model for Multi-Class Anomaly Detection", "weaknesses": "1. The theoretical analysis relies on several idealized assumptions. For example, the data are assumed to consist of P patches, each containing one dominant feature plus random noise. Real images are more complex, patches may not be independent and could contain multiple mixed features. Similarly, the definition of anomalies as “patch replacements” may not cover global distribution shifts or complex anomaly patterns. \n2. The paper only analyzes a two-layer convolutional autoencoder (single hidden convolutional layer + pooling). In practice, anomaly detection models are often deeper and structurally richer. The conclusions may not directly generalize to these more complex architectures, and care must be taken when extrapolating.\n3. The main limitation lies in the narrow scope of empirical validation. Although the paper includes some experiments, they are limited compared to the complexity of anomaly detection tasks. The synthetic experiments, while rigorous, remain toy settings far from real-world conditions. \n4. The paper barely discusses or compares with current state-of-the-art (SOTA) anomaly detection methods. While pure theory need not compete in performance, the omission creates a sense of detachment: the theory explains a relatively basic method, while the field’s focus has partially shifted toward more advanced ones. This gap could reduce confidence in the theory’s practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)