Exploring Invariance in Images through One-way Wave Equations
Authors: Yinpeng Chen
Dongdong Chen
Xiyang Dai
Mengchen Liu
Yinan Feng
Youzuo Lin
Lu Yuan
Zicheng Liu
Venue: ICML 2025 poster (2025)
Keywords: Mathematical invariance in images
One-way wave equation
Auto-regression
Primary area: applications->computer_vision

Abstract: In this paper, we empirically demonstrate that natural images can be reconstructed with high fidelity from compressed representations using a simple first-order norm-plus-linear autoregressive (FINOLA) process—without relying on explicit positional information. Through systematic analysis, we observe that the learned coefficient matrices ($\mathbf{A}$ and $\mathbf{B}$) in FINOLA are typically invertible, and their product, $\mathbf{AB}^{-1}$, is diagonalizable across training runs. This structure enables a striking interpretation: FINOLA’s latent dynamics resemble a system of one-way wave equations evolving in a compressed latent space. Under this framework, each image corresponds to a unique solution of these equations. This offers a new perspective on image invariance, suggesting that the underlying structure of images may be governed by simple, invariant dynamic laws. Our findings shed light on a novel avenue for understanding and modeling visual data through the lens of latent-space dynamics and wave propagation.

Exploring Invariance in Images through One-way Wave Equations
Yinpeng Chen
Dongdong Chen
Xiyang Dai Mengchen Liu 13 Zicheng Liu
Yinan Feng*
Youzuo Lin
Lu Yuan"
Abstract
one-way equations
In this paper, we empirically demonstrate that natural images can be reconstructed with high fi- delity from compressed representations using a simple first-order cattm--pus--inneee autoregressive (FINOLA) srsresss-wthhou relying on explicit positional information. Through systematic anal- ysis, we observe that the learned coefficient matri- ces (A and B) in FINOLA are typically invertible, and their product, AB is diagonalizable across training runs. This structure enables striking in- terpretation: FINOLA's '' latent dynamics resemble system of one-way wave equations evolving in a compressed latent space. Under this framework, each image corresponds to a unique solution of these equations. This offers new perspective on image invariance, suggesting that the underlying structure of images may be governed by simple, invariant dynamic laws. Our findings shed light on a novel avenue for understanding and model- ing visual data through the lens of latent dynamics and wave propagation.
0c oc dy
0x
A
diag(A
solution
W
H
conv
encoder
{(x,))
initial condition
upsample
FINOLA decoder
solution: { = 1/1 4 ... 1/ s.t.
H. M. H, p.. H1H+²=
Figure 1: Exploring invariance through one-way wave equations. Our empirical findings suggest that images may share set of one-way wave equations dz portation equations). Each image corresponds (to good approximation) to unique solution with an initial condition W. <(" 2 derived from the original image. The solution C(x, y) is a feature map (with resolutions of } or or full resolution the original image) facilitates image recon- struction using few upsampling and convolutional layers. The wave speeds, A1...., AC, are latent and learnable.
2 A ac (or trans-
Dy
often through deep Transformer architectures. Motivated by their success, we ask: can autoregressive modeling be simplified to a first-order process, while still retaining its expressive power?
1. Introduction
Motivation: Autoregressive language models, such as GPT (Radford et al., 2018; 2019; Brown et al., 2020), have achieved remarkable success in Natural Language Process- ing (NLP) by predicting each token based on its preceding context. This paradigm has also influenced Computer Vi- sion, inspiring models like iGPT (Chen et al., 2020a) for unsupervised representation learning, PixeICNN (van den Oord et al.. 2016a; Salimans al., 2017) for autoregressive image generation, and DALL-E (Ramesh et al., 2021) for text-to-image synthesis. These approaches typically rely on capturing high-order dependencies among multiple tokens,
First-order norm+linear autoregression (FINOLA): We propose simplified first-order autoregressive process for image reconstruction, which term FINOLA (First-Order Norm+L Linear Autoregression). Our key insight is that, with appropriate encoding, images can be autoregressed linearly after normalization. As illustrated in Figure 2, the pro- cess begins by encoding an image into a compact vector q E C.C This vector is placed at the center of latent feature map E W i.e., 2 4) q. We then recursively propagate values across the and y axes using: A,# =(x + 1, =(x, v) A y), and A az z(r,y+1) - 2(2, y) B2 y). where A, B E R RC are learnable matrices, and 2 y) is the channel- wise normalized version of 2(x, y): 2(x, v) x(2,8)-z
Work done while working at Microsoft. Google DeepMind "Microsoft 'Meta "University of North Carolina at Chapel Hill *AMD. Correspondence Yinpeng Chen <yin- Lungo@goolle.com:
Mx = //2 Σ. 2, y). NOLA can generate high-resolution feature maps (e.g., 4' 2. or full resolution of the original image). The final image
V - P/C. FI-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
1
is reconstructed using a lightweight decoder composed of a

---

Exploring Invariance in Images through One-way Wave Equations
RUK
FINOLA
f
conv upsample
encoder
d:
FINOIA
encoder
FINOLA
upsample
Share
QN
parameters
decoder
Original
FINOIA
FINOLA decoder
30 HRUM
Figure 3: Multi-path FINOLA:
The input image is en- coded into M vectors q1..... 9M. Then the shared FI- NOLA is applied on each q: to generate feature maps (1 y). which are aggregated (= = Σ, 0,) to pass through upsampling and convolution layers to reconstruct image pixels.
Reconstruction
30
Figure 2: FINOLA for image reconstruction. Each image is firstly encoded into single vector q. Then, FINOLA is applied to q to iteratively generate the feature map (xx y) through a first-order norm+linear autoregression. Finally, few upsampling and convolutional layers are used to recon- struct image pixels. Best viewed in color.
x- and y-axes, respectively. Since AB-1 is diagonalizable, we can express this relation transformed feature space ((x, v) 1y-12(x (x; y) as: A AAyC. In this space, the channels Ça are decorrelated, and each follows sepa- rable form: A = Ay. which can be interpreted as finite-difference approximation of a one-way wave equa- da
8ç* As illustrated in Figure 1, each image tion: 8C. = > ay
corresponds to a unique solution of this system of equations, fully determined by its initial condition the center of the feature map: <( C(W. H key insight:
few upsampling and convolutional layers.
Reconstruction based on local structure alone: The learned coefficient matrices A and B are invariant not only across spatial positions (z, v) within a single image but also across different images. These matrices encode con- sistent local relationship between a feature vector 2(x, y) and its rate of change (z y). enabling FINOLA to re- construct an entire image from a single central vector. Re- markably, this reconstruction relies solely on local prop- agation rules-without requiring explicit position encod- ing or global structure. On ImageNet (Deng al., 2009) (256x256 resolution), FINOLA achieves promising recon- struction performance. With a latent dimension of C = 128, we obtain a PSNR of 23.2 on the validation set. Increasing the dimension to C = 2048 improves the PSNR to 29.1. Compared to traditional encoding/decoding methods under the same latent size, FINOLA outperforms the discrete co- sine transform (DCT), discrete wavelet transform (DWT), and convolutional autoencoders (AEs).
V-lq. This observation leads to
Natural images may share a common system of one-way wave equations in latent space; each image is uniquely characterized by its initial con- dition, from which its full structure can be recon- structed.
FINOLA for self-supervised pre-training: Beyond re- construction, FINOLA can be adapted for self-supervised pre-training. Specifically, applying FINOLA to a single unmasked quadrant block and training it to predict the sur- rounding masked regions yields competitive performance compared established approaches such as MAE (He et al., 2021) and SimMIM (Xie al., 2022) even lightweight architectures like Mobile-Former (Chen et al., 2022). Comparing encoders trained with and without mask- ing reveals a trade-off: while masked prediction reduces reconstruction fidelity, significantly enhances the learned semantic representations. Interestingly, we observe that masking leads an increase in Gaussian curvature on the surfaces of critical feature maps, suggesting that the model develops more abstract and topologically complex represen- tations under masking constraints.
even when using
Empirical properties of the coefficient matrices A and B: Our empirical analysis reveals that the learned FINOLA matrices A and B are typically invertible and that their product, AB is diagonalizable across multiple training runs. That is, AB-1 VAV-1, where A is diagonal matrix and V contains the eigenvectors.
Interpretation via one-way wave equations: Given the in- vertibility of A and B, FINOLA'S dynamics can be reformu- lated as a linear difference equation: A, = AB AB''A,z, where Z z and 4y denote first-order differences along the
Acknowledging theoretical limitations: While our empiri- cal results strongly support the wave equation interpretation of FINOLA, we acknowledge that this perspective currently
2

---

Exploring Invariance in Images through One-way Wave Equations
Horizontal regression first
Regular vs. Parallel autoregression. End-to- end runtime of encoder/decoder for a 256x256 image MacBook Air with Apple M2 CPU is reported.
Feature Resolution
Vertical regression first
Autoregression
average
16x16
64x64
Regular
1.7s
7.7s
Parallel
1.2s
2.6s
Figure 4: Parallel implementation of FINOLA: Horizontal and vertical regressions separated. The top approach performs horizontal regression first, enabling parallel vertical regression. Similarly, the bottom approach starts with vertical regression, enabling parallel horizontal regression. The results of these approaches are averaged, corresponding to the two autoregression paths from the initial position marked by q. Best viewed in color.
lacks a rigorous theoretical foundation. We hope that our findings inspire future work to establish a deeper theoretical understanding of the dynamics uncovered in this study.
to perform predictions in the same manner as for right and down directions. Specifically, prediction toward the left is expressed as z(x 1, v) zax, v) + A 2(r, v). For brevity, we omit A_ and B_ in the rest of the paper.
2. First-order Norm+Linear Autoregression
Finally, image pixels are reconstructed by passing the fea- ture map through upsampling and convolutional layers, as depicted in Figure 2. Remarkably, FINOLA exhibits the ability generate the feature map z high resolutions, including 4 2. or full resolution of the original image. In the most extreme scenario, where the feature map matches the resolution of the original image, merely three 3x3 con- volutional layers are required to generate the image pixels.
In this section, we introduce first-order norm+linear au- toregressive process in the latent space, known as FINOLA, which is able to reconstruct the entire image from sin- position-invariant
gle vector the center. It unveils image-inaarian invariant relationship between the feature values =xx, y) (at any (Ir, y) position for any image) and its spatial rate of changes Z1; y) and (z, y).
and image-ing
The entire FINOLA framework, comprising the encoder, FINOLA, and the subsequent upsamplig/convolutiona lay- ers, can be trained in an end-to-end manner. This is achieved by minimizing the L2 distance between the original and the reconstructed images as the training loss.
FINOLA: FINOLA is a first-order norm+linear autoregres- sive process that generates W H feature map 2(x, y) by predicting each position using only its immediate pre- vious neighbor. As depicted in Figure 2, it places single embedding q (generated by an encoder) at the center, i.e., 2( H = q, and recursively regresses the entire feature 2
Note that the matrices A and B, once learned from data, remain invariant not only across spatial positions (zx, y) per image but also across images. They capture the local relation between the feature values v) and their spatial derivatives (A,z, Ayz).
Position and image invariance:
map using the following equations:
<(x 1,9) = =xx,1) v) + AZ(1.v)
=(x, + 1) = = + B2(x,)) ()) $(x,8) 2(2, 3) Ps
Ox (1)
Interpretation of matrices A and B: Imagine points in the latent feature map connected by "springs", Matrices A and B define the stiffness of these horizontal and vertical springs, while their eigenvectors VA and VB indicate their directions. We confirm A and B full rank and diagonal- izable across multiple training runs. Diagonalizing A and B simplifies this, projecting latent space so each dimen- sion (eigenvector) is an independent spring with stiffness (cigenvalue). The projected horizontal change V '', and vertical change V AA, become scaling operations, with eigenvalues from A and Ag determining scaling
The matrices A and B are learnable with dimensions C x C. {zx, y) is the normalized 2(x, y) over C' channels at posi- tion (z. y): the mean , _ 2x zu (z. v) and the standard deviation x2 = Ex 2* )2 are computed per po- sition (I. 3) over C channels. Due to the normalization, this process is a first-order non-linear process.
Eq. 1 provides a solution to predict towards the right and down (assuming the y axis points down). For predicting towards the left and up (with negative values of offset), we introduce two additional learnable matrices, A_ and B_,
3

---

Exploring Invariance in Images through One-way Wave Equations
strength along each eigenvector. In essence, A and B en- code directional "stretching/ccompresinng factors governing local feature map changes, with eigenvalues representing the strength of these factors.
optimal solution can be attained by relaxing the FINOLA constraint (A, = Aλ, Ay = Bž) through aggregating series of FINOLA solutions:
M
Autoregression can be compu- tationally intensive due to its sequential nature. FINOLA mitigates this by capitalizing on the independence of the x and y axes, enabling parallel execution, significantly boost- ing efficiency. As shown in Figure 4, performing horizontal regression first allows for parallel execution of subsequent vertical regression, and vice versa. In practice, both ap- proaches (horizontal first and vertical first) are combined by averaging their results. The prediction at each position represents the average of the two autoregression paths orig- inating from the initial position, marked as q. Figure 4 (table on the right) demonstrates the superior speed of the parallel implementation, compared to the regular AR setting. It achieves a 30% speedup at a resolution of 16x 16 and a threefold increase in speed at a higher resolution of 64 x 64.
Parallel implementation:
2xx, v) = Σ V) s.t. A. = A = B
(3)
where all FINOLA solutions {Φr} share the matrices A and
B. The resulting feature map z satisfies A = QÂ (Eq. 2). but it no longer adheres to the FINOLA constraint (A,z + A2, A + B2). Notably, the vanilla FINOLA corresponds to a special case M 1.
This approach can be implemented by expanding FINOLA from single path to multiple paths. As illustrated in Figure 3, image undergoes encoding into M vectors (q1---M)) ), with each vector subjected to the FINOLA process. Each path corresponds to a special solution Φ; in Eq. 3. Sub- sequently, the resulting feature maps are aggregated to re- construct the original image. Importantly, all these paths share the same set of parameters. Our experiments have validated the effectiveness of this approach, showing that the reconstruction PSNR improves as the number of paths increases.
Importance of NormaLinear: In Section 4.1, experiments support the significance of Norm+Linear by showing that (a) simpler processes such as repetition or linear without normalization lead to significant degradation, (b) per- per-sample normalization is crucial, as seen in poor performance of Batch-Norm during validation, and (c) the gain from more complex non-linear models (e.g. MLP) is negligible.
One-way wave equations after diagonalization: Empiri- cally, we consistently observed that the learned matrix Q is diagonalizable (Q= = VAV across various training con- figurations. As a result, channels become decorrelated when projecting the feature map z by the inverse of eigenvectors: <(1, y) V V122(z, y), which modifies Eq. 2 to:
3. Interpretation via One-way Wave Equations
In this section, we interpret FINOLA from the perspective of one-way wave equations, empirically offering deeper insight into the inherent nature of images.
A = AA,S, where A -diag (A1, ......,, ........)).......... (4)
Linear partial difference equations: Let's denote the spa- tial increments of the feature = along x and y axes as A=
where channels in Ç are decorrelated. Each channel Co follows an independent linear partial difference equation AzC* = A It is a finite approximation of a one-way wave equation (or transportation equation) as follows:
z(x + 1, y) 2(x, y) and 4, = z(x; y + 1) - =<r, respectively. Then, FINOLA (Eq. I) can be expressed as linear partial difference equations:
OCk 8r
OCk AA Dy
4 = AB- = QA,z s.t. Q AB-! (2)
Here, the horizontal change A,= exhibits linear correla- tion with its vertical counterpart Azz.. When the matrix B is invertible, FINOLA stands as special solution to this equation, given that A and 4, not only exhibit linear correlation but are also linearly correlated with the normal- ization of the current feature values & (referred to as the FINOLA constraint). It's noteworthy that the learned matri- ces A and B are empirically found to be invertible across multiple training runs with various dimensions, ranging from x28x 128 to 4096x4096.
where d is the tth eigenvalue in A. For each channel Ču, the rate of change along the z-axis is Å times the rate of change along the y-axis. Its solution takes the form Fe(Axtty) where Fie+) can be any differentiable function. Typically, one-way wave equation involves time t as du Br.
o here we replace t with y.
Key insight: The amalgamation of Eqs. 1, 3, and 5 empiri- cally reveals an insight into understanding images: images may share set of one-way wave equations in the latent feature space. Each image corresponds to a distinct solution that can be generated from its associated initial condition, as illustrated in Figure 1.
Relaxing the FINOLA constraint through FINOLA se- ries: FINOLA represents a specific solution to Eq. 2, but it may not be the optimal one. We have discovered that a more
4

---

Exploring Invariance in Images through One-way Wave Equations
Table 1: Reconstruction PSNR across various resolutions. Per- formance drops slightly at higher resolutions which have significant fewer parameters in the following upsampling and convolution layers.
Table 2: Comparison with simpler autoregressive baselines. PSNR values for image reconstruction on the ImageNet- IK validation set are reported. Im- age size x56x256. Single-path FINOLA with C 3072 channels is used. denotes the use of position embedding.
upsample/cony Single-path
Multi-path
Resolution
#Params
3x3022
4x 1024
Resolution
8x8
25.3M
25.4
25.9
Autoregression
16x 16
64 x 64
16x 16
18.5M
25.8
26.2
32x32
9.6M
25.8
26.2
Repetition
16.1
13.3
64x64
7.9M
25.7
26.1
Repetition
20.2
21.2
128x 128
1.7M
25.3
25.4
Linear
25.4 not converge
256x256
1.2M
24.6
24.8
Norm+Lineer
25.8
25.7
Table 3: Comparison with norm+nonlinear. PSNR values for image reconstruction are reported. The norm+nonlinear baseline replaces the linear model in FINOLA with two MLP layers incorporating GELU activation in between.
Table 4: Comparison between normalization mod-
els. PSNR values for image reconstruction reported. Layer-norm is significantly better than batch-norm the validation set.
Autoregression C=512
C=1024
C=3072
Normalization
Training
Validation
Norm+Nonlinear
22.4
23.8
25.8
Batch-Norm
25.1
16.3
Norm+Linear 22.2
23.7
25.8
Layer-Norm
25.5
25.8
Both FINOLA solution and initial condition be easily transformed the new feature space The transformed initial condition is 2( W H
AB-1 VAV-1 Examination of the eigenvalues in A and eigenvectors in V across various trained models con- firms their complex nature (A, V E CCXC). Please refer to Appendix B.9 for enforcing real-valued wave speed.
2 :2/ q. The FINOLA solution in Eq. is transformed as follows:
M
Additionally, it's noteworthy that the diagonalizability of AB is nor guaranteed since matrices A and B are learned from training loss without imposed constraints. However, in practice, experiments indicate that non- diagonalizable matrices rarely occur. This observation sug- gests that the set of matrices resistant to diagonalization is sufficiently small through the learning process.
C(x, v) = Σ 2m1
(1),,,), Anb; HAMS ,,b, = H
(6)
where the transformed FINOLA series bb and matrices HA and HB are computed by multiplying the inverse of eigenvectors V-1 before di A, and B, respectively:
3b V Φi. H V A, H = =
4. Experiments on Image Reconstruction
J)
11 V
We evaluate our FINOLA (single and multiple paths) for im- age reconstruction ImageNet- 1K (Deng et al., 2009). The default image size is 256x256 Our models are trained on the training set and subsequently evaluated on the validation set. Please refer to Appendix B.2 for model and training details, and Appendix B. for additional ablations, experimental results and visualization.
where C represents the number of channels (2, y) E CC. and J are the identity and all-ones matrices respectively. Unlike the normalization of Φ; in Eq. 3, which simply divides the standard deviation after subtracting the mean, the derivation of normalization & shown in Appendix E.I.
Implementation clarification: We clarify that the inter- pretation via one-way wave equation does not guide train- ing, but reveals an insight through post-training processing. Specifically, wave speeds, denoted as A, are not explicitly learned during training. Instead, they are computed post- training by diagonalizing trainable matrices A and B as
4.1. Main Properties
FINOLA across various resolutions: Table shows con- sistent PSNR scores across various feature map resolutions for both single-path and multi-path FINOLA. Minor perfor- mance reduction occurs at 128 x 128 and 256x256 due to
5

---

Exploring Invariance in Images through One-way Wave Equations
29
-048
Reconstruction PSNR
11 3n C 3 PSNR
28
1024
27
26
C=512
M
2
8
16
256
25
24
2048
24.8
25.9
26.9
28.0
29.1
23
128
Rec 7
Recom
22
1024
23.4
24.8
26.1
27.1
27.8
32768
21
512
22.0
23.2
24.5
25.8
26.3
16384
20
256
20.6
21.8
22.9
24.4
25.5
8192
19
18
128
19.3
20.4
21.5
22.5
23.2
4096
1
2
4
8 16
Number of FINOLA paths (log-scale)
Latent
128
256
1024
Figure 5: Reconstruction PSNR for multi-path FINOLA. The generated feature map has a resolution of 64,64, and the image size is 256x Increasing the number of paths M, as defined in Eq. 3, consistently enhances reconstruction PSNR across various dimensions (C = 128 to C = 2048). The blue lines in the right table represent contour lines of the latent size (equal to MC). PSNR remains consistent along each latent size line. Best viewed in color.
smaller decoders (1.7M and 1.2M parameters, respectively). Notably, at resolution 256x256, FINOLA is followed by only three 3x3 convolutional layers, covers 7-pixel field of view (see Table 13 in Appendix B.2).
Table 5: Comparison with discrete cosine transform (DCT). PSNR values for image reconstruction are reported on the ImageNet-IK validation set. (2048 x 16) indicates C 2048 channels and M 16 FINOLA paths. denotes using multiple initial conditions q: at different positions instead of overlapping the center (see Appendix B.
Norm+Linear: Table underscores the irreplaceability of norm+linear, as simpler alternatives like repetition exhibit significantly lower PSNR, and linear model without nor- malization fails converge higher resolutions (64 64). Additionally, Table 3 demonstrates that replacing the lin- component with more complex 2-layer MLP yields negligible gain. Moreover, Table 4 emphasizes the impor- tant role of layer normalization, with a substantial drop in validation observed for batch normalization. These find- ings collectively establish that norm+linear necessary and sufficient.
Method
Latent 4
PSNR
DCT (top-left 1)
3072
20.6
FINOLA (multi-path)
2048 (1024x2)
24.8
DCT (top-left 3)
9216
23.5
FINOLA (multi-path)
8192 (1024x8)
27.1
DCT (top-left 6)
18432
25.6
FINOLA (multi-path
16384 (2048x8)
28.0
Multi-path FINOLA: Figure 5 shows the PSNR values for multi-path FINOLA. Increasing the number of paths M consistently improves PSNR. Visual comparisons in Figure 16 at Appendix B.8 emphasize the notably enhanced image quality from single-path to multi-path FINOLA, showcasing its ability to find superior solutions within the wave equation solution space. However, multi path also increases the latent size of initial conditions (E q MC). The right side of Figure 5 demonstrates consistent PSNR along the same latent size line. This suggests that reconstruction quality is influenced not solely by the number of wave equations C' or the number of FINOLA paths M but by their product MC (the latent size). This finding enables parameter efficiency in matrices A and B by decreasing the number of channels and increasing the number of paths, which is important for latent size of 16,384,
FINOLA (multi-path)'
16384 (2048x8)
28.9
DCT (top-left 10)
30720
27.5
FINOLA (multi-path)
32768 (2098x16)
29.1
FINOLA (multi-parhi)
32768 (2048 16)
30.0
observations about how images are distributed the space of the compressed vector q: (a) the reconstruction from the averaged q over 50k validation images results gray image (Figure 12 in Appendix B.7), (b) the space predom- inantly occupied by noisy images (Figure 11 in Appendix B.7), and (c) the reconstruction from an interpolation be- tween two embeddings, aqi + - a))2. yields mix-up of corresponding images (Figure 13 in Appendix B.7).
large latent size. For instance, single path requires 268 million parameters in matrices A and B, whereas aggregating 16 FINOLA paths incurs only 1 million parameters.
4.2. Comparison with Previous Techniques
We compare multi-path FINOLA with widely recognized en- coding/decoding methods, such as discrete cosine transform, discrete wavelet transform, auto-encodee The comparison is based on a similar number of latent coefficients.
Image distribution q space: We made three intriguing
6

---

Exploring Invariance in Images through One-way Wave Equations
Table 6: Comparison with discrete wavelet transform (DWT), PSNR values for image reconstruction are reported on the ImageNet-1K validation set. (2048 x 16) indicates C 2048 channels and M 16 FINOLA paths. f denotes using multiple initial conditions at different positions instead of overlapping at the center (see Appendix B.10).
Table 8: Comparison with JPEG on end-to-end com- pression. A single-path FINOLA model with C' = 3072 channels is compared to JPEG compression end-to-end on ImageNet (Deng et al., 2009) and Kodak (Company, 1999) datasets. FINOLA has much cheaper pipeline, i.e. uniform quantization per channel without additional coding of the quantized bits, but achieves superior performance compared to JPEG.
Method
Latent 4
PSNR
DWT [scale-3 subband)
3888
21.5
ImageNet Bit/Pixel PSNR1
Kodak
DTCWT (scale-3 LL subband)
12288
22.3
Method
Bit/Pixel_ PSNR1
FINOLA (multi-path)
2048 (1024x2)
24.8
JPEG
0.50
24.5
0.20
24.0
DWT (scale-3 all subbands)
15552
24.3
FINOLA
0.19
24.9
0.19
25.6
DTCWT (scale-3 all subbands)
49152
25.6
FINOLA (mulltipatt)
8192 (1024x8)
27.1
and DTCWT in terms of PSNR for the first two groups, achieving smaller latent size. In the last group, while FINOLA's PSNR is lower than DTCWT, its latent size is significantly smaller (more than 6 times smaller).
DWT [scale:2 all subhands)
55953
28.7
DTCWT (scalo-2 all subbands)
196608
30.8
FINOLA (multi-path)
32768
(2048x16)
29.1
FINOLA (multi-potts)
32768 (2048x16)
30.0
Comparison with convolutional auto-encoder (Masci et al., 2011; Ronneberger et al., 2015; Rombach et al., 2021): Table 7 presents a comparison between our method and convolutional autoencoder (Conv-AE) con- cerning image reconstruction, measured by PSNR. Both approaches share the same Mobile- (Chen al., 2022) encoder and have identical latent sizes (2048 or 8192). In our method, multi-path FINOLA is initially em- ployed to generate a 64 x4 feature map, followed by up- samplingtconvolution to reconstruct an image with size 256x 256. On the other hand, Conv-AE employs a deeper decoder that utilizes upsampling-coovolutin from the la- tent vector to reconstruct 15 in Appendix B.3 for details in architecture compari- son. Our method has significantly fewer parameters in the decoder. The results highlight the superior performance of our method over Conv-AE, indicating that a single- layer FINOLA more effective than multi-layer upsam- poin-tconvolution approach. The comparison with auto- encoding (first stage) in generative models (e.g. Stable Diffusion (Rombach et al., 2021)) is shown in Table 16 in Appendix B.4.
Table 7: Comparison with convolutional auto-encoder (Conv-AE). FINOLA mnulti-path) achieves higher PSNR compared to Conv-AE with the same latent size, while using significantly fewer parameters in the decoder. Both methods employ the same Mobile-Former encoder, and the same upsampling/convolutii layers after the feature map x is generated at resolution 64x64.
Method
Latent
Param.. PSNR
Conv-AE
2048
35.9M
24.6
image. Please Table
FINOLA
2048 (1024x2))
16.6M
24.8
Conv-AE
8192
61.9M
26.0
FINOLA
(192
16.6M
27.1
Comparison with discrete cosine transform (DCT) (Ahmed et al., 1974): Table compares FINOLA with DCT. DCT is conducted per 8x8 image block, and the top- left K coefficients (in zig-zag manner) are kept, while the rest are set to zero. We choose four K values (1, 3, 6, 10) for comparison. Clearly, multi path FINOLA achieves higher PSNR with a similar latent size.
4.3. Comparison with JPEG on Image Compression
In Table 8, compare FINOLA (single path with 3072 channels) with JPEG for image compression. Remarkably, by employing only uniform quantization per channel with- out further coding of the quantized bits, FINOLA achieves higher PSNR values with lower bits per pixel on both the ImageNet and Kodak (Company, 1999) datasets.
Comparison with discrete wavelet transform (DWT/DTWWT) (Strang, 1989; Daubechies, 1992; Vetterli & Kovacevie, 2013): We compare FINOLA with DWT and DTCWT in Table 6. Three scales are chosen for wavelet decomposition. The comparisons are organized into three groups: (a) using only the LL subband at the coarsest scale (scale 3), (b) using all subbands (LL,, LH, HL, HH) at the coarsest level. and (c) using all subbands the finer scale (scale 2). Our method outperforms DWT
4.4. Comparison with AE in Generative Models
Table 9 presents a comparison of FINOLA with the first stage (learning an autoencoder and vector quantization in

---

Exploring Invariance in Images through One-way Wave Equations
Table 9: Comparisons with the first stage of multiple generative methods (learning an autoencoder and vector quantization in the latent space), assessed using both PSNR and Fréchet Inception Distance (FID) metrics.
Loss
Method
Latent Size #Channels
FID 1 PSNR
Sogit-Laplacs L2 Perceptual GAN
DAIL-F (Ramesh al., 2021)
16 x 16
32.00
22.8
VQGAN (Esser et al., 2021)
16 x 16
256
4.98
19.9
ViT-VQGAN (Yu et al., 2022a)
32 x 32
32
1.28
Stable Diffusion (Rombach al., 2021)
16 x 16
16
0.87
24.1
FINOLA
1 X
3072
27.82
25.8
the latent space) of multiple generative methods, assessed using both PSNR and Fréchet Inception Distance (FID) met- rics. While FINOLA achieves good performance in terms of PSNR, its results are less competitive with respect to FID. This divergence is a consequence our deliberate choice to employ the L2 loss function. We intentionally prioritized straightforward loss function to emphasize that FINOLA's efficacy does not depend on complex reconstruction loss for- mulations. To further optimize FII) scores, we acknowledge the potential benefits of incorporating perceptual loss func- tions and Generative Adversarial Network (GAN) losses, as demonstrated in models such as VQGAN (Esser et al., 2021), ViT VQGAN (Yu et al., 2022a), Stable Diffusion (Rombach al., 2021). We also leave more extensive com- parisons with recent generative models (Shocher et al., 2024; Ismail al., 2024) to future work.
Table 10:
Comparison with previous self-supervised methods on ImageNet-IK fine-tuning. The baseline meth- ods includes MoCo-v3 (Chen et 2021), MAE-Lite (Wang et al., 2022), UMMAE (Li et al., 2022b), MAE (He et al., 2021), and SimMIM (Xie et al., 2022). Three Mobile- Former backbones of varying widths are used, followed by a decoder with transformer blocks.
Method
Model
MAdds #Params
Top-1
MoCo-v3 ViT-Tiny
1.2G
6M
76.8
MAE-Lite ViT-Tiny
1.2G
6M
78.0
FINOLA MF-W720
0.7G
7M
78.4
MoCo-v3 ViT-S
4.6G
22M
81.4
UM-MAE Swin-T
4.5G
29M
82.0
MAB-Lite ViT-S
4.6G
22M
82.1
SimMIM Swin-T
4.5G
29M
82.2
It is crucial to reiterate that FINOLA's primary objective dif- fers from that of generative model. Our focus is not on im- age generation per se, but rather on introducing a novel per- spective for understanding images (first-order norm+linear autoregression).
FINOLA MF-W1440
2.6G
20M
82.2
MoCo-v3 ViT-B
16.8G
86M
83.2
MAE
ViT-B
16.8G
86M
83.6
SimMIM
ViT-B
16.8G
86M
83.8
SimMIM
Swin-B
15.4G
88M
84.0
5. Application on Self-Supervised Learning
FINOLA MF-W2880
9.9G
57M
83.9
FINOLA can be applied to self-supervised learning through a straightforward masked prediction task, which we refer to as Masked FINOLA to distinguish from the vanilla FINOLA. Please refer to Appendix C for details of masked prediction, network structure, training setup, and additional experiments. Our key findings include:
ranging from to 6.4 AP. Table 26 in Appendix C.5 shows that FINOLA outperforms MoCo-v2 on various tasks such as image classification, object detection, and segmentation. Notably, the encoder is frozen without fine-tuning on detec- tion and segmentation tasks. Please refer to Appendix C.5 for additional experimental results.
Comparable performance: Masked FINOLA demon- strates comparable performance to established baselines, e.g. MAE (He et al., 2021) and SimMIM (Xie et al., 2022), on ImageNet fine-tuning (see Table 10), as well as linear probing (see Table 25 in Appendix C.4), while maintaining lower computational requirements.
FINOLA vs. Masked FINOLA: Table 30 in Appendix D. compares vanilla FINOLA and two masked FINOLA variants image reconstruction and linear probing. The introduction of masking masked FINOLA trades restora- tion accuracy for improved semantic representation. Geo- metrically, Figure 23 in Appendix D.2 illustrates masked FINOLA introduces substantial increase in Gaussian cur- vature on critical feature surfaces, suggesting enhanced cur- vature in the latent space for capturing semantics. Computa- tion details of Gaussian curvature are available in Appendix D.3. Additional comparisons can be found Appendix D.1.
Pre-training with Masked FINOLA, followed by fine-tuning on ImageNet-IK (IN-IK),
Robust task-agnostic encoders:
consistently outperforms IN- KK supervised pre-training in both ImageNet classification and COCO object detection (see Figure 6). The gains in object detection are substantial,
8

---

Exploring Invariance in Images through One-way Wave Equations
50
FINOLA vs. Supervised Pre-training as task-agnostic encoders
FINOLA+INIK-FT
COCO mAp
FINOLA
W1440
W288C
W1440
W2880
Encoder
Pre-training
IN-1K Top
COCO Det AP
W720
W720
W2880
supervised
75.7
35.2
W1.440
MF-W720
FINOLAHNNIKFFT
75.8 (+0.1)
(16.4)
Sup-IN1K
35
supervised
79.4
38.3
W720
MF-W1440
FINOLA+INIK-FT
80.5 (v0.5)
44.0 (15.7)
supervised
80.8
40.5
30
65
ImageNet Top-1 Accuracy
70
75
80
85
MF-2880
FINOLAHINIKFFT
82.5 (+L.7)
(65.0)
Figure 6: Task-agnostic encoders evaluated on ImageNet (IN-IK) classification and COCO object detection. We assess three IN-IK pretraining methods: (a) supervised (Sup-INIK), (b) FINOLA, and (c) FINOLA with fine-tuning on IN-IK (FINOLA+INIK-FT), The dots represent different Formle backbones. It is noteworthy that the backbone remains task-agnostic and is kept frozen during object detection. FINOLA performs lower than Sup-INIK in classification but surpasses it in object detection. After fine-tuning on IN-IK, FINOLA+INI-FTT shows improvements in both tasks, providing robust task-agnostic encoders.
6. Related Work
between negative samples by handling collapse carefully. (Chen et al., 2020c; Li et al., 2021a) show pre-trained mod- els work well for semi-supervised learning and few-shot transfer. Masked image modeling (MIM) is inspired by BERT (Devlin al., 2019) and ViT (Dosovitskiy al., 2021) to learn representation via masked prediction. BEÎT (Bao et al., 2021) and PeCo (Dong et al., 2021) predict on tokens, MaskFeat (Wei et al., 2022) predicts on HOG, and MAE (He et al., 2021) reconstructs original pixels. Recent works explore combining MIM and contrastive learning (Zhou et al., 2022; Dong et al., 2022; Huang et al., 2022: Tao et al., 2022: Assran et al., 2022; Jiang al., 2023) or techniques suitable for ConvNets (Gao et al., 2022; Jing et al., 2022; Fang et al., 2022). Different from using random masking in these works, FINOLA uses regular masking and simpler norm+linear prediction.
Autoregression has played a pivotal role in generating high-quality images (van den Oord et al., 201 6b;a; Salimans et al., 2017; Chen et al., 2018). These methods model conditional probability distributions of cur- rent pixels based on previously generated ones, evolving from pixel-level focus latent space modeling using vector quantization (van den Oord et al., 2017; Razavi al., 2019; Esser et al., 2021; Yu et al., 2022c). In contrast, we present a first-order norm+lineat autoregression to generate feature map and reveals new insights by generalizing FINOLA as a set of one-way way wave equations.
Image autoregression:
Image transforms: The Discrete Cosine Transform (DCT) (Ahmed et al., 1974) and Wavelet Transform (Strang, 1989; Daubechies, 1992: Vetterli & Kovacevic, 2013) are widely recognized signal processing techniques for image compres- sion. Both DCT and wavelet transforms project images into a complete space consisting of known wave functions, in which each image has compact coefficients, i.e., most coef- ficients are close to zero. In contrast, our method offers a distinct mathematical perspective for representing images. It encodes images into a compact space represented by set of one-wave equations with learnable speeds, with each image corresponding to unique initial condition. These differences are summarized in Table 11 at Appendix B.
7. Conclusion
In this paper, we empirically revealed a form of invariance in natural images through the lens of one-way wave equa- tions. Our findings suggest that all images may share a common set of such equations, defined by learnable propa- gation speeds, with each image corresponding to a unique solution determined by its initial condition. This dynamic is seamlessly implemented within an encoder-decoder frame- work, where the wave equations are reformulated order norm-plus- liner autoregressive (FINOLA) process. The proposed approach demonstrates strong performance in image reconstruction and shows promising potential for self-supervised learning, offering a novel perspective on the underlying structure of visual data. Looking forward, we believe that further theoretical investigations into the dynamics uncovered in this work could provide valuable insights and foundations for future research.
first-
Self-supervised learning: Contrastive methods (Becker & Hinton, 1992; Hadsell et al., 2006; van den Oord al., 2018; Wu al., 2018; He et al., 2019; Chen & He, 2020: Caron al., 2021) achieve significant progress. They are most applied to Siamese architectures (Chen et al., 2020b; He et al., 2019; Chen et al., 2020d; 2021) to contrast image similarity and dissimilarity and rely on data augmentation. (Chen& & He, 2020; Grill et al., 2020) remove dissimilarity
9

---

Exploring Invariance in Images through One-way Wave Equations
Impact Statement
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and
Hinton, G. Big self-supervised models strong semi- supervised learners. arXiv preprint 2kIv:22006110022 2020c.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Chen, X. and He, K. Exploring simple siamese representa- tion learning. arXiv preprint arXi::2011. 2020.
References
Chen, X., Mishra, N., Rohaninejad, M., and Abbeel, Pix- elSNAIL: An improved autoregressive generative model. In Proceedings of the 35th International Conference Machine Learning, pp. 864 072-872,-5 2018.
Ahmed, N., Natarajan, T., and Rao, K. Discrete cosine transform. IEEE Transactions Computers, C-23(1): 90-93, 1974. doi: 10. 10.1099T-C.19742233888
Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P., Joulin, A., Rabbat, M., and Ballas, N.
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020d.
Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07141, 2022.
Chen, X., Xie, S., and He, K. An empirical study of train- ing self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
Bao, H., Dong, L., and Wei, F. BETT: BERT pre- pre-training of image transformers, 2021.
Becker, S. and Hinton, G. E. A self-organizing neural net- work that discovers surfaces in random-dot stereograms. Nature, 355:161-163, 1992.
Chen, Y., Dai, X., Chen, D., Liu, M., Dong, X., Yuan, L., and Liu, Mobile-former: Bridging mobilenet and transformer. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C.,, Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpps://procceedngggg neurips. oc cc/pappe_fille//pppr////////1//1 1457 4457ccdbb1cc09667111110aaaaaaa. pdf.
Company, E. K. Kodak lossless true color image suite, 1999. URL htw.h:ttps:.:tm http:/rrkkkus/ http::::////...uugrphhhhhhs
Daubechies, I. Ten Lectures on Wavelets. Society for In- dustrial and Applied Mathematics, USA, 1992. ISBN 0898712742.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee. 2009.
Devlin, J., Chang, M.-W., Lec, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan- guage Technologies, pp. 1171-4186, Minneapolis, Min- nesota, June 2019. doi: 10. 181186653 1/N19- URL https: http::://acaanhhott aee.lanthoocsccc ogy. org/N1gg 9-1199---4 9-19---44
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In ECCV, 2020.
Caron, M., Touvron, H.. Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-superviser vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., and Yu, N. Peco: Perceptual codebook for BERT pre-training of vision transformers. abs/2111.127DD 2021.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Dhariwal, P., Luan, D., and Sutskever, I. Generative pretraining from pixels, 2020a.
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., and Yu, N. Bootstrapped masked autoencoders for vision bert pretraining. arXiv preprint arXiv:2207.07116, 2022.
Chen, T., Korbllihh, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. arXiv preprint arXiv:2002.05709 2020b.
10

---

Exploring Invariance in Images through One-way Wave Equations
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn,
Jiang, Z., Chen, Y., Liu, M., Chen, D., Dai, X., Yuan, L., Liu, Z., and Wang, Z. Layer grafted pre-training: Bridg- ing contrastive learning and masked image modeling for better representations. In International Conference on Learning Representations, 2023.
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Jing, L., Zhu, J., and LeCun, Y. Masked siamese con- vnets. CoRR, abs/2206.077000, 2022. doi: 10.48550/arXiv. 2206.07700. URL https: htppps//ddo/ /ttp//di.o/// org/1..r/10..00 48550/ axXiv.2206. 2206. 07700.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883 June 2021.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML,2000) pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Fang, Y., Dong, L.., Bao, H., Wang, X., and Wei, F. Cor- rupted image modeling for self-supervised visual pre- training, ArXiv, abs/2202.03382, 2022.
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. Set transformer: A framework for attention-based permutation- invariiat neural networks. In Proceedings of the 36th International Conference on Machine Learning pp. 3744-3753, 2019.
Gao, P., Ma, T., Li, H., Dai, J., and Qiao, Y. Convmae: Masked convolution meets masked autoencoders. arXiv preprint arXiv:2205.03888 2022.
Grill, J.-B., Strub, F,, Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, A., Guo, Z. D., Azar, M. G., Piot, B., Kavukeuoglu, K., Munos, R., and Valko, M. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., and Zhang, L. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13619- 13627, 2022a.
Hadsell, R., Chopra, S., and LeCun, Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06 volume 2. pp. 1735- 1742, 2006. doi: 10.1109/CVPR.2006..
Li, S., Chen, D., Chen, Y., Yuan, L., Zhang, L.., Chu, Q., Liu, B., and Yu, N. Improve unsupervised pretraining for few-label transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10201-10210, October 2021a.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Li, X., Wang. W., Yang, L., and Yang. J. Uniform mask- ing: Enabling mae pre-training for pyramid-based vision transformers with locality. arXiv:2205. 10063, 2022b.
Li, Y., Chen, Y., Dai, X., Chen, D., Liu, M., Yuan, L.,, Liu, Z., Zhang, L., and Vasconcelos, N. Micronet: Improving image recognition with extremely low flops. In Interna- tional Conference on Computer Vision, 2021b.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: /9X 0.0522, 2019.
Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., and Zhang, L. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In International Conference on Learning Representations, 2022. URL /ttps:// openreview net forum? i d= 9P job9J1.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. arXiv:2111.06377, 2021.
Huang, Z., Jin, X., Lu, C., Hou, Q., Cheng, M.-M., Fu, D., Shen, X., and Feng, J. Contrastive masked au- toencoders are stronger vision learners. arXiv preprint arXiv:2207.13532. 2022.
Masci, J., Meier, U., Ciresan, D. C., and Schmidhuber, J. Stacked convolutional auto-encoders for hierarchical feature extraction. In International Conference on Ar- tificial Neural Networks, 2011. URL https: //api. semanticscholas org/ Corpus 1ate::2241119
Ismail, A. A., Adebayo, J., Bravo, H. C., Ra, S., and Cho, K. Concept bottleneck generative models. In The Twelfth International Conference on Learning Representations, 2024. URL httpt:tttp//peeee http///pppeeeeeeeeeeeeeeeeeeeeee Forum? 1 1d=1.9U5MJJ1eF.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre- training, 2018.

---

Exploring Invariance in Images through One-way Wave Equations
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models unsupervised multitask learners, 2019.
http::s/ppooceedigg..neeeeee cc/ cc//appe__iilee//pppr////ff// b1 1.30114ffeffbbccc$e11f9ffeeeee Paper.. pdf.
2016a. URL
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation, 2021. URL http: waw.vv.ivvvv.../.. abs/ 2092. cite arxiv2210..20092.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In Proceedings The 33rd International Conference Machine Learning, pp. 1747-1756, 2016b.
Razavi, A., den Oord, A., and Vinyals, O. Generating di- verse high-fidelity images with vq-vae-2. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alcéé-Buc, F., Fox, E.,
van den Oord, A., Vinyals, O., and kavukcuoglu, Neural discrete representation learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vish- wanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL ht.::::::t httpttpppp//pp//eeeeens...sse peu..pps.ccc/aaeee//0/////// T.9861776530000022222443332----0 pdf.
and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,
2019. URL httpp::/prroceeiinggg.... neurips. cc cc/paper hte/ppper__iles/pppeeeeeeeee oc//appee_filee//pper/////////// steezall/88lbbcaef1cc0277515 Paper. pdf.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models, 2021.
van den Oord, A., Li, Y., and Vinyals, O. Representa- tion learning with contrastive predictive coding. ArXiv, his/1807.0322 2018
Ronneberger, O.,
Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmentation. ArXiv, abs/1505.04597, 2015. URL https::t////....... semanticschon org/ Corp/orrrp:DD:
Vetterli, M. and Kovacevic, J. Wavelets and subband coding. In Prentice Hall Signal Processing Series, 2013. URL https: /app://pppp/pp///// semant.ccccoole... srg//CorpusDD:100653
ott.:cchhoooo..ooooooooooooaa/a. 371:29::3:111
Salimans, T., Karpathy, A., Chen, X., and Kingma, Pixelcn+++: A pixelcnn implementation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017.
Wang, S., Gao, J., Li, 7., Sun, J., and Hu, W. A closer look at self-supervised lightweight vision transformers. arXiv preprint arXiv:2205. 14443, 2022.
Sandler, M., Howard, A., Zhu, M.. Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510- 4520, 2018.
Wei, C., Fan, H. Xie, S., Wu, C.-Y.. Yuille, A., and Feicht- enhofer, C. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14668-14678, June 2022.
Shocher, A., Dravid, A., Gandelsman, Y., Mosseri, I., Ru- binstein, M., and Efros, A. A. Idempotent generative network. In ICLR, 2024.
Wu, Z., Xiong, Y., Stella, X. Y., and Lin, D. Unsupervised feature learning via non-parametric instance discrimina- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Strang, G. Wavelets and dilation equations: A brief intro- duction. SIAM Rev. 31,614-627, 1989. URL https: wew....smmattsschol..... wwt.:hoo.a.....oooouoossl...l 120403677.
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: A simple framework for masked im- age modeling. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022..
Tao, C., Zhu, X., Huang, G., Qiao, Y., Wang, X., and Dai, J. Siamese image modeling for self-supervise vision representation learning, 2022.
den Oord, A., Kalchbrenner, N., Espeholt, L., kavukcuoglu, k., Vinyals, O., and Graves, A. Condi- tional image generation with pixelenn decoders. In Lee, D., Sugiyama, M., Luxburg, U. Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.,
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector- quantized image modeling with improved VQGAN, In International Conference on Learning Representations, 2022a. URL httts:: : id=pfNyExj74 http://pppppeiieeeeeee net/ forum?
12

---

Exploring Invariance in Images through One-way Wave Equations
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners image- text foundation models. Transactions Machine Learn- ing Research, 2022b. ISSN 2835-88566 8856. URL https: / //pperreview net/ / forum? d=Ee277P.AYC..
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Vasudevan, V., Ku, A., Yang, Y. Ayan, B. K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., and Wu, Y. Scaling autoregressive mod- els for content-rich text-to-image generation. Transac- tions on Machine Learning Research, 2022c. ISSN 2835- 8856. URL https: / opw..evieeeeeeeeeee/ee/ee net/ forum? id=AFDcYJKhND, Featured Certification.
Zhang, H., Li, F., Liu, S.. Zhang. L., Su, H., Zhu, J., Ni, L. M., and Shum, H.-Y. Dino: Detr with improved denois- ing anchor boxes for end-to-end object detection, 2022.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre- pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. De- formable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13

---

Exploring Invariance in Images through One-way Wave Equations
Table 11: Comparison between DCT/Wavelet transform and FINOLA.
DCT or Wavelet Transform
FINOLA
Representation
Cosine/Wavelae functions
One-way wave equations
Parameters
Fixed parameters
Learnable speeds
Encoding Compactness
Image + coefficients
Image initial conditions
Compact coefficients per image
Compact space representation
Table 12: Specification of Mobile-Former encoders. "bneck-lite" denotes the lite bottleneck block (Li et al., 2021b).
"M-F" denotes the Mobile-Former block and "M-FU" denotes the Horile-Former block for downsampling.
MF-W2880
MF-W1440
MF-W720
Stage
Resolution
Block
#exp
#out
#exp
#out
#exp
#out
token
6x256
6x 656
6x
192
stem
256
3x3
64
32
16
1
128
bneck- lite
128
64
64
32
32
16
M-F
384
112
192
56
96
28
2
642
M-F
336
112
168
56
84
28
M-F
672
192
336
96
168
48
3
32
M-F
576
192
288
96
144
48
M-F
576
192
288
96
144
48
M-Ft
1152
352
288
96
240
80
M-F
1408
352
704
176
320
88
M-F
1408
352
704
176
480
4
162
M-F
2112
480
1056
240
528
120
M-F
2880
480
1440
240
720
120
M-F
2880
480
1440
240
720
120
conv 1x1
2880
1440
720
A. Limitations
The major limitation of our method that the invariance (encoded in matrices and B) is revealed empirically without theoretical proof. Additionally, this paper focuses on multi-path FINOLA, which represents only subspace of the solutions the one-way wave equations. future work, we plan to explore the theoretical analysis of the revealed invariance and the complete solution space of the one-way equations.
B. FINOLA for Image Reconstruction
In this section, we list implementation details and additional experimental results of FINOLA (single or multiple paths).
B.1. Conceptual Comparison with DCT/Wavelet Transforms
Both DCT and wavelet transforms project images into a complete space consisting of known wave functions, in which each image has compact coefficients, i.e., most coefficients are close to zero. In contrast, our method offers a distinct mathematical perspective for representing images. It encodes images into a compact space represented by a set one-way wave equations with learnable speeds. Each image corresponds to a unique initial condition. These differences are summarized in Table 11.
14

---

Exploring Invariance in Images through One-way Wave Equations
Table 13: Upsampling and convolutional layers in FINOLA decoder. The complexity of upsampling and convolution layers decreases as the spatial resolution of feature map (generated by FINOLA) increases from 8x8 to 256x256) "res- conv" represents a residual block (He et al., 2016) consisting of two 3x3 convolutional layers, while "up-conv" performs upsampling followed by a 3x3 convolutional layer.
8x8
16x 16
x2x32
64 x 64
128x x 128
256 x 256
Resolution
block
#out
block
#out
block
#out
hlock
#out
block
#out
block
#out
8"
res- res-conv 512
up-conv 512
16a
res-conv 512
res-conv
512
32
up-conv
up-conv 512
res-conv
256
res-conv
256
res-conv
256
64
up-conv
256
up-conv
256
up-conv
256
res-conv
256
res-conv
256
res-conv
256
res-conv
256
1282
up-conv
256
up-conv
256
up-conv
256
up-conv
256
res-conv
128
res-conv
128
res-conv
128
res-conv
128
res-conv 128
up-conv
128
up-conv
128
up-conv
128
up-conv
128
up-conv
128
256
res-conv
128
res-conv
128
res-conv
128
res-conv
res-conv
128
res-conv
128
conv3 x3
3
conv3 x3
conv3
3
conv3x3
3
conv3x3
3
#param
25.3M
18.5M
9.6M
7.9M
1.7M
1.2M
B.2. Implementation Details
B.2.1. NETWORK ARCHITECTURES
In this subsection, provide detailed information on the network architecture components used in our study. Specifically, describe (a) the Mobile-Former encoders, (b) the pooler to compress the feature map into single vector, (c) the upsampling and convolutional layers employed in FINOLA decoder.
Mobile-Former encoders: Mobile-Former (Chen et al., 2022) is used as the encoder in our approach. It is CNN-based network that extends MobileNet (Sandler et al., 2018) by adding 6 global tokens in parallel. To preserve spatial details, we increase the resolution of the last stage from 32 to 16 We evaluate three variants of Mobile-Former, which are detailed in Table 12. Each variant consists of 12 blocks and global tokens, but they differ in width (720, 1440, 2880). These models serve as the encoders (or backbones) for image reconstruction, self-supervised pre-training, and evaluation in image classification and object detection tasks. For image reconstruction, which increase the number of channels from W2880 by 1.5 and 2 times, respectively. It's important to note that these models were manually designed without an architectural search for optimal parameters such as width or depth.
also explore two wider models, W4320 and W5760,
Pooling the compressed vector q: In both FINOLA and element-wise masked FINOLA, the compressed vector q obtained by performing attentional pooling (Lee al., 2019; Yu et al., 2022b) the feature map. This pooling operation involves single multi-head attention layer with learnable queries, where the encoder output serves as both the keys and values.
Table 14: Training setting for FINOLA.
Config
FINOLA
optimizer
AdamW
base learning rate
1.5e-4
weight decay batch size
0.1
FINOLA decoders: Table 13 provides the architecture details of upsampling and covolutional layers after apply- ing FINOLA to generate feature maps z. The complexity of decreases the spatial resolution increases, going from 8x8 to 256 256. FINOLA is trained for 100 epochs ImageNet.
128
learning rate schedule
cosine decay
warmup epochs 10
training epochs
100
image size augmentation
2562
RandomResizeCrop
B.2.2. TRAINING SETUP
The FIOLA training settings for image reconstruction are provided in Table 14. The learning rate is scaled as Ir base Irx batchsize / 256.
15

---

Exploring Invariance in Images through One-way Wave Equations
Table 13: Architecture comparison between convolutional Auto-Encoder and FINOLA.
Auto-Encoder
FINOLA
Encoder
same
same
Pooling
2x2x 512 (2x2 grid)
2 1024 (overlap at center)
Upsampling 64x 64x 1024 5 conv blocks (from 2x2 64x64)
FINOLA
Upsampling 256x256x 13
Training setup
same
same
B.2.3. TRAINING AND INFERENCE TIME
Training the FINOLA model involves regressing dense feature maps, with computational requirements increasing with feature map size. For instance, training FINOLA to generate a 16x 16 feature map with 3072 latent channels for 100 epochs on ImageNet takes approximately 8 days with 8 V100 GPUs. Extending to a larger feature map, such as 64x64, increases the training time to 18 days using the same GPU setup.
Training time:
Inference time: In addition to training time, the runtime evaluation includes the complete inference pipeline, encompassing encoding, autoregression, and decoding, conducted MacBook Air with Apple M2 CPU. We evaluated FINOLA for generating feature maps of sizes 16x 16 and 64 x6,, with running times of 1.2 seconds and 2.6 seconds, respectively.
B.3. Architecture Comparison with Convolutional Auto-Encoder (Convv-AE)
Table 15 presents comparison of the architectural components between the Conv-AE and FINOLA, while their performance comparison is reported in Table 7 in Section 4.2). Both models share identical (a) encoder, (b) upsampling from resolution 64x64 to 256x256, and (c) training setup (hyper-parameters). However, they differ in their approaches to pooling and upsampling toward the resolution 64x64.
Auto-encoder pools a 2x2 grid with 512 channels, while FINOLA pools two vectors with dimension 1024, both yielding the same latent size (2048). Auto-encoder pooling retains spatial infor- mation within the 2x2 grid, whereas FINOLA has no explicit spatial information as both vectors are positioned centrally for the FINOLA process.
Pooling:
Table 16: Comparison with auto-encoding (first stage) in genera- tive models. PSNR values for image reconstruction are reported on the ImageNet-I JK validation set. (2048 x 16) indicates C 2048 channels and M 16 FINOLA paths. denotes using multiple initial conditions q. at different positions instead of overlapping at the center (see Appendix B.10).
Upsampling to Resolution 64x64: The auto- encoder utilizes a stack of five convolutional blocks generate features resolution of 64x64 Each block consists of three 3x3 convolutional layers followed by upsampling layer to double the reso- lution. In contrast, our method employs multi-pati FINOLA to generate the feature map from center- placed vectors. Since FINOLA utilizes only four matrices (A, B, A_, and B.), significantly re- duces the number of parameters compared to the five convolutional blocks used in the auto-encoder.
Method
Latent +
PSNR
DALL-E VQGAN
32x32x-
22.8
65536 (16x16x256)
19.9
Stable Diffusion
4096 (16x 16x
24.1
FINOLA (multi-path)
4096 (1024x4)
26.1
FINOLA (multi--path)
4096 (1024x4)
26.7
Stable Diffusion
12288 (64x64x3)
27.5
FINOLA (multi-path)
8192 (1024x8)
27.1
FINOLA (multi kmulti-pathh
8192 (1024x8)
28.0
Stable Diffusion
32768
(128x 128x2)
30.9
Engineering techniques: FINOLA does not rely on any additional engineering techniques. Despite this, slightly outperforms the auto-encoder while utilizing significantly fewer parameters. We at-
FINOLA (multi-path)
32768 (2048x 16)
29.1
FINOLA (multi-path)
32768
x2048x 16)
30.0
tribute this performance FINOLA'S efficient and effective modeling of spatial transitions.
16

---

Exploring Invariance in Images through Onc-way Wave Equations
Original
Reconstruction
Original
Reconstruction
Original
Reconstruction
Figure 7: Image reconstruction examples. FINOLA works well to reconstruct diverse categories such natural scenes, human portraits, facial images, animal photographs, air and land transportation, and medical images. Best viewed in color.
B.4. Comparison with Auto-Encoder in Generative Models
Table 16 provides a detailed comparison of FINOLA's performance against the first stage (autoencoding) of VQGAN and Stable Diffusion in image reconstruction, evaluated on ImageNet- Val with 256x256 images. It's important note that the first stage of VQGAN and Stable Diffusion focuses solely auto-encoding and does not involve the generation process (e.g., diffusion process).
This comparison underscores FINOLA's performance across varying latent dimensions and its effectiveness in comparison to other methods. Although FINOLA falls behind Stable Diffusion at the largest latent dimension (32768), operates in a more challenging setup. While FINOLA outputs single vector after encoding, positioned the center to generate feature maps through the FINOLA process, spatial information is not explicitly retained. contrast, the encoder in Stable Diffusion produces high-resolution grid (128x128) where spatial information is highly preserved.
Introducing spatial information in FINOLA by scattering the initial positions of multiple FINOLA paths (rather than overlapping at the center) enhances the reconstruction quality by 0.6-0.9 PSNR. However, due to scattering initial positions at only 16 locations, the preservation of spatial information remains constrained compared to Stable Diffusion's 128x 128 grid. Consequently, while this enhancement closes the gap in performance (PSNR 30.0 vs 30.9), it still falls short Stable Diffusion's 's spatial fidelity.
B.5. Additional Reconstructed Examples
Additional FINOLA reconstructed images, encompassing diverse categories such as natural scenes, human portraits, facial images, animal photographs, air and land transportation, and medical images, are shown in Figure
17

---

Exploring Invariance in Images through One-way Wave Equations
Table 17: Image reconstruction ablation experiments on ImageNet-IK, We report PSNR on the validate set. The reconstruction quality correlates to (a) the number of channels in the latent space and (b) complexity of encoder. Default settings are marked by! t.
#Channels 4096 3072 2048 1024 512 256 128 64
Encoder 67.6M 43.5M
25.0M
12.0M
5.0M
PSNR
25.9 25.8 25.1 23.7 22.2 20.8 19.4 18.2
PSNR 26.1
26.0
25.8
25.1
24.4
(a) Number of channels in latent space.
(b) Model size of encoders.
Original
d3072
d2048
d1024
d512
Nzedles
Needles
Meedles
Hord
40
95
95
40
95
95
Figure 8: Impact of the number of channels on image reconstruction quality. The leftmost column shows the original images. The number channels in the latent space, decreasing from 4096 to 64 from the left to right, controls the reconstruction quality. Best viewed in color.
B.6. Ablation Studies of Single Path FINOLA
The number of channels in the latent space is crucial. Table 17-(a) presents the PSNR values for various latent space dimensions, while Figure showcases the corresponding reconstructed examples. The image quality is noticeably poor when using only 64 channels, resulting in significant loss of details. However, as the number of channels increases, more details are successfully recovered. Using more than 3072 channels yields reasonably good image quality, achieving a PSNR of 25.8.
The model size of encoder is less critical but also related. As shown in Figure and Table 17-(b), the larger model has better image quality. But the gap not significant. When increasing model size by 13 times from 5.0M to 67.66 67.6M, the PSNR slightly improved from 24.4 2..1. Note all encoders share similar architecture (Mobilc-Former with 12 blocks), but have different widths.
The position of is not critical: Figure 10 showcases the reconstructed samples obtained by placing the compressed vector q at different positions, including the center and four corners. The corresponding peak signal-to-noise ratio (PSNR) values on the ImageNet validation set are provided at the bottom. While placing q at the center yields slightly better results compared to corner positions, the difference is negligible. It is important to note that each positioning corresponds to its own pre-trained model with non-shared parameters.
18

---

Exploring Invariance in Images through One-way Wave Equations
Original
67.6M
43.5M
25.0M
12.0M
5.0M
40
95
40
95
40
95
40
95
40
95
95
PSNR
26.1
26.0
25.8
25.1
24.4
Figure 9: Impact of encoder size on image reconstruction quality: The image reconstruction quality shows slight improvement as the size of the encoder increases. Even with a small encoder containing 5 million parameters (right column), effectively compresses an image into a single vector capable of reconstructing the entire image. Best viewed in color.
B.7. Inspecting the Image Distribution in q Space
In this subsection, we list main observations and analysis in the space of the compressed vector (named embedding space). This will help understand how images distributed in the embedding space. In this subsection, use single-path FINOLA with C 3072 channels.
Three observations: Below we list three observations that reveal properties of the embedding space.
Dominance of noisy images in the space: To analyze the distribution of images in the embedding space, we collected q vector for all 50,000 images from the ImageNet validation set and computed their statistics (mean and covariance). By sampling embeddings based on these statistics and reconstructing images, we consistently observed the emergence of similar noisy patterns, depicted in Figure 11. This observation highlights the prevalence of noisy images throughout the space, with good images appearing as isolated instances surrounded by the abundance of noise.
Averaged embedding q yields gray image: In Figure 12, we observe that the reconstructed image obtained from the averaged embedding 7. computed over 50,000 images from the ImageNet validation set, closely resembles a gray image. We further investigate the relationship between real image embeddings and the averaged embedding q through interpolations along the embedding space. As depicted the left figure, the reconstructed images maintain their content while gradually fading into a gray image. Additionally, we extend this connection to mirror embeddings the right figure, represented by 2q 7. which correspond to images with reversed colors. These findings suggest that despite the prevalence of noisy images, the line segment connecting an image embedding to the average embedding encompasses different color transformations of the same image.
Reconstruction from interpolated embeddings: In Figure 13, we present the reconstructed images obtained by interpolating between two image embeddings using the equation aqı + (1 - a)q2. This process of embedding mixup results in a corresponding mixup of the images, allowing for smooth transition between the two original images by varying the value
19

---

Exploring Invariance in Images through One-way Wave Equations
Original
Center
Top-left
Top-right
Bottom-left
Bottom-right
Neodles US
Needies
Necdles
Needles
Needles
Noscles
95
95
40
95
95
40
95
40
PSNR IN-1K val
25.8
25.7
25.6
25.6
25.4
Figure 10: Comparison of difffernt positions compressed vector q: The quality of image reconstruction shows minimal sensitivity the position of q. Placing it the center yields slightly better results compared to comner positions. It is worth noting that each positioning has its own pre-trained model with non-shared parameters. Best viewed in color.
Figure 11: Reconstruction from random samples: The reconstructed images generated by sampling from the statistics (mean and covariance) of compressed embeddings q obtained from the ImageNet validation set, consisting of 50,000 images. Although the samples are not similar to images of Gaussian noise, they lack semantic meaning and appear as noisy images. Multiple samplings consistently yield similar noisy patterns. Best viewed in color.
20

---

Exploring Invariance in Images through One-way Wave Equations
Mean 4
Mean
0.75q
0.5q+
0.25q+0.75q
2q
Original
Seedits
SUEGIES
Nentins
Hendles
Neadite
Reconstruction from the average embedding q: The reconstructed image corresponding to the average embedding 7 computed from 50,000 ImageNet validation images closely resembles a gray image (shown in the right column of the left figure). In the left figure, we demonstrate the interpolation along a line connecting embeddings from different images to the average embedding. Notably, the reconstructed images progressively fade into a gray image. In the right figure, we extend the connection between an image embedding q and the average embedding q to a mirror embedding 2q q, corresponding to image with reversed colors. This comparison provides insights into the nature of the embedding space. Best viewed in color.
Figure 12:
0.75q +0.25q 0.5q, +0.542 0.25q:+0.75qz
Headlet
95
95
Figure 13: Reconstruction from interpolated embeddings: The images are reconstructed by interpolating embeddings of two images, aqi + (1 a2)2. Although the mixed embedding passes through non-linear network that includes FINOLA and multi-layer decoder, it leads to mixing up images output. Best viewed in color.
21

---

Exploring Invariance in Images through One-way Wave Equations
Original Images
Reconstruction with Perturbations
Perturbation level
0.0
p 0.1
p 0.2
p 0.3
p 0.4
p 0.5
p 0.6
P=
p 0.8
0.9
1-0
Figure 14: Sensitivity to small perturbations: Perturbed representations (qp) were generated by linear interpolation between an image representation (q:) and a randomly sampled noise vector (q.): qp = (1 p)q: + pqp. The interpolation parameter, p, was varied from 0 (no perturbation) 1.0 (full perturbation) increments of 0.1. The perturbed vectors (qp) were then processed by FINOLA and the decoder reconstruct the images. Best viewed in color.
of a. However, it is important to note that the resulting reconstruction may not precisely match the simple mixup of the original images, represented by aT, + +1 T2.
Combining the three observations discussed above, our findings suggest that the presence of noisy images Figure 11 indicates the mixing of multiple surrounding images. As the number of image embeddings involved in the mixing process increases, the resulting reconstructions tend to resemble a gray image, depicted in Figure 12.
Sensitivity to small perturbations: To assess FINOLA's sensitivity to perturbations, conducted an experiment where perturbations were introduced into the compressed representation space. Specifically, perturbed representations (qp) were generated by linear interpolation between an image representation (q:) and a randomly sampled noise vector (q.): q, = (1 p)q: + pqn- The interpolation parameter, p, was varied from 0 (no perturbation) to 1.0 (full perturbation) in increments of 0. The perturbed vectors (qp) were then processed by FINOLA and the decoder to reconstruct the images. The result images are shown in Figure 14, demonstrate that FINOLA exhibits robustness small perturbations (p 0.1 or p 0.2). However, the quality of the reconstruction degrades the perturbation level increases.
Principle component analysis (PCA): The reconstruction results shown in Figure 15 are obtained using PCA with the top-K principle components. These components correspond to the largest K eigenvalues of the covariance matrix computed from 50,000 image embeddings in the ImageNet validation set. The principle components capture essential information, starting with color and layout, and gradually encoding finer image details as more components are included in the reconstruction process.
22

---

Exploring Invariance in Images through One-way Wave Equations
Original
All-3072
Top-1536
Top-768
Top-384
Top-192
Figure 15: Reconstruction from top principle components: The top-K principle components correspond to the largest K cigenvalues the covariance matrix computed from 50,000 image embeddings in the ImageNet validation set. With a selection of top-192 192 components (the right column), the color and layout of the images are primarily determined, but the resulting reconstructions appear blurred with noticeable loss of details. As more principle components are incorporated, the finer details gradually restored. Best viewed in color.
Original
2048 Channels
1024 Channels
512 Channels
256 Channels
128 Channels
Image
Multiple Paths Single Path
Multiple Paths Single Path
Multiple Paths Single Path
Multiple Paths Single Path
Multiple Paths Single Path
256x256
M=1
SHIES
STICIT
95
95
Figure 16: Multiple paths vs. Single path: Summing M = 8 FINOLA solutions Φi (as in Eq. 3) yields superior image reconstruction quality compared to the single path counterpart. This trend holds across various dimensions (from C 128 to C' = 2048). Resolution of feature map is set to 64 with an image size of 256 256. Best viewed in color.
23

---

Exploring Invariance in Images through Onc-way Wave Equations
C=2048
Original
Original
C=1024
C-256
C-128
M=1
M=2
M=4
M=8
M=1
M=2
M=4
M=8
Newdles
Needles
Heedies
95
95
C=
Original
Original
4:
C=0024
40
$5
95
40
95
C=512
95
C-256
C-128
Reconstruction examples for varying numbers of channels (C) and FINOLA paths (MM): Increasing the number of paths, per Eq. 3, consistently enhances image quality across different dimensions (C 128 C 2048), affirming the relaxation of FINOLA constraints. Feature resolution (z) is 64x64, and image size is 256 Best viewed in color.
Figure 17:
B.8. Visual Comparison between Single-path and Multi-path FINOLA
Single vs. Multiple paths: Figure 16 visually demonstrates that multiple paths M exhibit markedly superior image quality compared to the single path counterpart (M = 1).
Reconstruction examples for varying number of channels and paths M: Figure 17 illustrates the reconstruc- tion examples obtained for different combinations of channel counts (or number of one-way wave equations C 128, 256, 512, 1024, 2048) and the number of FINOAL paths (M 1, 4, 8). These results correspond to the exper- iments in Figure 5, as discussed in Section 4.1.
Notably, consistent trend emerges where increasing the value of M consistently enhances image quality. This trend remains consistent across various equation counts, ranging from C 128 to 2048. This observation underscores the efficacy of relaxing the FINOLA constraint by FINOLA series, as detailed Section 3.
24

---

Exploring Invariance in Images through One-way Wave Equations
Original 256x256
Complex d EC
Real A
All-One
Repetition
E R
Aa
Table 18: Inspection of real-valued wave speeds: (a) PSNR values for image reconstruction with varying wave speeds (complex, real, all-one) on the ImageNet- IK validation set, with the symbol denoting the use of position embedding. The number of wave equations (or feature map dimension) is set C 1024, and the number of FINOLA paths is set M = 4. (b) A comparison between all-one speed waves and feature map generation through repetition with position embedding to ensure position embedding isn't the sole dominant factor.
Needies
SPECIRE
Nesties
Nandimn
Wave Speed
Dimension
PSNR
95
95
95
95
Complex ÅE C
1024x4
26.1
Real A E
1024x4
25.1
All-one λu
1$
1024x4
23.9
(a) Special cases: real and all-one speeds.
Feature Map Gen
Dimension
PSNR
Repetition
4096
21.6
All-one waves
1024x4
23.9
Figure 18: Reconstructed examples for varying wave speeds (com- plex, real, all-one).
(b) Using position embedding.
B.9. Real-valued Wave Speeds
is worth noting that the speeds of the wave equations are generally complex numbers Å E C, which is also validated in the
experiments. This arises because we do not impose constraints on the coefficient matrices (A, B) in Eq. 2. Consequently, during the diagonalization process, AB-1 VAV-1, it is highly likely that the eigenvalues and eigenvectors will be complex numbers.
Here, we introduce two interesting cases by constraining the speeds of the one- one-way equations as follows: (a) real numbers A E IR, and (b) as all equal to one X
Ac 1.
Real speed A E R: This is achieved by constraining matrices HA and Hn in Eq. 6 as real diagonal matrices:
HA d2..(01,02............ 02. .., ac)) H = (di. (e,,,,,,,,,,,,,,,.. = PHA. B = PHB-
(8)
Here, the coefficient matrices A and B in FINOLA are implemented by multiplying a real projection matrix P with diagonal matrices HA and Hn, respectively. Consequently, the speeds of the wave equations are real numbers, denoted as A = a
All-one speed A1 ... Ac = 1: By further constraining HA and Hy as identity matrices, all wave equations have identical speed d 1.
HA HA=HB=I, A-B-P, =
Here, the coefficient matrices A and B in FINOLA are also identical and denoted as P.
Experimental results for real-valued wave speeds: Table 18-(a) provides the results for real-valued and all-one wave speed, while Figure 18 displays corresponding reconstruction examples. comparison the default scenario using
25

---

Exploring Invariance in Images through One-way Wave Equations
complex-valued wave speeds, enforcing wave speeds real numbers setting them uniformly one shows slight decline in performance. Nonetheless, both real-valued speed cases still deliver reasonably good PSNR scores. Notably, the all-one wave speed configuration achieves PSNR of 23.9. This specific configuration shares the coefficient matrix for autoregression across all four directions (up, down, left, right), creating symmetry in the feature map. To account for this symmetry, we introduced position embedding before entering the decoder.
In an effort to determine whether position em- bedding the dominant factor for all-one wave speed, we conducted experiments by generating feature maps using both repetition and position embedding, with the same dimention (4096). This approach falls short of the all-one wave speed configuration by 2.3 PSNR (as detailed in Table 18-()).. Its reconstruction quality sig- nificantly lags behind that of all-one waves, as depicted in the last two columns of Figure 18.
Table 19: Position initial conditions. PSNR values for image recon- struction on the ImageNet-IK validation set is reported. Scattering of initial positions spatially boosts performance.
Position
#Paths M #Channels C PSNR t
Overlapping at Center
1024
26.1
Scattering Uniformly
1024
26.7
Overlapping at Center Scattering Uniformly
1024
27.1
1024
28.0
B.10. Scattering Initial Conditions Spatially
Overlapping at Center
16
1024
27.7
Scattering Uniformly
16
1024
29.1
To enhance reconstruction further, we can adjust spatial positions place the initial conditions, without introducing additional parameters FLOPs. This concept is straig straightforward to im- plement through multi-path FINOLA (refer to Figure 3), where different paths employ scat- tered initial positions rather than overlapped at the center. Table 19 demonstrates that further improvements reconstruction achieved by scattering the initial conditions uniformly compared to placing them the center, regardless of whether we use 4, 8 or 16 FINOLA paths.
Overlapping at Center
2048
28.0
Scattering Uniformly
2048
28.9
Overlapping at Center Scattering Uniformly
16
2048
29.1
16
2048
30.0
C. Masked FINOLA for Self-supervised Pre-training
C.1. Masked FINOLA
FINOLA can be applied to self-supervised learning through a straightforward masked prediction task, which we refer to as Masked FINOLA to distinguish it from the vanilla FINOLA. Unlike vanilla FINOLA that sup- port various resolutions of feature map, masked FINOLA performs mask prediction at resolution 1 which is con- sistent with established baselines like MAE (He et al., 2021), SimMIM (Xie et al., 2022). In this paper, we only use single path for masked FINOLA.
Table 20: Mobile-Former decoder specifications for COCO object detection: 100 object queries with dimension 256 are used. "down-conv" includes a 3x3 depthwise convolution (stride=2) and a pointwise convolution (256 channels). "up- conv" uses bilinear interpolation, followed by a 3x3 depthwise and a pointwise convolution. "M-F+" replaces the Mobile sub- block with transformer block, while "M-F-" uses the lite bottleneck (Li et al., 2021b) to replace the Mobile sub-block.
Simple block masking: FINOLA is applied through a simple masked prediction design that involves using single unmasked image block (see Figure 19) to predict the surrounding masked region. Specifically, out the unmasked block and pass it through the encoder, leveraging the power of FINOLA to generate a full-size feature map. Finally, a decoder is applied to recover the pixels in masked region. Unlike vanilla FINOLA, the reconstruction loss computed only from the masked region. Please note that the unmasked block floats around the image randomly.
Stage
MF-Dec-522
MF-Dec-211
query
100x256
100x256
crop
down-conv M-F+
down-conv M-F+
32
x5
x2
up-conv M-F
up-conv M-F
16
x2
x1
up-conv
up-conv
M-F
x2
M-F
x1
Masked FINOLA variants: Masked FINOLA comprises two variants: the element-wise approach (Masked-FINOLL-EE)
26

---

Exploring Invariance in Images through One-way Wave Equations
Element-wise Masked FINOLA
Block-wise Masked FINOLA
Corner case: single group
Edge case: groups
Middle case: groups
Unmasked R(2:10,3 10,3:11)
Unmasked (0: 8.0:8) B)
Unmasked x(3 11.11.0:) B)
(x-3, y-2) [240. y-2
11, 10)
1X90, (195
F.5. V40
B- P
(X-3
(95,0900
R18. P
X
Y40
CO VRE
(x- 546
8
240. (98)
Figure 19: Two Masked FINOLA variants: clement-wise (left) and block-wise (right) approaches. In the element-wise approach, autoregression performed similarly to vanilla FINOLA, with the compressed vector q observing only the unmasked block rather than the entire image. Conversely, the block-wise approach does not compress the unmasked block. Each unmasked position exclusively predicts three masked positions, as indicated by arrows, using Eq. 1. Assignments grouped together, with shared offsets within each group. The grouping varies depending the location of the unmasked quadrant, resulting in 1, 2, and 4 groups for corner, edge, and middle locations, respectively. Best viewed in color.
and the block-wise approach (MAsAc--FINOLL--D), as depicted in Figure 19.
The element-wise variant (Masked-FINOLA-E) operates similarly to vanilla FINOLA, with the compressed vector q only observing the unmasked block rather than the entire image (see Figure 19-1eft). To accommodate the longer training required in masked FINOLA (e.g., 1600 epochs), we follow (He al., 2021) replace the convolutional decoder with a simple linear layer, transforming C-channel token into a 16x 16x 3 image patch.
In contrast, the bioek-wise variant (Masked-FINOL.AN)) preserves the unmasked block in its entirety, without compression. It requires the unmasked block to have a quadrant size. As shown in Figure 19-right, each unmasked position is tasked with predicting three masked positions, denoted by arrows and computed using Eq. 1. These assignments are organized into groups, and within each group, all unmasked positions share common offsets for reaching their assigned masked positions. The configuration of these groups dynamically adapts based on the location of the unmasked quadrant, resulting in 1, 2, or groups for corner, edge, or middle positions, respectively. To promote communication across these groups, transformer blocks are integrated into the decoder.
Relation to MAE (He et al., 2021): Masked FINOLA shares a similar architecture with MAE but differs notably in masking and prediction strategies. Firstly, masked FINOLA adopts a regular masking design, grouping all unmasked patches into a single block, in contrast to MAE's utilization of random unmasked patches. This design choice suits efficient CNN-based networks. Secondly, masked FINOLA employs a first-order norm+linear autoregression approach for predicting the masked region, whereas MAE utilizes masked tokens within an attention model.
C.2. Implementation Details
C.2.1. DECODER ARCHITECTURES
Below, we describe (a) the decoders employed in masked FINOLA, (b) the decoders designed for image classification, and (c) the decoders tailored for object detection.
Decoders for FINOLA pre-training: Unlike vanilla FINOLA, which employs stacked upsampling and convolution blocks, the masked FINOLA variants utilize simpler architectures linear layer for transforming features into 16x 16 image patches. This choice facilitates longer training. The decoder of Masked-FINOLAA incorporates transformer blocks (without positional embedding) to enable spatial communication. Masked FINOLA undergoes training for 1600 epochs.
Decoders for ImageNet classification: We utilize three decoders to evaluate the pre-trained encoders in FINOLA. These
decoders are as follows:
27

---

Exploring Invariance in Images through Onc-way Wave Equations
Table 22: Settings for linear probing and tran-1 probing on ImageNet-IK: The encoders are frozen during both tasks.
Config
Linear probing
tran-1 probing
optimizer
SGD
A
base learning rate
0.1
0.0005
weight decay
0
0.1
batch size
4096
4096
learning rate schedule
cosine decay
cosine decay
warmup epochs
10
10
training epochs
90
200
augmentation
RandomResizeCrop
RandAug (9, 0.5)
label smoothing
dropout random erase
0.1 (MF-W720) 0.2 (MF-W1440/W2880)
(MF-W720/W 1440) 0.25 (MF-W2880)
1 li decoder:
consists of single linear layer and used for linear probing.
tran-1 decoder: It incorporates a shallower transformer decoder with a single transformer block followed by a linear classifier and is employed for t tran-1 probing and fine-tuning.
rran-4 decoder: This decoder is composed of four transformer blocks followed by linear classifier and is utilized for fine-tuning alone.
The transformer decoders designed with different widths (192, 384, 768) correspond with the three Mobile-Formed encoders, which have widths of 720, 1440, and 2880, respectively.
The decoders used in the DETR framework with Mobile-Former (Chen et al., 2022) are described in Table 20. Both decoders consist of 100 object queries with dimension of 256. While they share similar structure across three scales, they differ in terms of their depths. Since the backbone network ends at a resolution of T6 the decoder incorporates downsampling step to further reduce the resolution to 32 This enables the decoder to efficiently process the features for object detection.
Decoders for object detection:
C.2.2. TRAINING SETUP
In this section, provide detailed training setups for different tasks, including:
Table 21: Pre-training setting for masked FINOLA.
Config
Masked FINOLA
Masked FINOLA pre-training on ImageNet- 1K.
optimizer
AdamW
Linear probing on ImageNet-1K.
base lcarning rate
1.5e-4
weight decay
0.1
tran-1 probing ImageNet-1K
batch size
1024
learning rate schedule
cosine decay
Fine-tuning on ImageNet-IK.
warmup epochs
10
COCO object detection.
training epochs
1600
image size
2562
augmentation
RandomResizeCropp
Masked FINOLA pre-training: Similar the vanilla FINOLA, masked FINOLA also follows the training setup described in Table 21, but with larger batch size due to the simpler decoder architecture that requires less memory consumption.
Linear probing: our linear probing, we follow the approach described in (He et al., 2021) by incorporating an additional NarchNorm layer without affine transformation (afineeFalsee) Detailed settings can be found in Table 22.
28

---

Exploring Invariance in Images through One-way Wave Equations
Table 23: Setting for end-to-end fine-tuning on ImageNet-IK.
Config
Value
optimizer
AdamW
base learning rate
0.0005
weight decay
0.05
layer-wise Ir decay
0.90 (MF-W720/W1440) 0.85 (MF-W2880)
batch size
512
learning rate schedule
cosine decay
warmup epochs
5
training epochs
200 (MF- (720) 150 (MF-W1440) 100 (MF-W2880)
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
(MF-W720) 0.2 (MF-W1440) 0.8 (MF-W2880)
cutmix
0 (MF-W720) 0.25 (MF-W1440) 1.0 (MF-W288)
dropout
0.2
random erase
0.25
Mobile-Former-W720
Mobile-Former--1440
Mobile-Former-W2880
90
90
90
60.4
81.9
Top-1 Accuracy
60
80
70
62:0
75,5
63.6
64.9
65.5
Top 1 Accuracy
80
70
69.6
58.0
72.2
73.9
60.5
74,8
62.0
75.2
62.8
Top-1 Accuracy
60
80
70
59.0
74.5
76.6
62.4
77.9
76.3
65.0
78.7
89.5
53.8
50
13.8
46.4
18.8
50.3
50
50
tran-1 probing probing
tran-1 finetuning
40
100
200
400
800
1600
100
200
400
800
1600
40
100
200
400
800
1600
epochs (log-scale)
epochs (log-scale)
epochs (log-scale)
Figure 20: Training schedules of Masked-FINOLAS Longer training schedule provides consistent improvement for linear and tran-1 probing over different models, while fine-tuning performance is not sensitive to training schedule. Best viewed color.
tran-1 probing: The settings for tran-1 decoder probing are presented in Table 22. It is important to note that the default decoder widths 192, 384, and 768 for MF-W720, MF-W1440, and MF- N2F80, respectively.
End-to-end fine-tuning on ImageNet-1K:: The settings for the end-to-end fine-tuning of both the encoder and tran-1 decoder are presented in Table 23. The decoder weights are initialized from the tran-1 probing stage.
Decoder probing on COCO object detection: In this configuration, the backbone pre-trained on ImageNet- IK is frozen, and only the decoders are trained for 500 epochs on GPUs with images per GPU. We employ AdamW optimizer with an initial learning rate of le-4. The learning rate is decreased by factor of 10 after 400 epochs. The weight decay is Ic-4, and the dropout rate is 0.1.
Fine-tuming on COCO object detection: In this setting, both the encoder and decoder are fine-tuned. The fine-tuning process consists of an additional 200 epochs following the decoder probing stage. The initial learning rate for both the encoder and decoder is set to le-5, which decreases to le-6 after 150 epochs.
C.3. Ablation Studies
Ablation on training schedule: The impact of training schedule length on three Mobile-Former encoders is depicted in Figure 20. Notably, the accuracies of both linear and tran-1 probings demonstrate a consistent improvement as the training duration increases. Interestingly, even with a pre-training of just 100 epochs, fine-tuning with tran-1 -1 achieves commendable performance. This finding diverges from the observations MAE (He al., 2021), where longer training
29

---

Exploring Invariance in Images through One-way Wave Equations
Table 24: Ablation on the number of transformer blocks in the decoder: Evaluation is conducted on ImageNet using Mobile-Former-WV -28t as the encoder. Each transformer block consists of 512 channels. Each model is pre-trained for 800 epochs. Increasing the de- coder depth exhibits consistent improvement for linear and tran-1 probing, while lin-ttnnig performance shows limited sensitivity to decoder depth.
Table 25: Comparison with masked encoding methods on ImageNet-1K using linear probing. The baseline methods include iGPT (Chen et al., 2020a), BEiT (Bao et al., 2021), SimMIM (Xie al., 2022), MAE (He al., 2021) and MAE- Lite (Wang al., 2022). Three Mobile-Former backbones of varying widths are used. FINOLA pre-training demonstrates the ability to learn effective representations for small models. denotes our implementation.
Method
Model
Params
Top-1
#Blocks 1in tran-1 tran- -f
iGPT
iGPT-L
1362M
69.0
61.1
74.4
82.2
BEiT
VIT-B
86M
56.7
62.6
76.5
82.3
SimMIM
VÎT-B
86M
56.7
63.5
77.3
82.2
MAE
VIT-B
86M
68.0
63.8
78.0
82.3
MAE
VIT-S
22M
49.2
5
64.0
78.1
82.3
MAE-Lite
ViT-Tiny
6M
23.3
6
65.0
78.3
82.4
FINOLA
MF-W720
6M
51.3
FINOLA
MF-W 1440
14M
62.8
FINOLA
MF-W2880
28M
66.4
essential for fine-tuning improvements.
Ablation on the number of transformer blocks in the decoder: We investigate the impact of the number of transformer blocks in the decoder on FINOLA pre-training using the Mobile-Formes W2880 encoder. Each transformer block in the decoder consists of 512 channels, but does not use positional embedding. The results, shown in Table 24, demonstrate that adding more transformer blocks leads to consistent improvements in both linear and t tranl1 probing tasks. However, we observe that the performance of fine-tuning is less sensitive to changes in the decoder depth.
C.4. Comparable Performance with Established Baselines on Linear Probing
As shown in Table 25, FINOLA achieves comparable performance with well known baselines on linear probing while requiring lower FLOPs. The comparison conducted end-to-end manner (combining encoder and pre-training method). For example, we compare FINOLA+Mobile with MAE+ in the context of ImageNet classification.
C.5. Robust Task Agnostic Encoders
FINOLA provides a robust task-agnostic encoders: Pre-training with FINOLA followed by fine-tuning on Image 1K (IN-IK) consistently outperforms IN-IK supervised pre-training in both ImageNet classification and COCO object detection (see Figure 6). The gains in object detection are substantial, ranging from 6.4 AP. Remarkably, even without IN-1K fine-tuning, FINOLA pre-training alone outperforms the supervised counterpart in object detection by clear margin (3 to 4.5 AP). This highlights FINOLA's ability to encode spatial structures.
Comparisons with MoCo-v2: As shown in Table 26, FINOLA demonstrates comparable performance to MoCo-V V2 in linear probing, while surpassing MoCo-V2 in tran-1 probing that uses single transformer block as decoder for classification, IN-IK fine-tuning, object detection and segmentation. The backbone is frozen for both COCO object detection and segmentation. FINOLA's superior performance suggests it learns more effective intermediate features, contributing to more representative decoder features. Furthermore, the improved performance in object detection emphasizes FINOLA's ability encode spatial structures effectively.
These experiments demonstrate that the proposed masked FINOLA is able to learn task-agnostic representation by using simple masking design. This supports that the underling PDEs capture the intrinsic spatial structures present in images.
Comparison with the IN-1K supervised pre-training on transferring to COCO object detection: Table 27 presents the results of COCO object detection using frozen backbones. The evaluation utilizes three Mobile-Former encoders with different widths and two Mobile-Former decoders with different depths. Notably, FINOLA pre-training followed
30

---

Exploring Invariance in Images through One-way Wave Equations
Table 26: Comparisons with MoCo-v2 (Chen et al., 2020d) on ImageNet classification, COCO object detection and instance segmentation. Three Mobile-Formmr backbones with different widths are used. In tran-1, 1, the encoder frozen while a transformer block is trained as a decoder using class labels. In tran-1-ft. encoders are fine-tuned. Encoders are frozen in both COCO object detection and instance segmentation. DETR framework is used for object detection, while Mask-RCNN (1x) used for segmentation. FINOLA outperforms MoCo-V2 in most evaluations, except par in linear probing.
IN-1K Top-1
COCO Det (Box-AP)
COCO Seg (Mask-AP) w/o IN-ft
Pre-training
Encoder
1in
tran-1 tran 1-ft
w/o IN-ft
with IN-ft
with IN-ft
MoCo-V2 FINOLA
51.6
52.9
74.3
31.8
39.9
23.2
25.3
MF-W720
51.3
65.5
75.6
40.0
41.6
26.3
28.4
MoCo-V V2
60.4
58.5
79.2
30.3
39.0
25.6
25.7
MF-W1440
FINOLA
62.8
75.2
80.5
42.6
44.0
30.6
32.7
MoCo-V2 FINOLA
66.5
63.8
80.0
25.5
31.7
27.8
25.2
MF-W2880
66.4
78.7
82.5
43.3
45.5
33.3
35.1
Table 27: COCO object detection results on the va 2017 dataset using a frozen backbone pre-trained on ImageNet- IK Evaluation is conducted over three backbones and heads that use Mobile-Former (Chen et al., 2022) end-to-end in DETR (Carion al., 2020) framework. Our FINOLA consistently outperform the supervised counterpart. Notably, fine-tuning ImageNet-1K I (denoted as "IN-ft") yields further improvements. The initial "MF" (e.g., MF-Dec- 522) denotes Mobile-Former, The madds metric is based on an image size of 800x 1333.
Head
Backbone
model madds param model (G)
madds (G)
param (M)
pre-train
IN-ft AP
AP3o
AP15
APs
APM
APL
(M)
supervised
40.5
58.5
43.3
21.1
43.4
56.8
34.6
19.4
77.5
25.0
FINOLA
43.3 (+28)
61.5
46.8
23.7
46,9
60.1
W2880
FINOLA
45,5 (450)
63.8
49.5
25.1
49.1
63.5
MF
supervised
38.3
56.0
40.8
19.0
40.9
54.3
Dec
32.3
18.6
20.4
11.7
FINOLA
X
42.6
60.3
46.1
22.6
46.2
60.0
522
W1440
FINOLA
44.0.
62.3
47.3
23.8
47.6
61.0
supervised
35.2
52.1
37.6
16.9
37.2
51.7
31.1
18.2
5.6
4.9
FINOLA
40.0
57.9
42.9
20.6
43.3
56.8
#720
FINOLA
41.6 +0.4)
59.4
45.0
21.2
45.0
58.9
supervised FINOLA
34.1
51.3
36.1
15.5
36.8
50.0
15.7
9.2
77.5
25.0
X
36.7,z2.)
53.7
39.3
18.2
39.7
52.2
W2880
FINOLA
41.0
59.2
44.4
20.9
44.6
58.3
MF
supervised
31.2
47.8
32.8
13.7
32.9
46.9
Dec
13.4
8.4
20.4
11.7
FINOLA
36.0
52.7
38.7
16.6
39.1
52.5
211
W1440
FINOLA
39.2
56.9
42.0
19.7
42.8
56.2
supervised FINOLA
27.8
43.4
28.9
11.3
29.1
41.6
12.2
8.0
5.6
4.9
33.0
49.3
35.0
15.3
35.1
48.9
W720
FINOLA
35.8 +8.0)
52.6
38.3
16.4
38.3
52.0
by ImageNet-Ie IK (IN-1K) fine-tuning consistently outperforms the IN-1K supervised pre-training across all evaluations, demonstrating the effectiveness of task-agnostic encoders. Impressively, even FINOLA pre-training alone, without IN-IK fine-tuning, surpasses the supervised counterpart on object detection by a significant margin of 2.6-5.2 AP. This showcases FINOLA's ability to encode spatial structures.
C.6. Fine-tuning on COCO
Furthermore, fine-tuning the backbone on COCO further enhances detection performance. Table 28 provides a comprehensive comparison of fine-tuning results using the Mobile-Former (Chen et al., 2022) in the DETR (Carion et al., 2020) framework. Unlike the frozen backbone configuration, where FINOLA outperforms supervised pre-training significantly (as shown in

---

Exploring Invariance in Images through One-way Wave Equations
Table 28: COCO object detection results on the va12017 dataset after fine-tuning both the backbone and head on COCO. Evaluation is performed on three different backbones and two heads, utilizzng the Mobile-Former (Chen et al., 2022) end-to-end in the DETR (Carion et al., 2020) framework. Our approach, which involves FINOLA pre-training followed by ImageNet- KK fine-tuning, surpasses the performance of the supervised baselines. The initial "MF" (e.g., MF-Dec-522) denotes Fobile-Former, while "IN-ft" indicates fine-tuning ImageNet-1K K. The reported madds values based on the image size of 800 1333.
Head model madds param model madds (G)
Backbone
param (M)
pre-train IN-ft AP
APso
APts
APs
APM
APt.
(G) (M)
supervised
48.1
66.6
52.5
29.7
51.8
64.0
34.6
19.4
77.5
25.0
FINOLA
X
48.0.am
66.2
52.3
28.2
51.4
64.1
W2880
FINOLA
V
49.0 (40.))
67.7
53.4
30.1
52.9
65.5
supervised
46.2
64.4
50.1
27.1
49.8
62.4
Dec
32.3
18.6
20.4
11.7
FINOLA
X
(6.8
64.9
51.0
26.6
50.6
63.4
522
W1 410
FINOLA
(7.3
65.6
51.4
27.3
50.7
63.9
supervised
42.5
60.4
46.0
23.9
46.0
58.5
31.1
18.2
5.6
4.9
FINOLA
X
(3.3
61.0
47.0
23.1
46.6
61.0
W720
FINOLA
44.4
62.1
48.1
24.3
47.8
61.5
supervised
44.0
62.8
47.7
25.8
47.3
60.7
15.7
9.2
MF W2880
77.5
25.0
FINOLA
44.4(.0.)
62.5
48.2
24.7
47.6
60.7
FINOLA
46.0.2.
64.8
49.9
26.2
50.0
62.7
MF
supervised
42.5
60.6
46.0
23.6
45.9
57.9
Dec 211
13.4
8.4
MF
20.4
11.7
FINOLA
42.4.om
60.2
45.9
21.9
45.7
60.0
W1440
FINOLA
43.8
61.8
47.5
23.9
47.1
60.8
supervised FINOLA
37.6
55.1
40.4
18.9
40.6
53.8
MF
12.2
8.0
5.6
4.9
37.2 (-0.)
54.3
39.7
18.7
39.8
53.4
M720
FINOLA
39.3
56.7
42.4
19.4
42.1
56.5
Table 29: Comparison with DETR- -ased models on COCO detection. All baselines are fine-tuned on COCO. FINOLA- DETR utilizes Fobile-Former (2F-28800 as the backbone, which has similar FLOPs and model size to the ResNet-50 used in other methods. MAdds are calculated based image size of 800x 1333.
MAdds
Param
Model
Query
AP
APs0
AP
APs
APM
APt
(G)
(M)
DETR-DCS(Carion et al., 2020)
100
43.3
63.1
45.9
22.5
47.3
61.1
187
41
Deform-DEL (Zhu et al., 2020)
300
46.2
65.2
50.0
28.8
49.2
61.7
173
40
DAB-DETR(LiP al., 2022)
900
46.9
66.0
50.8
30.1
50.4
62.5
195
48
DN- AN-DETR(Li al., 2022a)
48.6
67.4
52.7
31.0
52.0
63.7
195
DINO(Zhang al., 2022)
900
50.9
69.0
55.3
34.6
54.1
64.6
279
47
FINOLA-DETR (frozen)
45.5
63.8
49.5
25.1
49.1
63.5
100
112
44
FINOLA-DETR (fine-tune)
49.0
67.7
53.4
30.1
52.9
65.5
Table 27), they achieve similar performance in COCO fine-tuning. This because the advantage FINOLA pre-training on spatial representation diminishes when object labels in COCO provide strong guidance. However, FINOLA maintains its leading position by leveraging fine-tuning on IN-1K to improve semantic representation and transfer to object detection. Compared to the supervised baseline, FINOLA pre-training followed by IN-IK fine-tuning achieves gain of 0.9-2.0 AP for all three encoders and two decoders.
Table 29 compares FINOLA-DETR (in which the backbone is fine-tuned in the DETR framework) with existed DETR baselines. FINOLA-DETR achieves an AP of 49.0, outperforming most DETR- based detectors except DINO (Zhang et al., 2022). Remarkably, our method achieves these results while using significantly fewer FLOPs (112G vs. 279G) and object queries (100 vs. 900). When compared to DETR-DG with fine-tuned backbone, FINOLA- -ETR with a arroeen backbone achieves a 2.2 AP improvement while reducing MAdds by 40%.
32

---

Exploring Invariance in Images through One-way Wave Equations
Table 30: Comparing FINOLA and Masked FINOLA on ImageNet-IK, Masked FINOLA variants trade restoration accuracy for enhanced semantic representation. The block-wise masked FINOLA outperforms the element-wise variant in linear probing (1in) probing with a single transformer block (t.ann1] ), and fine-tuning (t (tran-l-ft).
Model
Compress Autoregression Decoder
Recon-PSNR
1in tran-1 tran-1-ft
FINOLA
element
up+cony
25.8
17.9
46.8
81.9
Masked FINOLA-E
element
linear
16.7
54.1
67.8
82.2
Masked FINOLA-B
block
trans+lincar
17.3
66.4
78.7
82.5
Masked FINOLA-E
Masked
Original
FINOLA
FINOLA-B
Table 31: Comparison between FINOLA and Masked FINOLA on ImageNet (Deng et al., 2009) classification: Compared to masked FI- NOLA variants, FINOLA performs poorly both linear probing (lin) and probing with single transformer block (t (an-1) with clear margins. Even we search over the dimension of latent space from 64 to 3072, the gap is still large, i.e. more than 20%. Block-wise masked FINOLA (Masked-FINOLA-D) out- performs the element-wise variant (Masked- FINOLA-E), achieving higher accuracy. Please note that the encoders are frozen when perform- ing linear and tran 1 probing.
Needles
Needies
40
95
40
95
Pre-training
Dim of q
lin
tran-1
64
10.2
20.2
128
11.5
24.0
256
15.0
29.0
FINOLA
512
20.1
34.1
1024
23.0
39.6
PSNR
25.8
16.7
17.3
2048
23.2
41.1
IN-1K val
3072
17.9
46.8
Figure 21: FINOLA vs. masked FINOLA on image reconstruction: In this comparison, the encoders of the two masked FINOLA variants are frozen, and their attentional pooling and FINOLA components are fine-tuned. To ensure a fair comparison, we replace the decoders in the masked FINOLA variants with the same architecture as FINOLA, trained from scratch. When compared to vanilla FINOLA, the masked variants preserve color and shape information but exhibit loss of texture details.
Masked
512
54.1
67.8
FINOLA-E
Masked
66.4
78.7
FINOLA-B
These results showease the efficacy of FINOLA in capturing rich image representations even with more compact models, offering promising approach for efficient self-supervised learning.
D. Comparison between FINOLA and Masked FINOLA
D.I. Detailed Experimental Results
Table 30 presents a comparison between vanilla FINOLA and two masked FINOLA variants, assessing both their architectural distinctions and performance image reconstruction and classification tasks. The introduction of masking, a characteristic
33

---

Exploring Invariance in Images through One-way Wave Equations
linear probing
tran-1 probing
tran-1 fine-tuning
90
32.2
02.2
82.4
82
lop-1 Accuracy
80
top-1 Accuracy
80
766
72.9
70.3
Top-1 Accuracy
80
81.8
82.0
70
70
70
65.0
66.5
68.2
66.0
60
60
60
Block vise Masked FIN-LL
50
51.5
52.6
50
50
Clemenit wise Masiedd FINOLA
49.0
40
40
40
200
400
800
1600
200
400
800
1600
200
400
800
1600
epochs (log scale)
epochs (log-scale)
epochs (log-scale)
Comparison of element-wise and biock-wise Masked FINOLA. The evaluation includes linear probing, tran-1 probing, and tran-1 fine-tuning. Block-wise masked FINOLA consistently outperforms the element-wiss counterpart across all evaluations. Notably, the performance gap in fine-tuning is smaller compared to linear and tran- probing. Best viewed in color.
Figure 22:
of masked FINOLA, entails a trade-off between restoration accuracy and enhanced semantic representation.
Comparison of FINOLA and Masked FINOLA on ImageNet classification: Table 31 presents the results of linear and raa--1 probing applied to the vanilla FINOLA across various dimensions of the latent space. Notably, even the highest accuracy achieved by the vanilla FINOLA falls significantly behind both masked FINOLA variants (element-wise or bloek-wise). This stark difference highlights the remarkable power of masked prediction in learning semantic representations.
Comparison of FINOLA and Masked FINOLA image reconstruction: Figure 21 presents comparison of re- constructed samples obtained using FINOLA and masked FINOLA. In the case of the two masked FINOLA variants (element-wise and block-wise), the encoders are frozen, and only their attentional pooling and FINOLA components are fine-tuned. To ensure a fair comparison, we utilize the same architecture for the decoders in the masked FINOLA variants as in FINOLA, training them from scratch. The corresponding peak signal-to-noise ratio (PSNR) values on the ImageNet validation set are provided at the bottom. While the masked variants preserve color and shape information, they exhibit a loss of texture details compared the vanilla FINOLA. Notably, as demonstrated in the main paper, the masked FINOLA variants demonstrate stronger semantic representation. This comparison highlights that FINOLA and masked FINOLA adhere to the same mathematical principles (involving partial differential equations) but strike different balances between semantic representation and preserving fine details.
Comparison between two Masked FINOLA variants: Figure 22 showcases the results of linear probing, tran-1 -1 probing, and fine-tuning for two masked FINOLA variants trained with different schedules. The block-wise masked FINOLA consistently outperforms its element-wise counterpart across all evaluations. These findings demonstrate the effectiveness of directly applying FINOLA on the unmasked features predict the masked region, opposed performing compression before applying FINOLA.
D.2. Geometric Insight
Geometrically, Figure 23 illustrates masked FINOLA introduces a substantial increase in Gaussian curvature on critical feature surfaces, suggesting enhanced curvature in the latent space for capturing semanties.
D.3. Calculation of Gaussian Curvature
To compute the Gaussian curvature, we consider the feature map per channel set of W H surfaces 24(2, y) in 3D space, where x, y, and 2 denote the coordinates. At each position (x. y). the Gaussian curvature for the ttth channel can be determined using the following equation:
22 8² 0z
Fz Jisyy
Kx v)
(10)
AEL 0x
)2
+
Ozg Dyy
34

---

Exploring Invariance in Images through Onc-way Wave Equations
Gaussian curvature of critical features
250
FINOLA
Masked-FINOLAE Masked-FINOLA-8
200
150
100
50
Top-1
Top-5
Top-10
Top-20
Top-50
All
Figure 23: FINOLA vs. Masked FINOLA on Gaussian eurvature critical features. Masked FINOLA demonstrates significantly larger curvature critical features than vanilla FINOLA, highlighting the effectiveness of masked prediction in curving the latent space to capture semantics. Best viewed in color.
Gaussian curvature is computed for all channels at each grid element. Subsequently, channels within each image are sorted based on the root mean square of the peak positive curvature (K.+) and the peak negative curvature (K-) over the surface.
E. Mathematical Derivation
E.1. Normalization after Diagonalization
Below, we provide the derivation of &b; in Eq. 7.
1b (CI- - J)Vbb
(11)
V JJV
FINOLA path is described as ΛzØ; Aδ: (Eq. in the paper), where òi is normalized di i.e. Φi 9- After diagonalization of AB-1 VAV-1 the FINOLA vectors o; projected into th: V-l where each satisfies one-way wave equation.
a
We attempt to rewrite ui; in the FINOLA format as A = HAV. (similar to o; before projection A2Ø: = Add). &r is not a simple normalization. The derivation of HA and iú; is shown below step by step:
A = V- AzΦ: Ø;
(shs V- (b))
V- V- = =V-1 A O -
(A.0; = Ado)
6
Φ: Jo:
=V-1 A
(n, in matrix format, J is all one matrix)
1 10 65 02 of JΦ:
(12)
= =V A (CI-J)dr
JVV-
( (v))
VV
Thus, we have:
(CI - JJV
HA =-=V-1A
-
(13)
V"
35


Reviews:
[{"review_id": "t1uxwaeVro", "reviewer": "Reviewer_Wm1J", "summary": "This paper explores the invariance in images and proposes an encoder-decoder framework based on the first-order wave equation. It works by encoding each image into an initial condition vector and then passing it to a special decoder that transforms the first-order wave equation into a linear autoregressive process to generate high-resolution feature maps and reconstruct image pixels. This approach reveals a new perspective on image understanding and provides promising directions for further exploration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.\n\n$\\color{blue}{\\textbf{[Weakness 1]:}}$\n\n**This method requires a lot of computing resources, especially when the feature map is high resolution. Therefore, it may be limited in practical applications.**\n\nWhile we acknowledge that generating high-resolution feature maps with FINOLA is computationally demanding, our parallel implementation enables its application in practical scenarios. As demonstrated in Figure 4, the entire process—including encoding, generating a 64x64 feature map using FINOLA, and decoding—can be completed in just 2.6 seconds on a MacBook Air equipped with an Apple M2 CPU.\n\nFurthermore, we emphasize that a key contribution of this work lies in offering a novel perspective on understanding images. FINOLA transforms an image into a compressed representation ($\\mathbf{q}$) that facilitates a remarkably simple autoregressive process for image reconstruction, relying solely on local relationships that are first-order and linear after normalization.\n\nIn essence, the FINOLA encoder learns a compressed representation that effectively discards explicit spatial information while preserving essential information about the image's inherent local relationships. This allows FINOLA to reconstruct the image by exploiting these consistent local structures.\n\n---\n\n$\\color{blue}{\\textbf{[Weakness 2]:}}$\n\n**This method is only applicable to a specific type of image, that is, images with linear structures. For other types of images, this method may not be applicable.**\n\nEmpirically, we have demonstrated the versatility of this method across a wide range of image types. Additional FINOLA reconstructed images, encompassing multiple categories such as natural scenes, human portraits, facial images, animal photographs, air and land transportation, and medical images, are available at https://tinyurl.com/4fdw47xm.\n\nFurthermore, our approach exhibits good performance with scientific images, such as seismic data and tomographic (CT) scans. For instance, in the domain of CT imaging (predicting a CT image from corresponding X-ray projection data), FINOLA not only performs well for both modalities (CT projection data and CT images) but also reveals a notable finding: the corresponding pairs of CT projection data and CT images exhibit a simple linear relationship between their compressed representations ($\\mathbf{q}$). Please see more details in our reply to reviewer tmjQ (Question 1).\n\n---\n\n$\\color{blue}{\\textbf{[Weakness 3]:}}$\n\n**The explanation in the article is not detailed enough, for example, it does not go into details about how to choose the initial conditions and how to train the model. These details may affect the actual effect of this method.**\n\nThanks for the suggestion. We will add more training details. The initial condition (or compressed vector $\\mathbf{q}$) is generated by the encoder. The encoder, FINOLA parameters (matrices $\\mathbf{A}$ and $\\mathbf{B}$), and decoder are learned end to end by using $L_2$ loss between the reconstructed and original images.\n\nThank you for pointing out the need for more detailed explanations. Most of details are included in Appendix B.2. We will ensure clear cross-referencing between the main paper and Appendix B.2.\n\nTo clarify the points you raised:\n\n- Initial Condition: The initial condition, represented by the compressed vector q, is generated by the encoder network. This network maps the input image to the compressed representation.\n\n- Training: The model is trained end-to-end using backpropagation. The encoder, FINOLA parameters (matrices $\\mathbf{A}$ and $\\mathbf{B}$), and the decoder network are jointly optimized to minimize the $L_2$ loss between the reconstructed and original images. Additional details are listed in Appendix B.2.", "claims_and_evidence": "The claims in this submission are clearly and convincingly supported. The paper provides detailed experimental results and presents comparative results in the form of tables and graphs. In addition, the paper introduces new methods and techniques and tests them on multiple datasets to demonstrate their effectiveness. Therefore, it can be considered that the claims of the paper are well-founded and well-supported.", "methods_and_evaluation_criteria": "The model and evaluation criteria proposed in this paper are very reasonable for the problem and application being solved. This paper proposes a new image reconstruction framework that uses a wave equation-based method to generate high-resolution feature maps and reconstruct image pixels. At the same time, this paper also conducts extensive experimental validation of the proposed model, including comparison with other existing methods and performance analysis under different parameter settings.", "theoretical_claims": "This paper proposes a new image reconstruction framework that uses a wave equation-based approach to generate high-resolution feature maps and reconstruct image pixels. Specifically, it is experimentally demonstrated that each image has a unique set of one-way wave equation solutions, and these solutions can be uniquely determined by initial conditions. This process is further transformed into a first-order norm plus linear autoregressive process, which enables efficient image reconstruction. The theoretical claims made in this paper are verified in experiments, so its theoretical correctness can be considered to be guaranteed.", "experimental_designs_or_analyses": "The experimental design and analysis of this paper are reasonable. This paper adopts an intuitive encoder-decoder framework for image reconstruction and compares different models. In addition, this paper adjusts different hyperparameters, such as training time, number of layers, etc., to evaluate their impact on model performance. Finally, this paper also provides detailed experimental results and visualization charts to help readers better understand and evaluate the performance of the model.", "relation_to_broader_scientific_literature": "The main contribution of this paper is to reveal the invariance in images and propose an image reconstruction method based on the one-way wave equation. This discovery is closely related to existing research in the fields of image processing and computer vision. In these fields, people have been exploring how to extract useful features and structures from raw data and how to use these features to achieve various tasks such as classification, detection and segmentation. The wave equation theory proposed in this paper provides a new perspective that can help us better understand the nature and structure of images and provide new ideas and directions for future image processing and computer vision research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nA new image coding method is proposed by converting the image into a one-dimensional vector and using a special decoder to generate a high-resolution feature map and reconstruct the image pixels. This method has a high reconstruction quality.\n\nThe fact that images share a set of one-way wave equations is discovered, which provides a completely new perspective to understand images and opens up the possibility for further exploration.\n\nWeaknesses:\n\nThis method requires a lot of computing resources, especially when the feature map is high resolution. Therefore, it may be limited in practical applications.\n\nThis method is only applicable to a specific type of image, that is, images with linear structures. For other types of images, this method may not be applicable.\n\nThe explanation in the article is not detailed enough, for example, it does not go into details about how to choose the initial conditions and how to train the model. These details may affect the actual effect of this method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kSCM4k86Ra", "reviewer": "Reviewer_tmjQ", "summary": "This paper introduces an encoder-decoder framework that autoregressively reconstructs images using a first-order difference equation. The method achieves high-fidelity reconstruction, outperforms traditional encoding techniques, and is effective for self-supervised learning. The key contribution of the paper is the discovery that images share a common latent wave equation structure, offering a new perspective on image representation and reconstruction.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions, which have significantly improved the quality of our paper.\n\n$\\color{blue}{\\textbf{[Question 1]:}}$\n\n**While ImageNet is a comprehensive dataset, could the authors evaluate FINOLA on other datasets (e.g., ADE20K, or medical images) to assess generalization?**\n\nThank you for suggesting the importance of assessing generalization. To address this, we conducted a novel experiment applying FINOLA to computed tomography (CT) data.\n\n**Task:** The objective of this experiment is to predict a CT image from its corresponding CT projection data. Both CT images and CT projection data are represented as 2D images.\n\n**Dataset:** The CT dataset, comprised of brain CT scans, consists of 47,000 training samples (CT image and CT projection pairs) and 6,000 test samples. The images have a resolution of 256x256.\n\n**FINOLA Implementation:** In this implementation, the CT projection data is initially encoded into a single vector, denoted as $\\mathbf{q}_p$. Subsequently, two decoding branches are employed. The first branch utilizes FINOLA to generate a feature map from $\\mathbf{q}_p$, followed by a decoder to reconstruct the CT projection data. The second branch linearly transforms $\\mathbf{q}_p$ into a vector $\\mathbf{q}_i$, which is then processed by FINOLA and a decoder to predict the corresponding CT image. Notably, the two decoding branches share the core processing components (FINOLA and the decoder) but operate on distinct, linearly correlated compressed representations ($\\mathbf{q}_p$ and $\\mathbf{q}_i$).\n\n**Result:** The following table presents the results of the CT image prediction task. FINOLA outperforms the two baseline methods, demonstrating its capacity to generalize to a new task and dataset.\n\n|Method| Mean Absolute Error $\\downarrow$|\n|---|---|\n|InversionNet [1]|63.27|\n|SIRT[2]|45.67|\n|**FINOLA**|**31.95**|\n\n[1] InversionNet: An efficient and accurate data-driven full waveform inversion. IEEE Transactions on Computational Imaging, 2019.\n\n[2] Fast and flexible x-ray tomograph using the astra toolbox. Optics Express, 2016.\n\n---\n$\\color{blue}{\\textbf{[Question 2]:}}$\n\n**The evaluation primarily relies on PSNR, which does not fully capture perceptual quality. Have the authors considered using other metrics?**\n\n\nThe table below presents a comparison of FINOLA with the ***first stage*** (learning an autoencoder and vector quantization in the latent space) of multiple generative methods, assessed using both PSNR and Fréchet Inception Distance (FID) metrics.\n\n|Method|Latent Size| Channel|logit-laplace loss|$L_2$ loss|Perceptual loss|GAN loss|FID$\\downarrow$|PSNR$\\uparrow$|\n|---|---|---|---|---|---|---|---|---|\n| DALL-E |16x16|--|&check;||||32.0|22.8|\n| VQGAN |16x16|256|||&check;|&check;|4.98|19.9|\n| ViT-VQGAN |32x32|32|&check;|&check;|&check;|&check;|1.28|--|\n|Stable Diffustion|16x16|16|||&check;|&check;|**0.87**|24.1|\n| **FINOLA (our)**|1x1|3072||&check;|||27.8|**25.8**|\n\nWhile FINOLA achieves good performance in terms of PSNR, its results are less competitive with respect to FID. This divergence is a consequence of our deliberate choice to employ the $L_2$ loss function. We intentionally prioritized a straightforward loss function to emphasize that FINOLA's efficacy does not depend on complex reconstruction loss formulations. To further optimize FID scores, we acknowledge the potential benefits of incorporating perceptual loss functions and Generative Adversarial Network (GAN) losses, as demonstrated in models such as VQGAN, ViT_VQGAN, and Stable Diffusion.\n\nIt is crucial to reiterate that FINOLA's primary objective differs from that of a generative model. Our focus is not on image generation per se, but rather on introducing a novel perspective for understanding images. FINOLA transforms an image into a compressed representation ($\\mathbf{q}$) that enables a remarkably simple autoregressive process for image reconstruction, relying exclusively on first-order and linear (after normalization) local relationships.\n\n---\n$\\color{blue}{\\textbf{[Question 3]:}}$\n\n**Qualitative results are reported on a limited number of images, could the authors provide results on different images?**\n\nAdditional FINOLA reconstructed images, encompassing diverse categories such as natural scenes, human portraits, facial images, animal photographs, air and land transportation, and medical images, are available at\nhttps://tinyurl.com/4fdw47xm.\n\n---\n$\\color{blue}{\\textbf{[Other Comments]:}}$\n\nThanks for the helpful comments on writing and typos. We will address all these points in the final draft, including correcting the typos you noted, reducing the table sizes, moving the key results from the appendix, and improving the clarity of Figure 5.", "claims_and_evidence": "The claims are supported by the experiments presented in the paper; however, some experiments could be strengthened by adding additional results. For example, while the paper discusses parameter efficiency, it does not analyze training time or inference speed.", "methods_and_evaluation_criteria": "The proposed methods and benchmarks align well with the problem of image reconstruction, compression, and self-supervised learning. The baseline and evaluation criteria make sense for the problem at hand; however, there is a lack of perceptual metrics for image reconstruction quality. The paper primarily uses PSNR, however alternative metrics, such as SSIM or LPIPS, could provide better insight into the realism of the reconstructed images. Additionally, while ImageNet-1K is indeed a comprehensive and widely used benchmark for evaluating image reconstruction, compression, and self-supervised learning, testing on additional datasets could still provide valuable insights.", "theoretical_claims": "The paper does not provide explicit proofs for the theoretical claims made. Instead, the claims are primarily supported by experimental results, such as image reconstruction performance and comparisons with other methods like DCT, DWT, and convolutional autoencoders.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper generally appear sound. The authors evaluate the proposed method (FINOLA) on  ImageNet-1K for image reconstruction and compression tasks, using standard metrics like PSNR. They also compare FINOLA against established methods like DCT, DWT, and convolutional autoencoders. However, as mentioned previously, some potential concerns include the lack of perceptual metrics (e.g., SSIM, LPIPS) in the image reconstruction evaluations, which could better capture visual quality. Additionally, the experiments don't address training time or inference speed. Finally, while ImageNet is a solid starting point, incorporating additional datasets would help strengthen the claim of robustness across different real-world scenarios. For example, datasets like COCO or ADE20K, which include diverse scenes and object categories, might help assess the method's performance in more complex, real-world settings. Additionally, datasets with different image qualities (e.g., lower resolution or noisy images) could offer insights into how well the model handles such variations.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on prior advancements in image reconstruction, compression, and self-supervised learning by introducing a novel method for efficient image representation through multi-path networks. It extends ideas from traditional image processing techniques, improving upon them by offering better PSNR performance at smaller latent sizes. Additionally, it advances the field of self-supervised learning by showing that Masked FINOLA can compete with existing methods like MAE and SimMIM.", "essential_references_not_discussed": "I am not aware of essential references not discussed.", "other_strengths_and_weaknesses": "The paper's strengths lie in its originality and in the experiments that demonstrate strong performance in image reconstruction while maintaining parameter efficiency, representing an improvement over existing approaches. However, the paper's readability could be improved. The flow of information is sometimes dense and mixed, making it challenging to follow. Furthermore, many important results and insights are placed in the appendix; relocating some of them to the main paper would enhance clarity and accessibility. A more structured presentation with clearer transitions between key ideas would further strengthen the work. Additionally, as previously noted, a key limitation is the lack of perceptual quality evaluation, as the paper primarily relies on PSNR. Finally, experimenting with additional datasets beyond ImageNet could help validate the method’s robustness across different domains.", "other_comments_or_suggestions": "* There are some typos in the paper. For example in the abstract:\n    * \"empirically reveals\" → \"empirically reveal\" \n    * \"to reconstruct image pixels\" → \"to reconstruct the image pixels\"\n* Reduce the size of tables (e.g. 1 to 4) and move some important results from the appendix to the main text\n* It's not clear to me how to read the table in Figure 5", "questions_for_authors": "* Q1:  While ImageNet is a comprehensive dataset, could the authors evaluate FINOLA on other datasets (e.g., ADE20K, or medical images) to assess generalization? \n* Q2: The evaluation primarily relies on PSNR, which does not fully capture perceptual quality. Have the authors considered using other metrics?\n* Q3: Qualitative results are reported on a limited number of images, could the authors provide results on different images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fS6OnfKXQ9", "reviewer": "Reviewer_vqWQ", "summary": "The authors introduce a new image encoder–decoder architecture called First Order Norm + Linear Autoregression (FINOLA). They compare it with several other image representations—including the Discrete Cosine Transform, the Discrete Wavelet Transform, and various generative models—and report that FINOLA achieves favorable results. They also test their encoder on a range of additional tasks, such as image inpainting, outpainting and self-supervised learning on Imagenet-1k dataset. However, the reviewer is not fully understood few things: (1) Image invariances (2) Line 260 contradicts the proposed method (3) the effect of perturbation on first order nonlinear differential equation", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's valuable feedback.\n\n$\\color{blue}{[Question-1]:}$\n\n**Contradiction Between Lines 260 and 098: Does the assertion that $\\bf{AB}$$^{-1}$ is not diagonalizable conflict with the assumption that is diagonal (line 102 and Equation 5), especially when Figure~15 shows failing to decorrelate and yielding nearly identical outputs for both real and complex domains?**\n\nWe appreciate the opportunity to clarify this point. The confusion between Lines 260 and 098 is resolved as follows:\n\n- Line 098: This line describes an empirical observation: \"we empirically observed that $\\bf{Q=AB}$$^{-1}$ matrices are diagonalizable after multiple trials of training.\" Subsequent derivations (Line 102 and Equation 5) are based on this observation.\n\n- Line 260: This line reiterates that the observed diagonalizability of $\\bf{AB}$$^{-1}$ is a natural outcome of the training process, not an enforced constraint. Consequently, while $\\bf{AB}$$^{-1}$ is not theoretically guaranteed to be diagonalizable, it consistently exhibits this property in practice across training runs.\n\nFurthermore, we wish to clarify the interpretation of Figure 15. The figure does not depict a failure to decorrelate. Instead, it illustrates two distinct variants of diagonalizable $\\bf{AB}$$^{-1}$: one possessing complex eigenvalues and the other real eigenvalues. The complex eigenvalue variant demonstrates marginally superior performance (PSNR 26.1 vs. 25.1). The real eigenvalue case is achieved by constraining matrices $\\bf{A}$ and $\\bf{B}$ to be products of a shared real projection matrix $\\bf{P}$ and distinct real diagonal matrices $\\bf{H}$$_A$ and $\\bf{H}$$_B$, respectively (i.e., $\\bf{A=PH}$$_A$, $\\bf{B=PH}$$_B$). This configuration ensures that $\\mathbf{AB}^{-1}=\\mathbf{P(H}_A\\mathbf{H}_B^{-1})\\mathbf{P}^{-1}$, where $\\mathbf{H}_A\\mathbf{H}_B^{-1}$ is a real diagonal matrix.\n\n---\n$\\color{blue}{[Question-2]:}$\n\n**Sensitivity to Small Perturbations (Equation 1): given the reliance on non-linear transformations where even small perturbations could significantly alter the final results?**\n\nThis is an excellent point. To assess the method's sensitivity to perturbations, we conducted an experiment where perturbations were introduced into the compressed representation space. Specifically, perturbed representations ($\\bf{q}$$_p$) were generated through linear interpolation between an image's representation ($\\bf{q}$$_i$) and a randomly sampled noise vector ($\\bf{q}$$_n$):\n\n$\\mathbf{q}_p=(1-p)\\mathbf{q}_i+p\\mathbf{q}_n$\n\nThe interpolation parameter, $p$, was varied from 0 (no perturbation) to 1.0 (full perturbation) in increments of 0.1. The perturbed vectors ($\\mathbf{q}_p$) were then processed by FINOLA and the decoder to reconstruct images.\n\nThe result images, available at https://tinyurl.com/47whcsve, demonstrate that FINOLA exhibits robustness to small perturbations ($p$=0.1 or $p$=0.2). However, reconstruction quality degrades as the perturbation level increases.\n\n---\n$\\color{blue}{[Question-3]:}$\n\n**Role and Interpretability of Matrices $\\bf{A}$ and $\\bf{B}$: including potential geometric or statistical interpretations, and how might elaborating on their function enhance the reader’s understanding of the proposed approach?**\n\nThank you for this question! Understanding the meaning of matrices $\\bf{A}$ and $\\bf{B}$ is key to grasping FINOLA's core mechanism.\n\n**Definition:** Matrices $\\bf{A}$ and $\\bf{B}$ model the local structure of the latent vector space $\\bf{z}$$(x,y)$, representing horizontal and vertical spatial transformations of the feature map:\n\n* $\\Delta_x\\mathbf{z}(x,y)=\\mathbf{A}\\hat{\\mathbf{z}}(x,y)=\\mathbf{P}_A\\mathbf{\\Lambda}_A\\mathbf{P}^{-1}_A\\hat{\\mathbf{z}}(x,y)$\n\n* $\\Delta_y\\mathbf{z}(x,y)=\\mathbf{B}\\hat{\\mathbf{z}}(x,y)=\\mathbf{P}_B\\mathbf{\\Lambda}_B\\mathbf{P}^{-1}_B\\hat{\\mathbf{z}}(x,y)$\n\nwhere $\\bf{P}$$_A$ and $\\bf{P}$$_A$ are the eigenvector matrices, $\\bf{\\Lambda}$$_A$ and $\\bf{\\Lambda}$$_B$ are the corresponding diagonal eigenvalue matrices. We confirm $\\bf{A}$ and $\\bf{B}$ full rank and diagonalizable across multiple training runs. \n\n**Interpretation:** Imagine points in the latent space connected by \"springs\". Matrices $\\bf{A}$ and $\\bf{B}$ define the stiffness of these horizontal and vertical springs, while eigenvectors ($\\bf{P}$$_A$, $\\bf{P}$$_B$) indicate their directions.\n\n* Diagonalization: Diagonalizing $\\bf{A}$ and $\\bf{B}$ simplifies this, projecting latent space so each dimension (eigenvector) is an independent spring with stiffness (eigenvalue).\n\n* Horizontal and Vertical Changes: The projected horizontal change ($\\Delta_x\\bf{z}$) and vertical change ($\\Delta_y\\bf{z}$) become scaling operations, with eigenvalues from $\\bf{\\Lambda}$$_A$ and $\\bf{\\Lambda}$$_B$ determining scaling strength along each eigenvector.\n\nIn essence, $\\bf{A}$ and $\\bf{B}$ encode directional \"stretching/compressing\" factors governing local feature map changes, with eigenvalues representing the strength of these factors.", "claims_and_evidence": "The reviewer observes a significant discrepancy between the theoretical claims made in the manuscript and their implementation in the accompanying code. Below are several specific points of concern:\n\nDiagonalizability and Contradiction (Line 260 vs. Line 098):\nThe claim that 𝐴𝐵^{-1} is non-diagonalizable directly conflicts with the assumption that \\Gamam is diagonal (as stated in line 102 and Equation (5)). In Figure 15, the \\Gamma parameters fail to decorrelate, producing nearly identical outputs for both real and complex domains. This inconsistency undermines the theoretical foundation presented in the paper.\n\nImpact of Small Perturbations (Equation 1):\nThe discussion on Equation (1) omits a key consideration: introducing a small perturbation or noise could potentially disrupt the entire generation process. Given that the model incorporates non-linearities, even minimal perturbations might significantly affect the final outputs. An analysis of the robustness of the method in the presence of noise would strengthen the manuscript.\n\nInterpretability of Matrices \n𝐴 and B:\nThe role and interpretability of matrices A and B remain unclear. The paper does not offer sufficient explanation or insight into how these matrices might be understood from either a theoretical or empirical perspective. Providing additional clarity—e.g., whether have specific geometric, statistical, or functional interpretations—would be beneficial.\n\nOverall, while the paper addresses an interesting problem, the points above highlight critical gaps between the stated theory and the practical implementation. Further clarification or resolution of these issues is necessary to substantiate the claims made in the submission.", "methods_and_evaluation_criteria": "Yes, the method showcased their result on Imagenet-1k. The quantitative matrices are correct.", "theoretical_claims": "The reviewer observes significant inconsistencies between the theoretical claims and their actual implementation. Specifically:\n\n1. Contradiction Between Lines 260 and 098:\n    The manuscript asserts that $AB^{-1}$ is not diagonalizable, yet subsequently assumes that $\\\\Gamma$ is diagonal (line 102 and Equation (5)). Figure~15 further illustrates that the $\\\\Gamma$ values fail to decorrelate, yielding nearly identical outputs for both real and complex domains. This discrepancy raises questions about the validity of the stated theoretical framework.\n\n2. Sensitivity to Small Perturbations (Equation 1):}\n    The paper does not sufficiently address the potential impact of noise or minor perturbations on the generation process. Given the reliance on non-linear transformations, even small perturbations could significantly alter the final results. An in-depth robustness analysis would strengthen the manuscript.\n\n3. Role and Interpretability of Matrices $A$ and $B$:}\n    The purpose of matrices $A$ and $B$ remains unclear. The paper offers no clear explanation of how these matrices might be interpreted, either geometrically or statistically. Elucidating their function and providing interpretability would enhance the reader’s understanding of the proposed approach.", "experimental_designs_or_analyses": "Comparison with Recent Methods\nThe proposed approach is fundamentally an autoencoder, yet it does not provide any comparison with other recent techniques, such as [1]. In particular, [1, 2, 3] have been shown to achieve lower parameter counts and better FID scores than the proposed model, making a direct empirical comparison crucial for a fair evaluation.\n\nReconstruction Quality\nWhile the method appears capable of capturing lower-frequency components, it struggles to preserve high-frequency details in the reconstructed images. This is especially problematic for applications in medical imaging or scenarios involving small object generation (e.g., crowds, satellite imagery), where fine-grained details are important.\n\nRobustness to Adversarial Attacks\nIt remains unclear whether the proposed model’s performance holds up under various dataset/domain shifts. For instance, color or rotational shifts, as well as operations like cutmix, may significantly alter the model’s outputs. Demonstrating robustness across these perturbations would strengthen the paper’s contributions.\n\nEvidence of Image Invariances\nThe discussion of image invariances is not supported by sufficiently clear experiments or evidence. Additional empirical results demonstrating these invariances under different transformations or conditions would help substantiate this claim.\n\nReferences\n[1] Idempotent Generative Network, The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=XIaS66XkNA\n[2] Swapping Autoencoder for Deep Image Manipulation, NeurIPS 2020.\n[3] Concept Bottleneck Generative Models, ICLR 2024.", "relation_to_broader_scientific_literature": "The work can be viewd in line of Idempotent Generative Network, The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=XIaS66XkNA, Swapping Autoencoder for Deep Image Manipulation, NeurIPS 2020., Concept Bottleneck Generative Models, ICLR 2024. \n\nThe method by nature is an image generator.", "essential_references_not_discussed": "[1] Idempotent Generative Network, The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=XIaS66XkNA\n[2] Swapping Autoencoder for Deep Image Manipulation, NeurIPS 2020.\n[3] Concept Bottleneck Generative Models, ICLR 2024.", "other_strengths_and_weaknesses": "The paper is well written\nDataset consider in this paper is good", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Contradiction Between Lines 260 and 098:\nDoes the assertion that $AB^{-1}$ is not diagonalizable conflict with the assumption that $\\\\Gamma$ is diagonal (line 102 and Equation (5)), especially when Figure~15 shows $\\\\Gamma$ failing to decorrelate and yielding nearly identical outputs for both real and complex domains?\n\n2. Sensitivity to Small Perturbations (Equation 1):\nDoes the manuscript sufficiently address the impact of minor perturbations or noise on the generation process, given the reliance on non-linear transformations where even small perturbations could significantly alter the final results?\n\n3. Role and Interpretability of Matrices $A$ and $B$:\nIs there enough clarity about the purpose of matrices $A$ and $B$, including potential geometric or statistical interpretations, and how might elaborating on their function enhance the reader’s understanding of the proposed approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ASfmOJbuzj", "reviewer": "Reviewer_D8KQ", "summary": "This paper draws connections between recurring regression using first-order norm + linear autoregression and the discretized one-way wave equation. Through this framework the paper proposes a model to embed images and provides empirical evidence supporting the model performance for image reconstruction. Moreover, the paper provides supporting evidence that this model is more memory efficient than existing methods and provides improved speed under parallelization.\n\n## update after rebuttal\n\nIn my opinion, the rebuttal has done a great job in the following aspects:\n- The acknowledgement of overstatements and willingness to revise these statements\n- Providing additional analysis on the diagonalizability and invertibility of the matrices key to their approach.\n\nGiven the good PSNR and poor FID scores along with a lack of theoretical connections, it remains unclear if traveling waves reveal a promising new avenue for autoregression on images. However, the paper makes a good case of empirical evidence supporting this approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's valuable feedback.\n\n$\\color{blue}{[Question-1]:}$\n**Can you provide more detailed theoretical insights or formal proofs that justify the assumption that images naturally conform to a one-way wave equation in the latent feature space?**\n\nWe acknowledge the challenge of providing a rigorous theoretical proof for this assumption. Therefore, our initial approach has been to establish strong empirical support. The successful reconstruction achieved by FINOLA, relying solely on local structure (first-order and linear after normalization), serves as compelling empirical evidence.\n\nFurthermore, we have empirically validated key mathematical properties associated with the model. Specifically, across multiple training runs, we have confirmed:\n\n* The invertibility of matrices $\\bf{A}$ and $\\bf{B}$.\n* The diagonalizability of $\\mathbf{AB}^{-1}$.\n\nWe believe that these empirical observations warrant further investigation and encourage the community to pursue theoretical proofs that can provide a deeper understanding of the underlying principles.\n\n---\n$\\color{blue}{[Question-2]:}$\n\n**Beyond PSNR, have you evaluated the model using perceptually motivated metrics (e.g., SSIM, FID, or IS)?**\n\nThis question is also addressed in our response to Reviewer tmjQ's Question 2. Please refer to that response for a detailed discussion of FID results.\n\nTo summarize, while FINOLA demonstrates strong performance in terms of PSNR, its results are less competitive with respect to FID. We attribute this discrepancy to our intentional use of the $L_2$ loss function, which was prioritized for its simplicity to emphasize that FINOLA's effectiveness does not rely on complex reconstruction loss formulations. We acknowledge that incorporating perceptual loss functions and Generative Adversarial Network (GAN) losses, as employed by models such as VQGAN, ViT_VQGAN, and Stable Diffusion, could potentially enhance FID scores.\n\n---\n$\\color{blue}{[Question-3]:}$\n\n**Is it possible to report the wave speed itself? Have you considered a framework where the wave speed varies spatially or by image rather than being invariant?**\n\nThe magnitude of the wave speed (represented by the eigenvalues of $\\mathbf{AB}^{-1}$) for dimension 512 is provided at \nhttps://tinyurl.com/2r4s8xs3. This speed value ranges from 0.5 to 1.4.\n\nWe appreciate the reviewer's suggestion. While we acknowledge that reconstruction quality could potentially be enhanced by allowing the wave speed to vary spatially or across different images, this is beyond the scope of the present study. Our current investigation focuses on the scenario where wave speed remains invariant across spatial positions and images, with image variations limited to the initial condition (or compressed representation $\\bf{q}$). Exploring the incorporation of spatially or image-dependent wave speeds represents a promising avenue for future research.\n\n---\n$\\color{blue}{[Question-4]:}$\n\n**Given that the derivation relies on assumptions such as the invertibility and diagonalizability of weight matrices, how sensitive is the model’s performance to these assumptions? Specifically, have you examined the condition numbers of matrices B or Q in practice, and what impact do they have on the stability and performance of the model?**\n\nWhile the derivation relies on the assumptions of matrix invertibility and diagonalizability, these are not explicitly enforced during model training but are validated through post-training analysis. The derivations offer a mathematical interpretation of FINOLA but do not directly affect model performance.\n\n**Empirical Validation:** We have empirically validated the invertibility of matrices $\\bf{A}$ and $\\bf{B}$ across multiple training runs. Furthermore, we have confirmed the diagonalizability of $\\mathbf{AB}^{-1}$. \n\n**Condition Number:** The eigenvalue spectra of matrices $\\bf{A}$ and $\\bf{B}$, related to their condition numbers (92 and 151, respectively), are available at https://tinyurl.com/3xz2e5am. \n\n**Sensitivity Analysis:** Despite the large condition number, FINOLA is stable for the perturbations in the compressed representation space ($\\bf{q}$).  To evaluate perturbation sensitivity, we performed a controlled experiment where the compressed representation ($\\bf{q}$) was perturbed. Perturbed representations ($\\bf{q}$$_p$) were generated by linearly interpolating between the image representation ($\\bf{q}$$_i$) and and a random noise vector ($\\bf{q}$$_n$):  \n\n$\\mathbf{q}_p=(1-p)\\mathbf{q}_i+p\\mathbf{q}_n$\n\nThe perturbation level, $p$, ranged from 0 (no perturbation) to 1.0 (full perturbation) in steps of 0.1. FINOLA and the decoder then processed these perturbed representations to reconstruct images.\n\nThe result images (see https://tinyurl.com/47whcsve) show that FINOLA is robust to small perturbations ($p$=0.1 or 0.2), but reconstruction quality diminishes with increasing perturbation.", "claims_and_evidence": "The central claim of the paper is the following: *Images share a set of one-way wave equations in the latent feature space*. This is evidenced through empirical comparison of on peak signal-to-noise ratio on the ImageNet and Kodak datasets. However, **this strong statement lacks strong theoretical support.** Moreover, simple image regression models achieve high PSNR by blurring the image which appears to be the evidence illustrated in Figure 2 as well as the reconstructions in the appendix. Without additional standard metrics (FID/IS/SSIM) for comparison, these empirical results are also insufficient to support the papers central claim.\n\nIn addition, the paper makes claims that this model is computationally more efficient in both speed and memory than existing methods. This is evidenced by comparing the dimension of the latent state, number of latent parameters and the bits per pixel across several datasets. This paper makes a claim that the model is capable of providing computational speed ups using parallelization. This is evidenced by inference time comparisons on a single image.", "methods_and_evaluation_criteria": "The proposed evaluation criteria and methods make sense and are necessary for evaluating the current model. However, they are insufficient in supporting the papers claim.", "theoretical_claims": "I have checked the derivation of the one-way wave equation from the FINOLA architecture. This derivation relies on several key assumptions such as the invertibility and diagonalizability of general weight matrices. As a note, these derivations lack some small but important details such as the commutativity of $V^{-1}$ with $\\nabla_x,\\nabla_y$,. The larger issue is that this derivation demonstrates a property of the network, that it learns a fixed wave speed, rather than a property of images. This would be significantly improved if the claim were relaxed or proper theoretical propositions were provided about the dynamics of the underlying latent state.", "experimental_designs_or_analyses": "I did not verify the experimental analysis as I did not find any supplementary material provided.", "relation_to_broader_scientific_literature": "This work demonstrating a network which admits a latent traveling wave solution mirrors some related works which demonstrate the capacity to generate traveling wave solutions in the latent state of neural networks [1]. This paper offers some novelty by using the FINOLA method and discretizing solely about the spatial differences. However, the central claim that all images should admit a common spatial wave-equation with spatially invariant weights appears at odds with recent techniques which perform image segmentation with traveling waves and demonstrate that unique spatiotemporal patterns emerge for each object in an input image [2].\n\n[1] Keller, T. Anderson, et al. _TRAVELING WAVES ENCODE THE RECENT PAST AND ENHANCE SEQUENCE LEARNING_. 2024.\n\n[2] Liboni, Luisa H. B., et al. _Image Segmentation with Traveling Waves in an Exactly Solvable Recurrent Neural Network_. arXiv:2311.16943, arXiv, 28 Nov. 2023. _arXiv.org_, [https://doi.org/10.48550/arXiv.2311.16943](https://doi.org/10.48550/arXiv.2311.16943).", "essential_references_not_discussed": "The paper thoroughly explores the use and impact of autoregressive models, but under explores of the importance or impact of traveling waves in neural networks and image processing. See above for some example references.", "other_strengths_and_weaknesses": "**Strengths:** To the best of my knowledge the derivation of the one-way wave equation from FINOLA is a novel derivation. In addition there are a substantial number of tasks and results presented.\n\n**Weaknesses:** Strong central claim without appropriate theoretical support. Insufficient metrics to adequately determine the full performance of the architecture.", "other_comments_or_suggestions": "Some typos like line one of the abstract: *we empirically reveals an invariance over images*.", "questions_for_authors": "Can you provide more detailed theoretical insights or formal proofs that justify the assumption that images naturally conform to a one-way wave equation in the latent feature space?\n\nBeyond PSNR, have you evaluated the model using perceptually motivated metrics (e.g., SSIM, FID, or IS)?\n\nIs it possible to report the wave speed itself? Have you considered a framework where the wave speed varies spatially or by image rather than being invariant? \n\nGiven that the derivation relies on assumptions such as the invertibility and diagonalizability of weight matrices, how sensitive is the model’s performance to these assumptions? Specifically, have you examined the condition numbers of matrices B or Q in practice, and what impact do they have on the stability and performance of the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three reviewers out of four recommend acceptation. Despite a reminder I am not sure that the skeptical reviewer has read the authors response.

After the discussion period two reviewers have increased their score demonstrating that the rebuttal was useful in resolving the concerns that were initially raised.

In particular, the following main strengths were identified :
- this method has a high reconstruction quality,
- new perspective to understand images and opens up the possibility for further exploration,
- the approach is original
- the experiments demonstrate strong performance in image reconstruction while maintaining parameter efficiency, representing an improvement over existing approaches
- it provides empirical evidence demonstrating that the model learns a traveling wave solution common among images
Decision: Accept (poster)