Exploring Invariance in Images through One-way Wave Equations

## Abstract

Abstract In this paper, we empirically reveal an invariance over images – images share a set of one-way wave equations with latent speeds. Each image is uniquely associated with a solution to these wave equations, allowing for its reconstruction with high fidelity from an initial condition. We demonstrate it using an intuitive encoder-decoder framework where each image is encoded into its corresponding initial condition (a single vector). Subsequently, the initial condition undergoes a specialized decoder, transforming the one-way wave equations into a first-order norm+linear autoregressive process. This process propagates the initial condition along the x and y directions, generating a high-resolution feature map (up to the image resolution), followed by a few convolutional layers to reconstruct image pixels. The revealed invariance, rooted in the shared wave equations, offers a fresh perspective for comprehending images, establishing a promising avenue for further exploration.

## 1 Introduction

Autoregressive language models, as exemplified by GPT Radford et al. (; ); Brown et al. (), have achieved remarkable success in Natural Language Processing (NLP). These models generate text by predicting the probability distribution of the next word in a sequence, based on the preceding words. This success has not been confined to NLP; it has extended into Computer Vision, witnessed in the form of innovations like iGPT Chen et al. () for unsupervised learning, PixelCNN van den Oord et al. (); Salimans et al. () for image generation, and DALL-E Ramesh et al. () for text-to-image synthesis. These autoregressive approaches rely on capturing complex relationships, typically implemented using Transformer blocks, among multiple tokens, often up to the $k^{th}$ order.

In contrast, our research unveils a simpler first-order approach for image reconstruction. Particularly, our investigation reveals that, with appropriate encoding, images can undergo first-order autoregression in a linear manner following normalization, termed FINOLA (First-Order Norm+Linear Autoregression). As depicted in Figure 2, our approach begins by encoding the input image into a single vector $\bm{q}$ with $C$ channels. Subsequently, we generate the feature map $\bm{z}\in\mathbb{R}^{W\times H\times C}$ in two steps: (a) placing $\bm{q}$ at the center, i.e., $\bm{z}(\frac{W}{2},\frac{H}{2})=\bm{q}$, and (b) applying FINOLA recursively to autoregress the entire feature map $\bm{z}$ along the $x$ and $y$ axes separately as $\Delta_{x}\bm{z}=\bm{z}(x+1,y)-\bm{z}(x,y)=\bm{A}\hat{\bm{z}}(x,y)$, and $\Delta_{y}\bm{z}=\bm{z}(x,y+1)-\bm{z}(x,y)=\bm{B}\hat{\bm{z}}(x,y)$. The matrices $\bm{A}$ and $\bm{B}$ are both learnable, with dimensions $C\times C$. Here, $\hat{\bm{z}}(x,y)$ normalizes $\bm{z}(x,y)$ over $C$ channels at position $(x,y)$ by subtracting the mean $\mu_{\bm{z}}=\frac{1}{C}\sum_{k}z_{k}(x,y)$ and dividing by the standard deviation $\sigma_{\bm{z}}=\sqrt{\sum_{k}(z_{k}-\mu_{\bm{z}})^{2}/C}$. FINOLA can generate feature maps at high resolutions (e.g. $\frac{1}{4}$, $\frac{1}{2}$, or full resolution of the original image). Finally, image pixels are reconstructed using a few upsampling and convolutional layers.

Figure: Figure 2: FINOLA for image reconstruction. Each image is firstly encoded into a single vector $\bm{q}$. Then, FINOLA is applied to $\bm{q}$ to iteratively generate the feature map $\bm{z}(x,y)$ through a first-order norm+linear autoregression. Finally, a few upsampling and convolutional layers are used to reconstruct image pixels. Best viewed in color.
Refer to caption: x2.png

An intriguing aspect is that the coefficient matrices $\bm{A}$ and $\bm{B}$ (once learned from data) are invariant not only across different spatial positions $(x,y)$ within an image but also across all images. This underscores an intrinsic property in the latent feature space $\bm{z}$: the relationship between the feature $\bm{z}(x,y)$ and its rate of change $\Delta\bm{z}(x,y)$ is position invariant and image invariant.

Furthermore, we extend FINOLA to a linear difference equation $\Delta_{x}\bm{z}=\bm{Q}\Delta_{y}\bm{z}$, where $\Delta_{x}\bm{z}$ and $\Delta_{y}\bm{z}$ represent differences along the $x$ and $y$ axes, respectively. Here, $\bm{Q}$ is a $C\times C$ matrix ($\bm{Q}=\bm{A}\bm{B}^{-1}$). This generalization provides a solution space to find a more optimal solution for reconstructing images (FINOLA corresponds to a specific solution). One improved solution is achieved by aggregating a series of FINOLA solutions as $\bm{z}=\sum_{i}\bm{\phi}_{i}$ where $\Delta_{x}\bm{\phi}_{i}=\bm{A}\hat{\bm{\phi}}_{i}$, $\Delta_{y}\bm{\phi}_{i}=\bm{B}\hat{\bm{\phi}}_{i}$. Figure 3 illustrates the extension of FINOLA from a single path to multiple paths with shared parameters.

Upon inspecting multiple instances of matrix $\bm{Q}$ learned with different configurations, we empirically observed that all $\bm{Q}$ matrices are diagonalizable ($\bm{Q}=\bm{V}\bm{\Lambda}\bm{V}^{-1}$) with complex eigenvalues ($\lambda_{k}\in\mathbb{C}$).
This reveals a nice property $\Delta_{x}\bm{\zeta}=\bm{\Lambda}\Delta_{y}\bm{\zeta}$ in a new feature space $\bm{\zeta}$ which projects feature map $\bm{z}$ by the inverse of eigenvectors as $\bm{\zeta}(x,y)=\bm{V}^{-1}\bm{z}(x,y)$. Since $\bm{\Lambda}$ is a diagonal matrix, channels $\zeta_{k}$ in $\bm{\zeta}$ are decorrelated. Each channel follows $\Delta_{x}\zeta_{k}=\lambda_{k}\Delta_{y}\zeta_{k}$, which is the finite approximation of a one-way wave equation $\frac{\partial\zeta_{k}}{\partial x}=\lambda_{k}\frac{\partial\zeta_{k}}{
\partial y}$. This is illustrated in Figure 1, where $\bm{\psi}_{i}$ corresponds to the projection of a FINOLA path $\bm{\psi}_{i}=\bm{V}^{-1}\bm{\phi}_{i}$. It empirically offers an interesting insight:

> images share a set of one-way wave equations in the latent feature space, with each image corresponding to a distinct solution that can be generated from its associated initial condition.

The entire framework (encoder and FINOLA decoder) is easy to implement and learns in an end-to-end manner. Experiments on ImageNet Deng et al. () (with an image size of 256$\times$256) demonstrate promising results. We achieved a PSNR of 23.2 for image reconstruction on the validation set when employing only $C=128$ wave equations. As the number of equations increases to 2048, the reconstruction PSNR boosts to 29.1. When compared with previous encoding/decoding techniques under the same latent size, our method outperforms discrete cosine transform (DCT), discrete wavelet transform (DWT), and convolutional auto-encoder (AE). Notably, our method reconstructs the entire image (without partitioning into blocks) from a single position (center).

Figure: Figure 3: Multi-path FINOLA: The input image is encoded into $M$ vectors $\bm{q}_{1},\dots,\bm{q}_{M}$. Then the shared FINOLA is applied on each $\bm{q}_{i}$ to generate feature maps $\bm{\phi}_{i}(x,y)$, which are aggregated ($\bm{z}=\sum_{i}\bm{\phi}_{i}$) to pass through upsampling and convolution layers to reconstruct image pixels.
Refer to caption: x3.png

In addition, FINOLA can serve for self-supervised pre-training. Applying FINOLA to a single unmasked quadrant block to predict the surrounding masked region yields performance comparable to established techniques, e.g. MAE He et al. (), SimMIM Xie et al. (), but using lightweight networks like Mobile-Former Chen et al. (). The comparison of encoders trained with and without masking reveals that introducing masked prediction sacrifices restoration accuracy for enhanced semantic representation. This is accompanied by an interesting observation: masking significantly increases the Gaussian curvature on the surfaces of critical features.

In outlining our research goals, it’s crucial to emphasize that our aim isn’t state-of-the-art performance but to empirically reveal a property inherent in images: the sharing of one-way wave equations within a latent space. We hope this encourage deeper understanding of images within the research community.

## 2 First-order Norm+Linear Autoregression

In this section, we introduce a first-order norm+linear autoregressive process in the latent space, known as FINOLA, which is able to reconstruct the entire image from a single vector at the center. It unveils a position-invariant and image-invariant relationship between the feature values $\bm{z}(x,y)$ (at any $(x,y)$ position for any image) and its spatial rate of changes $\Delta_{x}\bm{z}(x,y)$ and $\Delta_{y}\bm{z}(x,y)$.

FINOLA:
FINOLA is a first-order norm+linear autoregressive process that generates a $W\times H$ feature map $\bm{z}(x,y)$ by predicting each position using only its immediate previous neighbor. As depicted in Figure 2, it places a single embedding $\bm{q}$ (generated by an encoder) at the center, i.e., $\bm{z}(\frac{W}{2},\frac{H}{2})=\bm{q}$, and recursively regresses the entire feature map using the following equations:

$$ $\begin{aligned} \bm{z}(x+1,y)&=\bm{z}(x,y)+\bm{A}\hat{\bm{z}}(x,y)\\ \bm{z}(x,y+1)&=\bm{z}(x,y)+\bm{B}\hat{\bm{z}}(x,y)\end{aligned}\qquad where \qquad\begin{aligned} \hat{\bm{z}}(x,y)=\frac{\bm{z}(x,y)-\mu_{\bm{z}}}{\sigma _{\bm{z}}}.\end{aligned}$ (1) $$

The matrices $\bm{A}$ and $\bm{B}$ are learnable with dimensions $C\times C$. $\hat{\bm{z}}(x,y)$ is the normalized $\bm{z}(x,y)$ over $C$ channels at position $(x,y)$:
the mean $\mu_{\bm{z}}=\frac{1}{C}\sum_{k}z_{k}(x,y)$ and the standard deviation $\sigma_{\bm{z}}=\sqrt{\sum_{k}(z_{k}-\mu_{\bm{z}})^{2}/C}$ are computed per position $(x,y)$ over $C$ channels. Due to the normalization, this process is a first-order non-linear process.

Eq. 1 provides a solution to predict towards the right and down (assuming the $y$ axis points down). For predicting towards the left and up (with negative values of offset), we introduce two additional learnable matrices, $\bm{A}_{-}$ and $\bm{B}_{-}$, to perform predictions in the same manner as for right and down directions. Specifically, prediction toward the left is expressed as $\bm{z}(x-1,y)=\bm{z}(x,y)+\bm{A}_{-}\hat{\bm{z}}(x,y)$. For brevity, we omit $\bm{A}_{-}$ and $\bm{B}_{-}$ in the rest of the paper.

Finally, image pixels are reconstructed by passing the feature map $\bm{z}$ through upsampling and convolutional layers, as depicted in Figure 2. Remarkably, FINOLA exhibits the ability to generate the feature map $\bm{z}$ at high resolutions, including $\frac{1}{4}$, $\frac{1}{2}$, or full resolution of the original image. In the most extreme scenario, where the feature map matches the resolution of the original image, merely three 3$\times$3 convolutional layers are required to generate the image pixels.

The entire FINOLA framework, comprising the encoder, FINOLA, and the subsequent upsampling/convolutional layers, can be trained in an end-to-end manner. This is achieved by minimizing the $L_{2}$ distance between the original and the reconstructed images as the training loss.

Position and image invariance: Note that the matrices $\bm{A}$ and $\bm{B}$, once learned from data, remain invariant not only across spatial positions $(x,y)$ per image but also across images. They capture the consistent relationship between the feature values $\bm{z}(x,y)$ and their spatial derivatives ($\Delta_{x}\bm{z}$, $\Delta_{y}\bm{z}$).

Figure: Figure 4: Parallel implementation of FINOLA: Horizontal and vertical regressions are separated. The top approach performs horizontal regression first, enabling parallel vertical regression. Similarly, the bottom approach starts with vertical regression, enabling parallel horizontal regression. The results of these approaches are averaged, corresponding to the two autoregression paths from the initial position marked by $\bm{q}$. Best viewed in color.
Refer to caption: x4.png

Parallel implementation: Autoregression can be computationally intensive due to its sequential nature. FINOLA mitigates this by capitalizing on the independence of the $x$ and $y$ axes, enabling parallel execution, significantly boosting efficiency. As shown in Figure 4, performing horizontal regression first allows for parallel execution of subsequent vertical regression, and vice versa. In practice, both approaches (horizontal first and vertical first) are combined by averaging their results. The prediction at each position represents the average of the two autoregression paths originating from the initial position, marked as $\bm{q}$. Figure 4 (table on the right) demonstrates the superior speed of the parallel implementation, compared to the regular AR setting. It achieves a 30% speedup at a resolution of 16$\times$16 and a threefold increase in speed at a higher resolution of 64$\times$64.

Importance of Norm+Linear:
In Section 4.1, experiments support the significance of Norm+Linear by showing that (a) simpler processes such as repetition or linear without normalization lead to significant degradation, (b) per-sample normalization is crucial, as seen in poor performance of Batch-Norm during validation, and (c) the gain from more complex non-linear models (e.g. MLP) is negligible.

## 3 Generalization to One-way Wave Equations

In this section, we illustrate the generalization from FINOLA to a set of one-way wave equations, empirically offering a deeper insight into the inherent nature of images.

Linear partial difference equations:
Let’s denote the spatial increments of the feature $\bm{z}$ along $x$ and $y$ axes as $\Delta_{x}\bm{z}=\bm{z}(x+1,y)-\bm{z}(x,y)$ and $\Delta_{y}\bm{z}=\bm{z}(x,y+1)-\bm{z}(x,y)$, respectively. Then, the generalized form of FINOLA (Eq. 1) can be expressed as linear partial difference equations:

$$ $\displaystyle\Delta_{x}\bm{z}=\bm{A}\bm{B}^{-1}\Delta_{y}\bm{z}=\bm{Q}\Delta_{ y}\bm{z}\;\;s.t.\;\bm{Q}=\bm{A}\bm{B}^{-1}.$ (2) $$

Here, the horizontal change $\Delta_{x}\bm{z}$ exhibits a linear correlation with its vertical counterpart $\Delta_{y}\bm{z}$. When the matrix $\bm{B}$ is invertible, FINOLA stands as a special solution to this equation, given that $\Delta_{x}\bm{z}$ and $\Delta_{y}\bm{z}$ not only exhibit linear correlation but are also linearly correlated with the normalization of the current feature values $\hat{\bm{z}}$ (referred to as the FINOLA constraint). It’s noteworthy that despite the absence of specific regulations during training, the learned matrices $\bm{A}$ and $\bm{B}$ are empirically found to be invertible across various dimensions, ranging from 128$\times$128 to 4096$\times$4096.

Relaxing the FINOLA constraint through FINOLA series:
FINOLA represents a specific solution to Eq. 2, but it may not be the optimal one. We have discovered that a more optimal solution can be attained by relaxing the FINOLA constraint ($\Delta_{x}\bm{z}=\bm{A}\hat{\bm{z}}$, $\Delta_{y}\bm{z}=\bm{B}\hat{\bm{z}}$) through aggregating a series of FINOLA solutions:

$$ $\displaystyle\bm{z}(x,y)=\sum_{i=1}^{M}\bm{\phi}_{i}(x,y)\;\;s.t.\;\Delta_{x} \bm{\phi}_{i}=\bm{A}\hat{\bm{\phi}}_{i},\Delta_{y}\bm{\phi}_{i}=\bm{B}\hat{\bm {\phi}}_{i},$ (3) $$

where all FINOLA solutions $\{\bm{\phi}_{i}\}$ share the matrices $\bm{A}$ and $\bm{B}$. The resulting feature map $\bm{z}$ satisfies $\Delta_{x}\bm{z}=\bm{Q}\Delta_{y}\bm{z}$ (Eq. 2), but it no longer adheres to the FINOLA constraint ($\Delta_{x}\bm{z}\neq\bm{A}\hat{\bm{z}}$, $\Delta_{y}\bm{z}\neq\bm{B}\hat{\bm{z}}$).
Notably, the vanilla FINOLA corresponds to a special case $M=1$.

This approach can be implemented by expanding FINOLA from a single path to multiple paths. As illustrated in Figure 3, an image undergoes encoding into $M$ vectors, with each vector subjected to the FINOLA process. Each path corresponds to a special solution $\bm{\phi}_{i}$ in Eq. 3. Subsequently, the resulting feature maps are aggregated to reconstruct the original image. Importantly, all these paths share the same set of parameters. Our experiments have validated the effectiveness of this approach, showing that the reconstruction PSNR improves as the number of paths increases.

One-way wave equations after diagonalization:
Empirically, we consistently observed that the learned matrix $\bm{Q}$ is diagonalizable ($\bm{Q}=\bm{V}\bm{\Lambda}\bm{V}^{-1}$) across various training configurations. As a result, channels become decorrelated when projecting the feature map $\bm{z}$ by the inverse of eigenvectors: $\bm{\zeta}(x,y)=\bm{V}^{-1}\bm{z}(x,y)$, which modifies Eq. 2 to:

$$ $\displaystyle\Delta_{x}\bm{\zeta}=\bm{\Lambda}\Delta_{y}\bm{\zeta},\;where\; \bm{\Lambda}=\text{diag}(\lambda_{1},\lambda_{2},\dots,\lambda_{C}).$ (4) $$

where channels in $\bm{\zeta}$ are decorrelated. Each channel $\zeta_{k}$ follows an independent linear partial difference equation $\Delta_{x}\zeta_{k}=\lambda_{k}\Delta_{y}\zeta_{k}$. It is a finite approximation of a one-way wave equation (or transportation equation) as follows:

$$ $\displaystyle\frac{\partial\zeta_{k}}{\partial x}=\lambda_{k}\frac{\partial \zeta_{k}}{\partial y},$ (5) $$

where $\lambda_{k}$ is the $k^{th}$ eigenvalue in $\bm{\Lambda}$. For each channel $\zeta_{k}$, the rate of change along the $x$-axis is $\lambda_{k}$ times the rate of change along the $y$-axis. Its solution takes the form $\mathcal{F}_{k}(\lambda_{k}x+y)$, where $\mathcal{F}_{k}(\cdot)$ can be any differentiable function. Typically, a one-way wave equation involves time $t$ as $\frac{\partial u}{\partial x}=c\frac{\partial u}{\partial t}$; here, we replace $t$ with $y$.

Key insight:
The amalgamation of Eqs. 1, 3, and 5 empirically reveals an insight into understanding images: images share a set of one-way wave equations in the latent feature space. Each image corresponds to a distinct solution that can be generated from its associated initial condition, as illustrated in Figure 1.

Both FINOLA solution and initial condition can be easily transformed to the new feature space $\bm{\zeta}$. The transformed initial condition is $\bm{z}(\frac{W}{2},\frac{H}{2})=\bm{q}$. The FINOLA solution in Eq. 3 is transformed as follows:

$$ $\displaystyle\bm{\zeta}(x,y)=\sum_{i=1}^{M}\bm{\psi}_{i}(x,y),\;\Delta_{x}\bm{ \psi}_{i}=\bm{H}_{A}\hat{\bm{\psi}}_{i},\;\Delta_{y}\bm{\psi}_{i}=\bm{H}_{B} \hat{\bm{\psi}}_{i},$ (6) $$

where the transformed FINOLA series $\bm{\psi}_{i}$ and matrices $\bm{H}_{A}$ and $\bm{H}_{B}$ are computed by multiplying the inverse of eigenvectors $\bm{V}^{-1}$ before $\bm{\phi}_{i}$, $\bm{A}$, and $\bm{B}$, respectively:

$$ $\displaystyle\bm{\psi}_{i}=\bm{V}^{-1}\bm{\phi}_{i},\;\;\bm{H}_{A}=\bm{V}^{-1} \bm{A},\;\;\bm{H}_{B}=\bm{V}^{-1}\bm{B},\;\;\hat{\bm{\psi}}_{i}=\frac{(C\bm{I} -\bm{J})\bm{V\psi}_{i}}{\sqrt{\bm{\psi}_{i}^{T}\bm{V}^{T}(C\bm{I}-\bm{J})\bm{V }\bm{\psi}_{i}}}.$ (7) $$

where $C$ represents the number of channels $\bm{\psi}_{i}(x,y)\in\mathbb{C}^{C}$, $\bm{I}$ and $\bm{J}$ are the identity and all-ones matrices respectively. Unlike the normalization of $\hat{\bm{\phi}}_{i}$ in Eq. 3, which simply divides the standard deviation after subtracting the mean, the derivation of normalization $\hat{\bm{\psi}}_{i}$ is shown in Appendix D.1.

Implementation clarification:
We clarify that the generalization to one-way wave equation does not guide training, but reveals an insight through post-training processing. Specifically, wave speeds, denoted as $\bm{\Lambda}$, are not explicitly learned during training. Instead, they are computed post-training by diagonalizing trainable matrices $\bm{A}$ and $\bm{B}$ as $\bm{AB}^{-1}=\bm{V}\bm{\Lambda}\bm{V}^{-1}$. Examination of the eigenvalues in $\bm{\Lambda}$ and eigenvectors in $\bm{V}$ across various trained models confirms their complex nature ($\bm{\Lambda},\bm{V}\in\mathbb{C}^{C\times C}$). Please refer to Appendix A.8 for enforcing real-valued wave speed.

Additionally, it’s noteworthy that the diagonalizability of $\bm{A}\bm{B}^{-1}$ is not guaranteed since matrices $\bm{A}$ and $\bm{B}$ are learned from training loss without imposed constraints. However, in practice, our experiments indicate that non-diagonalizable matrices rarely occur. This observation suggests that the set of matrices resistant to diagonalization is sufficiently small through the learning process.

## 4 Experiments on Image Reconstruction

We evaluate our FINOLA (single and multiple paths) for image reconstruction on ImageNet-1K Deng et al. (). The default image size is 256$\times$256. Our models are trained on the training set and subsequently evaluated on the validation set. Please refer to Appendix A.2 for model and training details, and Appendix A.5–A.9 for additional ablations, experimental results and visualization.

### 4.1 Main Properties

FINOLA across various resolutions:
Table 2 shows consistent PSNR scores across various feature map resolutions for both single-path and multi-path FINOLA. Minor performance reduction occurs at 128$\times$128 and 256$\times$256 due to smaller decoders (1.7M and 1.2M parameters, respectively). Notably, at resolution 256$\times$256, FINOLA is followed by only three 3x3 convolutional layers, covers a 7-pixel field of view (see Table 12 in Appendix A.2).

Norm+Linear: Table 2 underscores the irreplaceability of norm+linear, as simpler alternatives like repetition exhibit significantly lower PSNR, and a linear model without normalization fails to converge at higher resolutions (64$\times$64). Additionally, Table 4 demonstrates that replacing the linear component with a more complex 2-layer MLP yields negligible gain. Moreover, Table 4 emphasizes the important role of layer normalization, with a substantial drop in validation observed for batch normalization. These findings collectively establish that norm+linear is necessary and sufficient.

**Table 1: Reconstruction PSNR across various resolutions. Performance drops slightly at higher resolutions which have significant fewer parameters in the following upsampling and convolution layers.**
| Resolution | upsample/conv | Single-path | Multi-path |
| --- | --- | --- | --- |
| #Params | 1$\times$3072 | 4$\times$1024 |  |
| 8$\times$8 | 25.3M | 25.4 | 25.9 |
| 16$\times$16 | 18.5M | 25.8 | 26.2 |
| 32$\times$32 | 9.6M | 25.8 | 26.2 |
| 64$\times$64 | 7.9M | 25.7 | 26.1 |
| 128$\times$128 | 1.7M | 25.3 | 25.4 |
| 256$\times$256 | 1.2M | 24.6 | 24.8 |

**Table 3: Comparison with norm+nonlinear. PSNR values for image reconstruction are reported. The norm+nonlinear baseline replaces the linear model in FINOLA with two MLP layers incorporating GELU activation in between.**
| Autoregression | $C$=512 | $C$=1024 | $C$=3072 |
| --- | --- | --- | --- |
| Norm+Nonlinear | 22.4 | 23.8 | 25.8 |
| Norm+Linear | 22.2 | 23.7 | 25.8 |

Multi-path FINOLA:
Figure 5 shows the PSNR values for multi-path FINOLA. Increasing the number of paths $M$ consistently improves PSNR. Visual comparisons in Figure 13 at Appendix A.7 emphasize the notably enhanced image quality from single-path to multi-path FINOLA, showcasing its ability to find superior solutions within the wave equation solution space. However, multi-path also increases the latent size of initial conditions ($\sum|\bm{q}_{i}|=MC$). The right side of Figure 5 demonstrates a consistent PSNR along the same latent size line. This suggests that reconstruction quality is influenced not solely by the number of wave equations $C$ or the number of FINOLA paths $M$ but by their product $MC$ (the latent size). This finding enables parameter efficiency in matrices $\bm{A}$ and $\bm{B}$ by decreasing the number of channels and increasing the number of paths, which is important for large latent size. For instance, at a latent size of 16,384, single path requires 268 million parameters in matrices $\bm{A}$ and $\bm{B}$, whereas aggregating 16 FINOLA paths incurs only 1 million parameters.

Image distribution in $\bm{q}$ space:
We made three intriguing observations about how images are distributed in the space of the compressed vector $\bm{q}$: (a) the reconstruction from the averaged $\bm{\bar{q}}$ over 50k validation images results in a gray image (Figure 10 in Appendix A.6), (b) the space is predominantly occupied by noisy images (Figure 9 in Appendix A.6), and (c) the reconstruction from an interpolation between two embeddings, $\alpha\bm{q}_{1}+(1-\alpha)\bm{q}_{2}$, yields a mix-up of corresponding images (Figure 11 in Appendix A.6).

Figure: Figure 5: Reconstruction PSNR for multi-path FINOLA. The generated feature map has a resolution of 64$\times$64, and the image size is 256$\times$256. Increasing the number of paths $M$, as defined in Eq. 3, consistently enhances reconstruction PSNR across various dimensions ($C=128$ to $C=2048$). The blue lines in the right table represent contour lines of the latent size (equal to $MC$). PSNR remains consistent along each latent size line. Best viewed in color.
Refer to caption: x5.png

### 4.2 Comparison with Previous Techniques

We compare multi-path FINOLA with widely recognized encoding/decoding methods, such as discrete cosine transform, discrete wavelet transform, and auto-encoders. The comparison is based on a similar number of latent coefficients.

**Table 5: Comparison with discrete cosine transform (DCT). PSNR values for image reconstruction are reported on the ImageNet-1K validation set. $(2048\times 16)$ indicates $C=2048$ channels and $M=16$ FINOLA paths. ^† denotes using multiple initial conditions $\bm{q}_{i}$ at different positions instead of overlapping at the center (see Appendix A.9).**
| Method | Latent $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- |
| DCT (top-left 1) | 3072 | 20.6 |
| FINOLA (multi-path) | 2048 (1024$\times$2) | 24.8 |
| DCT (top-left 3) | 9216 | 23.5 |
| FINOLA (multi-path) | 8192 (1024$\times$8) | 27.1 |
| DCT (top-left 6) | 18432 | 25.6 |
| FINOLA (multi-path) | 16384 (2048$\times$8) | 28.0 |
| FINOLA (multi-path)^† | 16384 (2048$\times$8) | 28.9 |
| DCT (top-left 10) | 30720 | 27.5 |
| FINOLA (multi-path) | 32768 (2048$\times$16) | 29.1 |
| FINOLA (multi-path)^† | 32768 (2048$\times$16) | 30.0 |

**Table 7: Comparison with convolutional auto-encoder (Conv-AE). FINOLA (multi-path) achieves a higher PSNR compared to Conv-AE with the same latent size, while using significantly fewer parameters in the decoder. Both methods employ the same Mobile-Former encoder, and the same upsampling/convolution layers after the feature map $\bm{z}$ is generated at resolution 64$\times$64.**
| Method | Latent | Param$\downarrow$ | PSNR$\uparrow$ |
| --- | --- | --- | --- |
| Conv-AE | 2048 | 35.9M | 24.6 |
| FINOLA | 2048 (1024$\times$2) | 16.6M | 24.8 |
| Conv-AE | 8192 | 61.9M | 26.0 |
| FINOLA | 8192(1024$\times$8) | 16.6M | 27.1 |

Comparison with discrete cosine transform (DCT) Ahmed et al. (): Table 6 compares FINOLA with DCT. DCT is conducted per 8$\times$8 image block, and the top-left $K$ coefficients (in zig-zag manner) are kept, while the rest are set to zero. We choose four $K$ values (1, 3, 6, 10) for comparison. Clearly, multi-path FINOLA achieves a higher PSNR with a similar latent size.

Comparison with discrete wavelet transform (DWT/DTCWT) Strang (); Daubechies (); Vetterli & Kovacevic ():
We compare FINOLA with DWT and DTCWT in Table 6. Three scales are chosen for wavelet decomposition. The comparisons are organized into three groups: (a) using only the LL subband at the coarsest scale (scale 3), (b) using all subbands (LL, LH, HL, HH) at the coarsest level, and (c) using all subbands at the finer scale (scale 2). Our method outperforms DWT and DTCWT in terms of PSNR for the first two groups, achieving at a smaller latent size. In the last group, while FINOLA’s PSNR is lower than DTCWT, its latent size is significantly smaller (more than 6 times smaller).

Comparison with convolutional auto-encoder Masci et al. (); Ronneberger et al. (); Rombach et al. (): Table 8 presents a comparison between our method and convolutional autoencoder (Conv-AE) concerning image reconstruction, measured by PSNR. Both approaches share the same Mobile-Former Chen et al. () encoder and have identical latent sizes (2048 or 8192). In our method, multi-path FINOLA is initially employed to generate a 64$\times$64 feature map,

**Table 9: Comparison with previous self-supervised methods on ImageNet-1K fine-tuning. The baseline methods includes MoCo-v3 Chen et al. (), MAE-Lite Wang et al. (), UMMAE Li et al. (), MAE He et al. (), and SimMIM Xie et al. (). Three Mobile-Former backbones of varying widths are used, followed by a decoder with 4 transformer blocks.**
| Method | Model | MAdds$\downarrow$ | #Params$\downarrow$ | Top-1$\uparrow$ |
| --- | --- | --- | --- | --- |
| MoCo-v3 | ViT-Tiny | 1.2G | 6M | 76.8 |
| MAE-Lite | ViT-Tiny | 1.2G | 6M | 78.0 |
| FINOLA | MF-W720 | 0.7G | 7M | 78.4 |
| MoCo-v3 | ViT-S | 4.6G | 22M | 81.4 |
| UM-MAE | Swin-T | 4.5G | 29M | 82.0 |
| MAE-Lite | ViT-S | 4.6G | 22M | 82.1 |
| SimMIM | Swin-T | 4.5G | 29M | 82.2 |
| FINOLA | MF-W1440 | 2.6G | 20M | 82.2 |
| MoCo-v3 | ViT-B | 16.8G | 86M | 83.2 |
| MAE | ViT-B | 16.8G | 86M | 83.6 |
| SimMIM | ViT-B | 16.8G | 86M | 83.8 |
| SimMIM | Swin-B | 15.4G | 88M | 84.0 |
| FINOLA | MF-W2880 | 9.9G | 57M | 83.9 |

followed by upsampling+convolution to reconstruct an image with size 256$\times$256. On the other hand, Conv-AE employs a deeper decoder that utilizes upsampling+convolution from the latent vector to reconstruct an image. Please see Table 14 in Appendix A.3 for details in architecture comparison. Our method has significantly fewer parameters in the decoder. The results highlight the superior performance of our method over Conv-AE, indicating that a single-layer FINOLA is more effective than a multi-layer upsampling+convolution approach. The comparison with auto-encoding (first stage) in generative models (e.g. Stable Diffusion Rombach et al. ()) is shown in Table 15 in Appendix A.4.

### 4.3 Comparison with JPEG on Image Compression

In Table 8, we compare FINOLA (single path with 3072 channels) with JPEG for image compression. Remarkably, by employing only uniform quantization per channel without further coding of the quantized bits, FINOLA achieves higher PSNR values with lower bits per pixel on both the ImageNet and Kodak Company () datasets.

## 5 Application on Self-Supervised Learning

FINOLA can be applied to self-supervised learning through a straightforward masked prediction task, which we refer to as Masked FINOLA to distinguish it from the vanilla FINOLA. Please refer to Appendix B for details of masked prediction, network structure, training setup, and additional experiments. Our key findings include:

Comparable performance:
Masked FINOLA demonstrates comparable performance to established baselines, e.g. MAE He et al. () and SimMIM Xie et al. (), on ImageNet fine-tuning (see Table 9), as well as linear probing (see Table 24 in Appendix B.4), while maintaining lower computational requirements.

Robust task-agnostic encoders:
Pre-training with Masked FINOLA, followed by fine-tuning on ImageNet-1K (IN-1K), yields a robust encoder applicable to various tasks such as image classification, object detection, and segmentation (shown in Table 25 in Appendix B.5). Notably, the encoder is frozen without fine-tuning on detection and segmentation tasks. Please refer to Appendix B.5 for additional experimental results.

FINOLA vs. Masked FINOLA:
Table 29 in Appendix C.1 compares vanilla FINOLA and two masked FINOLA variants in image reconstruction and linear probing. The introduction of masking in masked FINOLA trades restoration accuracy for improved semantic representation. Geometrically, Figure 21 in Appendix C.2 illustrates masked FINOLA introduces a substantial increase in Gaussian curvature on critical feature surfaces, suggesting enhanced curvature in the latent space for capturing semantics. Computation details of Gaussian curvature are available in Appendix C.3. and additional comparisons can be found in Appendix C.1.

## 6 Related Work

Image autoregression: Autoregression has played a pivotal role in generating high-quality images van den Oord et al. (; ); Salimans et al. (); Chen et al. (). These methods model conditional probability distributions of current pixels based on previously generated ones, evolving from pixel-level focus to latent space modeling using vector quantization van den Oord et al. (); Razavi et al. (); Esser et al. (); Yu et al. (). In contrast, we present a first-order norm+linear autoregression to generate feature map and reveals new insights by generalizing FINOLA as a set of one-way wave equations.

Image transforms: The Discrete Cosine Transform (DCT) Ahmed et al. () and Wavelet Transform Strang (); Daubechies (); Vetterli & Kovacevic () are widely recognized signal processing techniques for image compression.
Both DCT and wavelet transforms project images into a complete space consisting of known wave functions, in which each image has compact coefficients, i.e., most coefficients are close to zero. In contrast, our method offers a distinct mathematical perspective for representing images. It encodes images into a compact space represented by a set of one-wave equations with learnable speeds, with each image corresponding to a unique initial condition. These differences are summarized in Table 10 at Appendix A.1.

Self-supervised learning:
Contrastive methods Becker & Hinton (); Hadsell et al. (); van den Oord et al. (); Wu et al. (); He et al. (); Chen & He (); Caron et al. () achieve significant progress. They are most applied to Siamese architectures Chen et al. (); He et al. (); Chen et al. (; ) to contrast image similarity and dissimilarity and rely on data augmentation. Chen & He (); Grill et al. () remove dissimilarity between negative samples by handling collapse carefully. Chen et al. (); Li et al. () show pre-trained models work well for semi-supervised learning and few-shot transfer.
Masked image modeling (MIM) is inspired by BERT Devlin et al. () and ViT Dosovitskiy et al. () to learn representation via masked prediction. BEiT Bao et al. () and PeCo Dong et al. () predict on tokens, MaskFeat Wei et al. () predicts on HOG, and MAE He et al. () reconstructs original pixels. Recent works explore combining MIM and contrastive learning Zhou et al. (); Dong et al. (); Huang et al. (); Tao et al. (); Assran et al. (); Jiang et al. () or techniques suitable for ConvNets Gao et al. (); Jing et al. (); Fang et al. (). Different from using random masking in these works, FINOLA uses regular masking and simpler norm+linear prediction.

## 7 Limitations

The major limitation of our method is that the invariance (encoded in matrices $\bm{A}$ and $\bm{B}$) is revealed empirically without theoretical proof. Additionally, this paper focuses on multi-path FINOLA, which represents only a subspace of the solutions to the one-way wave equations. In future work, we plan to explore the theoretical analysis of the revealed invariance and the complete solution space of the one-way wave equations.

## 8 Conclusion

In this paper, we have revealed a fundamental mathematical invariance present in images through the lens of one-way wave equations. All images share a common set of one-way wave equations characterized by learnable speeds, each uniquely tied to a specific solution associated with an initial condition. The entire process is seamlessly implemented within an encoder-decoder framework, wherein the wave equations undergo transformation into a first-order norm+linear autoregressive process. Our proposed method excels in image reconstruction and shows promising potential in self-supervised learning, offering a distinctive mathematical perspective on the inherent nature of images. Looking ahead, future investigations exploring non-FINOLA based solutions to these wave equations hold the promise of delving even deeper into this intriguing realm.

## Appendix A FINOLA for Image Reconstruction

In this section, we list implementation details and additional experimental results of FINOLA (single or multiple paths).

### A.1 Conceptual Comparison with DCT/Wavelet Transforms

Both DCT and wavelet transforms project images into a complete space consisting of known wave functions, in which each image has compact coefficients, i.e., most coefficients are close to zero. In contrast, our method offers a distinct mathematical perspective for representing images. It encodes images into a compact space represented by a set of one-way wave equations with learnable speeds. Each image corresponds to a unique initial condition. These differences are summarized in Table 10.

**Table 10: Comparison between DCT/Wavelet transform and FINOLA.**
|  | DCT or Wavelet Transform | FINOLA |
| --- | --- | --- |
| Representation | Cosine/Wavelet functions | One-way wave equations |
| Parameters | Fixed parameters | Learnable speeds |
| Encoding | Image $\rightarrow$ coefficients | Image $\rightarrow$ initial conditions |
| Compactness | Compact coefficients per image | Compact space representation |

### A.2 Implementation Details

#### A.2.1 Network Architectures

In this subsection, we provide detailed information on the network architecture components used in our study. Specifically, we describe (a) the Mobile-Former encoders, (b) the pooler to compress the feature map into a single vector, (c) the upsampling and convolutional layers employed in FINOLA decoder.

**Table 11: Specification of Mobile-Former encoders. “bneck-lite” denotes the lite bottleneck block Li et al. (). “M-F” denotes the Mobile-Former block and “M-F^↓” denotes the Mobile-Former block for downsampling.**
| Stage | Resolution | Block | MF-W2880 | MF-W1440 | MF-W720 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #exp | #out | #exp | #out | #exp | #out |  |  |  |
| token |  |  | 6$\times$256 | 6$\times$256 | 6$\times$192 |  |  |  |
| stem | 256^2 | conv 3$\times$3 | – | 64 | – | 32 | – | 16 |
| 1 | 128^2 | bneck-lite | 128 | 64 | 64 | 32 | 32 | 16 |
| 2 | 64^2 | M-F^↓ | 384 | 112 | 192 | 56 | 96 | 28 |
| M-F | 336 | 112 | 168 | 56 | 84 | 28 |  |  |
| 3 | 32^2 | M-F^↓ | 672 | 192 | 336 | 96 | 168 | 48 |
| M-F | 576 | 192 | 288 | 96 | 144 | 48 |  |  |
| M-F | 576 | 192 | 288 | 96 | 144 | 48 |  |  |
| 4 | 16^2 | M-F^↓ | 1152 | 352 | 288 | 96 | 240 | 80 |
| M-F | 1408 | 352 | 704 | 176 | 320 | 88 |  |  |
| M-F | 1408 | 352 | 704 | 176 | 480 | 88 |  |  |
| M-F | 2112 | 480 | 1056 | 240 | 528 | 120 |  |  |
| M-F | 2880 | 480 | 1440 | 240 | 720 | 120 |  |  |
| M-F | 2880 | 480 | 1440 | 240 | 720 | 120 |  |  |
| conv 1$\times$1 | – | 2880 | – | 1440 | – | 720 |  |  |

Mobile-Former encoders:
Mobile-Former Chen et al. () is used as the encoder in our approach. It is a CNN-based network that extends MobileNet Sandler et al. () by adding 6 global tokens in parallel. To preserve spatial details, we increase the resolution of the last stage from $\frac{1}{32}$ to $\frac{1}{16}$. We evaluate three variants of Mobile-Former, which are detailed in Table 11. Each variant consists of 12 blocks and 6 global tokens, but they differ in width (720, 1440, 2880). These models serve as the encoders (or backbones) for image reconstruction, self-supervised pre-training, and evaluation in image classification and object detection tasks. For image reconstruction, we also explore two wider models, W4320 and W5760, which increase the number of channels from W2880 by 1.5 and 2 times, respectively. It’s important to note that these models were manually designed without an architectural search for optimal parameters such as width or depth.

**Table 12: Upsampling and convolutional layers in FINOLA decoder. The complexity of upsampling and convolution layers decreases as the spatial resolution of feature map (generated by FINOLA) increases from 8$\times$8 to 256$\times$256). “res-conv” represents a residual block He et al. () consisting of two 3x3 convolutional layers, while “up-conv” performs upsampling followed by a 3x3 convolutional layer.**
| Resolution | 8$\times$8 | 16$\times$16 | 32$\times$32 | 64$\times$64 | 128$\times$128 | 256$\times$256 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| block | #out | block | #out | block | #out | block | #out | block | #out | block | #out |  |
| 8^2 | res-conv | 512 |  |  |  |  |  |  |  |  |  |  |
| 16^2 | up-conv | 512 |  |  |  |  |  |  |  |  |  |  |
| res-conv | 512 | res-conv | 512 |  |  |  |  |  |  |  |  |  |
| 32^2 | up-conv | 512 | up-conv | 512 |  |  |  |  |  |  |  |  |
| res-conv | 256 | res-conv | 256 | res-conv | 256 |  |  |  |  |  |  |  |
| 64^2 | up-conv | 256 | up-conv | 256 | up-conv | 256 |  |  |  |  |  |  |
| res-conv | 256 | res-conv | 256 | res-conv | 256 | res-conv | 256 |  |  |  |  |  |
| 128^2 | up-conv | 256 | up-conv | 256 | up-conv | 256 | up-conv | 256 |  |  |  |  |
| res-conv | 128 | res-conv | 128 | res-conv | 128 | res-conv | 128 | res-conv | 128 |  |  |  |
| 256^2 | up-conv | 128 | up-conv | 128 | up-conv | 128 | up-conv | 128 | up-conv | 128 |  |  |
| res-conv | 128 | res-conv | 128 | res-conv | 128 | res-conv | 128 | res-conv | 128 | res-conv | 128 |  |
| conv3$\times$3 | 3 | conv3$\times$3 | 3 | conv3$\times$3 | 3 | conv3$\times$3 | 3 | conv3$\times$3 | 3 | conv3$\times$3 | 3 |  |
| #param | 25.3M | 18.5M | 9.6M | 7.9M | 1.7M | 1.2M |  |  |  |  |  |  |

**Table 13: Training setting for FINOLA.**
| Config | FINOLA |
| --- | --- |
| optimizer | AdamW |
| base learning rate | 1.5e-4 |
| weight decay | 0.1 |
| batch size | 128 |
| learning rate schedule | cosine decay |
| warmup epochs | 10 |
| training epochs | 100 |
| image size | 256^2 |
| augmentation | RandomResizeCrop |

Pooling the compressed vector $\bm{q}$: In both FINOLA and element-wise masked FINOLA, the compressed vector $\bm{q}$ is obtained by performing attentional pooling Lee et al. (); Yu et al. () on the feature map. This pooling operation involves a single multi-head attention layer with learnable queries, where the encoder output serves as both the keys and values.

FINOLA decoders: Table 12 provides the architecture details of upsampling and covolutional layers after applying FINOLA to generate feature maps $\bm{z}$. The complexity of decreases as the spatial resolution increases, going from 8$\times$8 to 256$\times$256. FINOLA is trained for 100 epochs on ImageNet.

#### A.2.2 Training Setup

The FIOLA training settings for image reconstruction are provided in Table 13. The learning rate is scaled as lr = base_lr$\times$batchsize / 256.

#### A.2.3 Training and Inference Time

Training time:
Training the FINOLA model involves regressing dense feature maps, with computational requirements increasing with feature map size. For instance, training FINOLA to generate a 16$\times$16 feature map with 3072 latent channels for 100 epochs on ImageNet takes approximately 8 days with 8 V100 GPUs. Extending to a larger feature map, such as 64$\times$64, increases the training time to 18 days using the same GPU setup.

Inference time:
In addition to training time, the runtime evaluation includes the complete inference pipeline, encompassing encoding, autoregression, and decoding, conducted on a MacBook Air with an Apple M2 CPU. We evaluated FINOLA for generating feature maps of sizes 16$\times$16 and 64$\times$64, with running times of 1.2 seconds and 2.6 seconds, respectively.

### A.3 Architecture Comparison with Convolutional Auto-Encoder (Convv-AE)

**Table 14: Architecture comparison between convolutional Auto-Encoder and FINOLA.**
|  | Auto-Encoder | FINOLA |
| --- | --- | --- |
| Encoder | same | same |
| Pooling | 2$\times$2$\times$512 (2$\times$2 grid) | 2$\times$1024 (overlap at center) |
| Upsampling to 64$\times$64$\times$1024 | 5 conv blocks (from 2$\times$2 to 64$\times$64) | FINOLA |
| Upsampling to 256$\times$256$\times$13 | same | same |
| Training setup | same | same |

Table 14 presents a comparison of the architectural components between the Conv-AE and FINOLA, while their performance comparison is reported in Table 8 in Section 4.2). Both models share identical (a) encoder, (b) upsampling from resolution 64x64 to 256$\times$256, and (c) training setup (hyper-parameters). However, they differ in their approaches to pooling and upsampling toward the resolution 64$\times$64.

**Table 15: Comparison with auto-encoding (first stage) in generative models. PSNR values for image reconstruction are reported on the ImageNet-1K validation set. $(2048\times 16)$ indicates $C=2048$ channels and $M=16$ FINOLA paths. ^† denotes using multiple initial conditions $\bm{q}_{i}$ at different positions instead of overlapping at the center (see Appendix A.9).**
| Method | Latent $\downarrow$ | PSNR $\uparrow$ |
| --- | --- | --- |
| DALL-E | 32$\times$32$\times$– | 22.8 |
| VQGAN | 65536 (16$\times$16$\times$256) | 19.9 |
| Stable Diffusion | 4096 (16$\times$16$\times$16) | 24.1 |
| FINOLA (multi-path) | 4096 (1024$\times$4) | 26.1 |
| FINOLA (multi-path)^† | 4096 (1024$\times$4) | 26.7 |
| Stable Diffusion | 12288 (64$\times$64$\times$3) | 27.5 |
| FINOLA (multi-path) | 8192 (1024$\times$8) | 27.1 |
| FINOLA (multi-path)^† | 8192 (1024$\times$8) | 28.0 |
| Stable Diffusion | 32768 (128$\times$128$\times$2) | 30.9 |
| FINOLA (multi-path) | 32768 (2048$\times$16) | 29.1 |
| FINOLA (multi-path)^† | 32768 (2048$\times$16) | 30.0 |

Pooling:
Auto-encoder pools a 2$\times$2 grid with 512 channels, while FINOLA pools two vectors with dimension 1024, both yielding the same latent size (2048). Auto-encoder pooling retains spatial information within the 2$\times$2 grid, whereas FINOLA has no explicit spatial information as both vectors are positioned centrally for the FINOLA process.

Upsampling to Resolution 64$\times$64:
The auto-encoder utilizes a stack of five convolutional blocks to generate features at a resolution of 64$\times$64. Each block consists of three 3$\times$3 convolutional layers followed by an upsampling layer to double the resolution. In contrast, our method employs multi-path FINOLA to generate the feature map from center-placed vectors. Since FINOLA utilizes only four matrices ($\bm{A}$, $\bm{B}$, $\bm{A}_{-}$, and $\bm{B}_{-}$), it significantly reduces the number of parameters compared to the five convolutional blocks used in the auto-encoder.

Engineering techniques:
FINOLA does not rely on any additional engineering techniques. Despite this, it slightly outperforms the auto-encoder while utilizing significantly fewer parameters. We attribute this performance to FINOLA’s efficient and effective modeling of spatial transitions.

### A.4 Comparison with Auto-Encoder in Generative Models

Table 15 provides a detailed comparison of FINOLA’s performance against the first stage (autoencoding) of VQGAN and Stable Diffusion in image reconstruction, evaluated on ImageNet-Val with 256x256 images. It’s important to note that the first stage of VQGAN and Stable Diffusion focuses solely on auto-encoding and does not involve the generation process (e.g., diffusion process).

This comparison underscores FINOLA’s performance across varying latent dimensions and its effectiveness in comparison to other methods. Although FINOLA falls behind Stable Diffusion at the largest latent dimension (32768), it operates in a more challenging setup. While FINOLA outputs a single vector after encoding, positioned at the center to generate feature maps through the FINOLA process, spatial information is not explicitly retained. In contrast, the encoder in Stable Diffusion produces a high-resolution grid (128x128) where spatial information is highly preserved.

Introducing spatial information in FINOLA by scattering the initial positions of multiple FINOLA paths (rather than overlapping at the center) enhances the reconstruction quality by 0.6-0.9 PSNR. However, due to scattering initial positions at only 16 locations, the preservation of spatial information remains constrained compared to Stable Diffusion’s 128x128 grid. Consequently, while this enhancement closes the gap in performance (PSNR 30.0 vs 30.9), it still falls short of Stable Diffusion’s spatial fidelity.

### A.5 Ablation Studies of Single Path FINOLA

**Table 16: Image reconstruction ablation experiments on ImageNet-1K. We report PSNR on the validate set. The reconstruction quality correlates to (a) the number of channels in the latent space and (b) complexity of encoder. Default settings are marked by ^†.**
| #Channels | 4096 | 3072^† | 2048 | 1024 | 512 | 256 | 128 | 64 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR | 25.9 | 25.8 | 25.1 | 23.7 | 22.2 | 20.8 | 19.4 | 18.2 |
| (a) Number of channels in latent space. |  |  |  |  |  |  |  |  |

Figure: Figure 6: Image reconstruction examples. The leftmost column shows the original images. The number of channels in the latent space, decreasing from 4096 to 64 from the left to right, controls the reconstruction quality. Best viewed in color.
Refer to caption: x6.png

Figure: Figure 7: Impact of encoder size on image reconstruction quality: The image reconstruction quality shows a slight improvement as the size of the encoder increases. Even with a small encoder containing 5 million parameters (right column), it effectively compresses an image into a single vector capable of reconstructing the entire image. Best viewed in color.
Refer to caption: x7.png

Figure: Figure 8: Comparison of different positions of compressed vector $\bm{q}$: The quality of image reconstruction shows minimal sensitivity to the position of $\bm{q}$. Placing it at the center yields slightly better results compared to corner positions. It is worth noting that each positioning has its own pre-trained model with non-shared parameters. Best viewed in color.
Refer to caption: x8.png

The number of channels in the latent space is crucial.
Table 16-(a) presents the PSNR values for various latent space dimensions, while Figure 6 showcases the corresponding reconstructed examples. The image quality is noticeably poor when using only 64 channels, resulting in significant loss of details. However, as the number of channels increases, more details are successfully recovered. Using more than 3072 channels yields reasonably good image quality, achieving a PSNR of 25.8.

The model size of encoder is less critical but also related. As shown in Figure 7 and Table 16-(b), the larger model has better image quality. But the gap is not significant. When increasing model size by 13 times from 5.0M to 67.6M, the PSNR is slightly improved from 24.4 to 26.1. Note all encoders share similar architecture (Mobile-Former with 12 blocks), but have different widths.

The position of $\bm{q}$ is not critical: Figure 8 showcases the reconstructed samples obtained by placing the compressed vector $\bm{q}$ at different positions, including the center and four corners. The corresponding peak signal-to-noise ratio (PSNR) values on the ImageNet validation set are provided at the bottom. While placing $\bm{q}$ at the center yields slightly better results compared to corner positions, the difference is negligible. It is important to note that each positioning corresponds to its own pre-trained model with non-shared parameters.

### A.6 Inspecting the Image Distribution in q 𝑞 q italic_q Space

In this subsection, we list main observations and analysis in the space of the compressed vector $\bm{q}$ (named embedding space). This will help us to understand how images are distributed in the embedding space. In this subsection, we use single-path FINOLA with $C=3072$ channels.

Figure: Figure 9: Reconstruction from random samples: The reconstructed images are generated by sampling from the statistics (mean and covariance) of compressed embeddings $\bm{q}$ obtained from the ImageNet validation set, consisting of 50,000 images. Although the samples are not similar to images of Gaussian noise, they lack semantic meaning and appear as noisy images. Multiple samplings consistently yield similar noisy patterns. Best viewed in color.
Refer to caption: x9.png

Figure: Figure 10: Reconstruction from the average embedding $\bm{\bar{q}}$: The reconstructed image corresponding to the average embedding $\bm{\bar{q}}$ computed from 50,000 ImageNet validation images closely resembles a gray image (shown in the right column of the left figure). In the left figure, we demonstrate the interpolation along a line connecting embeddings from different images to the average embedding. Notably, the reconstructed images progressively fade into a gray image. In the right figure, we extend the connection between an image embedding $\bm{q}$ and the average embedding $\bm{\bar{q}}$ to a mirror embedding $2\bm{q}-\bm{\bar{q}}$, corresponding to an image with reversed colors. This comparison provides insights into the nature of the embedding space. Best viewed in color.
Refer to caption: x10.png

Figure: Figure 11: Reconstruction from interpolated embeddings: The images are reconstructed by interpolating embeddings of two images, $\alpha\bm{q}_{1}+(1-\alpha)\bm{q}_{2}$. Although the mixed embedding passes through a non-linear network that includes FINOLA and a multi-layer decoder, it leads to mixing up images as output. Best viewed in color.
Refer to caption: x11.png

Figure: Figure 12: Reconstruction from top principle components: The top-$K$ principle components correspond to the largest $K$ eigenvalues of the covariance matrix computed from 50,000 image embeddings in the ImageNet validation set. With a selection of top-192 components (the right column), the color and layout of the images are primarily determined, but the resulting reconstructions appear blurred with noticeable loss of details. As more principle components are incorporated, the finer details are gradually restored. Best viewed in color.
Refer to caption: x12.png

Figure: Figure 13: Multiple paths vs. Single path: Summing $M=8$ FINOLA solutions ${\bm{\phi}_{i}}$ (as in Eq. 3) yields superior image reconstruction quality compared to the single path counterpart. This trend holds across various dimensions (from $C=128$ to $C=2048$). Resolution of feature map $\bm{z}$ is set to 64$\times$64, with an image size of 256$\times$256. Best viewed in color.
Refer to caption: x13.png

Three observations: Below we list three observations that reveal properties of the embedding space.

Dominance of noisy images in the space: To analyze the distribution of images in the embedding space, we collected $\bm{q}$ vector for all 50,000 images from the ImageNet validation set and computed their statistics (mean and covariance). By sampling embeddings based on these statistics and reconstructing images, we consistently observed the emergence of similar noisy patterns, as depicted in Figure 9. This observation highlights the prevalence of noisy images throughout the space, with good images appearing as isolated instances surrounded by the abundance of noise.

Averaged embedding $\bm{\bar{q}}$ yields a gray image:
In Figure 10, we observe that the reconstructed image obtained from the averaged embedding $\bm{\bar{q}}$, computed over 50,000 images from the ImageNet validation set, closely resembles a gray image. We further investigate the relationship between real image embeddings $\bm{q}$ and the averaged embedding $\bm{\bar{q}}$ through interpolations along the embedding space. As depicted in the left figure, the reconstructed images maintain their content while gradually fading into a gray image. Additionally, we extend this connection to mirror embeddings in the right figure, represented by $2\bm{q}-\bm{\bar{q}}$, which correspond to images with reversed colors. These findings suggest that despite the prevalence of noisy images, the line segment connecting an image embedding to the average embedding encompasses different color transformations of the same image.

Reconstruction from interpolated embeddings:
In Figure 11, we present the reconstructed images obtained by interpolating between two image embeddings using the equation $\alpha\bm{q}_{1}+(1-\alpha)\bm{q}_{2}$. This process of embedding mixup results in a corresponding mixup of the images, allowing for a smooth transition between the two original images by varying the value of $\alpha$. However, it is important to note that the resulting reconstruction may not precisely match the simple mixup of the original images, represented by $\alpha\bm{I}_{1}+(1-\alpha)\bm{I}_{2}$.

Combining the three observations discussed above, our findings suggest that the presence of noisy images in Figure 9 indicates the mixing of multiple surrounding images. As the number of image embeddings involved in the mixing process increases, the resulting reconstructions tend to resemble a gray image, as depicted in Figure 10.

Principle component analysis (PCA): The reconstruction results shown in Figure 12 are obtained using PCA with the top-$K$ principle components. These components correspond to the largest $K$ eigenvalues of the covariance matrix computed from 50,000 image embeddings in the ImageNet validation set. The principle components capture essential information, starting with color and layout, and gradually encoding finer image details as more components are included in the reconstruction process.

Figure: Figure 14: Reconstruction examples for varying numbers of channels ($C$) and FINOLA paths ($M$): Increasing the number of paths, as per Eq. 3, consistently enhances image quality across different dimensions ($C=128$ to $C=2048$), affirming the relaxation of FINOLA constraints. Feature resolution ($\bm{z}$) is 64$\times$64, and image size is 256$\times$256. Best viewed in color.
Refer to caption: x14.png

### A.7 Visual Comparison between Single-path and Multi-path FINOLA

Single vs. Multiple paths:
Figure 13 visually demonstrates that multiple paths $M=8$ exhibit markedly superior image quality compared to the single path counterpart ($M=1$).

Reconstruction examples for varying number of channels $C$ and paths $M$:
Figure 14 illustrates the reconstruction examples obtained for different combinations of channel counts (or number of one-way wave equations $C=128,256,512,1024,2048$) and the number of FINOAL paths ($M=1,2,4,8$). These results correspond to the experiments in Figure 5, as discussed in Section 4.1.

Notably, a consistent trend emerges where increasing the value of $M$ consistently enhances image quality. This trend remains consistent across various equation counts, ranging from $C=128$ to $2048$. This observation underscores the efficacy of relaxing the FINOLA constraint by FINOLA series, as detailed in Section 3.

### A.8 Real-valued Wave Speeds

**Table 17: Inspection of real-valued wave speeds: (a) PSNR values for image reconstruction with varying wave speeds (complex, real, all-one) on the ImageNet-1K validation set, with the symbol ^‡ denoting the use of position embedding. The number of wave equations (or feature map dimension) is set $C=1024$, and the number of FINOLA paths is set $M=4$. (b) A comparison between all-one speed waves and feature map generation through repetition with position embedding to ensure position embedding isn’t the sole dominant factor.**
| Wave Speed | Dimension | PSNR |
| --- | --- | --- |
| Complex $\lambda_{k}\in\mathbb{C}$ | 1024$\times$4 | 26.1 |
| Real $\lambda_{k}\in\mathbb{R}$ | 1024$\times$4 | 25.1 |
| All-one $\lambda_{k}=1^{\ddagger}$ | 1024$\times$4 | 23.9 |
| (a) Special cases: real and all-one speeds. |  |  |
| Feature Map Gen | Dimension | PSNR |
| Repetition | 4096 | 21.6 |
| All-one waves | 1024$\times$4 | 23.9 |
| (b) Using position embedding. |  |  |

It is worth noting that the speeds of the wave equations are generally complex numbers $\lambda_{k}\in\mathbb{C}$, which is also validated in the experiments. This arises because we do not impose constraints on the coefficient matrices ($\bm{A}$, $\bm{B}$) in Eq. 2. Consequently, during the diagonalization process, $\bm{AB}^{-1}=\bm{V\Lambda V}^{-1}$, it is highly likely that the eigenvalues and eigenvectors will be complex numbers.

Here, we introduce two interesting cases by constraining the speeds of the one-way wave equations as follows: (a) as real numbers $\lambda_{k}\in\mathbb{R}$, and (b) as all equal to one $\lambda_{1}=\dots=\lambda_{C}=1$.

Real speed $\lambda_{k}\in\mathbb{R}$:
This is achieved by constraining matrices $\bm{H}_{A}$ and $\bm{H}_{B}$ in Eq. 6 as real diagonal matrices:

$$ $\displaystyle\bm{H}_{A}=\text{diag}(\alpha_{1},\alpha_{2},\ldots,\alpha_{C}), \quad\bm{H}_{B}=\text{diag}(\beta_{1},\beta_{2},\ldots,\beta_{C}),\quad\bm{A}= \bm{P}\bm{H}_{A},\quad\bm{B}=\bm{P}\bm{H}_{B}.$ (8) $$

Here, the coefficient matrices $\bm{A}$ and $\bm{B}$ in FINOLA are implemented by multiplying a real projection matrix $\bm{P}$ with diagonal matrices $\bm{H}_{A}$ and $\bm{H}_{B}$, respectively. Consequently, the speeds of the wave equations are real numbers, denoted as $\lambda_{k}=\alpha_{k}/\beta_{k}$.

All-one speed $\lambda_{1}=\dots=\lambda_{C}=1$:
By further constraining $\bm{H}_{A}$ and $\bm{H}_{B}$ as identity matrices, all wave equations have identical speed $\lambda_{k}=1$.

$$ $\displaystyle\bm{H}_{A}=\bm{H}_{B}=\bm{I},\quad\bm{A}=\bm{B}=\bm{P},\quad \lambda_{1}=\lambda_{2}=\dots=\lambda_{C}=1.$ (9) $$

Here, the coefficient matrices $\bm{A}$ and $\bm{B}$ in FINOLA are also identical and denoted as $\bm{P}$.

Experimental results for real-valued wave speeds:
Table 17-(a) provides the results for real-valued and all-one wave speed, while Figure 15 displays corresponding reconstruction examples. In comparison to the default scenario using complex-valued wave speeds, enforcing wave speeds as real numbers or setting them uniformly to one shows a slight decline in performance. Nonetheless, both real-valued speed cases still deliver reasonably good PSNR scores. Notably, the all-one wave speed configuration achieves a PSNR of 23.9. This specific configuration shares the coefficient matrix for autoregression across all four directions (up, down, left, right), creating symmetry in the feature map. To account for this symmetry, we introduced position embedding before entering the decoder.

**Table 18: Position of initial conditions. PSNR values for image reconstruction on the ImageNet-1K validation set is reported. Scattering of initial positions spatially boosts performance.**
| Position | #Paths $M$ | #Channels $C$ | PSNR $\uparrow$ |
| --- | --- | --- | --- |
| Overlapping at Center | 4 | 1024 | 26.1 |
| Scattering Uniformly | 4 | 1024 | 26.7 |
| Overlapping at Center | 8 | 1024 | 27.1 |
| Scattering Uniformly | 8 | 1024 | 28.0 |
| Overlapping at Center | 16 | 1024 | 27.7 |
| Scattering Uniformly | 16 | 1024 | 29.1 |
| Overlapping at Center | 8 | 2048 | 28.0 |
| Scattering Uniformly | 8 | 2048 | 28.9 |
| Overlapping at Center | 16 | 2048 | 29.1 |
| Scattering Uniformly | 16 | 2048 | 30.0 |

In an effort to determine whether position embedding is the dominant factor for all-one wave speed, we conducted experiments by generating feature maps using both repetition and position embedding, with the same dimention (4096). This approach falls short of the all-one wave speed configuration by 2.3 PSNR (as detailed in Table 17-(b)). Its reconstruction quality significantly lags behind that of all-one waves, as depicted in the last two columns of Figure 15.

### A.9 Scattering Initial Conditions Spatially

To enhance reconstruction further, we can adjust spatial positions to place the initial conditions, without introducing additional parameters or FLOPs. This concept is straightforward to implement through multi-path FINOLA (refer to Figure 3), where different paths employ scattered initial positions rather than overlapped at the center. Table 18 demonstrates that further improvements in reconstruction is achieved by scattering the initial conditions uniformly compared to placing them at the center, regardless of whether we use 4, 8 or 16 FINOLA paths.

## Appendix B Masked FINOLA for Self-supervised Pre-training

### B.1 Masked FINOLA

FINOLA can be applied to self-supervised learning through a straightforward masked prediction task, which we refer to as Masked FINOLA to distinguish it from the vanilla FINOLA. Unlike vanilla FINOLA that support various resolutions of feature map, masked FINOLA performs mask prediction at resolution $\frac{1}{16}$, which is consistent with established baselines like MAE He et al. (), SimMIM Xie et al. (). In this paper, we only use single path for masked FINOLA.

Figure: Figure 16: Two Masked FINOLA variants: element-wise (left) and block-wise (right) approaches. In the element-wise approach, autoregression is performed similarly to vanilla FINOLA, with the compressed vector $\bm{q}$ observing only the unmasked block rather than the entire image. Conversely, the block-wise approach does not compress the unmasked block. Each unmasked position exclusively predicts three masked positions, as indicated by arrows, using Eq. 1. Assignments are grouped together, with shared offsets within each group. The grouping varies depending on the location of the unmasked quadrant, resulting in 1, 2, and 4 groups for corner, edge, and middle locations, respectively. Best viewed in color.
Refer to caption: x16.png

Simple block masking: FINOLA is applied through a simple masked prediction design that involves using a single unmasked image block (see Figure 16) to predict the surrounding masked region. Specifically, we crop out the unmasked block and pass it through the encoder, leveraging the power of FINOLA to generate a full-size feature map. Finally, a decoder is applied to recover the pixels in masked region. Unlike vanilla FINOLA, the reconstruction loss is computed only from the masked region. Please note that the unmasked block floats around the image randomly.

Masked FINOLA variants: Masked FINOLA comprises two variants: the element-wise approach (Masked-FINOLA-E) and the block-wise approach (Masked-FINOLA-B), as depicted in Figure 16.

The element-wise variant (Masked-FINOLA-E) operates similarly to vanilla FINOLA, with the compressed vector $\bm{q}$ only observing the unmasked block rather than the entire image (see Figure 16-left). To accommodate the longer training required in masked FINOLA (e.g., 1600 epochs), we follow He et al. () to replace the convolutional decoder with a simple linear layer, transforming a $C$-channel token into a 16$\times$16$\times$3 image patch.

**Table 19: Mobile-Former decoder specifications for COCO object detection: 100 object queries with dimension 256 are used. “down-conv” includes a 3$\times$3 depthwise convolution (stride=2) and a pointwise convolution (256 channels). ”up-conv” uses bilinear interpolation, followed by a 3$\times$3 depthwise and a pointwise convolution. ”M-F^+” replaces the Mobile sub-block with a transformer block, while ”M-F^-” uses the lite bottleneck Li et al. () to replace the Mobile sub-block.**
| Stage | MF-Dec-522 | MF-Dec-211 |  |  |
| --- | --- | --- | --- | --- |
| query | 100$\times$256 | 100$\times$256 |  |  |
| $\frac{1}{32}$ | down-conv |  | down-conv |  |
| M-F^+ | $\times$5 | M-F^+ | $\times$2 |  |
| $\frac{1}{16}$ | up-conv |  | up-conv |  |
| M-F^- | $\times$2 | M-F^- | $\times$1 |  |
| $\frac{1}{8}$ | up-conv |  | up-conv |  |
| M-F^- | $\times$2 | M-F^- | $\times$1 |  |

In contrast, the block-wise variant (Masked-FINOLA-B) preserves the unmasked block in its entirety, without compression. It requires the unmasked block to have a quadrant size. As shown in Figure 16-right, each unmasked position is tasked with predicting three masked positions, denoted by arrows and computed using Eq. 1. These assignments are organized into groups, and within each group, all unmasked positions share common offsets for reaching their assigned masked positions. The configuration of these groups dynamically adapts based on the location of the unmasked quadrant, resulting in 1, 2, or 4 groups for corner, edge, or middle positions, respectively. To promote communication across these groups, transformer blocks are integrated into the decoder.

Relation to MAE He et al. ():
Masked FINOLA shares a similar architecture with MAE but differs notably in masking and prediction strategies. Firstly, masked FINOLA adopts a regular masking design, grouping all unmasked patches into a single block, in contrast to MAE’s utilization of random unmasked patches. This design choice suits efficient CNN-based networks. Secondly, masked FINOLA employs a first-order norm+linear autoregression approach for predicting the masked region, whereas MAE utilizes masked tokens within an attention model.

### B.2 Implementation Details

#### B.2.1 Decoder Architectures

Below, we describe (a) the decoders employed in masked FINOLA, (b) the decoders designed for image classification, and (c) the decoders tailored for object detection.

Decoders for FINOLA pre-training: Unlike vanilla FINOLA, which employs stacked upsampling and convolution blocks, the masked FINOLA variants utilize simpler architectures — a linear layer for transforming features into 16$\times$16 image patches. This choice facilitates longer training. The decoder of Masked-FINOLA-B incorporates transformer blocks (without positional embedding) to enable spatial communication. Masked FINOLA undergoes training for 1600 epochs.

Decoders for ImageNet classification: We utilize three decoders to evaluate the pre-trained encoders in FINOLA. These decoders are as follows:

- •
lin decoder: It consists of a single linear layer and is used for linear probing.
- •
tran-1 decoder: It incorporates a shallower transformer decoder with a single transformer block followed by a linear classifier and is employed for tran-1 probing and fine-tuning.
- •
tran-4 decoder: This decoder is composed of four transformer blocks followed by a linear classifier and is utilized for fine-tuning alone.

The transformer decoders are designed with different widths (192, 384, 768) to correspond with the three Mobile-Former encoders, which have widths of 720, 1440, and 2880, respectively.

Decoders for object detection:
The decoders used in the DETR framework with Mobile-Former Chen et al. () are described in Table 19. Both decoders consist of 100 object queries with a dimension of 256. While they share a similar structure across three scales, they differ in terms of their depths. Since the backbone network ends at a resolution of $\frac{1}{16}$, the decoder incorporates a downsampling step to further reduce the resolution to $\frac{1}{32}$. This enables the decoder to efficiently process the features for object detection.

**Table 20: Pre-training setting for masked FINOLA.**
| Config | Masked FINOLA |
| --- | --- |
| optimizer | AdamW |
| base learning rate | 1.5e-4 |
| weight decay | 0.1 |
| batch size | 1024 |
| learning rate schedule | cosine decay |
| warmup epochs | 10 |
| training epochs | 1600 |
| image size | 256^2 |
| augmentation | RandomResizeCrop |

#### B.2.2 Training Setup

In this section, we provide detailed training setups for different tasks, including:

- •
Masked FINOLA pre-training on ImageNet-1K.
- •
Linear probing on ImageNet-1K.
- •
tran-1 probing on ImageNet-1K.
- •
Fine-tuning on ImageNet-1K.
- •
COCO object detection.

**Table 21: Settings for linear probing and tran-1 probing on ImageNet-1K: The encoders are frozen during both tasks.**
| Config | Linear probing | tran-1 probing |
| --- | --- | --- |
| optimizer | SGD | AdamW |
| base learning rate | 0.1 | 0.0005 |
| weight decay | 0 | 0.1 |
| batch size | 4096 | 4096 |
| learning rate schedule | cosine decay | cosine decay |
| warmup epochs | 10 | 10 |
| training epochs | 90 | 200 |
| augmentation | RandomResizeCrop | RandAug (9, 0.5) |
| label smoothing | – | 0.1 |
| dropout | – | 0.1 (MF-W720) 0.2 (MF-W1440/W2880) |
| random erase | – | 0 (MF-W720/W1440) 0.25 (MF-W2880) |

**Table 22: Setting for end-to-end fine-tuning on ImageNet-1K.**
| Config | Value |
| --- | --- |
| optimizer | AdamW |
| base learning rate | 0.0005 |
| weight decay | 0.05 |
| layer-wise lr decay | 0.90 (MF-W720/W1440) 0.85 (MF-W2880) |
| batch size | 512 |
| learning rate schedule | cosine decay |
| warmup epochs | 5 |
| training epochs | 200 (MF-W720) 150 (MF-W1440) 100 (MF-W2880) |
| augmentation | RandAug (9, 0.5) |
| label smoothing | 0.1 |
| mixup | 0 (MF-W720) 0.2 (MF-W1440) 0.8 (MF-W2880) |
| cutmix | 0 (MF-W720) 0.25 (MF-W1440) 1.0 (MF-W2880) |
| dropout | 0.2 |
| random erase | 0.25 |

Masked FINOLA pre-training:
Similar to the vanilla FINOLA, masked FINOLA also follows the training setup described in Table 20, but with a larger batch size due to the simpler decoder architecture that requires less memory consumption.

Linear probing: In our linear probing, we follow the approach described in He et al. () by incorporating an additional BatchNorm layer without affine transformation (affine=False). Detailed settings can be found in Table 21.

tran-1 probing: The settings for tran-1 decoder probing are presented in Table 21. It is important to note that the default decoder widths are 192, 384, and 768 for MF-W720, MF-W1440, and MF-W2880, respectively.

End-to-end fine-tuning on ImageNet-1K: The settings for the end-to-end fine-tuning of both the encoder and tran-1 decoder are presented in Table 22. The decoder weights are initialized from the tran-1 probing stage.

Figure: Figure 17: Training schedules of Masked-FINOLA-B. Longer training schedule provides consistent improvement for linear and tran-1 probing over different models, while fine-tuning performance is not sensitive to training schedule. Best viewed in color.
Refer to caption: x17.png

**Table 23: Ablation on the number of transformer blocks in the decoder: Evaluation is conducted on ImageNet using Mobile-Former-W2880 as the encoder. Each transformer block consists of 512 channels. Each model is pre-trained for 800 epochs. Increasing the decoder depth exhibits consistent improvement for linear and tran-1 probing, while fine-tuning performance shows limited sensitivity to decoder depth.**
| #Blocks | lin | tran-1 | tran-1-ft |
| --- | --- | --- | --- |
| 1 | 61.1 | 74.4 | 82.2 |
| 2 | 62.6 | 76.5 | 82.3 |
| 3 | 63.5 | 77.3 | 82.2 |
| 4 | 63.8 | 78.0 | 82.3 |
| 5 | 64.0 | 78.1 | 82.3 |
| 6 | 65.0 | 78.3 | 82.4 |

Decoder probing on COCO object detection: In this configuration, the backbone pre-trained on ImageNet-1K is frozen, and only the decoders are trained for 500 epochs on 8 GPUs with 2 images per GPU. We employ AdamW optimizer with an initial learning rate of 1e-4. The learning rate is decreased by a factor of 10 after 400 epochs. The weight decay is 1e-4, and the dropout rate is 0.1.

Fine-tuning on COCO object detection: In this setting, both the encoder and decoder are fine-tuned. The fine-tuning process consists of an additional 200 epochs following the decoder probing stage. The initial learning rate for both the encoder and decoder is set to 1e-5, which decreases to 1e-6 after 150 epochs.

### B.3 Ablation Studies

Ablation on training schedule: The impact of training schedule length on three Mobile-Former encoders is depicted in Figure 17. Notably, the accuracies of both linear and tran-1 probings demonstrate a consistent improvement as the training duration increases. Interestingly, even with a pre-training of just 100 epochs, fine-tuning with tran-1 achieves commendable performance. This finding diverges from the observations in MAE He et al. (), where longer training is essential for fine-tuning improvements.

Ablation on the number of transformer blocks in the decoder:
We investigate the impact of the number of transformer blocks in the decoder on FINOLA pre-training using the Mobile-Former-W2880 as encoder. Each transformer block in the decoder consists of 512 channels, but does not use positional embedding. The results, shown in Table 24, demonstrate that adding more transformer blocks leads to consistent improvements in both linear and tran-1 probing tasks. However, we observe that the performance of fine-tuning is less sensitive to changes in the decoder depth.

### B.4 Comparable Performance with Established Baselines on Linear Probing

As shown in Table 24, FINOLA achieves comparable performance with well known baselines on linear probing while requiring lower FLOPs. The comparison is conducted in end-to-end manner (combining encoder and pre-training method). For example, we compare FINOLA+MobileFormer with MAE+ViT in the context of ImageNet classification.

### B.5 Robust Task Agnostic Encoders

FINOLA provides a robust task-agnostic encoders:
Pre-training with FINOLA followed by fine-tuning on ImageNet-1K (IN-1K) consistently outperforms IN-1K supervised pre-training in both ImageNet classification and COCO object detection (see Figure 18). The gains in object detection are substantial, ranging from 5 to 6.4 AP. Remarkably, even without IN-1K fine-tuning, FINOLA pre-training alone outperforms the supervised counterpart in object detection by a clear margin (3 to 4.5 AP). This highlights FINOLA’s ability to encode spatial structures.

Figure: Figure 18: Task-agnostic encoders evaluated on ImageNet (IN-1K) classification and COCO object detection. We assess three IN-1K pretraining methods: (a) supervised (Sup-IN1K), (b) FINOLA, and (c) FINOLA with fine-tuning on IN-1K (FINOLA+IN1K-FT). The dots represent different Mobile-Former backbones. For classification, we add a tran-1 decoder (with a single transformer block) trained with class supervision. It’s important to note that the backbone remains task-agnostic, frozen during object detection. FINOLA performs lower than Sup-IN1K in classification but surpasses it in object detection. After fine-tuning on IN-1K, FINOLA+IN1K-FT shows improvements in both tasks, providing robust task-agnostic encoders.
Refer to caption: x18.png

Comparisons with MoCo-v2:
As shown in Table 25, FINOLA demonstrates comparable performance to MoCo-V2 in linear probing, while surpassing MoCo-V2 in tran-1 probing that uses a single transformer block as a decoder for classification, IN-1K fine-tuning, object detection and segmentation. The backbone is frozen for both COCO object detection and segmentation. FINOLA’s superior performance suggests it learns more effective intermediate features, contributing to more representative decoder features.
Furthermore, the improved performance in object detection emphasizes FINOLA’s ability to encode spatial structures effectively.

These experiments demonstrate that the proposed masked FINOLA is able to learn task-agnostic representation by using a simple masking design. This supports that the underling PDEs capture the intrinsic spatial structures present in images.

**Table 25: Comparisons with MoCo-v2 Chen et al. () on ImageNet classification, COCO object detection and instance segmentation. Three Mobile-Former backbones with different widths are used. In tran-1, the encoder is frozen while a transformer block is trained as a decoder using class labels. In tran-1-ft, encoders are fine-tuned. Encoders are frozen in both COCO object detection and instance segmentation. DETR framework is used for object detection, while Mask-RCNN (1$\times$) is used for segmentation. FINOLA outperforms MoCo-V2 in most evaluations, except on par in linear probing.**
| Pre-training | Encoder | IN-1K Top-1 | COCO Det (Box-AP) | COCO Seg (Mask-AP) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| lin | tran-1 | tran-1-ft | w/o IN-ft | with IN-ft | w/o IN-ft | with IN-ft |  |  |
| MoCo-V2 | MF-W720 | 51.6 | 52.9 | 74.3 | 31.8 | 39.9 | 23.2 | 25.3 |
| FINOLA | 51.3 | 65.5 | 75.6 | 40.0 | 41.6 | 26.3 | 28.4 |  |
| MoCo-V2 | MF-W1440 | 60.4 | 58.5 | 79.2 | 30.3 | 39.0 | 25.6 | 25.7 |
| FINOLA | 62.8 | 75.2 | 80.5 | 42.6 | 44.0 | 30.6 | 32.7 |  |
| MoCo-V2 | MF-W2880 | 66.5 | 63.8 | 80.0 | 25.5 | 31.7 | 27.8 | 25.2 |
| FINOLA | 66.4 | 78.7 | 82.5 | 43.3 | 45.5 | 33.3 | 35.1 |  |

**Table 26: COCO object detection results on the val2017 dataset using a frozen backbone pre-trained on ImageNet-1K. Evaluation is conducted over three backbones and two heads that use Mobile-Former Chen et al. () end-to-end in DETR Carion et al. () framework. Our FINOLA consistently outperform the supervised counterpart. Notably, fine-tuning on ImageNet-1K (denoted as ”IN-ft”) yields further improvements. The initial ”MF” (e.g., MF-Dec-522) denotes Mobile-Former. The madds metric is based on an image size of 800$\times$1333.**
|  | Head |  | Backbone | AP | AP50 | AP75 | APS | APM | APL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | madds | param | model | madds | param | pre-train | IN-ft |  |  |  |  |  |  |
|  | (G) | (M) |  | (G) | (M) |  |  |  |  |  |  |  |  |
| MF<br>Dec<br>522 | 34.6 | 19.4 | MF<br>W2880 | 77.5 | 25.0 | supervised | – | 40.5 | 58.5 | 43.3 | 21.1 | 43.4 | 56.8 |
| FINOLA | ✗ | 43.3 (+2.8) | 61.5 | 46.8 | 23.7 | 46.9 | 60.1 |  |  |  |  |  |  |
| FINOLA | ✓ | 45.5 (+5.0) | 63.8 | 49.5 | 25.1 | 49.1 | 63.5 |  |  |  |  |  |  |
| 32.3 | 18.6 | MF<br>W1440 | 20.4 | 11.7 | supervised | – | 38.3 | 56.0 | 40.8 | 19.0 | 40.9 | 54.3 |  |
| FINOLA | ✗ | 42.6(+4.3) | 60.3 | 46.1 | 22.6 | 46.2 | 60.0 |  |  |  |  |  |  |
| FINOLA | ✓ | 44.0(+5.7) | 62.3 | 47.3 | 23.8 | 47.6 | 61.0 |  |  |  |  |  |  |
| 31.1 | 18.2 | MF<br>W720 | 5.6 | 4.9 | supervised | – | 35.2 | 52.1 | 37.6 | 16.9 | 37.2 | 51.7 |  |
|  | FINOLA | ✗ | 40.0(+4.8) | 57.9 | 42.9 | 20.6 | 43.3 | 56.8 |  |  |  |  |  |
|  | FINOLA | ✓ | 41.6(+6.4) | 59.4 | 45.0 | 21.2 | 45.0 | 58.9 |  |  |  |  |  |
| MF<br>Dec<br>211 | 15.7 | 9.2 | MF<br>W2880 | 77.5 | 25.0 | supervised | – | 34.1 | 51.3 | 36.1 | 15.5 | 36.8 | 50.0 |
| FINOLA | ✗ | 36.7(+2.6) | 53.7 | 39.3 | 18.2 | 39.7 | 52.2 |  |  |  |  |  |  |
| FINOLA | ✓ | 41.0(+6.9) | 59.2 | 44.4 | 20.9 | 44.6 | 58.3 |  |  |  |  |  |  |
| 13.4 | 8.4 | MF<br>W1440 | 20.4 | 11.7 | supervised | – | 31.2 | 47.8 | 32.8 | 13.7 | 32.9 | 46.9 |  |
| FINOLA | ✗ | 36.0(+4.8) | 52.7 | 38.7 | 16.6 | 39.1 | 52.5 |  |  |  |  |  |  |
| FINOLA | ✓ | 39.2(+8.0) | 56.9 | 42.0 | 19.7 | 42.8 | 56.2 |  |  |  |  |  |  |
| 12.2 | 8.0 | MF<br>W720 | 5.6 | 4.9 | supervised | – | 27.8 | 43.4 | 28.9 | 11.3 | 29.1 | 41.6 |  |
|  | FINOLA | ✗ | 33.0(+5.2) | 49.3 | 35.0 | 15.3 | 35.1 | 48.9 |  |  |  |  |  |
|  | FINOLA | ✓ | 35.8(+8.0) | 52.6 | 38.3 | 16.4 | 38.3 | 52.0 |  |  |  |  |  |

Comparison with the IN-1K supervised pre-training on transferring to COCO object detection:
Table 26 presents the results of COCO object detection using frozen backbones. The evaluation utilizes three Mobile-Former encoders with different widths and two Mobile-Former decoders with different depths. Notably, FINOLA pre-training followed by ImageNet-1K (IN-1K) fine-tuning consistently outperforms the IN-1K supervised pre-training across all evaluations, demonstrating the effectiveness of task-agnostic encoders. Impressively, even FINOLA pre-training alone, without IN-1K fine-tuning, surpasses the supervised counterpart on object detection by a significant margin of 2.6–5.2 AP. This showcases FINOLA’s ability to encode spatial structures.

**Table 27: COCO object detection results on the val2017 dataset after fine-tuning both the backbone and head on COCO. Evaluation is performed on three different backbones and two heads, utilizing the Mobile-Former Chen et al. () end-to-end in the DETR Carion et al. () framework. Our approach, which involves FINOLA pre-training followed by ImageNet-1K fine-tuning, surpasses the performance of the supervised baselines. The initial ”MF” (e.g., MF-Dec-522) denotes Mobile-Former, while ”IN-ft” indicates fine-tuning on ImageNet-1K. The reported madds values are based on the image size of 800$\times$1333.**
|  | Head |  | Backbone | AP | AP50 | AP75 | APS | APM | APL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| model | madds | param | model | madds | param | pre-train | IN-ft |  |  |  |  |  |  |
|  | (G) | (M) |  | (G) | (M) |  |  |  |  |  |  |  |  |
| MF<br>Dec<br>522 | 34.6 | 19.4 | MF<br>W2880 | 77.5 | 25.0 | supervised | – | 48.1 | 66.6 | 52.5 | 29.7 | 51.8 | 64.0 |
| FINOLA | ✗ | 48.0(-0.1) | 66.2 | 52.3 | 28.2 | 51.4 | 64.1 |  |  |  |  |  |  |
| FINOLA | ✓ | 49.0 (+0.9) | 67.7 | 53.4 | 30.1 | 52.9 | 65.5 |  |  |  |  |  |  |
| 32.3 | 18.6 | MF<br>W1440 | 20.4 | 11.7 | supervised | – | 46.2 | 64.4 | 50.1 | 27.1 | 49.8 | 62.4 |  |
| FINOLA | ✗ | 46.8(+0.6) | 64.9 | 51.0 | 26.6 | 50.6 | 63.4 |  |  |  |  |  |  |
| FINOLA | ✓ | 47.3(+1.1) | 65.6 | 51.4 | 27.3 | 50.7 | 63.9 |  |  |  |  |  |  |
| 31.1 | 18.2 | MF<br>W720 | 5.6 | 4.9 | supervised | – | 42.5 | 60.4 | 46.0 | 23.9 | 46.0 | 58.5 |  |
|  | FINOLA | ✗ | 43.3(+0.8) | 61.0 | 47.0 | 23.1 | 46.6 | 61.0 |  |  |  |  |  |
|  | FINOLA | ✓ | 44.4(+1.9) | 62.1 | 48.1 | 24.3 | 47.8 | 61.5 |  |  |  |  |  |
| MF<br>Dec<br>211 | 15.7 | 9.2 | MF<br>W2880 | 77.5 | 25.0 | supervised | – | 44.0 | 62.8 | 47.7 | 25.8 | 47.3 | 60.7 |
| FINOLA | ✗ | 44.4(+0.4) | 62.5 | 48.2 | 24.7 | 47.6 | 60.7 |  |  |  |  |  |  |
| FINOLA | ✓ | 46.0(+2.0) | 64.8 | 49.9 | 26.2 | 50.0 | 62.7 |  |  |  |  |  |  |
| 13.4 | 8.4 | MF<br>W1440 | 20.4 | 11.7 | supervised | – | 42.5 | 60.6 | 46.0 | 23.6 | 45.9 | 57.9 |  |
| FINOLA | ✗ | 42.4(-0.1) | 60.2 | 45.9 | 21.9 | 45.7 | 60.0 |  |  |  |  |  |  |
| FINOLA | ✓ | 43.8(+1.3) | 61.8 | 47.5 | 23.9 | 47.1 | 60.8 |  |  |  |  |  |  |
| 12.2 | 8.0 | MF<br>W720 | 5.6 | 4.9 | supervised | – | 37.6 | 55.1 | 40.4 | 18.9 | 40.6 | 53.8 |  |
|  | FINOLA | ✗ | 37.2 (-0.4) | 54.3 | 39.7 | 18.7 | 39.8 | 53.4 |  |  |  |  |  |
|  | FINOLA | ✓ | 39.3(+1.7) | 56.7 | 42.4 | 19.4 | 42.1 | 56.5 |  |  |  |  |  |

**Table 28: Comparison with DETR-based models on COCO detection. All baselines are fine-tuned on COCO. FINOLA-DETR utilizes Mobile-Former (MF-W2880) as the backbone, which has similar FLOPs and model size to the ResNet-50 used in other methods. MAdds are calculated based on an image size of 800$\times$1333.**
| Model | Query | AP | AP50 | AP75 | APS | APM | APL | MAdds | Param |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (G) | (M) |  |  |  |  |  |  |  |  |
| DETR-DC5Carion et al. () | 100 | 43.3 | 63.1 | 45.9 | 22.5 | 47.3 | 61.1 | 187 | 41 |
| Deform-DETRZhu et al. () | 300 | 46.2 | 65.2 | 50.0 | 28.8 | 49.2 | 61.7 | 173 | 40 |
| DAB-DETRLiu et al. () | 900 | 46.9 | 66.0 | 50.8 | 30.1 | 50.4 | 62.5 | 195 | 48 |
| DN-DETRLi et al. () | 900 | 48.6 | 67.4 | 52.7 | 31.0 | 52.0 | 63.7 | 195 | 48 |
| DINOZhang et al. () | 900 | 50.9 | 69.0 | 55.3 | 34.6 | 54.1 | 64.6 | 279 | 47 |
| FINOLA-DETR (frozen) | 100 | 45.5 | 63.8 | 49.5 | 25.1 | 49.1 | 63.5 | 112 | 44 |
| FINOLA-DETR (fine-tune) | 49.0 | 67.7 | 53.4 | 30.1 | 52.9 | 65.5 |  |  |  |

### B.6 Fine-tuning on COCO

Furthermore, fine-tuning the backbone on COCO further enhances detection performance. Table 27 provides a comprehensive comparison of fine-tuning results using the Mobile-Former Chen et al. () in the DETR Carion et al. () framework. Unlike the frozen backbone configuration, where FINOLA outperforms supervised pre-training significantly (as shown in Table 26), they achieve similar performance in COCO fine-tuning. This is because the advantage of FINOLA pre-training on spatial representation diminishes when object labels in COCO provide strong guidance. However, FINOLA maintains its leading position by leveraging fine-tuning on IN-1K to improve semantic representation and transfer it to object detection. Compared to the supervised baseline, FINOLA pre-training followed by IN-1K fine-tuning achieves a gain of 0.9–2.0 AP for all three encoders and two decoders.

Table 28 compares FINOLA-DETR (in which the backbone is fine-tuned in the DETR framework) with existed DETR baselines. FINOLA-DETR achieves an AP of 49.0, outperforming most DETR-based detectors except DINO Zhang et al. (). Remarkably, our method achieves these results while using significantly fewer FLOPs (112G vs. 279G) and object queries (100 vs. 900). When compared to DETR-DC5 with a fine-tuned backbone, FINOLA-DETR with a frozen backbone achieves a 2.2 AP improvement while reducing MAdds by 40%.

These results showcase the efficacy of FINOLA in capturing rich image representations even with more compact models, offering a promising approach for efficient self-supervised learning.

**Table 29: Comparing FINOLA and Masked FINOLA on ImageNet-1K. Masked FINOLA variants trade restoration accuracy for enhanced semantic representation. The block-wise masked FINOLA outperforms the element-wise variant in linear probing (lin), probing with a single transformer block (tran-1), and fine-tuning (tran-1-ft).**
| Model | Compress | Autoregression | Decoder | Recon-PSNR | lin | tran-1 | tran-1-ft |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FINOLA | ✓ | element | up+conv | 25.8 | 17.9 | 46.8 | 81.9 |
| Masked FINOLA-E | ✓ | element | linear | 16.7 | 54.1 | 67.8 | 82.2 |
| Masked FINOLA-B | ✗ | block | trans+linear | 17.3 | 66.4 | 78.7 | 82.5 |

**Table 30: Comparison between FINOLA and Masked FINOLA on ImageNet Deng et al. () classification: Compared to masked FINOLA variants, FINOLA performs poorly on both linear probing (lin) and probing with a single transformer block (tran-1) with clear margins. Even we search over the dimension of latent space from 64 to 3072, the gap is still large, i.e. more than 20%. Block-wise masked FINOLA (Masked-FINOLA-B) outperforms the element-wise variant (Masked-FINOLA-E), achieving higher accuracy. Please note that the encoders are frozen when performing linear and tran-1 probing.**
| Pre-training | Dim of $\bm{q}$ | lin | tran-1 |
| --- | --- | --- | --- |
|  | 64 | 10.2 | 20.2 |
|  | 128 | 11.5 | 24.0 |
|  | 256 | 15.0 | 29.0 |
| FINOLA | 512 | 20.1 | 34.1 |
|  | 1024 | 23.0 | 39.6 |
|  | 2048 | 23.2 | 41.1 |
|  | 3072 | 17.9 | 46.8 |
| Masked | 512 | 54.1 | 67.8 |
| FINOLA-E |  |  |  |
| Masked | —- | 66.4 | 78.7 |
| FINOLA-B |  |  |  |

## Appendix C Comparison between FINOLA and Masked FINOLA

### C.1 Detailed Experimental Results

Table 29 presents a comparison between vanilla FINOLA and two masked FINOLA variants, assessing both their architectural distinctions and performance in image reconstruction and classification tasks. The introduction of masking, a characteristic of masked FINOLA, entails a trade-off between restoration accuracy and enhanced semantic representation.

Comparison of FINOLA and Masked FINOLA on ImageNet classification:
Table 30 presents the results of linear and tran-1 probing applied to the vanilla FINOLA across various dimensions of the latent space. Notably, even the highest accuracy achieved by the vanilla FINOLA falls significantly behind both masked FINOLA variants (element-wise or block-wise). This stark difference highlights the remarkable power of masked prediction in learning semantic representations.

Comparison of FINOLA and Masked FINOLA on image reconstruction:
Figure 19 presents a comparison of reconstructed samples obtained using FINOLA and masked FINOLA. In the case of the two masked FINOLA variants (element-wise and block-wise), the encoders are frozen, and only their attentional pooling and FINOLA components are fine-tuned. To ensure a fair comparison, we utilize the same architecture for the decoders in the masked FINOLA variants as in FINOLA, training them from scratch. The corresponding peak signal-to-noise ratio (PSNR) values on the ImageNet validation set are provided at the bottom. While the masked variants preserve color and shape information, they exhibit a loss of texture details compared to the vanilla FINOLA. Notably, as demonstrated in the main paper, the masked FINOLA variants demonstrate stronger semantic representation. This comparison highlights that FINOLA and masked FINOLA adhere to the same mathematical principles (involving partial differential equations) but strike different balances between semantic representation and preserving fine details.

Comparison between two Masked FINOLA variants:
Figure 20 showcases the results of linear probing, tran-1 probing, and fine-tuning for two masked FINOLA variants trained with different schedules. The block-wise masked FINOLA consistently outperforms its element-wise counterpart across all evaluations. These findings demonstrate the effectiveness of directly applying FINOLA on the unmasked features to predict the masked region, as opposed to performing compression before applying FINOLA.

Figure: Figure 20: Comparison of element-wise and block-wise Masked FINOLA. The evaluation includes linear probing, tran-1 probing, and tran-1 fine-tuning. Block-wise masked FINOLA consistently outperforms the element-wise counterpart across all evaluations. Notably, the performance gap in fine-tuning is smaller compared to linear and tran-1 probing. Best viewed in color.
Refer to caption: x20.png

### C.2 Geometric Insight

Geometrically, Figure 21 illustrates masked FINOLA introduces a substantial increase in Gaussian curvature on critical feature surfaces, suggesting enhanced curvature in the latent space for capturing semantics.

Figure: Figure 21: FINOLA vs. Masked FINOLA on Gaussian curvature of critical features. Masked FINOLA demonstrates significantly larger curvature on critical features than vanilla FINOLA, highlighting the effectiveness of masked prediction in curving the latent space to capture semantics. Best viewed in color.
Refer to caption: x21.png

### C.3 Calculation of Gaussian Curvature

To compute the Gaussian curvature, we consider the feature map per channel as a set of $W\times H$ surfaces $z_{k}(x,y)$ in 3D space, where $x$, $y$, and $z_{k}$ denote the coordinates. At each position $(x,y)$, the Gaussian curvature for the $k^{th}$ channel can be determined using the following equation:

$$ $\displaystyle\kappa_{k}(x,y)=\frac{\frac{\partial^{2}z_{k}}{\partial x^{2}} \frac{\partial^{2}z_{k}}{\partial y^{2}}-\left(\frac{\partial^{2}z_{k}}{ \partial x\partial y}\right)^{2}}{\left(1+(\frac{\partial z_{k}}{\partial x})^ {2}+(\frac{\partial z_{k}}{\partial y})^{2}\right)^{2}}.$ (10) $$

Gaussian curvature is computed for all channels at each grid element. Subsequently, channels within each image are sorted based on the root mean square of the peak positive curvature ($\kappa_{+}$) and the peak negative curvature ($\kappa_{-}$) over the surface.

## Appendix D Mathematical Derivation

### D.1 Normalization after Diagonalization

Below, we provide the derivation of $\hat{\bm{\psi}}_{i}$ in Eq. 7.

$$ $\displaystyle\hat{\bm{\psi}}_{i}=\frac{(C\bm{I}-\bm{J})\bm{V\psi}_{i}}{\sqrt{ \bm{\psi}_{i}^{T}\bm{V}^{T}(C\bm{I}-\bm{J})\bm{V}\bm{\psi}_{i}}}.$ (11) $$

A FINOLA path is described as $\Delta_{x}\phi_{i}=\bm{A}\hat{\phi}_{i}$ (Eq. 3 in the paper), where $\hat{\phi}_{i}$ is a normalized $\phi_{i}$, i.e. $\phi_{i}=\frac{\phi_{i}-\mu}{\sigma}$. After diagonalization of $\bm{AB}^{-1}=\bm{V\Lambda V}^{-1}$, the FINOLA vectors $\phi_{i}$ are projected into $\psi_{i}=\bm{V}^{-1}\phi_{i}$, where each $\psi_{i}$ satisfies a one-way wave equation.

We attempt to rewrite $\psi_{i}$ in the FINOLA format as $\Delta_{x}\psi_{i}=\bm{H}_{A}\hat{\psi}_{i}$ (similar to $\phi_{i}$ before projection $\Delta_{x}\phi_{i}=\bm{A}\hat{\phi}_{i}$). $\hat{\psi}_{i}$ is not a simple normalization. The derivation of $\bm{H}_{A}$ and $\hat{\psi}_{i}$ is shown below step by step:

$$ $\displaystyle\begin{aligned} \Delta_{x}\psi_{i}&=\bm{V}^{-1}\Delta_{x}\phi_{i} \qquad\qquad\qquad\qquad\qquad\qquad(\psi_{i}=\bm{V}^{-1}\phi_{i})\\ &=\bm{V}^{-1}\bm{A}\hat{\phi}_{i}=\bm{V}^{-1}\bm{A}\frac{\phi_{i}-\mu}{\sigma} \quad\qquad\qquad(\Delta_{x}\phi_{i}=\bm{A}\hat{\phi}_{i})\\ &=\bm{V}^{-1}\bm{A}\frac{\phi_{i}-\frac{1}{C}\bm{J}\phi_{i}}{\sqrt{\frac{1}{C} \phi_{i}^{T}\phi_{i}-\frac{1}{C^{2}}\phi_{i}^{T}\bm{J}\phi_{i}}}\quad\qquad\;( \mu,\sigma\;in\;matrix\;format,\;\bm{J}\;is\;all\;one\;matrix)\\ &=\bm{V}^{-1}\bm{A}\frac{(C\bm{I}-\bm{J})\phi_{i}}{\sqrt{\phi_{i}^{T}(C\bm{I}- \bm{J})\phi_{i}}}\\ &=\bm{V}^{-1}\bm{A}\frac{(C\bm{I}-\bm{J})\bm{V}\psi_{i}}{\sqrt{\psi_{i}^{T}\bm {V}^{T}(C\bm{I}-\bm{J})\bm{V}\psi_{i}}}\quad\qquad(\phi_{i}=\bm{V}\psi_{i})\\ \end{aligned}$ (12) $$

Thus, we have:

$$ $\displaystyle\bm{H}_{A}=\bm{V}^{-1}\bm{A},\qquad\hat{\psi}_{i}=\frac{(C\bm{I}- \bm{J})\bm{V}\psi_{i}}{\sqrt{\psi_{i}^{T}\bm{V}^{T}(C\bm{I}-\bm{J})\bm{V}\psi_ {i}}}.$ (13) $$

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - **Latent Speeds (Abstract, Line 2)**: What does "latent speeds" refer to in the second line of the abstract?

- **Optimality of FINOLA Solution (Eq. 2)**: Why might the solution of FINOLA in Equation 2 not be the optimal one? Isn’t it supposed to be unique?

- **Definition of $\widehat{\phi}$ (Equation 3)**: What is $\widehat{\phi}$ in Equation 3? If it represents a normalization of $\phi$, how is this performed? Is it similar to the normalization for $z$ in Equation 1?

- **Feature Map Resolution (Line 195)**: Is FINOLA unable to produce a feature map at a resolution of 1/8 of the original image?

- **Data Dependency of Learned A and B**: Are the learned matrices $A$ and $B$ data-dependent? If the test set changes, would the learned $A$ and $B$ no longer be applicable?
soundness: 3
strengths: + The proposed idea of the paper is very interesting. 
+ The authors have included numerous experimental results.
summary: This paper empirically investigates the invariance of images using wave equations. Interestingly, the paper showes that images share a wave-equations in latent space (Changes is vertical direction of the image is proportional to horizontal direction), where each individual image can be uniquely estimated by initial conditions.

 Initial condition is obtained using an encoder, where it goes through first-order norm+linear autoregressive process to produce a feature map (solutions of the wave equation). The image in pixel space is then  reconstructed using convolutional layer and upsampling layer from the feature map.
weaknesses: 1. The paper lacks a logical flow, making it challenging to read. While each individual section is well-written, the overall structure makes it difficult for readers to follow a coherent narrative. Here are some specific points:
- **Focus of the Paper**: The title and much of the abstract emphasize "Exploring Invariance," with a focus on demonstrating invariance in images using wave equations. However, the paper appears to focus on FINOLA as the main contribution, focusing on its performance and characteristics. 

- **Purpose and Structure of Sections**: If the primary goal is to demonstrate invariance through FINOLA, then sections on *Parallel Implementation* and *Importance of Norm+Linear* relate more to the efficient training of FINOLA and the utility of norm+linear components. These sections feel tangential to the paper’s message and might be better suited to the experiments section or supplementary material.

- **Rationale for Generalization**: The paper does not completely explain the rationale for generalizing to a set of wave equations (section 3) beyond performance improvements. If enhancing performance is not within the paper's scope, as claimed by the authors, then extra clarification to why we need the generalization  is necessary.

- **Unexplained or Disconnected Claims**: The paper includes unrelated discussions and unexplained claims. For example, in the statement:
  
  > Relaxing the FINOLA constraint through FINOLA series: FINOLA represents a specific solution to Eq. 2,....
  
  there is no justification provided as to why relaxing the constraint would lead to a more optimal solution.


- **Section 5**: The introduction of FINOLA’s application in self-supervised learning is not fully explained. 



2. The paper shares significant overlap with reference [1]. This would be acceptable if the authors explicitly referenced this prior work and clarified how their approach differs, yet there is no mention of it.

  > **[1]** Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Lu Yuan, Zicheng Liu, Youzuo Lin, "Image as First-Order Norm+ Linear Autoregression: Unveiling Mathematical Invariance," arXiv 2305.16319.



3. The authors suggest that the invariance property inherent to FINOLA could provide new insights into image analysis. However, the paper lacks a discussion on potential applications of this property or its implications for future research.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: My questions are related to the above mentioned weaknesess

* Can you elaborate on the meaning/interpretation of your result? [I acknowledge this may be difficult to do within the 9 page restriction, but you can do it in the appendices. I would understand the "that is matter for further work" response, but you would understand that the reader may be not satisfied with that response. Understanding is what the reader wants].

* What is the qualitative meaning of the encoder? (see above)
  
* What is the meaning of the matrices A and B? (see above)

* Why is relevant the normalization over features?. Can you show reconstructions -both z and in the image domain- with and without normalization?. Discuss possibly related normalizations in related work.

* Can you compare to other (more interpretable) entropy autoencoders? (see above). Note that the dimension of the representation for unbounded variance/entropy is misleading. I guess the results of your model will be good given the comparison with JPEG.
soundness: 3
strengths: * The ablation study is thorough and highlights interesting properties either of natural images or of the encoding domain… or, better said ;-), of that specific signal in that specific domain (natural images in this encoding domain).
Examples of the intriguing properties include: the validity of the matrices of coefficients A and B for may images, the relevance of the variance normalization of the features in the encoding domain, and the meaningful reconstructions obtained from linear combinations in the encoding domain. 

* The comparison with other image autoencoders is extensive and suggest the good properties of the proposed domain is better than linear DCT and Wavelet encoders and nonlinear convolutional autoenc. (for the same number of coefficents or dimensions), and better than JPEG (for the same entropy). 

* Interesting references are given in the related work section. Thanks for those!
summary: The authors have discovered a spatial auto-regression property of natural images in the inner domain of a specific autoencoder also proposed in this work.

The autoencoder is similar in spirit to U-net convolutional autoencoders, with the encoding part based on a previously reported Mobile-Former [Chen et al. CVPR 22]. The main differences of the proposed scheme are: (1) final part of the encoder (the bottleneck of reduced spatial resolution) is compressed to a single vector corresponding to a single spatial location, and (2) the initial decoding layers (instead of upscaling+convolution) are substituted by a process that spatially expands the encoding vector through a differential equation whose coefficients are learnt. After this (original) spatial expansion leading to an intermediate feature representation with low spatial resolution, the decoder consists of the usual (upscaling+convolution) layers.

The discovered property is that the coeffcients of the differential equation that govern the way the information of the encoding vector propagates to different spatial locations is valid for many different natural images. This represents a spatial-autoregression property of natural images (in that specific encoding domain), which is not intuitively illustrated in the work.
weaknesses: * The important problem of this work is that no intuition is given, and all proposals and findings have to be taken as-they-are. If the meaning of the encoding vector is not explained (if it can be explained at all), then, the property found is of no use because one does not know how to understand the (otherwise interesting) constancy of the coefficients of the differential equation. For instance, when one derives classical autoregressive models of the images in the spatial domain, one can interpret the coefficients of these models as autocorrelation functions that describe the (prediction) relations between the luminances at different points of the images. The above sentence can be illustrated both visually and in predictive coders and in conditional probability markov models. However, what is the meaning of things here?. Specifically:

* What is the qualitative meaning of the encoder? (and hence of the encoding vector). Incidentally, this is not very much clear in the original paper on Mobile-Former [Chen et al. CVPR 22]. Can you show receptive fields of the coeficients of the encoding vector?. Can you compare vectors corresponding to different spatial locations?. Do they contain different seeds that are related using the differential equation?
  
* What is the meaning of the matrices A and B?. Can you visualize and interpret them? (this interpretation depends on the previous question). When you diagonalize, and you find the transformed coefficients, what (qualitative) information is transferred in the horizontal and vertical directions?

* You suggest that the normalization of the z features is critical for the wave equation. However, why is relevant the normalization over features?. In which way is this normalization similar or different from other normalization schemes sucha as the usual batch normalization, or the (more interesting) local normalization as in the biological Divisive Normalization [Heeger Science 92, Carandini & Heeger Nature Rev. Neurosci. 12, Malo et al. J.Nonlin.Sci. 24], also used in state-of-the-art end-to-end optimized autoencoders [Ballé et al. ICLR 17] and in the local normalization in Alexnet [Krizhevsky et al. NeurIPS 12]. A possibility is that this normalization captures predictable information [Schwartz & Simoncelli Nat Neurosci. 01, Malo & Laparra Neur. Comp. 10] and hence transforms the (complicated) nonlinear relations into (easier to describe) linear relations, which are more invariant [Hernandez et al. Patt.Rec.Lett.24] and hence, it may make the linear differential formulation possible.

* The comparison of the performance with other autoencoders is illustrative, however, I am curious about how this will compare with more interpretable autoencoders using not only the number of coefficients but the information they contain (i.e. entropy, as in the comparison with JPEG)... what about JPEG2000 [Marcellin Kluwer 02], or end-to-end divisive-normalized optimized linear transforms [Balle et al. ICLR 17].

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: What do you think your results teach us about natural images? 

How do you think the matrices A and B encode spatial information? Have you examined the learned matrices?

Does your network manage to compress other signals that are not natural images?

Do you think that the autoregressive approach is better than simply reshaping q so that it is a small image (e.g. reshape q which is 2048 into a 16x16x8 feature map)? 

Equation (1) is similar (although not identical) to the power method for finding eigenvectors of (I+A). If you iterate equation (1) many times, does it not forget the initial value of q and converge to some vector that does not depend on q?
soundness: 3
strengths: I like the author's goal of focusing on finding new properties of natural images and I find the approach original.

I also find the results intriguing: at first glance it seems as if the network is managing to encode the image without any spatial information. This is because the vector q is placed at a single location, and then all other locations are generated by two fixed matrices A, B which do not depend on the location. Upon further consideration, one can imagine the matrices A and B implicitly encoding a mapping that maps certain coordinates in q to certain locations.
summary: The paper presents a novel encoder-decoder architecture for images and uses that architecture to argue for a new insight about natural images. The encoder consists of two stages: (1) an  encoder that produces a single vector q and (2) a linear+normalization autoregressor that converts q into a feature map that is the size of the image (or of smaller sizes). A relatively simple decoder can then reconstruct the original image from the feature map. Extensive experiments compare the psnr of the reconstruction to other methods but the authors point out that their main interest is in " it’s crucial to emphasize that our aim isn’t state-of-the-art performance but to empirically reveal a property inherent in images"
weaknesses: At the end of the day, the proposed method is an auto-encoder and there is no inherent reason that this particular architecture for the encoder should be better than any other architecture. Indeed some of the other auto-encoder architectures are shown to outperform FINOLA according to the authors (e.g. stable diffusion in the appendix), and I am not sure what to make of the comparisons: surely the performance of all the architectures is highly dependent on hyperparameters.

Although the authors say that their main goal is to reveal "a property inherent in natural images", they do not say much in the paper about what they have learned from this property or what are the consequences of having such a property (see question below):

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please respond to my criticisms in the weaknesses section. If the authors can persuade me of their core theoretical claims, I will revise my review.
soundness: 1
strengths: Experiments are very thorough, paper looks professional in formatting and presentation. The appendix contains an incredible 10 additional pages of results with 21 tables and 16 figures. Clearly an extensive amount of work has gone into this paper.
summary: The authors introduce a novel image encoder/decoder architecture, First Order Norm + Linear Autoregression (FINOLA). They compare against several other image representations, such as Discrete Cosine Transform, Discrete Wavelet Transform, several other autoencoders, and find that their method compares favorably. They perform many experiments on this encoder for lots of other applications; such as image inpainting/outpainting, self-supervised learning, and much more. However, their central thesis is situated around two claims: 1) that their latent representations are solutions to one-way wave equations and 2) that their experiments somehow reveal "an invariance over images". I am suspicious of both claims.
weaknesses: I am dubious of this paper’s core narrative. I don’t think this paper is about image invariance or one-way wave equations.

First, I don’t think the authors have discovered a meaningful image invariance. As I see it, the authors have devised an autoencoder with an unusual architecture. They claim that, since the autoencoder’s learned parameters are constant across all images, this represents some kind of invariance inherent in images. At best, I think this stretches the definition of invariance. At worst I think it is philosophically confused. The relationship to wave equations feels like an arbitrary constraint on the autoencoder's latent representations. If this characterization is unfair, feel free to debate me on this.

Second, I am also dubious that the FINOLA encodings really are solutions to a 1-way wave equation. Have you empirically checked that equation 1 actually holds for all x and y values in your solution $z(x,y)$, when $z(x,y)$ is constructed as shown in figure 4? Because I suspect this property may not hold, which would then invalidate your discussion about wave equations. 

While the experimental results from this autoencoder architecture seem promising, I am too suspicious of the paper’s central claims to recommend it for publication.

Meta-review: The paper studies image representation. It develops an encoder-decoder pair with a special structure: the encoder maps the input image to a C dimensional feature vector q. Meanwhile, the decoder applies learned spatially invariant operations as follows: the vector q is placed at the center pixel, and then propagated outward via a learned difference equation (in feature space), in which increments are generated by applying learned matrices A and B to statistically normalized features. The paper observes that in this scheme, the matrices A and B are not only position-independent — they are in fact image-independent. This entire scheme is called FINOLA (first order norm + linear autoregression). The fact that these difference equations can be chosen in an image-independent fashion can be considered an approximate invariance associated with natural images — these images satisfy a common system of equations. The paper explores a multi-path variant which satisfies a one-way wave equation relating the horizontal and vertical increments in feature space. Experiments compare this image representation to classical transform coding (DCT/DWT/JPEG) and convolutional autoencoders, and apply the proposed representation to self-supervised learning. 

On the positive side, the paper introduces a novel approach to autoencoding through a learned autoregressive process in feature space. This approach outperforms classical transform coding baselines. As the authors note, it is interesting that the spatial structure of images can be reproduced by applying a space invariant learned operation in feature space. At the same time, the paper frames its main contribution as discovering an invariance satisfied by natural images (rather than producing SOTA results on image representation); this contribution would be stronger with clearer intuitions for the nature of this invariance, i.e., what the matrices A and B are capturing. Reviewers produced a mixed evaluation of this core contribution, ranging from skepticism to enthusiasm. As noted by one of the reviewers, it is not clear if this particular encoder-decoder structure outperforms simpler baselines such as reshaping.

Decision: Reject