Building Vision Models upon Heat Conduction

## Abstract

Abstract A fundamental problem in learning robust and expressive visual representations lies in efficiently estimating the spatial relationships of visual semantics throughout the entire image.
In this study, we propose vHeat, a novel vision backbone model that simultaneously achieves both high computational efficiency and global receptive field.
The essential idea, inspired by the physical principle of heat conduction, is to conceptualize image patches as heat sources and model the calculation of their correlations as the diffusion of thermal energy.
This mechanism is incorporated into deep models through the newly proposed module, the Heat Conduction Operator (HCO), which is physically plausible and can be efficiently implemented using DCT and IDCT operations with a complexity of 𝒪 ⁢ ( N 1.5 ) 𝒪 superscript 𝑁 1.5 \mathcal{O}(N^{1.5}) caligraphic_O ( italic_N start_POSTSUPERSCRIPT 1.5 end_POSTSUPERSCRIPT ) .
Extensive experiments demonstrate that vHeat surpasses Vision Transformers (ViTs) across various vision tasks, while also providing higher inference speeds, reduced FLOPs, and lower GPU memory usage for high-resolution images. The code will be released at https://github.com/MzeroMiko/vHeat and https://openi.pcl.ac.cn/georgew/vHeat .

## 1 Introduction

Convolutional Neural Networks (CNNs) [, ] have been the cornerstone of visual representation since the advent of deep learning, exhibiting remarkable performance across vision tasks.
However, the reliance on local receptive fields and fixed convolutional operators imposes constraints, particularly in capturing long-range and complex dependencies within images [].
These limitations have motivated significant interest in developing alternative visual representation models, including architectures based on ViTs [, ] and State Space Models [, ].
Despite their effectiveness, these models continue to face challenges, including relatively high computational complexity and a lack of interpretability.

When addressing these limitations, we draw inspiration from the field of heat conduction, where spatial locality is crucial for the transfer of thermal energy due to the collision of neighboring particles.
Notably, analogies can be drawn between the principles of heat conduction and the propagation of visual semantics within the spatial domain, as adjacent image regions in a certain scale tend to contain related information or share similar characteristics.
Leveraging these connections, we introduce vHeat, a physics-inspired vision backbone model that conceptualizes image patches as heat sources and models the calculation of their correlations as the diffusion of thermal energy.

To integrate the principle of heat conduction into deep networks, we first derive the general solution of heat conduction in 2D space and extend it to multiple dimensions, corresponding to various feature channels.
Based on this general solution, we design the Heat Conduction Operator (HCO), which simulates the propagation of visual semantics across image patches along multiple dimensions.
Notably, we demonstrate that HCO can be approximated through 2D DCT and IDCT operations, effectively reducing the computational complexity to $\mathcal{O}(N^{1.5})$.
This improvement boosts both training and testing efficiency due to the high parallelizability of DCT and IDCT operations.
Furthermore, as each element in the frequency domain obtained by DCT incorporates information from all patches in the image space, vHeat can establish long-range feature dependencies and achieve global receptive fields.
To enhance the representation adaptability of vHeat, we propose learnable frequency value embeddings (FVEs) to characterize the frequency information and predict the thermal diffusivity of visual heat conduction.

Figure: Figure 1: Comparison of information conduction mechanisms: self-attention vs. heat conduction. (a) The self-attention operator uniformly “conducts” information from a pixel to all other pixels, resulting in $\mathcal{O}(N^{2})$ complexity. (b) The heat conduction operator (HCO) conceptualizes the center pixel as the heat source and conducts information propagation through DCT ($\mathcal{F}$) and IDCT ($\mathcal{F}^{-1}$), which enjoys interpretability, global receptive fields, and $\mathcal{O}(N^{1.5})$ complexity.
Refer to caption: x1.png

We develop a family of vHeat models (i.e., vHeat-Tiny/Small/Base), and extensive experiments are conducted to demonstrate their effectiveness in diverse visual tasks.
Compared to benchmark vision backbones with various architectures (e.g., ConvNeXt [], Swin [], and Vim []), vHeat consistently achieves superior performance on image classification, object detection, and semantic segmentation across model scales.
Specifically, vHeat-Base achieves a $83.9\%$ top-1 accuracy on ImageNet-1K, surpassing Swin by $0.4\%$, with a throughput exceeding that of Swin by a substantial margin over $40\%$ ($661$ vs. $458$).
Besides, due to the $\mathcal{O}(N^{1.5})$ complexity of HCO, vHeat enjoys considerably lower computational cost compared to ViT-based models, demonstrating significantly reduced FLOPs and GPU memory requirements, and higher throughput as image resolution increases.
In particular, when the input image resolution increases to $768\times 768$, vHeat-Base achieves a 3$\times$ throughput compared to Swin, with over 70% less GPU memory allocation and more than 30% fewer computational FLOPs.

The contributions of this study are summarized as follows:

- •
We propose vHeat, a vision backbone model inspired by the physical principle of heat conduction, which simultaneously achieves global receptive fields, low computational complexity, and high interpretability.
- •
We design the Heat Conduction Operator (HCO), a physically plausible module conceptualizing image patches as heat sources, predicting adaptive thermal diffusivity by FVEs, and transferring information following the principles of heat conduction.
- •
Without bells and whistles, vHeat achieves promising performance in vision tasks including image classification, object detection, and semantic segmentation. It also enjoys higher inference speeds, reduced FLOPs, and lower GPU memory usage for high-resolution images.

## 2 Related Work

Convolution Neural Networks.
CNNs have been landmark models in the history of visual perception  [, ].
The distinctive characteristics of CNNs are encapsulated in the convolution kernels, which enjoy high computational efficiency given specifically designed GPUs.
With the aid of powerful GPUs and large-scale datasets [], increasingly deeper [, , , ] and efficient models [, , , ] have been proposed for higher performance across a spectrum of vision tasks.
Numerous modifications have been made to the convolution operators to improve its capacity [], efficiency [, ] and adaptability [, ].
Nevertheless, the born limitation of local receptive fields remains. Recently developed large convolution kernels [] took a step towards large receptive fields, but experienced difficulty in handling high-resolution images.

Vision Transformers.
Built upon the self-attention operator [], ViTs have the born advantage of building global feature dependency. Based on the learning capacity of self-attention across all image patches, ViTs has been the most powerful vision model ever, given a large dataset for pre-training [, , ]. The introduction of hierarchical architectures [, , , , , , , , ] further improves the performance of ViTs.
The Achilles’ Heel of ViTs is the $\mathcal{O}(N^{2})$ computational complexity, which implies substantial computational overhead given high-resolution images. Great efforts have been made to improve model efficiency by introducing window attention, linear attention and cross-covariance attention operators [, , , ], at the cost of reducing receptive fields or non-linearity capacity. Other studies proposed hybrid networks by introducing convolution operations to ViTs [, , ], designing hybrid architectures to combine CNN with ViT modules [, , ].

State Space Models and RNNs.
State space models (SSMs) [, , ], which have the long-sequence modeling capacity with linear complexity, are also migrated from the natural language area (Mamba []). Visual SSMs were also designed by adapting the selective scan mechanism to 2-D images [, ].
Nevertheless, SSMs built upon the selective scan mechanism lack the advantages of high parallelism, which limit its potential.

Recent receptance weighted key value (RWKV) and RetNet models [, ] improved the parallelism while retaining the linear complexity. They combine the efficient parallelizable training of transformers with the efficient inference of RNNs, leveraging a linear attention mechanism and allowing formulation of the model as either a Transformer or an RNN, thus parallelizing computations during training and maintaining constant computational and memory complexity during inference. Despite the advantages, modeling a 2-D image as a sequence impairs interpretability.

Biology and Physics Inspired Models.
Biology and physics principles have long been the fountainhead of creating vision models.
Diffusion models [, , ], motivated by Nonequilibrium thermodynamics [], are endowed with the ability to generate images by defining a Markov chain for the diffusion step.
QB-Heat [] utilizes physical heat equation as supervision signal for masked image modeling task. Spiking Neural Network (SNNs) [, , ] claims better simulation on the information transmission of biological neurons, formulating models for simple visual tasks [].
The success of these models encourages us to explore the principle of physical heat conduction for the development of vision representation models.

## 3 Methodology

### 3.1 Preliminaries: Physical Heat Conduction

Let $u(x,y,t)$ denote the temperature of point $(x,y)$ at time $t$ within a two-dimensional region $D\in\mathbb{R}^{2}$, the classic physical heat equation [] can be formulated as

$$ $\frac{\partial{u}}{\partial{t}}=k\left(\frac{\partial^{2}{u}}{\partial{x^{2}}} +\frac{\partial^{2}{u}}{\partial{y^{2}}}\right),$ (1) $$

where $k>0$ is the thermal diffusivity [] measuring the rate of heat transfer in a material.
By setting the initial condition $u(x,y,t)|_{t=0}$ to $f(x,y)$, the general solution of Eq. (1) can be derived by applying the Fourier Transform (FT, denoted as $\mathcal{F}$) to both sides of the equation, which gives

$$ $\mathcal{F}\left(\frac{\partial{u}}{\partial{t}}\right)=k\mathcal{F}\left( \frac{\partial^{2}{u}}{\partial{x^{2}}}+\frac{\partial^{2}{u}}{\partial{y^{2}} }\right).$ (2) $$

Denoting $\widetilde{u}(\omega_{x},\omega_{y},t)$ as the FT-transformed form of $u(x,y,t)$, i.e., $\widetilde{u}(\omega_{x},\omega_{y},t)\coloneqq\mathcal{F}(u(x,y,t))$, the left-hand-side of Eq. (2) can be written as

$$ $\mathcal{F}\left(\frac{\partial{u}}{\partial{t}}\right)=\frac{\partial{ \widetilde{u}(\omega_{x},\omega_{y},t)}}{\partial{t}}.$ (3) $$

and by leveraging the derivative property of FT, the right-hand-side of Eq. (2) can be transformed as

$$ $\mathcal{F}\left(\frac{\partial^{2}{u}}{\partial{x^{2}}}+\frac{\partial^{2}{u} }{\partial{y^{2}}}\right)=-(\omega_{x}^{2}+\omega_{y}^{2})\widetilde{u}(\omega _{x},\omega_{y},t).$ (4) $$

Therefore, by combining the expression of both sides of the equation, Eq. (2) can be formulated as an ordinary differential equation (ODE) in the frequency domain, which can be written as

$$ $\frac{d{\widetilde{u}(\omega_{x},\omega_{y},t)}}{dt}=-k(\omega_{x}^{2}+\omega_ {y}^{2})\widetilde{u}(\omega_{x},\omega_{y},t).$ (5) $$

By setting the initial condition $\widetilde{u}(\omega_{x},\omega_{y},t)|_{t=0}$ to $\widetilde{f}(\omega_{x},\omega_{y})$ ($\widetilde{f}(\omega_{x},\omega_{y})$ denotes the FT-transformed $f(x,y)$), $\widetilde{u}(\omega_{x},\omega_{y},t)$ in Eq (5) can be solved as

$$ $\widetilde{u}(\omega_{x},\omega_{y},t)=\widetilde{f}(\omega_{x},\omega_{y})e^{ -k(\omega_{x}^{2}+\omega_{y}^{2})t}.$ (6) $$

Finally, the general solution of heat equation in the spatial domain can be obtained by performing inverse Fourier Transformer ($\mathcal{F}^{-1}$) on Eq. (6), which gives the following expression

$$ $\displaystyle u(x,y,t)$ $\displaystyle=\mathcal{F}^{-1}(\widetilde{f}(\omega_{x},\omega_{y})e^{-k( \omega_{x}^{2}+\omega_{y}^{2})t})$ (7) $\displaystyle=\frac{1}{4\pi^{2}}{\int_{\widetilde{D}}}\widetilde{f}(\omega_{x} ,\omega_{y})e^{-k(\omega_{x}^{2}+\omega_{y}^{2})t}e^{i({\omega_{x}}x+{\omega_{ y}}y)}d{\omega_{x}}d{\omega_{y}}.$ (8) $$

Figure: Figure 2: HCO and HCO layer. FVEs, FFN, LN, DWConv respectively denote frequency value embeddings, feed-forward network, layer normalization, and depth-wise convolution(^2^22Please refer to Sec. E.3 in Appendix, where we demonstrate that while depth-wise convolution aids in feature extraction, the primary improvements are attributed to the proposed HCO.).
Refer to caption: x2.png

### 3.2 vHeat: Visual Heat Conduction

Drawing inspiration from the analogies between the principles of physical heat conduction and the propagation of visual semantics within the spatial domain (i.e., ‘visual heat conduction’), we propose vHeat, a physics-inspired deep architecture for visual representation learning.
The vHeat model is built upon the Heat Conduction Operator (HCO), which is designed to integrate the principle of heat conduction into handling the discrete feature of vision data.
We also leverage the thermal diffusivity in the classic physical heat equation (Eq (1)) to improve the adaptability of vHeat to vision data.

#### 3.2.1 Heat Conduction Operator (HCO)

To extract visual features, we design HCO to implement the conduction of visual information across image patches in multiple dimensions, following the principle of physical heat conduction.
To this end, we first extend the 2D temperature distribution $u(x,y,t)$ along the channel dimension and denote the resultant multi-channel image feature as $U(x,y,c,t)$ ($c=1,\cdots,C$).
Mathematically, considering the input as $U(x,y,c,0)$ and the output as $U(x,y,c,t)$, HCO simulates the general solution of physical heat conduction (Eq. (7)) in visual data processing, which can be formulated as

$$ $U^{t}=\mathcal{F}^{-1}(\mathcal{F}(U^{0})e^{-k(\omega_{x}^{2}+\omega_{y}^{2})t }),$ (9) $$

where $U^{t}$ and $U^{0}$ are abbreviations for $U(x,y,c,t)$ and $U(x,y,c,0)$, respectively.

For applying $\mathcal{F}(\cdot)$ and $\mathcal{F}^{-1}(\cdot)$ to discrete image patch features, it is necessary to utilize the discrete version of the (inverse) Fourier Transform (i.e., DFT and IDFT).
However, since vision data is spatially constrained and semantic information will not propagate beyond the border, we additionally introduce a common assumption of Neumann boundary condition [], i.e., $\partial{u(x,y,t)}/\partial{\mathbf{n}}=0,\forall{(x,y)\in{\partial}D,t\geq 0}$, where $\mathbf{n}$ denotes the normal to the image boundary $\partial{D}$.
As vision data is typically rectangular, this boundary condition enables us to replace the 2D DFT and IDFT with the 2D discrete cosine transformation, $\mathbf{DCT_{2D}}$, and the 2D inverse discrete cosine transformation, $\mathbf{IDCT_{2D}}$ [].
Therefore, the discrete implementation of HCO can be expressed as

$$ $U^{t}=\mathbf{IDCT_{2D}}(\mathbf{DCT_{2D}}(U^{0})e^{-k(\omega_{x}^{2}+\omega_{ y}^{2})t}),$ (10) $$

and its internal structure is illustrated in Fig. 2(a).
Particularly, the parameter $k$ stands for the thermal diffusivity in physical heat conduction and is predicted based on the features within the frequency domain (explained in the following subsection).

Notably, due to the computational efficiency of $\mathbf{DCT_{2D}}$, the overall complexity of HCO is $\mathcal{O}(N^{1.5})$, where $N$ denotes the number of input image patches.
Please refer to Sec. B in Appendix for the detailed implementation of HCO using $\mathbf{DCT_{2D}}$ and $\mathbf{IDCT_{2D}}$.

Figure: Figure 3: Illustration of temperature distribution $U^{t}$ $w.r.t.$ thermal diffusivity $k$, given a heat source as the initial condition. The predicted $k$ leads to nonuniform visual heat conduction, which facilitates the adaptability of visual representation. (Best viewed in color)
Refer to caption: x3.png

#### 3.2.2 Adaptive Thermal Diffusivity

In physical heat conduction, thermal diffusivity represents the rate of heat transfer within a material.
While in visual heat conduction, we hypothesize that more representative image contents contain more energy, resulting in higher temperatures in the corresponding image features within $U(x,y,c,t)$.
Therefore, it is suggested that the thermal diffusivity parameter $k$ should be learnable and adaptive to image content, which facilitates the adaptability of heat condution to visual representation learning.

Given that the output of $\mathbf{DCT}$ (i.e., $\mathbf{DCT_{2D}}(U^{0})$ in Eq. (10)) lies in the frequency domain, we also determine $k$ based on frequency values ($k\coloneqq k(\omega_{x},\omega_{y})$).
Since different positions in the frequency domain correspond to different frequency values, we propose to represent these values using learnable Frequency Value Embeddings (FVEs), which function similarly to the widely used absolute position embeddings in ViTs[] (despite in the frequency domain).
As shown in Figure 2 (a), FVEs are fed to a linear layer to predict the thermal diffusivity $k$, allowing it to be non-uniform and adaptable to visual representations.

Practically, considering that $k$ and $t$ (the conduction time) are multiplied in Eq. (10), we empirically set a fixed value for $t$ and predict the values of $k$.
Specifically, FVEs are shared within each network stage of vHeat to facilitate the convergence of the training process.

#### 3.2.3 vHeat Model

Network Architecture.
We develop a vHeat model family including vHeat-Tiny (vHeat-T), vHeat-Small (vHeat-S), and vHeat-Base (vHeat-B). An overview of the network architecture of vHeat is illustrated in Fig. 4, and the detailed configurations are provided in Sec. C in Appendix.
Given an input image with the spatial resolution of $H\times W$, vHeat first partitions it to image patches through a stem module, yielding a 2D feature map with $\frac{H}{4}\times\frac{W}{4}$ resolution.
Subsequently, multiple stages are utilized to create hierarchical representations with gradually decreased resolutions of $\frac{H}{4}\times\frac{W}{4}$, $\frac{H}{8}\times\frac{W}{8}$, $\frac{H}{16}\times\frac{W}{16}$ and increasing channels.
Each stage is composed of a down-sampling layer followed by multiple heat conduction layers (except for the first stage).

Figure: Figure 4: The network architecture of vHeat.
Refer to caption: x4.png

Heat Conduction Layer.
The heat conduction layer, Fig. 2 (b), is similar to the ViTs block while replacing self-attention operators with HCOs and retaining the feed-forward network (FFN).
It first utilizes a $3\times 3$ depth-wise convolution layer.
The depth-wise convolution is followed by two branches: one maps the input to HCO and the other computes the multiplicative gating signal like  [].
HCO plays a crucial role in each heat conduction layer, Fig. 2 (b), where the mapped features from a linear layer are first processed by the $\mathbf{DCT_{2D}}$ operator to generate features in the frequency domain.
Additionally, HCO takes FVEs as input for frequency representation to predict adaptive thermal diffusivity $k$ through a linear layer.
By multiplying the coefficient matrix $e^{-k{\omega^{2}}t}$ and performing $\mathbf{IDCT_{2D}}$, HCO implements the discrete solution of the visual heat equation, Eq. (10).

### 3.3 Discussion

$\bullet$ What is role of the thermal diffusivity coefficient $e^{-k(\omega_{x}^{2}+\omega_{y}^{2})t}$?
When multiplying with $\mathbf{DCT_{2D}}(U^{0})$, $e^{-k(\omega_{x}^{2}+\omega_{y}^{2})t}$ acts as an adaptive filter in the frequency domain to perform visual heat conduction.
Different frequency values correspond to distinct image patterns, $i.e.,$ high frequency corresponds to edges and textures while low frequency corresponds to flat regions.
With adaptive thermal diffusivity, HCO can enhance/depress these patterns within each feature channel. Aggregating the filtered features from all channels, vHeat achieves a robust feature representation.

$\bullet$ Why does temperature $U(x,y,c,t)$ correspond to visual features?
Visual features are essentially the outcome of the feature extraction process, characterized by pixel propagation within the feature map. This process aligns with the properties of existing convolution, self-attention, and selective scan operators, exemplifying a form of information conduction. Similarly, visual heat conduction embodies this concept of information conduction through temperature, denoted as $U(x,y,c,t)$.

$\bullet$ What is the relationship/difference between HCO and self-attention? HCO dynamically propagates energy via heat conduction, enabling the perception of global information within the input image. This positions HCO as a distinctive form of attention mechanism. The distinction lies in its reliance on interpretable physical heat conduction, in contrast to self-attention, which is formulated through token similarity. Furthermore, HCO works in the frequency domain, implying its potential to affect all image patches through frequency filtering. Consequently, HCO exhibits greater efficiency compared to self-attention, which necessitates computing the relevance of all pairs across image patches.

**Table 1: Performance comparison of image classification on ImageNet-1K. Test throughput values are measured with an A100 GPU, using the toolkit released by [], following the protocol proposed in []. The batch size is set as 128, and the PyTorch version is 2.2. Please refer to Sec. D in Appendix for complete comparisons.**
| Method | Image<br>size | #Param. | FLOPs | Test Throughput<br>(img/s) | ImageNet<br>top-1 acc. (%) |
| --- | --- | --- | --- | --- | --- |
| ConvNeXt-T [] | 224^2 | 29M | 4.5G | 1198 | 82.1 |
| ConvNeXt-S [] | 224^2 | 50M | 8.7G | 684 | 83.1 |
| ConvNeXt-B [] | 224^2 | 89M | 15.4G | 436 | 83.8 |
| Swin-T [] | 224^2 | 28M | 4.6G | 1244 | 81.3 |
| Swin-S [] | 224^2 | 50M | 8.7G | 718 | 83.0 |
| Swin-B [] | 224^2 | 88M | 15.4G | 458 | 83.5 |
| vHeat-T | 224^2 | 29M | 4.6G | 1514 | 82.2 |
| vHeat-S | 224^2 | 50M | 8.5G | 945 | 83.6 |
| vHeat-B | 224^2 | 87M | 14.9G | 661 | 83.9 |

**Table 2: Left: Results of object detection and instance segmentation on COCO dataset. FLOPs are calculated with input size $1280\times 800$. $AP^{b}$ and $AP^{m}$ denote box AP and mask AP, respectively. The notation ‘$1\times$’ indicates models fine-tuned for 12 epochs, while ‘$3\times$MS’ denotes the utilization of multi-scale training for 36 epochs. Right: Results of semantic segmentation on ADE20K using UperNet []. FLOPs are calculated with the input size of $512\times 2048$.**
| Mask R-CNN 1$\times$ schedule on COCO |  |  |  |  |
| --- | --- | --- | --- | --- |
| Backbone | AP${}^{\text{b}}$ | AP${}^{\text{m}}$ | #param. | FLOPs |
| Swin-T | 42.7 | 39.3 | 48M | 267G |
| ConvNeXt-T | 44.2 | 40.1 | 48M | 262G |
| vHeat-T | 45.1 | 41.0 | 53M | 286G |
| Swin-S | 44.8 | 40.9 | 69M | 354G |
| ConvNeXt-S | 45.4 | 41.8 | 70M | 348G |
| vHeat-S | 46.8 | 42.3 | 74M | 377G |
| Swin-B | 46.9 | 42.3 | 107M | 496G |
| ConvNeXt-B | 47.0 | 42.7 | 108M | 486G |
| vHeat-B | 47.7 | 43.0 | 115M | 526G |
| Mask R-CNN 3$\times$ MS schedule on COCO |  |  |  |  |
| Swin-T | 46.0 | 41.6 | 48M | 267G |
| ConvNeXt-T | 46.2 | 41.7 | 48M | 262G |
| vHeat-T | 47.2 | 42.4 | 53M | 286G |
| Swin-S | 48.2 | 43.2 | 69M | 354G |
| ConvNeXt-S | 47.9 | 42.9 | 70M | 348G |
| vHeat-S | 48.8 | 43.7 | 74M | 377G |

## 4 Experiment

Experiments are performed to assess vHeat and compare it against popular CNN and ViT models. Visualization analysis is presented to gain deeper insights into the mechanism of vHeat. The evaluation spans three vision tasks including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K. Please refer to Sec. C for experimental settings.

### 4.1 Image Classification

The image classification results are summarized in Table  1. With similar FLOPs, vHeat-T achieves a top-1 accuracy of $82.2\%$, outperforming DeiT-S by $2.4\%$, and Swin-T by $0.9\%$, respectively.
Notably, the superiority of vHeat is also observed at both Small and Base scales. Specifically, vHeat-B achieves a top-1 accuracy of $83.9\%$, outperforming DeiT-B by $2.1\%$, and Swin-B by $0.4\%$, respectively.

In terms of computational efficiency, vHeat enjoys significantly higher inference speed across Tiny/Small/Base model scales compared to benchmark models. For instance, vHeat-T achieves a throughput of 1514 image/s, $87\%$ higher than Vim-S, $26\%$ higher than ConvNeXt-T, and $22\%$ higher than Swin-T, while maintaining a performance superiority, respectively.

### 4.2 Downstream Task

Object Detection and Instance Segmentation.
As a backbone network, vHeat is tested on the MS COCO 2017 dataset [] for object detection and instance segmentation. We load classification pre-trained vHeat weights for downstream evaluation. Considering the input image size is different from the classification task, the shape of FVEs or $k$ should be aligned to the target image size on downstream tasks. Please refer to Sec. E.1 for ablation of interpolation for downstream tasks.
The results for object detection are summarized in Table  2 (left), and vHeat enjoys superiority in box/mask Average Precision (AP^b and AP^m) in both of the training schedules ($12$ or $36$ epochs).
For example, with a $12$-epoch fine-tuning schedule, vHeat-T/S/B models achieve object detection mAPs of $45.1\%/46.8\%/47.7\%$, outperforming Swin-T/S/B by $2.4\%/2.0\%/0.8\%$ mAP, and ConvNeXt-T/S/B by $0.9\%/1.4\%/0.7\%$ mAP, respectively.
With the same configuration, vHeat-T/S/B achieve instance segmentation mAPs of $41.0\%/42.3\%/43.0\%$, outperforming Swin-T/S/B by $1.7\%/1.4\%/0.7\%$ mAP, and ConvNeXt-T/S/B by $0.9\%/0.5\%/0.3\%$ mAP, respectively.
The advantages of vHeat persist under the $36$-epoch ($3\times$) fine-tuning schedule with multi-scale training.
These results showcase vHeat’s potential to achieve promising performance in downstream tasks with dense prediction.

Semantic Segmentation.
The results on ADE20K are summarized in Table  2 (right), and vHeat consistently achieves superior performance. For example, vHeat-B respectively outperform Swin-B [] and ConvNeXt-B [] by 1.5% and 0.5% mIoU.

### 4.3 Visualization Analysis

Visual Heat Conduction.
In Fig. 5, we visualize the temperature $U^{t}$ defined in Eq. (10) under predicted $k$ when a random patch is taken as the heat source.
With a predicted $k$, vHeat delivers self-adaptive visual heat conduction.
With the increase of heat conduction time ($t$), the correlation of the selected patch to the whole image is enhanced. Please refer to Sec. F in Appendix for more visualization instances.

Figure: Figure 5: Temperature distribution ($U^{t}$) when using a randomly selected patch as the heat source. (Best viewed in color)
Refer to caption: x5.png

Figure: Figure 6: Visualization of the effective receptive fields (ERF) []. Pixels of higher intensity indicate larger responses with the central pixel.
Refer to caption: x6.png

Receptive Field.
The Effective Receptive Field (ERF) [] of an output unit denotes the region of input that contains elements with a non-negligible influence on that unit.
In Fig. 6, ResNet, ConNeXT, and Swin have local ERF. DeiT [], HiViT [], and vHeat exhibit global ERFs. The difference lies in that DeiT and HiViT have a $\mathcal{O}(N^{2})$ complexity while vHeat enjoys $\mathcal{O}(N^{1.5})$ complexity.

Figure: Figure 7: Left / Mid / Right: Throughput / GPU memory / FLOPs under different image resolutions. The throughput and GPU memory are tested on 80 GB Tesla A100 GPUs with batch size 64. Swin-B is tested with scaled window size here.
Refer to caption: x7.png

### 4.4 Computational Cost

The comparisons of throughput / GPU memory / FLOPs of vHeat-B and other ViTs are shown in Fig. 7. Thanks to HCO’s $\mathcal{O}(N^{1.5})$ computational complexity $w.r.t.$ $N$ image patches, vHeat-B has a significant superiority over other base-level ViT models on throughput / FLOPs.
Fig. 7 (right) shows that with the increase of input image resolution, vHeat enjoys the slowest increase of computational overhead.
Fig. 7 (middle) shows that vHeat requires 70% GPU memory less than Swin-Transformer given large input images. Given the larger image resolution, the superiority becomes larger.
These demonstrate vHeat’s great potential to handle high-resolution images.

### 4.5 Ablation of Thermal Diffusivity

**Table 3: Evaluating thermal diffusivity $k$.**
| Settings | Acc |
| --- | --- |
| Fixed $k=0.0$ | 81.0 |
| Fixed $k=1.0$ | 81.7 |
| Fixed $k=5.0$ | 81.8 |
| $k$ as a learnable parameter | 81.5 |
| Predicting $k$ using individual FVEs | 82.0 |
| Predicting $k$ using shared FVEs | 82.2 |

To show the effectiveness of shared FVEs, we conduct the following experiments on ImageNet-1K. (1) Fix the thermal diffusivity $k=0.0/1.0/10.0$. (2) Treat $k$ as a learnable parameter for each layer. (3) Use individual FVEs to predict $k$ for each layer.
As shown in Table  3, when $k=0.0$, the visual heat conduction doesn’t work. A larger fixed $k$ value, $e.g.$, $k=5.0$, enables HCO to work isotropically without considering the image content and the performance reaches $81.7\%$ top-1 accuracy.
Predicting $k$ by FVEs outperforms treating $k$ as a learnable parameter, which may be attributed to the strengthened prior knowledge of frequency values provided by FVEs. Please refer to Sec. E.5 in Appendix for the detailed analysis.
When $k$ is predicted by shared FVEs, the performance improves to 82.2%, which validates shared FVEs can effectively reduce the learning diffusivity and further improve the performance.

## 5 Conclusion

We propose vHeat, a visual backbone model that leverages the advantages of global receptive fields, low complexity, and interoperability. The effectiveness of the vHeat model family, including vHeat-T/S/B models, has been demonstrated through extensive experiments and ablation studies, significantly outperforming popular CNNs and ViTs. The results highlight the potential of vHeat as a new paradigm for vision representation learning, offering fresh insights for the development of physics-inspired vision models.

Limitations.
Based on the physical heat conduction, the learning process of vHeat may become challenging when distant information conduction is required, as it needs extensive training to effectively perceive long-range dependencies. Moreover, masked image modeling serves as an effective self-supervised learning paradigm for ViTs. By now, we have not yet developed a self-supervised learning method for vHeat, which is left for the future work.

## Appendix A Motivation

Modern visual representation models are built upon the attention mechanism inspired by biological vision systems. One drawback of it is the lack of a clear definition of the relationship between biological electrical signals and brain activity (energy). This drives us to break through the attention mechanism and attempt other physical laws. Heat conduction is a physical phenomenon in nature, characterized by the propagation of energy. The heat conduction process combines implicit attention computation with energy computation and has the potential to be a new mechanism for visual representation models.

## Appendix B HCO implementation using 𝐃𝐂𝐓 𝟐 ⁢ 𝐃 subscript 𝐃𝐂𝐓 2 𝐃 \mathbf{DCT_{2D}} bold_DCT start_POSTSUBSCRIPT bold_2 bold_D end_POSTSUBSCRIPT and 𝐈𝐃𝐂𝐓 𝟐 ⁢ 𝐃 subscript 𝐈𝐃𝐂𝐓 2 𝐃 \mathbf{IDCT_{2D}} bold_IDCT start_POSTSUBSCRIPT bold_2 bold_D end_POSTSUBSCRIPT

Assume a matrix denoted as $\mathbf{A}$ and the transformed matrix denoted as $\mathbf{B}$, the $\mathbf{DCT_{2D}}$ and the $\mathbf{IDCT_{2D}}$ can be performed by

$$ $\displaystyle\mathbf{DCT_{2D}}:\mathbf{B}_{pq}=\mathbf{\alpha_{p}}\mathbf{ \alpha_{q}}\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}\mathbf{A}_{mn}\mathbf{cos}\frac{(2 m+1)p\pi}{2M}\mathbf{cos}\frac{(2n+1)q\pi}{2N},$ (11) $\displaystyle\mathbf{IDCT_{2D}}:\mathbf{A}_{mn}=\sum_{m=0}^{M-1}\sum_{n=0}^{N- 1}\mathbf{\alpha_{p}}\mathbf{\alpha_{q}}\mathbf{B}_{pq}\mathbf{cos}\frac{(2m+1 )p\pi}{2M}\mathbf{cos}\frac{(2n+1)q\pi}{2N},$ $$

where $0{\leq}\{p,m\}{\leq}M-1$, $0{\leq}\{q,n\}{\leq}N-1$, $\mathbf{\alpha_{p}}=\left\{\begin{aligned} &\frac{1}{\sqrt{M}},p=0\\
&\frac{2}{\sqrt{M}},p>0\end{aligned}\right.$, and $\mathbf{\alpha_{q}}=\left\{\begin{aligned} &\frac{1}{\sqrt{N}},q=0\\
&\frac{2}{\sqrt{N}},q>0\end{aligned}\right.$.
$M$ and $N$ respectively denote the row and column sizes of $\mathbf{A}$.
Considering the matrix multiplication is GPU-friendly, we implement the $\mathbf{DCT_{2D}}$ and $\mathbf{IDCT_{2D}}$ in Eq.  (11) by

$$ $\displaystyle\mathbf{C}$ $\displaystyle=(\mathbf{C}_{mp})_{M\times{M}}=\left(\mathbf{\alpha_{p}}\mathbf{ cos}\frac{(2m+1)p\pi}{2M}\right)_{M\times{M}},$ (12) $\displaystyle\mathbf{D}$ $\displaystyle=(\mathbf{D}_{nq})_{N\times{N}}=\left(\mathbf{\alpha_{q}}\mathbf{ cos}\frac{(2n+1)q\pi}{2N}\right)_{N\times{N}},$ $\displaystyle\mathbf{B}$ $\displaystyle=\mathbf{CAD^{T}},$ $\displaystyle\mathbf{A}$ $\displaystyle=\mathbf{C^{T}BD}.$ $$

Suppose the number of total patches is $N$ and the image is square, the shapes of $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$ and $\mathbf{D}$ are all $\sqrt{N}\times\sqrt{N}$, which illustrates the computational complexity of (12) and HCO is $O(N^{1.5})$.

## Appendix C Experimental Settings

Model configurations. The configurations of vHeat-T/S/B models are shown in Table  4.

**Table 4: Configurations of vHeat. The contents in the tuples represent configurations for four stages.**
| Size | Tiny | Small | Base |
| --- | --- | --- | --- |
| Stem | 3$\times$3 conv with stride 2; Norm; GELU; 3$\times$3 conv with stride 2; Norm |  |  |
| Downsampling | 3$\times$3 conv with stride 2; Norm |  |  |
| MLP ratio | 4 |  |  |
| Classifier head | Global average pooling, Norm, MLP |  |  |
| Layers | (2, 2, 6, 2) | (2, 2, 18, 2) | (2, 2, 18, 2) |
| Channels | (96, 192, 384, 768) | (96, 192, 384, 768) | (128, 256, 512, 1024) |

Image Classification.
Following the standard evaluation protocol used in [], all vHeat series are trained from scratch for $300$ epochs and warmed up for the first $20$ epochs. We utilize the AdamW optimizer [] during the training process with betas set to $(0.9,0.999)$, a momentum of $0.9$, a cosine decay learning rate scheduler, an initial learning rate of $2\times 10^{-3}$, a weight decay of $0.08$, and a batch size of $2048$. The drop path rates are set to $0.1/0.3/0.5$ for vHeat-T/S/B, respectively. Other techniques such as label smoothing ($0.1$) and exponential moving average (EMA) are also applied. No further training techniques are employed beyond these for a fair comparison. The training of vHeat-T/S/B takes 5/7/8.5 minutes per epoch on Tesla 16$\times$V100 GPUs.

object Detection.
Following the settings in Swin [] with the Mask-RCNN detector, we build the vHeat-based detector using the MMDetection library [].
The AdamW optimizer [] with a batch size of $16$ is used to train the detector.
The initial learning rate is set to $1\times 10^{-4}$ and is reduced by a factor of $10\times$ at the 9th and 11th epoch.
The fine-tune process takes $12$ ($1\times$) or $36$ ($3\times$) epochs.
We employ the multi-scale training and random flip technique, which aligns with the established practices for object detection evaluations.

Semantic Segmentation.
Following the setting of Swin Transfomer [], we construct a UperHead [] on top of the pre-trained vHeat model to test its capability for semantic segmentation.
The AdamW optimizer [] is employed and the learning rate is set to $6\times 10^{-5}$ with a batch size of $16$.
The fine-tuning process takes a total of standard $160k$ iterations and the default input resolution is $512\times 512$.

## Appendix D Performance Comparison

The complete comparison of vHeat and other vision models on ImageNet-1K is shown in Table  5.

**Table 5: Performance comparison of image classification on ImageNet-1K.**
| Method | Image<br>size | #Param. | FLOPs | Test Throughput<br>(img/s) | ImageNet<br>top-1 acc. (%) |
| --- | --- | --- | --- | --- | --- |
| DeiT-S [] | 224^2 | 22M | 4.6G | 1761 | 79.8 |
| DeiT-B [] | 224^2 | 86M | 17.5G | 503 | 81.8 |
| ConvNeXt-T [] | 224^2 | 29M | 4.5G | 1198 | 82.1 |
| ConvNeXt-S [] | 224^2 | 50M | 8.7G | 684 | 83.1 |
| ConvNeXt-B [] | 224^2 | 89M | 15.4G | 436 | 83.8 |
| HiViT-T [] | 224^2 | 19M | 4.6G | 1393 | 82.1 |
| HiViT-S [] | 224^2 | 38M | 9.1G | 712 | 83.5 |
| HiViT-B [] | 224^2 | 66M | 15.9G | 456 | 83.8 |
| XCiT-S/12 [] | 224^2 | 26M | 4.8G | 1283 | 82.0 |
| XCiT-S/24 [] | 224^2 | 48M | 9.1G | 671 | 82.6 |
| XCiT-M/24 [] | 224^2 | 84M | 16.2G | 423 | 82.7 |
| Vim-S [] | 224^2 | 26M | - | 811 | 80.5 |
| Swin-T [] | 224^2 | 28M | 4.6G | 1244 | 81.3 |
| Swin-S [] | 224^2 | 50M | 8.7G | 718 | 83.0 |
| Swin-B [] | 224^2 | 88M | 15.4G | 458 | 83.5 |
| vHeat-T | 224^2 | 29M | 4.6G | 1514 | 82.2 |
| vHeat-S | 224^2 | 50M | 8.5G | 945 | 83.6 |
| vHeat-B | 224^2 | 87M | 14.9G | 661 | 83.9 |

## Appendix E Additional Ablation Studies

### E.1 Interpolation of FVEs/ k 𝑘 k italic_k for downstream tasks

We have tried several approaches to align the shape for ablation. (1) Directly interpolate FVEs to the target shape of the input image. (2) Add 0 to the lower right region of FVEs to align the target shape. (3) Add 0 to the lower right region of FVEs to $512\times 512$, and interpolate to the target shape. (4) Directly interpolate the predicted thermal diffusivity $k$ to the target shape. The results are summarized in Table  6. Through the comparison, we select adding 0, then interpolating FVEs to the target shape for all downstream tasks.

**Table 6: Evaluating different methods to align the shape of FVEs/$k$ when loading ImageNet-1K pre-trained vHeat-B weights for detection and segmentation on COCO.**
| Method | AP${}^{\text{b}}$ | AP${}^{\text{m}}$ |
| --- | --- | --- |
| Interpolating FVEs to predict $k$ | 47.4 | 42.9 |
| Adding 0 to FVEs | 47.4 | 42.7 |
| Adding 0, then interpolating FVEs | 47.7 | 43.0 |
| Interpolating the predicted $k$ | 47.2 | 42.7 |

### E.2 Plain vHeat model

We’ve tested the performance of plain vHeat-B on ImageNet-1K classification. Keeping the same as DeiT-B, plain vHeat-B has 12 HCO layers, 768 embedding channels and the patch size is set to 16. Results are shown in Table  7. The superiority of plain vHeat-B over DeiT-B also validates the effectiveness of vHeat model.

**Table 7: Evaluating different methods to align the shape of FVEs when loading ImageNet-1K pre-trained vHeat-B weights for detection and segmentation on COCO.**
| Model | #Param. | FLOPs | Acc |
| --- | --- | --- | --- |
| DeiT-B | 86M | 17.5G | 81.8 |
| Plain vHeat-B | 88M | 16.9G | 82.6 |

### E.3 Depth-wise convolution

We conduct experiments to validate the performance improvement from DWConv. We replace depth-wise convolution with layer normalization for vHeat-B. Results are summarized in Table  8, and vHeat-B achieves 83.7% Top-1 accuracy on ImageNet-1K classification, 0.2% lower than with DWConv, which validates the main gains come from the proposed HCO. Besides, when $k$ is fixed as a large value, e.g. $k=10.0$, replacing DWConv with layer normalization causes a significant performance drop (-0.7% top-1 accuracy). The comparison validates predicting $k$ by FVEs can effectively improve the robustness of vHeat.

Additionally, we train vHeat without DWConv with a different recipe from vHeat with DWConv. The batch size is set as 1024, the initial learning rate is set as $1\times 10^{-3}$, and the weight decay is set as 0.05.

**Table 8: Ablation experiments of depth-wise convolution (DWConv).**
| Model | DWConv | Acc |
| --- | --- | --- |
| vHeat-B | ✓ | 83.9 |
| vHeat-B | ✗ | 83.7 (-0.2) |
| vHeat-B (fix $k$=10.0) | ✓ | 83.5 |
| vHeat-B (fix $k$=10.0) | ✗ | 82.8 (-0.7) |

### E.4 Global filters

Considering HCO works as a global filter in the frequency domain for visual heat conduction, we compare vHeat with (1) GFNet [], and (2) replacing HCO with the operators proposed in GFNet for ablation. Results are summarized in Table  9, vHeat-S has a large superiority over GFNet-H-B under approximate model scale. Besides, replacing HCO with operations proposed in GFNet achieves lower performance, which validates the effectiveness of the proposed HCO and visual heat conduction modeling for representation.

**Table 9: Comparison of vHeat with global filters, where vHeat-B^⋆ denotes replacing HCO with operators proposed in GFNet.**
| Model | #Param. | FLOPs | Acc |
| --- | --- | --- | --- |
| vHeat-S | 50M | 8.5G | 83.6 |
| GFNet-H-B [] | 54M | 8.4G | 82.9 |
| vHeat-B | 87M | 14.9G | 83.9 |
| vHeat-B^⋆ | 87M | 14.9G | 83.5 |

### E.5 Predicting k 𝑘 k italic_k by FVEs vs. treating k 𝑘 k italic_k as a learnable parameter

After performing DCT, the features lack explicit frequency value, while FVEs provide the model with prior knowledge of frequency values. Similar to how the introduction of positional encoding can enhance performance even in models that include positional information [], predicting $k$ by FVEs, rather than treating $k$ as a learnable parameter, reinforces prior frequency information and more clearly represents the relationship between frequency and thermal diffusivity.

## Appendix F Heat Conduction Visualization

We visualize more instances of visual heat conduction, given a randomly selected patch as the heat source, Fig.  8, validating the self-adaptive visual heat conduction pattern through the prediction of $k$.

Figure: Figure 8: Temperature distribution ($U^{t}$) when using a randomly selected patch as the heat source. (Best viewed in color)
Refer to caption: x8.png

## Appendix G Feature Map Visualization

We visualize the feature before/after HCO in a random layer in stage 2 with randomly selected images as input, Fig.  9. Before HCO, only a few regions of the foreground object are activated. After HCO, almost the entire foreground object is activated intensively.

Figure: Figure 9: Visualization of the feature before/after HCO in a random layer in stage 2 with ImageNet-1K classification pre-trained vHeat-B. The images are randomly selected from ImageNet-1K.
Refer to caption: x9.png

## Appendix H Analysis of k 𝑘 k italic_k in each layer

We calculate average values of $k$ in each layer of ImageNet-1K classification pre-trained vHeat-Tiny, Fig.  10. In stage 2 and stage 3, average values of $k$ corresponding to deeper layers are larger, indicating that the visual heat conduction effect of deeper layers is stronger, leading to faster and farther overall content propagation.

Figure: Figure 10: Average values of $k$ in each layer.
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the DCT the only choice? Could we replace it with FFT? Could you discuss their potential differences in computational efficiency and numerical stability?
soundness: 3
strengths: 1. The motivation is reasonable. 
   2. The result is better than Swin.
   3. Compared with Transformer of O(N2) computational complexity, this method becomes O(N1.5).
summary: This paper introduces the Heat Conduction Operator (HCO) built upon the physical heat conduction principle, to obtain large receptive fields with less computational overhead. 
Its computational complexity is O(N1.5) and is a plug-and-play module to obtain global receptive fields. 
Experiments across vision tasks demonstrate its effectiveness.
weaknesses: 1. The main purpose is to obtain the global receptive fields, thus, some MLP-based Backbones [1-6] should be added to compare. 
Maybe, you could discuss computational efficiency and performance in ImageNet, and you can also visualize their receptive fields. 

   [1] Strip-MLP: Efficient Token Interaction for Vision MLP. ICCV 2023
   [2] RaMLP: Vision MLP via Region-aware Mixing. IJCAI 2023
   [3] Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition. TPAMI 2023.
   [4] ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training. TPAMI 2023.
   [5] DynaMixer: A Vision MLP Architecture with Dynamic Mixing. ICML 2022
   [6] MLP-Mixer: An all-MLP Architecture for Vision. NeurIPS 2021

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In physics, heat conduction is defined as the diffusion of energy within a material. Representing image features usually involves pixel or feature map propagation, and whether heat conduction can directly simulate the propagation of visual semantics warrants further discussion. For instance, how the thermal diffusivity coefficient 𝑘 corresponds to "feature similarity" or "semantic consistency" in images requires more specific explanation. The authors could elaborate on how this analogy improves the representation of visual features and explore the biological or physical foundation of this mapping to enhance the method's credibility.
2. While vHeat has improved computational efficiency, its performance in real-world industrial applications has not yet been fully validated. In particular, whether it can outperform existing self-attention models in practical deployments requires further experimental data to substantiate its effectiveness.
3. The manuscript could benefit from a more comprehensive comparison with models inspired by physics and biology, such as diffusion models. Expanding on relevant work in related fields would better highlight the novelty and positioning of this model.
4. Although vHeat's experimental performance is demonstrated, the paper lacks a detailed description of specific parameter settings and model hyperparameters. For instance, how is the thermal diffusivity coefficient 𝑘 adjusted across different levels or data resolutions? How do the learned frequency value embeddings (FVEs) adapt to image content? These settings significantly impact experimental outcomes, and a detailed account of the parameter tuning process and experimental setup is recommended to enhance reproducibility. Additionally, visualizing and comparing heat diffusion effects across different model layers could aid in understanding its feature extraction capabilities.
5. The manuscript primarily focuses on test data in laboratory settings and lacks experimentation in real-world applications. This may limit the model's practical applicability. Including case studies from various fields would be beneficial; if the model demonstrates stable performance in these applications, it would be more persuasive.
6. Although the introduction of physical heat conduction concepts provides a degree of interpretability to the model's computation, the explanation of how specific visual features are enhanced or attenuated through heat diffusion remains insufficient. For example, how certain frequency components of image features are retained or filtered could be clarified. Experimental visualizations showing the influence of heat diffusion on feature selection would be helpful. This improvement in interpretability could aid users in understanding the model's mechanism and applicability, making it more transparent and interpretable.
soundness: 2
strengths: 1. vHeat’s foundation in the physical principles of heat conduction offers a unique perspective on visual representation. By conceptualizing image patches as heat sources and modeling their interactions through thermal energy diffusion, it provides a fresh, interpretable approach to feature extraction that stands apart from traditional self-attention mechanisms.
2.Despite the lower complexity, vHeat still achieves global receptive fields, allowing it to capture long-range dependencies within images without the computational burden of standard attention mechanisms. This combination of global awareness and efficiency is a strong advantage in many visual tasks.
3.vHeat demonstrates impressive performance across a range of visual tasks, including image classification, object detection, and semantic segmentation. It consistently outperforms existing models like Swin Transformer and ConvNeXt in both accuracy and throughput, highlighting its effectiveness and versatility as a visual representation model.
summary: The manuscript introduces an approach that links spatial propagation of visual semantics with physical heat conduction. It conceptualizes image feature as heat sources and leverages heat conduction principles to extract image features by predicting heat transfer rates. A visual heat conduction operator (HCO) is proposed, resulting in a visual representation model called vHeat, which combines low complexity, global receptive fields, and physical interpretability. Experiments demonstrate the strong performance of vHeat across various visual tasks.
weaknesses: 1. Although vHeat performs well in experimental settings, the manuscript lacks substantial evidence of its effectiveness in real-world industrial applications. Validation in practical deployments, especially compared to established self-attention models, is necessary to confirm its utility beyond controlled environments.
2.While vHeat draws an analogy between heat conduction and feature propagation, certain aspects of this analogy could benefit from further elaboration. For example, how the thermal diffusivity coefficient 𝑘 corresponds to “feature similarity” or “semantic consistency” in images is not fully explained, which could limit the model’s interpretability.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: I would expect more discussions about W4 and W5.
soundness: 2
strengths: 1. A novel view of visual representation
2. Extensive experiments and good results.
summary: In this work, inspired by heat conduction, the authors propose a new method for visual representation called vHeat. vHeat treats each image patch as a source of heat that propagates from local to global, functioning as a bridge between convolution and attention mechanisms. Experimental results suggest that this approach leads to improved performance.
weaknesses: 1. I would like the authors to discuss the proposed method in relation to relative positional embeddings, as I believe they may share similar high-level concepts.

2. I would also like to understand the motivation behind the additional branch (Linear + SiLU) shown in Fig. 2(b).

3. I am curious about the introduction of DWConv in the block and how much it contributes to the overall performance. While the authors presented this in the supplementary materials, I still do not fully grasp the motivation for its inclusion. Additionally, I would like to know why a different training recipe is adopted (Line 892). Does this lead to an unfair comparison? I was expecting a larger performance gap.

4. While the proposed method achieves promising results, the baselines appear to be somewhat weak and outdated. For example, Table 5 in MetaFormer v2 [1] shows that vHeat performs significantly worse than UniFormer, iFormer, CAFormer, and others if necessary.

5. It’s acceptable if we are not targeting SOTA performance (as well as for downstream tasks), but what other advantages or applications does this method offer that previous work cannot achieve?