UNIP: Rethinking Pre-trained Attention Patterns for Infrared Semantic Segmentation
Authors: Tao Zhang
Jinyong Wen
Zhen Chen
Kun Ding
Shiming Xiang
Chunhong Pan
Venue: ICLR 2025 Poster (2025)
Keywords: Image Pre-training
Semantic Segmentation
Infrared Image
Attention Distillation
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A comprehensive analysis of attention patterns in pre-trained models, along with several enhancements to improve their transfer performance on infrared semantic segmentation tasks.

Abstract: Pre-training techniques significantly enhance the performance of semantic segmentation tasks with limited training data. However, the efficacy under a large domain gap between pre-training (e.g. RGB) and fine-tuning (e.g. infrared) remains underexplored. In this study, we first benchmark the infrared semantic segmentation performance of various pre-training methods and reveal several phenomena distinct from the RGB domain. Next, our layerwise analysis of pre-trained attention maps uncovers that: (1) There are three typical attention patterns (local, hybrid, and global); (2) Pre-training tasks notably influence pattern distribution across layers; (3) The hybrid pattern is crucial for semantic segmentation as it attends to both nearby and foreground elements; (4) The texture bias impedes model generalization in infrared tasks. Building on these insights, we propose UNIP, a UNified Infrared Pre-training framework, to enhance the pre-trained model performance. This framework uses the hybrid-attention distillation NMI-HAD as the pre-training target, a large-scale mixed dataset InfMix for pre-training, and a last-layer feature pyramid network LL-FPN for fine-tuning. Experimental results show that UNIP outperforms various pre-training methods by up to 13.5% in average mIoU on three infrared segmentation tasks, evaluated using fine-tuning and linear probing metrics. UNIP-S achieves performance on par with MAE-L while requiring only 1/10 of the computational cost. Furthermore, with fewer parameters, UNIP significantly surpasses state-of-the-art (SOTA) infrared or RGB segmentation methods and demonstrates the broad potential for application in other modalities, such as RGB and depth. Our code is available at https://github.com/casiatao/UNIP.

Published as a conference paper at ICLR 2025
UNIP: RETHINKING PRE-TRAINED ATTENTION PAT- TERNS FOR INFRARED SEMANTIC SEGMENTATION
Tao Zhang!. Jinyong Wen Zhen Chen' Kun Ding Shiming Xiang Chunhong Pan
'AAIS, Institute of Automation, Chinese Academy of Sciences, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Chima
Sccoool of Automation Science and Electrical Engineering, Beihang University, China (zhangtao2021, kun. di @ia aaw.f.aaaa..coo.aa.ommmoo.o....a
ABSTRACT
Pre-training techniques significantly enhance the performance of semantic seg- mentation tasks with limited training data. However, the efficacy under a large do- main gap between pre- wre-training (e.g. RGB) and fine-tuning (e.g. infrared) remains underexplored. In this study, we first benchmark the infrared semantic segmenta- tion performance of various pre-training methods and reveal several phenomena distinct from the RGB domain. Next, our layerwise analysis of pre-trained atten- tion maps uncovers that: (1) There are three typical attention patterns (local, hy- brid, and global): (2) Pre-training tasks notably influence the pattern distribution across layers; (3) The hybrid pattern is crucial for semantic segmentation tends to both nearby and foreground elements; (4) The texture bias impedes model generalization infrared tasks. Building on these insights, we propose UNIP, a UNified Infrared Pre-training framework, to enhance the pre-trained model per- formance. This framework uses the hybrid-attention distillation NMI-HAD as the pre-training target, large-scale mixed dataset InfMix for pre-training, and a last-layer feature pyramid network LL-FPN for fine-tuning. Experimental re- sults show that UNIP outperforms various pre-training methods by up to 13.5% in average mloU on three infrared segmentation tasks, evaluated using fine-tuning and linear probing metrics. UNIP-S achieves performance on par with MAE-L while requiring only 1/10 of the computational cost. Furthermore, UNIP signif- icantly surpasses state-of-the-art (SOTA) infrared or RGB segmentation methods and demonstrates broad potential for application in other modalities, such as RGB http:///gghuubb.oooccaaatttttttt
at-
and depth. Our code is available
INTRODUCTION
Pre fre-training is essential in computer vision, equipping models with fundamental feature extraction capabilities. Supervised methods (Touvron et al., 2021; 2022) and self-supervised methods, such as contrastive learning (CI.) (Chen et al., 2021b; Caron et al., 2021) and masked image modeling (MIM) (He et al., 2022; Fu et al., 2024), have demonstrated great potential in various visual tasks, particularly for small-scale datasets. Infrared images, widely used in road surveillance (Bondi al., 2020), autonomous driving (Xiong et al., 2021), and unmanned aerial vehicle (Sun et al., 2022), often lack labeled data for tasks like object detection and semantic segmentation (Li et al., 2021a). Therefore, having a strong pre-trained backbone is vital for these data- hata-limited scenarios.
However, the transfer performance infrared segmentation of different pre-training meth- ods remains considerably underexplored. Previous works (Xiong et al., 2021: Chen & Bai, 2023) aim to improve performance by designing specific architectures for infrared segmentation tasks, without assessing the impact of various pre-training methods on model performance. Additionally, mainstream pre-training methods (He et al., 2022; Zhou et al., 2022) usually evaluate performance on large-scale RGB datasets like ImageNet (Deng et al., 2009) and ADE20K (Zhou et al., 2017).
Kun Ding is the corresponding author. We use the term method-size denote the vision transformer (ViT) of specific size pre-trained by specific method. T, S, and I. refer the ViT-Tiny, ViT-Small, ViT-Base, and ViT-Large, respectively.

---

Published as a conference paper at ICLR 2025
Stepl: How do various pre-fruining methods perform?
Step2: Why do they perform distinctively?
Benchmark
Performance: Supervised/CI.
MIM
by
Pretraining Task
Texpe/eexture Bias
241
ogflcccce
RGB Pre-Trained
Models
Attention Pattern
Restteee by
DINO-S
MAE-S
Infrared
Segmentation Tasks
Sepanise
PIR
STal
Ease
NMI
Hybrid Patiern
Distribution Shift
Performance Cost Ratio; Large Small
Benefits Segmentatio
Step3: How to improve their performance?
mepping
UNTP
NMI-guided Hybrid Attention pattern
Last-Layer FPN (LL.FFNN
Distillation (NMI-HAD)
InfMix
1/16
1/32
Teacher
Hybved Punoon
Infrared Datasets
(BBB pre-trincel)
Student
1/16
1/16
Decoder
Layer aelocted
ImageNet Subaet
by NMI
1:16
1/8
Hybvid Pation
COCO Training
Student
1:16
1:4
Cnnddraly initialionds
Pre-training
Fine-tuming
The Choin-of-Thought (CoT) of our work. Step1 (Sec. 2): We benchmark the infrared segmentation performance of various pre-trained models and derive several insights. Step2 (Sec. 3): We explore the reasons for the varying behaviors of these models by analyzing the pre-trained atten- tion maps. Step3 (Sec. 4): Based on these findings, propose UNIP, a unified framework aimed to enhance the performance of small pre-trained models, focusing on three aspects: the pre-training dataset (InfMix), the pre-training task (NMI-HAD), and the fine-tumin fine-tuning architecture (LL-FPN).
Figure I:
Given the significant domain differences between RGB and infrared datasets, further study is neces- sary to evaluate the transfer performance of different pre-training methods infrared visual tasks and the validity of phenomena observed in RGB datasets for the infrared domain.
To this end, benchmark six popular supervised and supervidd (CL and MIM) pre-training methods on three infrared semantic segmentation datasets, across different model sizes and eval- uation metrics (see Sec. 2, Step1 in Fig. 1). Some valuable phenomena are discovered: (1) The ImageNet accuracy of models does not necessarily correlate with their performance on infrared seg- mentation tasks; (2) Supervised and CL methods exhibit better generalization than MIM methods, especially for small models like ViT-T and VīT-S; (3) The performance improvement of larger mod- els is marginal compared to the substantial increase in computational cost, making them unsuitable for rnfrared-related tasks that require fast processing speeds with limited computing resources.
To understand the distinct performance of these methods, we conduct a thorough analysis of atten- tion maps (see Sec. 3, Step2 in Fig. 1). Three attention patterns-local, hybrid, and global-are iden- tified in different layers pre-trained models. As shown in Fig. 3, local patterns focus on nearby tokens?, while global patterns prefer foreground tokens. Hybrid patterns attend to both types. The pre-training tasks significantly influence the pattern distributions: Supervised and CL models ex- hibit all patterns, whereas MIM models show only local and hybrid patterns. Importantly, the hybrid attention pattern found to be crucial for semantic segmentation as it can effectively capture both local and global information. To quantitatively distinguish these patterns, duce the normalized mutual information (NMI) between query and key tokens as an indicator, which aligns well with pattern distributions. Additionally, we find that the bias towards texture observed in attention maps can exacerbate distribution shifts and hinder model generalization in infrared tasks.
intro-
Based on the above analysis, UNified Infrared Pre-training framework called UNIP proposed to enhance the infrared segmentation performance of small models (see Sec. 4, Step3 in Fig. 1). First, we introduce the NMI -uilgdddedd Hybrid Attention pattern Distillation (NMI-HAD) as the pre-training target, which uses NMI to select the distillation layer and compresses hybrid patterns from teacher models to randomly initialized student models. Second, to bridge the gap between pre-training and infrared data and mitigate distribution shifts, we construct large mixed dataset called InfMix as the pre-training dataset. It comprises 859,375 images from 25 datasets, ensuring no overlap with the segmentation datasets used in our benchmark. Third, to utilize hybrid patterns in the last layer of
In this work, token is used to denote the 16x 16 patch in the image.
2

---

Published as a conference paper at ICLR 2025
MAE
CrossMAE DeiT
DeiT III
DINO
iBOT
*
UNIP (MAE-L)
UNIP (IBOT-L)
50
++3.7
mmggNNet Accuracy (FT)
86
Avera Average mlou (FT)
+9.0
Average (LP)
45
+12.8
84
40
56
++2.99
82
+13.6
35
52
Model
Param
FLOP:
WI-T
11.0M
36.7G
30
+13.6
80
WTTS
41.9M
146.3G
163.7M
WITL
441.3M
1192.6G
72
VIT-T
VIT-S
VIT-B
VIT-L
VIT-T
VIT-S
VIT-B
vIT-L
VIT-T
VIT-S
VIT-B
VIT-L
Figure 2: The performance of pre-trained models across various methods and sizes. Left: The
average fine-tuning (FT) performance on three infrared semantic segmentation datasets, along with the associated computational cost. Middle: The average linear probing (LP) performance infrared datasets. Right: The fine-tuning performance on ImageNet (Deng et al., 2009). The gray dotted lines and corresponding values highlight the performance gains of UNIP over other methods. Detailed results for each dataset are presented in Tab. 11.
three
distilled modes, we propose the Last-Layer Feature Pyramid Network (LL-FPN) for fine-tuning to enhance performance further. With these enhancements, the average segmentation mIoU of UNIP significantly surpasses their counterparts, as shown in Fig. 2 and Tab. 4. When using MAE-L (He al.. 2022) as the teacher, UNIP achieves improvements of 13.57 (T), 8.98% % (S). and 4.34% (B) in fine-tuning, and at least 12.79% in linear probing. With iBOT-L (Zhou et al., 2022) as the teacher, UNIP-S exceeds iBOT-S by 2.61 % fine-tuning, while UNIP-B surpasses iBOT-B by 3.74% in linear probing. Notably, the distilled models even outperform their teacher models across different pre-training methods. UNIP also substantially outperforms other SOTA infrared or RGB segmentation methods, such as TINN (Chen & Bai, 2023) and Mask2Former (Cheng et al., 2022), and exhibits effectiveness and application potential in other modalities like RGB and depth images.
Our main contributions consist of (1) A comprehensive benchmark of six pre- pre-training methods on three infrared semantic segmentation datasets, highlighting several key phenomena; (2) A detailed investigation of pre-trained attention patterns, emphasizing the critical importance of the hybrid pattern for semantic segmentation; (3) A unified infrared pre-training framework UNIP, including the NMI-HAD method, the InfMix dataset, and the LL-FPN architecture; (4) Extensive experimental results, demonstrating the effectiveness and efficiency of our method and dataset.
2 How DO PRE-TRAINING METHODS PERFORM ON INFRARED TASKS?
In this section, we benchmark six pre-training methods on three infrared semantic segmentation datasets and discuss several key phenomena.
2.1 INFRARED SEGMENTATION BENCHMARK OF RGB PRE-TRAINED MODELS
Pre-trained Backbone. Pre-training of the Vision Transformer (ViT) (Dosovitskiy et al., 2021) has gained widespread attention and demonstrated powerful performance in various fields. Many recent pre-training methods (Touvron et al., 2021; He et al., 2022; Zhou et al., 2022; Oquab et al., 2024) use ViT for experiments, making pre-trained ViT models readily available. Therefore, ViT models of various sizes are set as the evaluation backbone.
Pre-training Methods. Both supervised and self-supervised methods are investigated. For super- vised approaches, we use DeiT (Touvron et al., 2021) and DeiT III (Touvron et al., 2022), which perform image classification on ImageNet for pre-training. In supf-vsuee methods, we study contrastive learning (CL) and masked image modeling (MIM). CL methods like DINO (Caron et al., 2021) encourage features from different views of the same image to be close, while keeping features from different images distinct. MIM methods like MAE (He et al., 2022) and CrossMAE (Fu et al., 2024) focus on reconstructing masked image patches by learning context relations. Although iBOT (Zhou et al., 2022) combines CL with masked feature prediction, to its similar characteristics to DINO. The above methods are selected because they all pre-train vanilla ViT models on ImageNet, without additional pre-trained tokenizers like BeiT (Bao al.,
classify it as CL method due
3

---

Published as a conference paper at ICLR 2025
2022) MILAN (Hou et al., 2022), or larger datasets like EVA (Fang et al., 2023) and DINOv2 (Oquab et al., 2024). This allows us to focus on the impact of the pre-training tasks alone.
Evaluation Datasets. The evaluation is conducted three infrared semantic segmentation datasets: SODA (Li et al., 2021a), MFNet-T (Ha et al., 2017), and SCUT--egg (Xiong et al., 2021). Notably, MFNet is an RGB-Thermal paired dataset. The thermal part MFNet-T'is is used for benchmarks while the RGB part MFNet-RGB is employed in further investigations. Additionally, RGB datasets like ImageNet- IK (Deng al., 2009) and ADE20K (Zhou et al., 2017) are also used for comparison. Details about these datasets can be found in Appendix C.3.
Evaluation Metries. We employ two metries: fine-tuning (FT) and linear probing (LP). FT (Fig. Sa) is the primary metric, where both the pre-trained model and the decoder are tuned with the labeled datasets. In LP (Fig. 8b), only linear head is updated while all other parameters remain frozen. Average (Avg) FT or LP performance in subsequent sections denotes the mean mIoU across three infrared semantic segmentation datasets. More details are available in Appendix C.2.
Benchmark Results. In the benchmark, all models are trained for 100 epochs for both evaluation metries. Typical results are illustrated in Fig. 2, with ImageNet (Deng et al., 2009) fine-tuning performance included for comparison. The complete results of each dataset are detailed in Tab. 11.
2.2 WHAT INSIGHTS CAN WE GAIN FROM THIS BENCHMARK?
The infrared FT performance is strongly positively corre- lated with LP, but has no clear relationship with ImageNet FT. Tab. presents the Pearson (Pearson, 1896) correlation coefficients between different metries. For each metric pair, the coefficients are calculated across six pre-training methods in Fig. 2. Notably, the coefficients between average infrared LP and FT are close to 1, indicating that models with better LP performance generally exhibit better FT performance. Conversely, the ImageNet FT performance does not consistently correlate with infrared FT results across various model sizes, likely due to domain and task differences. Therefore, using ImageNet accuracy to predict transfer performance on infrared segmentation datasets is not reliable, underscoring the importance of benchmarking infrared segmentation datasets.
Table 1: Pearson coefficients be- tween average FT and other metrics.
Metric
Small
Base
Large
Mean
Arg FT Avg Avg FT
0.89
0.93
0.81
0.88
0.78
0.08
Supervised and CL methods outperform MIM methods, especially for small models. As de- picted in Fig. 2 and Tab. 11, the performance of supervised and CL methods like DeiT, DeiT III, DINO, and iBOT is similar across both metrics, except for the LP of DeiT-S. For the LP metric, MIM methods of various sizes consistently lag behind supervised and CL methods by significant margin, matching observations in the RGB domain (He et al., 2022) that MIM representations are less linearly separable. In terms of FT, smaller MIM models (ViT-T, S, and B) still underperform supervised and CL. methods, while larger models (ViT-L) are more competitive. For instance, MAE- S is far behind iBOT-S (55.39% vs 62.09% but MAE-L performs comparably iBOT-L (64.35% vs 64.97%). As we will discuss in Sec. 3. the discrepancy in the attention pattern distribution and texture bias between different models accounts for their distinct infrared segmentation performance.
Larger models perform better, but their computational cost increases sharply. As illustrated in Fig. larger MIM models bring considerable performance gains over smaller models. However, for supervised and CL methods, small models are already well-trained, and the performance improve- ment from larger models is marginal compared the significant increase in computational cost. For example, iBOT-L surpasses iBOT-S by only 2.85% (64.97% vs 62.09%), while the parameter count and FLOPs increase by 10.5x (441M vs 42M) and 8.2x (1193G vs 146G), respectively. Given that infrared images are often processed edge devices with limited computing budgets. using large models to pursue better performance is not cost-effective. Therefore, we believe improving small models is more effective approach. In Sec. 4, we propose several strategies to elevate the performance of small models to be on par with larger models.
WHAT MATTERS FOR INFRARED SEMANTIC SEGMENTATION?
To determine which characteristics of the pre-trained models are critical for infrared semantic seg- mentation, we analyze different models from multiple perspectives.
4

---

Published as a conference paper at ICLR 2025
Layer
Layer
Layer 12
Layer 5
Layer 9
Layer 12
Layer
Layer 1d
Layer 24
Quary
Query
Quiry
Local
Hybrid
Global
Local
Local
Local-Hfyrrid
Local
Hybrid
Local
(o) DINO-S
MAE-
MAE-L
Figure 3: Attention maps for different query tokens in three representative layers. Each query token's 's attention map corresponds to a row in the attention matrix, averaged over different heads.
3.1 THE PRE-TRAINING TASKS INFLUENCE ATTENTION PATTERNS
The self-attention mechanism a key component of VIT. semantic segmentation tasks, the spatial interactions between tokens crucial. Thus, visualize the attention maps of pre- rreitrained models.
Attention maps of supervised/CL and MIM methods differ significantly. As shown in Fig. 3a, DINO-S exhibits three distinct attention patterns: (1) Local: In shallow layers (Layer 5), different query tokens focus only on their spatially nearby key tokens; (2) Hybrid: middle layers (Layer 9), query tokens attend to both nearby tokens and foreground tokens; (3) Global: In deep layers (Layer 12), different query tokens all focus on foreground tokens with nearly identical attention maps, phenomenon known as atfention collapse (Park al., 2023). This attention pattern distribution is consistent across different sizes in CL and supervised methods, as shown in Fig. 10. However, in MIM methods like MAE, the distribution varies. In MAE-S (Fig. 3b), attention maps are mainly local, with slight hybrid patterns emerging in deep layers. Conversely, in MAE-L (Fig. 3c), shallow and deep layers exhibit local patterns, while middle layers show hybrid patterns. CKA (Kornblith et al., 2019) analysis in Appendix D.2 reveals similar phenomena regarding feature representation.
Differences in attention patterns stem from the pre-training tasks. CL methods, similar to su- pervised approaches, treat views from the same image as belonging to the same class. This setup encourages models to focus on foreground tokens, as images the same class often share simi- lar foreground objects but may differ in background. Consequently, attention maps in later layers present global patterns. The local and hybrid patterns can be regarded as the intermediate states in forming the global pattern. This high-level pre-training task causes models of different sizes and methods to have similar pattern distributions across layers. In contrast, pre-training tasks of MIM methods focus reconstructing features raw pixels of masked tokens, which is relatively low-level task relying heavily on spatially nearby tokens. Consequently, models are not compelled capture global image information, leading small models to primarily exhibit local patterns. In larger models like MAE-L, the increased representation capacity allows hybrid patterns to sponta- neously emerge in the middle layers capture broader context. In deep layers near the decoder, local patterns reappear to support the pre-training task of reconstructing nearby masked tokens.
As supplement, iBOT exhibits similar patterns with DINO in shallow and middle layers but shows less attention collapse in deep layers (see Fig. 10). This can be attributed to that iBOT combines DINO with masked feature prediction (Zhou al., 2022), which encourages the later layers to leverage spatial information predict features of masked tokens.
3.2 How TO QUANTITIVELY IDENTIFY DIFFERENT ATTENTION PATTERNS?
Three distinct attention patterns are qualitatively summarized in Sec. 3.1, prompting the question of whether metric can quantitatively measure them. Attention distance measures the average distance between the query and key tokens, while attention entropy implies the concentration of the attention distribution. However, both metrics depict the relationship between one query and multiple key tokens and are unable to reflect differences in the attention maps of various queries. We find that the normalized mutual information (NMI) between query and key tokens is an effective indicator. The calculation process is elaborated in Appendix D. Let A E Nx denote the attention matrix, where the number of tokens. The NMI a function of A, ranging from 0 to We highlight two
5

---

Published as a conference paper at ICLR 2025
MAE
CrossMAE
DeiT
DelT III
DINO
IBOT
t Peak
50
55
60
mlou
35
20
25
30
10
6
10
12
12
16
20
24
Depth (VIT-S)
Depth (VIT-B)
Depth (ViT-L)
Figure 5: The layerwise linear probing performance of different methods on SODA (Li et al., 2021a).
special cases to clarify NMI: (I) When query tokens focus solely on their spatially corresponding key tokens (an extreme local pat- tern), A becomes an identity matrix. Thus the joint probability of query and key tokens is equivalent their marginal probability and the NMI reaches its maximum value of I; (2) When all query tokens attend the same key tokens (an extreme global pattern),
0.5
DINO.S
Local
iBOT-S
MAE-S
0.3
MM
0.2
Hybrid
each row of A is identical. Consequently, the probability distri- bution of query and key tokens are independent, leading to the minimum NMI of O.
0.0
0.4
Global
0.7
1.0
Normalized Dopth
Figure 4: NMI on ImageNet.
Therefore, local patterns have larger NMI values, while global patterns exhibit lower ones. As illustrated in Fig. 4, the NMI of DINO-S and iBOT-S decreases with depth and consistently stays later layers, revealing
below that of MAE models. Especially, the NMI DINO-S approaches its atlention collapse. In contrast, the NMI of MAE-L first decreases and then increases, due to the hybrid patterns in middle layers and local patterns in later layers.
3.3 THE HYBRID PATTERN MATTERS FOR SEMANTIC SEGMENTATION
Semantic segmentation is a dense prediction task where all pixels in an image are classified into different semantic classes. Local information is crucial nearby pixels usually belong to the same class, while global cues are also essential since instances of the same class may appear in different positions within the image. Therefore, we hypothesize that hybrid patterns, which capture both local and global information, are more important for semantic segmentation than purely local or global patterns. To demonstrate this, we conduct the layerwise linear probing (LLP) experi- ments, where frozen features of only one layer are passed to the linear head, shown in Fig. 8d.
The LLP performance peaks where hybrid attention patterns emerge. As shown in Fig. 5, supervised and CL methods peak at about three-quarters of the model's depth. Large MIM models (ViT-L) perform better in the middle layers. These peaks commonly occur near the hybrid patterns. In contrast, the performance of small MIM models (VIT-S and ViT-B) gradually increases with depth, peaking in the last two layers. This is because, although all layers exhibit local patterns, deep layers focus more foreground tokens (Fig. 3b) and have smaller NMI values (Fig. 4), leading to better LLP performance. Additionally, the performance degradation in iBOT's deep layers less pronounced than that in DINO and supervised methods. This aligns with observations that iBOT's deep-layer attention maps contain more local information (Sec. 3.1) and have larger NMI values than those of DINO (Fig. 4), underscoring the importance of the hybrid attention pattern.
This hypothesis can explain the phenomena in Sec. 2. Small MIM models struggle to learn the hybrid pattern, resulting in a notable performance gap compared supervised and CL methods. Conversely, large MIM models successfully develop the hybrid pattern, making their fine-tuning performance comparable to other methods. iBOT performs best across different model sizes and evaluation metrics because the hybrid pattern occurs more frequently than in other methods.
3.4
THE TEXTURE BIAS HINDERS THE MODEL'S 'S GENERALIZATION ON INFRARED IMAGES
When transferring RGB pre-trained models to infrared tasks, the distribution shift between these modalities significantly impacts performance. A major difference between RGB and infrared im- ages is that RGB images can capture fine-grained textures, which are scarce in infrared images.

---

Published as a conference paper at ICLR 2025
Therefore, assume that the model's bias towards texture would exacerbate the distribution shift, thereby impairing the transfer performance on infrared tasks.
According to Park et al. (2023), MIM methods are texture-biased while CL and supervised methods are shape-biased. This bias is evident in attention maps in Fig. 3, where MAE mod- els focus on textures while DINO emphasizes edges. Tab. 2 investigates the bias's impact on infrared segmentation. MAE-B outperforms DeÎT-B and DINO-B on RGB datasets like ADE20K (Zhou et al., 2017) and MFNet-RGB (Ha et al., 2017), but consistently underperforms on infrared datasets. Notably, the paired MFNet-RGB and MFNet-T share the same scenario and image counts, differing only in modality. This indicates that MAE models pre-trained on ImageNet rely on low-level texture information to reconstruct masked patches, leading to poor generaliza- tion on texture-less infrared images. Therefore, reducing the texture bias is a promising way to enhance the transfer performance of RGB- RGB-pre-trained models on infrared tasks.
Table 2: The FT performance on RGB and infrared
semantic segmentation datasets.
RGB
Infrared
Methods
ADE20K MFNe1-RGB MFNet-T
SODA
SCUT-Sog
DeiT-B DINO-B
47,4
57.07
48.59
69.73
69.35
46.8
55.20
48.5c
69.79
69.82
MAE-B
48.1
57.29
46.78
68.18
67.86
HOW TO IMPROVE THE PERFORMANCE ON INFRARED SEGMENTATION?
As discussed in Sec. 2.2, scaling up model sizes for better performance impractical for resource- constrained scenarios. Therefore, we focus on enhancing small pre-trained models by introducing comprehensive framework, UNIP, and validating its effectiveness through extensive experiments.
4.1
UNIP: A UNIFIED INFRARED PRE-TRAINING FRAMEWORK
UNIP improves small pre-trained models by optimizing the pre-training task, constructing an appro- priate pre-training dataset, and refining the fine-tuning architecture, as depicted in Fig. 1.
NMI-Guided Hybrid Attention Pattern Distillation (NMI-HAD). Compressing knowledge from large models into smaller ones is an effective strategy to enhance performance without increasing parameter count. Previous works use various distillation targets like logits (Caron et al., 2021) and features (Xiong al., 2024). However, they often overlook the relationship between distillation targets and attention patterns. As revealed in Sec. 3.3, the hybrid attention pattern is crucial for semantic segmentation, with NMI values linked attention patterns. Therefore, we propose using hybird patterns as the distillation target and introduce the NMI-guided hybrid attention pattern distil- lation. First, the NMI value (MI( of each teacher model's layer is calculated on ImageNet- 1K:
1 M
QF (PM(KT)))
L 2
Σ NMI(A?"), NMI(A.) = M ===1
ATT softmax
1
L, (1)
where AT denotes the m-th head attention matrix in the 1-th layer. and d are the number of layers and the dimension of the teacher model. The method for calculating NMI is detailed in Appendix D. Note that we only consider layers in the latter half of the model, as shallow layers do not capture sufficient knowledge. Next, NMI values are used to identify the location of the hybrid attention pattern. The attention map of the layer whose NMI is closest an empirical value is utilized as the distillation target Ar- Finally, the attention map As in the last layer of the student model is forced imitate Ar by employing the Kullback-eeibler (KL) divergence constraints:
M
AT = arg max ANMI(A)) ANMI(A) = (M)((A) - -|, L M 10
KL (I
A,
(2)
Empirically, the NMI values of hybrid patterns range between 0.06 and 0.12. We find that setting within this range yields good results. In all our experiments, we set 0.09 by default.
InfMix Dataset. To alleviate the distribution shift and reduce texture bias when distilling RGB pre-trained models for infrared tasks, we develop InfMix, mixed dataset for distillation. InfMix comprises 859,375 images from both RGB and infrared modalities, constructed through four steps. (1) Infrared images play a key role mitigating the distribution shift. However, existing datasets
7

---

Published as a conference paper at ICLR 2025
Table 4: The infrared semantic segmentation performance of different models. Across various pre- trained methods, UNIP models significantly surpass pre-trained models of the same size. Remark- ably, they even outperform their teacher models, despite the latter having more parameters.
Methods
Params(M)
Fine luning (FT)
Linear Probing (LP)
SODA
MPNet-T
SCUT-Seg
Average FT
SODA
SCUT-Seg MPNei-T
Avenae I.P
MAE-L (Teacher)
441.3
71.04
51.17
70.83
64.35
52.20
31.21
43.71
42.37
MAE-T
52.85
35.93
23.75
15.79
27.18
22.24
UNIP-T
11.0
64,83
48.77
67.22
60.27 (+13.57)
44,12
28.26
38.09
35.82 (+ (3.58)
MAB-S
41.9
63.36
42.44
60.38
55,39
38.17
21.14
34.15
31,15
UNIP-S
41.9
70.99
51.32
70.79
64.37 (+8.98)
55.25
33,49
43.37
44.04 (+ (+.2289)
MAE-B
163.7
68.18
46.78
67.86
60,94
43.01
23:42
37.48
34.64
UNIP-B
71.47
52.55
65.28 (+4.34)
58.82
34.75
48.74
47.43 (+12.79)
DINO-B (Teacher)
163.7
69.79
48.
69.82
62.72
59.33
34.86
47.23
14
DINO-S
41.9
68.56
47.98
68.74
61.76
56.02
32.94
45.94
44,97
UNIP S
41.9
69.35
49.95
69.70
63.00 (+1.24)
57.76
34.15
46.37
46,09 (+1.12)
BBTT-L (Teacher)
441.3
71.75
51.66
71.49
64,97
61.73
36.68
50.12
49.51
69.33
47.15
57.10
33.87
45.82
45.60
UNIP-S
41.9
70.75
51.81
71.55
64.70 (+2.61)
60.28
37.16
47.68
48.37
(42.77)
:BOT-B
163.7
71.15
48.98
71.26
63,80
60.05
34.34
49.12
47.84
UNIP-B
163.7
71.75
51.46
72.00
65.07 (+27)
63.14
39.08
52.53
51.58 (+3.74)
often lack diversity and sufficient images, so we collect large and unlabelled infrared pre-training dataset called InfPre. It consists of 541,088 im- ages from 23 infrared-related datasets. Compared to the other two datasets in Tab. 3, InfPre offers a larger number of higher-resolution images sourced from more diverse datasets. Importantly, three segmentation datasets used in the benchmark are excluded from InfPre for fair comparison. Details on data collection and deduplication can be found in Appendix F.I. (2) A subset of ImageNet-IK (Deng et al., 2009) is used, comprising 200,000 images evenly sampled from 1,000 classes. Since these images are part of the teacher model's pre- training data, they can anchor the student representation space close to the teacher's, thereby aiding in transferring the teacher's 's general feature extraction capabilities to the student. (3) The training set of COCO (Lin et al., 2014), with 118,287 images, is also included to further enrich the pre-training dataset. Unlike single-object-centrie images in ImageNet, COCO images typically depict larger scenes with multiple objects, making them more similar to infrared images, as indicated in Tab. 21 in the appendix. (4) Images from ImageNet and COCO are converted to grayscale (three identical channels) to resemble infrared images more closely, as noted in Tab. 21.
Table 3: Comparisons of infrared pre-training
datasets. #Subset denotes the number of datasets from which the images are collected.
Dataset
#image
#Subset
Width Hoight
MSTP (Zhang al., 2023)
178,756
8
844
596
Int30 (Liu al., 2024a)
305,241
700
562
InfFrr (ours)
541,088
23
1,075
686
Last-Layer Feature Pyramid Network (LL-FPN). To adapt the non-bierarchical VIT to multi- scale decoders in dense prediction tasks, previous works (He et al., 2022; Zhou et al., 2022) typically generate multi-scale feature maps from different layers of VIT, shown in Fig. 8a. However, find this multi-layer design unnecessary for our distilled models. In these models, the hybrid patterns in later layers equip the final features with both local and global information, making them suitable for multi-scale feature map generation. Inspired by ViTDet (Li et al., 2022b), we propose using the last-layer feature pyramid network during fine-tuning. It constructs all feature maps of different scales upon the last layer's features, as illustrated in Fig. and Fig. 8c. As bonus, this approach enhances the representation capacity of each scale branch compared to the configuration in Fig. 8a, since they go through the entire backbone, leading to improved fine-tuning performance.
4.2 EXPERIMENTS
The MAE-L, DINO-B, and iBOT-L are utilized as teacher models for distillation, and the 18th, 9th, and 21st layers are used as the target layer, according to Eq. (1) and Eq. (2). Unless otherwise spec- ified, the distillation, fine-tuning, and linear probing processes are each conducted for 100 epochs. For ablation studies, we mainly focus on the fine-tu fine-tuning metric as it reflects the model's highest achievable performance. More details about experimental settings can be found in Appendix C.4.
Improvements of UNIP. As shown in Tab. 4, UNIP significantly enhances the performance of small models across both metrics, often exhibiting comparable even better performance than teacher
8

---

Published as a conference paper at ICLR 2025
Table 5: Comparisons with other segmentation methods (FT), All the compared results except PAD are borrowed from TINN. Training epochs of SODA and MFNet-T are 200 and 300.
Table 6: Impact of distillation targets (UNIP-S) -S).
Targot (Teacher)
Layer
SODA
MFNes-T
SCUT-Seg
Avg FT
Feature (MAE-L)
18
65.66
48.44
66.55
60.22
66.86
49.00
66.61
60.82
Methods
Params(M)
SODA
MINNetTT'
Attention (MAE-L)
70.99
51.32
24
67.74
50.39
69.00
62,38
DeepLab V3+ (Chon et al., 2018)
62.7
68.73
49.80
PSPNet (Zhao et 2017)
68.1
68.68
45.24
Table 7: Impact for the LL-FPN (UNIP-S), X and
UPerNet (Xiao ct ol., 2018)
72.3
67.45
48.56
SogFormer (Xie
2021)
84.7
67.86
V denote the ones in Fig. 8a and Fig. 8c.
Ada-pee (Chen et al.,, 2023)
99.8
68.12
50.62
Teacher (1 Lyer)
LI-F
SODA
MPNer-T SCUT--eg Aug FT
2ask2Former (Cheng et al., 2022)
216.0
67.58
51.30
MaskDINO (Li et al., 2023)
223.0
66.32
51,03
Hlyrrdd Pattern
X
69.54
50.18
70.63
63.45
MAE-L (Layer 18)
70.99
51.32
70.79
64.37
EC.CNN (1.i al., 2021a)
54.5
65.87
47.56
MCNet (Xiong ot al., 2021)
35.7
63.89
43.15
Local Pattern
X
67.64
49.93
68.68
62.08
PAD (MAE-B) (Zhang et 2023)
164.9
MAE-L (Layer 24)
67.74
50.39
69.00
62.38
TINN (Chen & Bai, 2023)
85.3
69:45
51,93
Hybrid Patter
X
68.56
49
68.49
62.16
UNIP-T (MAE.)
11.0
67.29
50.39
DINO-B B (Layer 9)
69.35
49.95
69.70
63,00
UNTP-S (MAF-L)
41.9
71.35
53.76
Global Pattern
x
68.62
47. 36
69.71
61.90
UNIP-B (MAE-L)
163.7
72.19
54.35
DINO-B (Layer 12)
68.50
48.40
69.67
models. With MAE-L as the teacher, UNIP-T, UNIP-S, and UNIP-B achieve average mIoU gains of 13.57% 8.98%, and 4.34% in fine-tuning, and 13.58%. %. 12.98%. and 12.79 in linear probing- Notably, UNIP-S performs comparably to MAE-L with only 1/10 of the computational cost. UNIP- even outperforms MAE-L by 0.93% FT and 5.06 % in LP. Using iBOT-L as the teacher, U transcends iBOT-S by 2.61 % in FT and 2.77 % in LP. Meanwhile, UNIP-B shows gains of 1.27 % in FT and 3.74% in LP, exceeding its teacher iBOT-L. Even with a smaller teacher like DINO-B, UNIP-S still enhances performance by as least 1.12%. Tab. 5 compares the fine-tuning performance of UNIP with other RGB or infrared segmentation methods. With fewer than half the parameters, UNIP-S, distilled from MAE-L, surpasses the universal segmentation method Mask2Former (Cheng et al., 2022) by 3.77% on SODA and 2.46% on MFNet-T. It also outperforms TINN (Chen & Bai, 2023), specially designed for infrared semantic segmentation, by 1.9% on SODA and 1.83% on MFNet-T. A larger model UNIP-B further widens this performance gap, indicating that UNIP can greatly unleash the potential of the vanilla VIT for infrared semantic segmentation.
Impact of Distillation Target Layers. Fig. 6 displays the average fine-tuning performance using different layers of MAE-L and DINO- B as the distillation target layer. Notably, both models exhibit strong positive correlation be- tween average FT performance and ANMI in Eq. (2), as indicated by large Pearson co- efficient. Furthermore, the peaks of FT and ANMI occur in the same layer, highlighting the effectiveness of the NMI-HAD.
11.0 Doud 8
ET (UNII-5
IT JUNIN-S
ZAI1 (MA4-4
ANVE (2IVO-5)
BI Figure 6: The average FT and NMI of each target layer. Each model is distilled for 20 epochs.
Impact of the Hyperparameter 8. The parameter 8 in Eq. (2) deter- mines the layer chosen for distillation. As presented in Fig. 7, when ranges from 0.06 to 0.12, the selected layer remains nearly constant:
Layer
Layer 18
Merraee motu 1 FT]
the 18th layer for MAE-L and the 9th layer for DINO-B. Therefore, the performance of UNIP is relatively stable with respect to
Layer
Layer
Comparison with the Feature Distillation. As compared in Tab, 6, the performance of feature distillation consistently lags behind attention distillation across different layers, implying the latter's superiority. We believe this is because attention distillation only restricts the relationship between tokens, whereas feature distillation imposes direct constraints on each token's 's features. Excessive constraints features may intensify the distribution shift and hinder the generalization of distilled models. Additionally, the performance of feature distillation across different layers similar, likely due to the skip connections in VIT, which enhance feature similarities between layers. In contrast, the attention maps of different layers differ significantly, as revealed Sec. 3.1.
Layer 8
UNIP (MAE-L)
UNIP-S (DINO-B)
Hyperparameters
Figure 7: The average FT when employing dif- ferent in Eq. (2).

---

Published as a conference paper at ICLR 2025
Impact of Pre-training Datasets. Tab. 8 il- lustrates the performance of different datasets. As anticipated, all components of the InfMix dataset are necessary, including the infrared dataset InfPre, the ImageNet subset (Deng et al., 2009), the COCO training set (Lin et al., 2014), and the grayscale operation. Remark- ably, InfMix significantly outperforms single- modality datasets like ImageNet and InfPre. This improvement can be attributed to the
Table 8: Ablations for components of the InfMix
dataset. The teacher and student models are MAE- L and UNIP-S. All datasets are distilled for the same number of iterations for fair comparison.
Dalasel
Almages
SODA
M SCUT Seg Avg FT
InlMix
859,375
70.99
51.32
70.79
64.37
wlo INIK
659,375
69.41
51.13
70,21
63.58
wlo COCO
741,088
69.62
51.29
69.58
wlo Grayscale
859,375
69.73
50.71
71.09
63,84
TeageNNt-1KK InfPre
1,281,167
69.39
49.11
69.63
62.71
complementary strengths of both modalities:
541,088
68.45
51.27
67.87
62:53
infrared images help mitigate the distribution shift issue, while RGB images enhance general fea- ture extraction capabilities. The mixed dataset effectively balances these two aspects. Moreover, Tab. 15 in the appendix displays the scaling characteristics of pre-training data, demonstrating the necessity of constructing the larger InfMix dataset.
Comparison with Contiunal Pre-training on Target Domain. We initialize MAE-S with RGB pre-trained weights and further pre-train it on InfMix for 100 epochs. As shown in Tab. 9, this continually pre-trained MAE-S (58.53%) exceeds the RGB pre-trained one (55.39% in Tab. 4). However, it still underperforms UNIP-S by 5.84% and requires more training time, highlighting the efficiency of UNIP over continual pre-training.
Table 9: Comparisons of pre-training methods.
Method
Avg FT
Training Time (b)
Continual Pre-trained (MAE-S) S)
$8.53
75.0 (Ix KTX3090)
UNIP-S distileed)
64.37
(Ix RTX3090)
Impact of the LL-FPN. Tab. 7 shows the performance of models distilled from various teachers. While LL-FPN enhances performance for all models, the improvements are much greater when using hybrid patterns as distillation targets than local or global patterns. This demonstrates LL- FPN's superiority and good compatibility with the hybrid pattern, supporting the analysis Sec. 4. 1.
Applicability to Other Modalities. We extend the LLP experiments in Sec. 3.3 to the RGB and depth modal- ities. As shown in Tab. 10, for both DINO-S and DeïT-S, the LLP per- formance of middle layers (the lo- cal pattern) surpasses that of deep layers (the global pattern) across all RGB and depth semantic segmenta- tion datasets. This mirrors the phenomenon in the infrared domain discussed in Sec. 3.3, underscor- ing the importance of hybrid patterns for semantic segmentation tasks, regardless of dataset size or modality. Therefore, we believe that UNIP can be effectively extended to other modalities.
Table 10: The LLP performance on RGB and depth datasets. Training epochs are 30 for ADE20K and 100 for others.
DINO-S
DaiT-S
(Modality) Dalaset
Layer 9 Layer 12 Layer Layer
(yyrrdd)
(Global)
(Hivrrid)
(Giobal)
(RGB) ADE20K (Zhon et al., 2017)
26.11
23.15
24.35
(RGB) MFNer RGB (Ha et al., 2017)
38.94
37,53
30.43
29.44
(Depph) NYUDep2 (Silherman et at., 2012)
17.25
15.29
5.55
5.15
(Depch) AUN-RGBD (Song 2015)
13.17
11.41
5.61
4.94
5 CONCLUSION AND DISCUSSION
In this work, we comprehensively benchmark the infrared segmentation performance of different pre-training methods and uncover several valuable insights. We further analyze the pre-trained at- tention maps and identify the importance of hybrid patterns for semantic segmentation. Finally, we propose the UNIP framework to improve the performance of small ViT models. Extensive exper- imental results demonstrate the effectiveness of our dataset and method. UNIP presents a viable approach for selective knowledge distillation in domain transfer settings. We hope our analysis can provide meaningful insights into the characteristies and differences among pre-training methods, ul- timately contributing the advancements of visual pre-training and downstream transfer learning.
Limitations and Future Work. Due to limited computing resources, we validate UNIP's effec- tiveness only in the infrared domain for semantic segmentation. However, we believe UNIP can be effectively extended to other modalities, such as RGB and depth images, as the superiority of hybrid patterns in these modalities is demonstrated in Tab. 10. Exploring its potential in other dense pre- diction tasks, like object detection and depth estimation, is also worthwhile. Moreover, combining hybrid patterns from different pre-trained methods could be a promising avenue.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No.XDA03703055 the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No.XDB0500103, and the National Natural Science Foundation of China under Grant 62306310.
REPRODUCIBILITY STATEMENT
Reproducibility is a priority our research. In this statement, we outline the measures taken to ensure our work can be reproduced.
Source Code. The source code our work is available at this link. Researchers can access and utilize our code to reproduce the experimental results in this paper. The source code and pre-trained model weights will made publicly available.
Experimental Setup and Details. In the main paper, the basic experimental configurations presented in Sec. 2.1 (benchmark) and in Sec. 4.2 (UNIP). In Appendix C. we provide the de- tailed settings, including the implementation details of the benchmark (Appendix C.I) and UNIP (Appendix C.4), the comparisons of different evaluation metrics and their hyperparameter settings (Appendix C.2), and the evaluation datasets usage (Appendix C.3).
Datasets. We outline the construction steps of our InfMix dataset in Sec. 4.1. In Appendix F, we further present more details about the dataset collection and preprocessing.
By highlighting these references, we intend to improve the reproducibility of our work, helping other researchers verify and build on our findings. We're We're open to any questions or requests for more information about our methods, as we aspire to ensure our research is transparent and reliable.
ETHICS STATEMENT
Our constructed dataset InfMix is derived from the extraction and integration of 25 publicly acces- sible RGB and infrared datasets. These datasets are available for public use. During the extraction process, we employ data deduplication methods only, ensuring that no personal biases were intro- duced. We adhere to all usage requirements of the original datasets and use them solely for scientific research purposes. Importantly, the dataset does not include any additional metadata or labels that could be used to identify individuals. Detailed information regarding the composition and con- struction process of our dataset is provided in Appendix F. We solely responsible for any legal violations with respect to our collected dataset and accept all the risks associated.
REFERENCES
CVC-14 pedestrian dataset, 2016a.
URL http://adas, eve. eab.es/elektrr/
onigma-portfflio// portfolio/vvc wowo-voo---iibbleee fir f-r--fir--------- pedestiad seeessttrrin--nsuunnn ce ne-dataset/.
CVC-9 pedestrian dataset, 2016b.
URL http:///das..oo..ub.eeeeeeeee
Besma Abidi. Iris thermal/visible face database. U/hhtt/p///eeell-okttttte.oooogp bench/.
Chris H. Bahnsen and Thomas B. Moeslund. Rain Removal in Traffic Surveillance: Does it matter? IEEE TITS, 2019.
Yutong Bai, Zeyu Wang, Junfei Xiao, Chen Wei, Huiyu Wang, Alan L. Yuille, Yuyin Zhou, and Cihang Xie. Masked autoencoders enable efficient knowledge distillers. In CVPR, 2023.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BeiT: BERT pre-training of image transform- In ICLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Elizabeth Bondi, Raghav Jain, Palash Aggrawal, Saket Anand, Robert Hannaford, Ashish Kapoor, Jim Piavis, Shital Shah, Lucas Joppa, Bistra Dilkina, and Milind Tambe. BIRDSAL: A dataset for detection and tracking in acrial thermal infrared videos. In WACV, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Junzhang Chen and Xiangzhi Bai. Atmospheric transmission and thermal inertia induced blind road segmentation with a large-scale dataset tbrsd. In ICCV, 2023.
Kai Chen, Chenglong Zhou, and Shuigen Wang. Infrared city database, 2021a. URL http:// openai raytronteks com/ apply/ppaeeeea. / www.weiiaiiivviddeeaeeee html/.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 2018.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self--uperrised vision transformers. In ICCV, 2021b.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
MMSegmentation Contributors. MMSegmentatiin:: Openmmlab semantic segmentation toolbox and benchmark, 2020. URL https: tattps://githui /github. com/openg wwwooppeeeaaamaameeeeeeeeeeeeeaa www.a//mnnneeetati..o..
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. arXiv preprint arXiv:2401. 143991 2024.
Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In IROS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint ar 100.:05030005331 2015.
Zejiang Hou, Fei Sun, Yen- Ken-Kuang Chen, Yuan Xie, and Sun-Yuan Kung. Milan: Masked image pretraining on language assisted representation. arXiv preprint moXiv:2208.06044 2022.
Soonmin Hwang. Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedes- trian detection: Benchmark dataset and baselines. In CVPR, 2015.
12

---

Published as a conference paper at ICLR 2025
Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. LLVIP: A visible-infrared paired dataset for low-light vision. ICCVW, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. TBD, 2021.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In /CML, 2019.
Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, and Jin Tang. RGB-T object tracking: Bench- mark and baseline. PR, 2019.
Chenglong Li, Wei Xia, Yan Yan, Bin Luo, and Jin Tang. Segmenting objects in day and night: Edge-conditioned cnn for thermal image semantic segmentation. TNNLS, 2021a.
Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, and Dengdi Sun. LasHeR: A large-scale high-diversity benchmark for rght tracking. IEEE TIP, 2022a.
ChunLiu Li and Shuigen Wang. Infrared ship database, 2021. URL http:
raytrontek.com
Satpaiteeeccceappp_____eeaaa__ae SaappplyE__easshiiapaapeeeeeeeee Shipppinshpppnngggg html.
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung- Yeung Shum. Mask DINO: Towards unified transformer-batt based framework for object detection and segmenta- tion. In CVPR, 2023.
Gangqiang Li, Jiansheng Wang, and Shuigen Wang. On-vehicle visible and infrared object de- tection database, 2021b. URL http::tttp//////e raattrntekk com/ Sapply/E_Dooo Light_vehicle html/.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In ECCV, 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
Fangeen Liu, Chenqiang Gao, Yaming Zhang, Junjie Guo, Jinhao Wang, and Deyu Meng. InfMAE: A foundation model in infrared modality, arXiv preprint arXiv:2402.00407, 2024a.
Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In CVPR, 2022.
Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, and Liang Lin. Cross-modal collaborative representation learning and a large-scale rght benchmark for crowd counting. In CVPR, 2021a.
Qiao Liu, Xin Li, Zhenyu He, Chenglong Li, Jun Li, Zikun Zhou, Di Yuan, Jing Li, Kai Yang, Nana Fan, and Feng Zheng. LSOTB-TIR, A large-scale high-diversity thermal infrared object tracking benchmark. In ACM MM, 2020.
Qing Liu, Zhaofei Xu. Ronglu Jin, and Shuigen Wang. General-purpose dual-sensor (in- frared/visible) video database, 2021b. URL http:/tt///// htttp://opente wwwppppea....mm/oooo raytrronee.ttt.ttee...... com apply/ F_Infraree__ecuraae html/.
Qing Liu, Zhaofei Xu, and Shuigen Wang. Infrared aerial photography database, 2021c. URL http::/ raytrontek. com/ apply/pppyyy/e///// Aerial mancar html.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target represen- tations for masked autoencoders. In ICLR, 2024b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
13

---

Published as a conference paper at ICLR 2025
Ivan Adriyanov Nikolov, Mark Philip Philipsen, Jinsong Liu, Jacob Velling Dueholm, An- ders Skaarup Johansen, Kamal Nasrollahi, and Thomas B Moeslund. Seasons Drift: long term thermal imaging dataset for studying concept drift. In NeurIPS, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wee Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. 7MLR, 2024.
Cristina Palmero, Albert Clapés, Chris Bahnsen, Andreas Møgelmose, Thomas B Moeslund, and Sergio Escalera. Multi-modal rgh- thee-ep human body segmentation. IJCV, 2016.
Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? In ICLR, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
Karl Pearson. Mathematical contributions to the theory of evolution. iii. regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, Containing Pa- pers of a Mathematical or Physical Character, 1896.
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. TinyMIM: An empirical study of distilling mim pre-trained models. In CVPR, 2023.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. ECCV, 2012.
Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D: A rgb-d scene understand- ing benchmark suite. In CVPR, 2015.
Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu, Drone-based rgb-infrared cross-modality vehicle detection via uncertainty-aware learning. IEEE TCSVT, 2022.
Karasawa Takumi, Kohei Watanabe, Qishen Ha, Antonio Sejero-De-Pabllee Yoshitaka Ushiku, and Tatsuya Harada. Multispectral object detection for autonomous vehicles. In MM, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. DeiT III: Revenge the vit. In ECCV, 2022.
Zhengzheng Tu, Yan Ma, Zhun Li, Chenglong Li, Jieming Xu, and Yongtao Liu. RGBT Salient Object Detection: large-scale dataset and benchmark. IEEE TMM, 2023.
Shaoru Wang, Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. A closer look at self- supervised lightweight vision transformers. In ICML, 2023.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Seg- Former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Haitao Xiong, Wenjie Cai, and Qiong Liu. MCNet: Multi-level correction network for thermal image semantic segmentation of nighttime driving scene. Infrared Physics & Technology, 2021.
14

---

Published as a conference paper at ICLR 2025
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang. Fanyi Xiao, Chenchen Zhu, Xiao- liang Dai, Dilin Wang, Fei Sun, Forrest landola, Raghuraman Krishnamoorthi, and Vikas Chan- dra. EfficientSAM: Leveraged masked image pretraining for efficient segment anything. In CVPR, 2024.
Zhenjie Yu, Kai Chen, Shuang Li, Bingfeng Han, Chi Harold Liu, and Shuigen Wang. ROMA: Cross-domain region similarity matching for unpaired nighttime infrared to daytime visible video translation. In ACM MM, 2022.
Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, and Xiang Ruan. Visible-Thermal UAV Track- ing: A large-scale benchmark and new baseline. In CVPR, 2022.
Tao Zhang, Kun Ding, Jinyong Wen, Yu Xiong, Zeyu Zhang, Shiming Xiang, and Chunhong Pan. PAD: Sule-supe pre-training with patchwise-scale adapter for infrared images. arXiv preprint arXiv:2312.08112, 2023.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In ICLR, 2022.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
In Sec. A, we discuss the related works. In Sec. B, we review the relationship between our mo- tivations and proposed methods. In Sec. C, we provide detailed descriptions of the experimental settings, including the complete benchmark results, evaluation metrics and datasets, and the exper- imental specifics of UNIP. Further analysis is conducted Sec. D, covering (1) the relationship between NMI and attention patterns, and (2) the CKA analysis of feature representation. Additional experimental results are presented in Sec. E. Finally, we provide more details of the pre-training dataset in Sec. F and offer additional visualization results in Sec. G.
A RELATED WORK
Visual pre-training aims to equip models with fundamental feature extraction capabilities using large-scale pre-training data, aiding their fine-tuning on downstream tasks. Supervised pre-training (He et al., 2016; Dosovitskiy et al., 2021), one of the earliest methods, typically involves image classification on labeled datasets like ImageNet (Deng et al., 2009). However, its reliance on labeled data limits its scalability, prompting the development of self-supervised pre-training, This approach utilizes various pretext tasks, such as contrastive learning and masked image modeling, to pre-train models, achieving results competitive with supervised counterparts. These methods are detailed in Sec. 2.1. In the infrared domain, Zhang et al. (2023) proposes the patchwise-scale adapter to adapt RGB pre-trained models for infrared tasks, and Liu et al. (2024a) constructs hierarchical model for infrared pre- training. However, previous works have not thoroughly analyzed the transfer performance of different pre- mre-trainng methods on infrared tasks. Our work aims to fill this gap.
Knowledge distillation (KD) is a widely used technique to improve the performance of small mod- els by extracting knowledge from well-trained large models. Initially developed for supervised learning (Hinton et al., 2015), it has recently gained popularity in self-supervised learning. Bai et al. (2023), Liu al. (2024b), and Xiong et al. (2024) focus on feature KD, while Caron et al. (2021), Zhou et al. (2022), and Oquab et al. (2024) employ self-relational KD. Similar to our work, Wang et al. (2023) and Ren et al. (2023) explore attention KD, but they only conduct empirical explorations on MAE in the RGB domain and do not explore the underlying mechanism of using different lay- ers for distillation. In contrast, our research systematically investigates which attention patterns are most advantageous for distillation in domain transfer settings and proposes the NMI metric to guide the process, demonstrating effectiveness across various pre-training methods.
Semantic segmentation is widely investigated visual task that aims to classify each pixel into different semantic categories. As one of the fundamental works, FCN (Long et al., 2015) employs a fully convolutional neural network for pixel-to-pixx classification. The following works (Chen et al., 2018; Zhao et al., 2017: Xiao et al., 2018) enhance FCN by constructing the feature pyramid network and improving the context fusion module. With the advancements of transformer-batt architectures in visual tasks, Xie et al. (2021) proposes the powerful SegFormer, featuring a hier- archical transformer encoder and lightweight decoder. Mask2Former (Cheng al., 2022) further unifies semantic segmentation with other segmentation tasks following the framework of DETR (Carion et al., 2020). For infrared semantic segmentation, Xiong et al. (2021) develops a multi-level correction network (MCNet) to capture the context in infrared images, while TINN (Chen & Bai, 2023) focuses on preserving the inherent radiation characteristic within the thermal imaging process. However, these methods do not explore the impact of different pre-trained models on segmentation performance. Our study utilizes semantic segmentation representative downstream visual task and systematically investigates the influence of various pre-trained models on this task.
B RELATIONSHIP BETWEEN MOTIVATIONS AND METHODS
Our primary motivation is to enhance the performance of models on infrared semantic segmentation tasks. From the model perspective, factors affecting the performance on specific tasks include not only the design of the model architecture but also the quality of the model's pre-training. Previous works (Li et al., 2021a; Chen & Bai, 2023) have aimed to improve performance by designing specific network architectures for infrared semantic segmentation tasks. However, in the infrared domain, where labeled data is limited, the quality of the pre-trained model is also crucial. Therefore, our
16

---

Published as a conference paper at ICLR 2025
work explores an alternative approach by emphasizing the optimization of pre-trained models specifically for infrared semantic segmentation tasks to enhance performance. To facilitate this exploration, our work is organized into three stages: benchmark establishment, cause analysis, and method proposal, as illustrated in Fig. 1.
Benchmark Establishment (Sec. 2). We establish benchmark for the transfer performance of six RGB pre-training methods, encompassing a total of 18 pre-trained models, on three infrared semantic segmentation datasets (Sec. 2.1). Our findings reveal several key phenomena (Sec. 2.2), such as the lack of correlation between model performance on ImageNet and infrared segmentation datasets (Tab. 1), and the superior generalization of supervised and contrastive learning methods over masked image modeling methods in the context of infrared segmentation tasks (Fig. 2).
Cause Analysis (Sec. 3). To analyze the performance discrepancies among various pre-training methods in infrared segmentation tasks, we conduct an in-depth analysis of the attention maps from the pre-trained models. Our findings indicate that the degree of focus on local and global information (Sec. 3.1 Sec. 3.3), well as on shape and texture information (Sec. 3 3,4), significantly impacts the performance of infrared semantic segmentation tasks. We further validate through corresponding experiments that the existence of hybrid attention patterns (Fig. 3 - Fig. 5) and the reduced bias towards texture (Tab, 2) both play crucial roles in enhancing the performance of pre-trained models in infrared segmentation tasks.
Method Proposal (Sec. 4.1). Based on the observations and analyses from the previous two sec- tions, propose UNIP, a framework designed to improve the infrared segmentation performance of pre-trained models through three key aspects: the pre-training objective (NMI-HAD), the pre- training data (InfMix), and the fine-tuning architecture (LL-FPN). Both NMI-HAD and LL-FPN enhance performance by effectively leveraging hybrid attention patterns, while InfMix enhances performance by reducing the pre-trained model's 's bias toward texture information. Importantly, our approach does not alter the structure of the backbone model or the decoder; instead, we focus on targeted pre-training specifically designed for infrared segmentation tasks. We believe this is one way in which the proposed method is specific to infrared semantic segmentation tasks. As a result, our pre-trained models significantly outperform RGB pre-trained models of compara- ble or even larger sizes in infrared semantic segmentation tasks (Tab. 4), and also achieve superior performance compared to other models specifically designed for infrared segmentation (Tab. 5).
C EXPERIMENTAL DETAILS
C.I BENCHMARK DETAILS
Reproduction of small MAE models. The MAE-T and MAE-S are reproduced following the settings in He et al. (2022). We make several adjustments to the decoder to make it suitable for small encoders. For both MAE-S and MAE-T, the decoder includes 8 transformer blocks, each with 8 attention heads. The decoder dimensions in MAE-S and MAE-T are 256 and 192, respectively.
Implementation details. The weights of all pre-trained models are downloaded from corresponding official repositories. The models are trained for 100 epochs using MMMSegmentation (Contributors, 2020). For different methods and model sizes, we keep the learning rate constant and sweep the layerwise decay rate across {0.5, 0.65, 0.75, 0.85, 1.0). To adapt models pre-trained on three- channel RGB images for single-channel infrared images, we duplicate the infrared images three times create pseudo-three--hhmeee images.
C.2 EVALUATION METRICS
Fine-tuning, Fine-tuning is the default evaluation metric in this work, which utilizes the pre-trained model as the backbone of existing semantic segmentation models. Following previous works (He et al., 2022; Zhou et al., 2022), we employ UperNet (Xiao et al., 2018) as the semantic segmentation model. As illustrated in Fig. 8a, to build the feature pyramid based on the non-hierarchical ViT model, features from different layers are passed through the MaxPooling layers or DeConv layers, to obtain features of different resolutions. These multi-scale features are then input into the decoder for segmentation results. Following He et al. (2022) and Zhou et al. (2022), we use features of the (4, 6, 8, 12) layers in ViT-T, VIT-S, and ViT-B, and the features of the {8, 12, 16, 24) layers in
17

---

Published as a conference paper at ICLR 2025
The performance of different pre-trained models TmageNet and infrared semantic seg- mentation datasets. The Scratch means the performance of randomly initialized models. The PT Epochs denotes the pre-training epochs while the INIK FT epochs represents the fine-tuning epochs on ImageNet (Deng et al., 2009). denotes models reproduced using official codes. refers to the effective epochs used in Zhou al. (2022). The top two results are marked in bold and underlined format. Supervised and CL methods, MIM methods, and UNIP models colored in orange
Table 11:
gray and cyan respectively.
IN
Linear Probing (LP)
Methods
Epochs
Epochs
MFNet-T SCUT-Seg
SCUT-Seg MFNeetT
ViT-Tiny/16
Seratch
31,34
19.50
41.09
30.64
MAE (He et 2022)
800
200
71.8
52.85
35.93
51.31
46.70
23.75
15.79
27.18
22.24
DeiT (Touvrom et 2021)
300
72.2
63.14
44.60
61.36
56.37
42.29
21.78
31.96
32.01
UNTP (MAE..)
100
64.83
48.77
67.22
60.27
44.12
28.26
35.09
35.82
UNIP (IBOT-L)
100
65.54
48.45
67.73
60.57
52.95
30.10
40.12
41.06
ViT-Small/16
Scratch
41.70
22.49
46.28
36.82
(Hle et 2022)
55.39
CrosMAE (Fu ct al, 2024)
800
200
80.5
63.95
43.99
63.53
57.16
39.40
23.87
34.01
32.43
DelT (Touvron al., 2021)
300
79.9
68.08
45.91
66.17
60.05
44.88
28.55
38.92
37.44
Deil III (Touvron
2022)
800
81.4
69.35
47.73
67.32
61.47
54.17
32.01
43.54
43.24
DINO (Caron ot al., 2021)
3200
200
82.0
68.56
47.98
68.74
61.76
56.02
32.94
45.94
44.97
2022)
3200
82.3
UNIP (DINO-BB B)
100
69.35
49.95
69.70
63.00
$7.76
34.15
46.37
46.09
(MAE-I.)
51.32
70.79
64.37
UNIP (IBOT-L)
100
20.75
51.81
71.55
64,70
60.28
37.16
47.68
48.37
VIT: UTT-Rase/16
Scratch
44,25
23.72
49, 44
39.
MAB 2022)
60.94
CrosMME (Fu al, 2024)
800
100
83.7
68.29
47.8 85
68.39
61.51
43.35
26.03
38.36
35.91
DeiT (Touvron al., 2021)
300
81.8
69.73
48.59
69.35
62.56
57.40
34.82
46.44
46,22
Deil III (Touvron
2022)
800
20
83.8
71.09
49.62
70.19
63.63
59.01
35.34
48.01
47.45
DINO (Caron et al., 2021)
1600
100
83.6
69.79
69.82
62.72
59.33
34,86
47.23
47.14
iBOT (Zhou 2022)
1600
100
84.0
71.15
48.98
71.26
63.
60.05
34.34
49.12
47.84
UNIP (MAE-L)
100
71.47
$2.55
71.82
65.28
38.82
34.75
48.74
47.43
(IBOT-L)
V
Seratch
44.70
23.68
49.55
39.31
MAE (He ot al., 2022)
1600
50
85.9
71.04
51.17
70.83
64.35
52.20
31.21
43.71
42.37
CrosMMAE 2024)
63.90
DeiT3 (Touvron al., 2022)
800
20
84.9
71.67
50.78
71.54
64.66
59.42
37.57
50.27
49.09
(Zhou et 2022)
VIT-L, to build the feature pyramid. Remarkably, fine-tuning, all parameters including the pre- trained model, the feature pyramid, and the decoder, tuned with the labeled downstream datasets. Hyperparameters are listed in Tab. 12.
Linear Probing. As mentioned above, fine-tuing introduces additional learnable parameters and alters the pre-trained feature representation. Its performance may not fully reflect the characteristics of the pre-trained features. Therefore, linear probing is also employed as an evaluation metric. As shown in Fig. 8b, features from different layers are resized to 1/4 4 of the input resolution and then concatenated together. Finally, a linear head (1 conv) utilizes these concatenated features to predict segmentation results. Notably, only the linear head is trainable, while all other parameters are frozen. The layer settings of output features are the same as fine-tuning.
Fine-tuning (LL-FPNN, This metric is discussed in Sec. 4, which aims to enhance the fine-tuning performance of UNIP models by using the last layer obtain features of different resolutions, as depicted in Fig. Sc. Specifically, we employ the features of the {12. 12, 12, 12) layers in ViT-T, ViT- S. and ViT-B, and the features the {24, 24, 24, 24} layers in ViT-L, to build the feature pyramid. Other settings remain the same fune-nuning.
Layerwise Linear Probing. This metric is a layerwise version of the linear probing metric. is designed to assess the pre- preitrained feature representation at each layer. As shown in Fig. 8d, only the features of a single layer are forwarded to the linear head following the resize operation. Other settings are the same linear probing.
18

---

Published as a conference paper at ICLR 2025
1/16
1/32
1/16
1/4
1/16
1/16
Decoder
1/16
1/4
Concat
1xt Conv
1/16
1/8
1/16
1/4
1/16
1/4
1/16
1/4
(a) Fine-tuning
(b) Linear Probing
1/16
1/32
1/16
1/4
1/16
Decoder
1/16
1/4
1x1 Conv
5
trainable
1/16
1/8
1/16
1/4
1/16
1/4
1/16
1/4
frooee
(c) Fine-tuning (LL-FPN)
(d) Layerwise Linear Probing
Figure 8: Illustrations of different transfer architectures for semantic segmentation tasks.
Table Settings of semantic segmentation.
Hyperparameters
SODA
MFNet-T
SCUT-seg
Input resolution
512 512
512 x 512
512 512
Training epochs
100 / 200
100 / 300
100
Training iterations
14400 28800
9800 29400
16800
Peak learning rate
le-4
le-4
le-4
Batch size
8 AdamW 0.05
Optimizer
AdamW
AdamW
Weight decay
0.05
0.05
Optimizer momentum
31,02
0.9, 0.999
81, B2 0.9, 0.999
BI,
0.999
Learning rate schedule
Linear decay
Linear decay 0 /000/3000
Linear decay
Minimal learning rate
Warmup steps
1500 3000
3000
1700
C.3 EVALUATION DATASETS
SODA (Li et al., 2021a). This dataset features variety of indoor and outdoor scenes. It comprises 1,168 training images and 1,000 test images, spanning 20 distinct semantic categories, including road, building, car, chair, lamp, table, monitor, and others.
MENet (Ha et al., 2017). This dataset focuses on RGBT semantic segmentation for automotive driving scenarios and includes 1,569 image pairs of infrared and RGB images. It is divided into 784 training images, 392 validation images, and 393 test images, covering semantic categories such as car, person, bike, curve, and others. When benchmarking the performance of different pre-training methods, we combine the validation set with the test set, resulting in a larger test set of 785 images. When comparing UNIP models with other SOTA semantic segmentation models, follow their settings, i.e., using the original 393 test images for evaluation.
SCUT-Seg (Xiong al., 2021). This dataset includes 1345 training images and 665 test images in nighttime driving scenes. It has 10 classes including road, person, fence, pole, and others.
ADE20K (Zhou et al., 2017). ADE20K is large-scale RGB semantic segmentation dataset, cov- ering variety of scenes from indoor to outdoor and nature to urban. It consists of 20,210 training images and 2,000 test images, with 150 different semantic categories.
ImageNet-Ile -K (Deng et al., 2009). ImageNet- IK is subset of the ImageNet database, consisting of 1,000 categories with roughly million training images, 50,000 validation images, and 100,000 test images. It is widely used in computer vision research like image classification and pre-training.
C.4 UNIP
Head Misalignment. solve the head misalignment between teacher and student models during distillation, experiment with two methods. (1) The first method is the adaptive block proposed
19

---

Published as a conference paper at ICLR 2025
Table 14: Settings of pre-training.
Hyperparameters
Value
Table 13: Configurations of ViT for semantic
Input resolution
224 x 224
segmentation tasks.
Training epochs
100
Warmup epochs Optimizer
5
Model Dimension
Head Num
Depth
AdamW
VIT-T 192
3
12
Base learning rate
le-4
VIT-S
384
6
12
Weight decay
0.05
ViT-B 768
12
12
Optimizer momentum
31, B2 = 0.9, 0.95
ViT-I 1024
16
24
Batch size
4096
Learning schedule
Cosine decay
Augmentation
Random resized cropping &
Random horizontal flipping
in Ren et al. (2023). Specifically, during distillation, the number of attention heads in the student model's last layer is adjusted to be the same that of the teacher model by changing the head dimension while keeping the overall dimension constant. When performing fine-tuning or linear probing on downstream tasks, the number of attention heads is reverted to the standard setting in Tab. 13. (2) The second method involves adding a self-attention layer at the end of the student model during distillation. The number of attention heads in the extra attention layer is equivalent to the teacher model's. This layer is removed when transferring to downstream tasks. These two methods achieve similar performance, but the latter consumes slightly more training time. Therefore, we use the first method in practice.
Feature Distillation. For the feature distillation in Tab. 6, we employ linear projection layer to match the dimension of the student model to that of the teacher model. The distillation and fine- tuning settings are the same UNIP. The loss function is the cosine similarity loss between the L2 normalized student feature 12( (F)) and teacher feature 12 (Fs)::
(F 122-222225 (FF))))
Implementation Details. All experiments are conducted using the PyTorch toolkit (Paszke et al., 2019) on NVIDIA RTX 3090 GPUs. The default settings are shown in Tab. 14. We use the linear learning rate scaling rule: Ir = base. Irx batchsize / 256, following He et al. (2022). The semantic segmentation settings of UNIP models are the same as those in Appendix C.2.
D ADDITIONAL ANALYSIS
D.1 NORMALIZED MUTUAL INFORMATION
The Normalized Mutual Information (NMI) is employed in Sec. 3.2 to measure the attention pat- terns. p(z) denote the marginal probability of the i-th query token and p(ky) denote the marginal probability of the j-th key token. Since query tokens are evenly distributed across every spatial co- ordinate, p(g) can be formulated as:
1 pla.) = N' 1 N.
(4)
Assume A E N represents the m-th head of the attention matrix after the softmax operation without the class token, where N is the number of spatial tokens. The attention scores from each query token to all key tokens sum to i.e., Σド A 1, = 1, N. Thus, each row of A can be viewed as the conditional probability distribution of key tokens given the query token:
p(k,lq:) A
(5)
Then the joint probability of qi and k, can be calculated as:
plg. ky) = p(kylas)p(aa AT
(6)
The marginal probability of ky is:
N
N Σ Am
p(ky) = Σ
plg, kz) = N
(7)
1=1
20

---

Published as a conference paper at ICLR 2025
The mutual information of query and key tokens can be formulated as:
N
N plas, Rig) log
p(gi, ki))
(":
K)
p(g))p(kg)
i=1
(8)
NV
N
NA
log
Σ
A
The entropy of query and key tokens can be calculated as:
N
N
(T) Σ p(a:) log p(g:))))
log
(9)
N
i=1
N
N
1 HT (K) = Σ p(k) = Σ
m 3
log
AM
(10)
N
N
j=1
1=1
1=1
Therefore, the NMI of the m head is:
(m
NMI (Q: K)
(11)
The final NMI is calculated by averaging all heads:
M Σ M
NMI(Q; K)
NMI K).
(12)
The value of NMI ranges from to 1. It reaches the maximum value of when the joint probability of the query and key tokens is the same their marginal probability:
p(g. ky) = p(4) = p(k:). i == 1,,,
(13)
According to Eq. (4), Eq. (6), Eq. (7), and Eq. (13), it can be derived that
1. if
AT
(14)
0, if
which implies that the attention matrix of each head is an identity matrix. This indicates that each query token focuses only on the key token at the same spatial position, which is particular case of the local attention pattern.
On the other hand, the NMI has a value of when the query and key tokens are independent:
p(g,, kj) = p(z))((;;) i= 1,2, N. = 1,2, .... N.
(15)
According to Eq. (4), Eq. (6), Eq. (7), and Eq. (15), we can derive that
A Ak.j i = 1,,
2, .... N.j 1,.,...
(16)
which indicates that every row of the attention matrix the same. This means that each query token has the same attention maps for all key tokens, which is a particular case of the global attention pattern. Therefore, a higher NMI value indicates stronger relationship between the query and key tokens and more local attention pattern. Conversely, lower NMI value means that different query tokens have more similar and global attention patterns for key tokens.
D.2 CENTERED KERNEL ALIGNMENT
In this section, we extend the analysis Sec. 3.1 from the attention pattern to the feature represen- tation. Let x denote the input features of the I-th block of the ViT model. The features of the next block !++1 can be formulated as:
Ptmp = x + Attention(LN(ze)
(17)
x 1+1 Timp + FFN(LN FFN(LN(2tmp))
(18)
21

---

Published as a conference paper at ICLR 2025
2.0
12
3.9
5.8
9-
0-
9
DINO-S
DINO-S
DINO.S
9.6
0.5
244
433
9
MAE-S
MAE-L
UNIP-S(MAE-L)
Figure 9: CKA representation analysis of different models. UNIP-S aligns well with DINO-S in the shallow and middle layers, indicating that the hybrid patterns are effectively distilled from MAE-L.
The FT performance of using differ- ent ratios of InfMix as the pre-training dataset. The images are evenly sampled from each sub-
Table 15:
Table 16:
The FT performance of using multiple layers of the teacher model for distillation. The attention maps of different layers are concate- nated along the channel dimension. The teacher
dataset. The teacher and student models are
MAE-L and UNIP-S.
and student models are MAE-L and UNIP-S.
Ratio #images
SODA
MFNet-T
SCUT-Seg
Avg
Layer
SODA
MPNet-T
SCUT-Seg
Avg
1%:
8,594
20.61
16.10
31.24
22.65
18
70.99
$1.32
70.79
64.37
10%
85,938
59.66
38.80
57.72
52,06
16+18
69.73
50.88
70.68
63.76
30%
257,813
68.29
50.39
69.04
62.57
17+18
69.59
51.33
69.47
63.46
100%
859,375
70.99
64.37
17+18+:19
69.13
67.73
62.27
where Attention, FFN, LN refer to the self-attention module, the feedforward module, and the Lay- erNorm layer, respectively. Obviously, the self-attention module plays crucial role in transforming the feature representation. The global attention pattern will bring the features of different tokens closer since different query tokens interact similarly with all key tokens. In contrast, the local atten- tion pattern will make the features of different tokens further apart.
To investigate the relationships between features of different layers and models, use the centered kernel alignment (CKA), a metric that measures the similarity between two feature maps. The details of CKA can refer to Kornblith et al. (2019). As shown in Fig. 9, features in the later layers of MAE-S, e.g., the 10th and 1 Ithh layers, are similar to features in the shallow layers of DINO-S, e,g., the 4th, 5th, and 6th layer, implying that the features of MAE-S are relatively lower level compared to DINO-S. This is consistent with observations in Sec. 3.1 that the local attention patterns are distributed in the shallow layers DINO-S, but are present in all layers of MAE.
For MAE-L, the features in the middle layers (13th to 20th) exhibit high similarity with the middle- layer features of DINO-S (9th and 10th), due the hybrid patterns in these layers. On the contrary, the features in the later layers (the 22nd and 23rd layers) gradually resemble the shallow layers of DINO-S, which can be attributed to the local patterns the later layers of MAE-L.
It is noteworthy that the UNIP model effectively imitates the features in the middle layers of MAE- L. Its features align more closely with DINO-S than those MAE-S, especially in the shallow and middle layers, demonstrating that attention distillation can implicitly change the features of distilled models like what feature distillation explicitly does.
E MORE EXPERIMENTS.
Pre-training important. We compare the average FT performance of pre-trained and randomly initialized mod- els. For pre-trained models, the performance averaged across six different methods in Tab. 11. As shown in Tab. 17, models without pre- pre-training consistently fall be- hind by 20.89% 25.03%, regardless of model size. This gap widens with larger models, highlighting the importance of pre-training and the necessity of studying different pre-training approaches on infrared tasks.
Table 17: Comparison of initialization.
Initialization
Tiny
Small
Base
Large
Random
30.64
36.82
39.14
39.31
Pre-training
51.53
59.65
62.53
64.47
+20.89
+25.16
22

---

Published as a conference paper at ICLR 2025
Table 18: The fine-tuning performance of using different training strategies on the pre-training dataset InfMix.
Model
Training Strategy
Epoch
SODA
MFNet-T
SCUT-Seg
Avg FT
MAE-Small
63.36
42.44
60.38
55,39
UNIP-Smal
Sepurate Truining (Stagel: RGB)
100
68.98
50.01
68.79
62.59
UNIP-Sman
Separate Training (Stage2: Infrared)
100
68.49
52.10
69.62
63.40
UNIP-Smal
Joint Training
100
70.99
51.32
70.79
64.37
Table 19:
The fine-tuning performance of using head-wise and layer-wise distillation.
Method
Layer (MAE-L)
Target
Avg NM1
SODA
MPNet.T
SCUT--eg
Avg FT
Layer-wise
18
All 16 Heads
0,1185
70.99
51.32
70.79
64.37
18
5 (b), 6 (h), (h), 10 (b), 13 (b), 15 (h)
0.0985
70.37
52.01
71.82
64.73
Head-wise
18
(e). 4 (g), (), ()) (4
0,1049
68.75
50.89
70.43
63.36
18
(g). 4 (g). 8 (b). 9 ()) 1b), 15 (b)
0,1077
70.07
51.65
69.76
63.83
Layer-wise
24
All 16 Heads 3 (h), 4 (b), 6 (b), 12(h), 15 (h), 16(b)
0,1882
67.74
50.39
69.00
62.38
Head-wise
24
0.1092
69,95
51.82
69.88
63,88
Impact of the Size of the Pre-training Dataset. Tab. 15 illustrates the fine-tuning performance with varying ratios of the InfMix dataset. clear data scaling law is observed, where the performance consistently improves as the pre-training dataset size increases. This demonstrates the necessity of constructing the InfMix dataset, much larger dataset than other infrared pre-training datasets like MSIP (Zhang et al., 2023) and Inf30 (Liu et al., 2024a). As we continue to expand the InfMix dataset, we can anticipate even greater advancements in model performance, potentially enabling breakthroughs in applications that rely on infrared data, such as autonomous driving (Xiong et al., 2021), and surveillance (Bondi et al., 2020).
Impact of the Training Strategy on the Pre-training Dataset. In Tab. 18, we conduct experi- ments involving a two-stage training process using MAE-L as the teacher model. In the first stage, the model is distilled using the RGB component of InfMix. In the second stage, the model is subse- quently distilled using the infrared component of InfMix. As indicated in Tab. 18, benefiting from the hybrid pattern distillation, the model of the RGB training stage surpasses MAE-S by a large margin. After the infrared training stage, the model's ' average segmentation performance improves further. However, we observe slight decline in performance on the SODA dataset. We attribute this to the problem of data distribution mismatch. Notably, half of the images in SODA depict indoor scenes, which are searce in our infrared pre-training dataset InfPre In contrast, such scenes are more prevalent in the ImageNet and COCO datasets. We believe this discrepancy also accounts for the inferior performance of the two-stage training compared to joint training. Joint training benefits from wider data distribution, which contributes to improved generalization performance.
Multi-layer Distillation. Tab. 16, we examine the use of attention maps from multiple layers of the teacher model for distillation. Interestingly, performance declines as more layers included. We hypothesize that requiring a single student layer to mimic multiple teacher layers' attention maps introduces excessive complexity and noise, which impedes the distillation process. An adaptive selection of attention maps to minimize noise and redundancy could be promising direction.
Head-wise Distillation. In Tab. 19, we explore the head- wise distillation, a more fine fine-grained dis- tillation method. Compared to the layer-wise distillation in NMI-HAD, it directly utilizes different heads for distillation. The experimental setup involves using MAE-L (16 attention heads for each layer) as the teacher to distill UNIP-S (6 attention heads for each layer). First, we calculate the NMI for attention maps of each attention head in MAE-L and observe that not all attention heads within the same layer exhibit the same attention pattern. Therefore, we categorize these attention heads into three patterns: local (I), hybrid (h), and global (g). We then select six attention heads (the total number of heads in UNIP-S) as distillation targets. For the 18th layer of MAE-L, there are 5 global heads, 4 local heads, and 7 hybrid heads. We experiment with three different combi- nations: one containing only hybrid patterns (row 2), one containing only local and global patterns (row 3), and one containing all three patterns (row 4). The average NMI values for these combina- tions comparable. Notably, the combination containing only hybrid attention patterns achieves the best performance, demonstrating the effectiveness of hybrid attention patterns even in head- wise distillation. Furthermore, using just 6 hybrid attention heads for distillation even surpasses the per-
23

---

Published as a conference paper at ICLR 2025
Table 20: Details of the InfPre dataset. #lmage and #Extraced image represent the number of original and extracted images from the dataset. Interval and Similarity denote the fixed-interval and similarity-based sampling methods, respectively. The value after the slash indicates the fixed interval or similarity threshold.
Task
Sccuario
#Image
#Extracted Image
Average Width
Average Height
Sampling
RGBT (Liu et al., 2021a)
Crowd Counting
Urban
2.030
2,030
636
484
KAIST (Hwang al., 2015)
Object Detection
Driving
95.328
9,546
640
512
Interval 10 Interval 10
Infrared City al,, 2022)
Video Translation
Driving
200,000+
20.187
256
256
CVC-09 (exe. 2016b)
Object Detection
Driving
13.184
13,184
640
480
CVC-14 (eve, 2016a)
Object Detection
Driving
8518
8518
640
471
VAP (Palmero 2016)
Semantic Segmentation
Indoors
23,080
2.309
640
480
Interval 10
RGBT-234 (Li ot al,,
Object Tracking
Surveillance
117,612
11,762
628
459
Interval 10
LTD (Nikolov et al., 2021)
Concept Drift
Surveillance
26,820.000
15,749
384
288
Similarity / 0.95
Rain (Bahasen Mocslond, 2019)
Semantic Sogmentation
Surveillanee
130,800
25,920
640
480
Interval/
Infrared Security (Liu
2021b)
Object Detection
Surveillance
8,999
8,999
495
386
LLVIP (lia al., 2021)
Object Detection
Surveillance
15,488
15,488
1,280
1,024
LSOTB-TIR (Liu et 2020)
Object Tracking
Diverse
600,000+
61.154
925
623
Interval / 10
Dual-Sensor (Chon at, 2021a)
Driving
73,638
14,728
384
288
Interval
LasHeR (Li et 2022a)
Object Tracking
Diverse
740,000+
74,035
$79
554
Interval 10
VT5000 (Tu al., 2023)
Salicat Object Detection
Diverse
5,000
5,000
640
480
Iafrared Vebicle (Li et 2021b)
Object Detection
Driving
13166
13.166
815
613
Infrared Ship (Li Wang, 2021)
Object Detection
Marine
9,402
9,402
772
591
Drone Vehicle (Sun 2023)
Object Detection
Acrial
28,439
28.439
640
512
Infrared Acrial (Lio ot al., 2021e)
Object Detection
Acrial
11,045
11,045
627
502
VTUAV (Zhang 2022)
Object Tracking
Acrial
1,700,000
166,986
1,920
1,080
Interval 10
M3FD et 2022)
Object Detection
Driving
4,200
4,200
1,00,1
744
OTCBVS IRIS Face (Abidii
Human Face
4.199
4,199
320
240
Multispoctral (Takumi et al., 2017)
Object Detection
Driving
15,042
15,042
480
368
IntPre
Pre-training
Diverse
$41,088
1,075
686
The cosine similarity between pre-training and infrared segmentation datasets. The em- beddings of images are extracted by DINO-B. The similarity is averaged over all pairwise images from different datasets.
Table 21:
Downstream dataset
Pre-training dataset
SODA
MFNet-T
SCUT-Scg
Mcan
Image-Net-IK (Deng et al., 2009)
0.083
0.074
0,081
0.079
COCO (Lin et al., 2014)
0.111
0.101
0.106
0.106
InfMix InfMix (gray)
0.200
0.227
0.236
0.221
0.216
0.246
0.254
0.239
formance of distilling all heads in the 18th layer (row 1). This phenomenon also observed in the 24th layer. This suggests that there may be redundancy in the attention maps within single layer. Therefore, we believe that more fine-grained distillation, such as head-wise wise distillation, is a highly promising research direction.
F PRE-TRAINING DATASET.
F.I THE INFPRE DATASET.
The IniPre dataset is constructed by collecting images from 23 infrared-related visual datasets. The details of the extracted datasets are presented Tab. 20. To reduce the redundancy im- ages with similar backgrounds, we employ two sampling methods: fixed-interval sampling and similarity-based sampling. For datasets containing diverse image sequences with different back- grounds, frames are sampled at fixed intervals (e.g. 2, 5, and 10) within each sequence. For datasets captured in the same location, we only sample frames that are less similar to each other. The cosine similarity of image embeddings extracted by DINO-B is used as the similarity metric. Images with high similarity to those already sampled images will be discarded. The Faiss (Johnson et al., 2021) library is utilized accelerate the sampling process.
24

---

Published as a conference paper at ICLR 2025
F.2 THE INFMIX DATASET.
The InfMix dataset combines the InfPre, the subset of ImageNet-Ik (Deng et al., 2009), and the training set of COCO (Lin al., 2014), totaling 859,375 images. Tab. 21 compares the similarity between various pre-training datasets and three infrared segmentation datasets used in our bench- mark. Notably, compared RGB datasets like ImageNet- Ik and COCO, the mixed dataset exhibits higher similarity with infrared downstream tasks, thereby mitigating the representation shift between pre-training and downstream data. Moreover, converting RGB images to grayscale further enhances this similarity, resulting in better fine-tuning performance, as shown in Tab. 8.
G MORE VISUALIZATIONS.
We present visualizations of distilled models in this section. Fig. 10 shows the attention maps of different supervised and CL methods of various sizes. As shown in Fig. 11, the deep layers of UNIP- S exhibit hybrid patterns, indicating that UNIP effectively transfers these patterns from the teacher to the student. Additionally, compared to MAE-L, attention maps in UNIP-S focus more on shape information than textures, evident in Fig. 12. The emergence of hybrid patterns in deep layers and the reduced bias towards texture both contribute to the excellent performance of UNIP. The attention maps of RGB image inputs are visualized in Fig. 13, exhibiting nearly identical attention pattern distribution with infrared images in Fig. 3.
Layer
Layer
Laver 12
Layer
Layer
Laver 12
Laver
Layer
Laver 12
Quiry
Quiry
Querg
DINO-S
DINO-B
IBOT-S
Layer
Laver
(6 DeiT II-B Figure 10: Visualizations of attention maps supervised and CL models. The attention maps averaged over different heads. All CL and supervised methods share similar attention pattern distribution across layers.
(d) IBOT-B
(c) IBOT-L
(8) DeiT III-I
25

---

Published as a conference paper at ICLR 2025
Layer 5
Loyer
Layer 12
Layer s
Layer 18
Layer 24
Layer
Layer
Layer 12
Query
Quiry 2
Query 3
Local
Loca!
Locul-Hybrii
Local
Hybrid
Local
Local
Hyyrid
Hybrid
(8) MAE-S
(b) MAE-L
UNIP-S
Figure 11: Visualizations layerwise attention maps in MAE and UNIP-S distilled from MAE-L. The hybrid patterns emerge in the later layers of UNIP-S but in the middle layers of MAE-L.
Query
Query
Quiry
Query
Qwiry
Quiry
MAS-L
Query
Quary
3 Query
Query Query
Figure 12: Visualizations attention maps in MAE and UNIP distilled from MAE-L. Attention maps from the 12th layer of MAE-S, the 18th layer of MAE-L, and the 12th layer of UNIP-S displayed, respectively. Compared to MAE-S and MAE-L, UNIP-S exhibits reduced texture bias, emphasizing shape information textures.
26

---

Published as a conference paper at ICLR 2025
Layer 5
Layer
Layer 12
Layer
Layer
Layer 12
Layer
Layer
Layer 24
Query 1
Quiry
Quiry
Local
Hybrid DINO-S
Global
Local
Local
Local
Local
Hybrid
Local
(b)
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Quiey 1
Quirry
Quefy
Local
Hyybid
Local
Hybhrd
DINO-S
(b) MAE-S
MAE-L
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Quaryy
Quiry 2
Query
Local
Hybrid
Global
Local
Local
Local
Local
Hybrid
Local
DINO-S
(b) MAESS
MAE-L
Figure 13: Attention maps of RGB image inputs for different query tokens in three representative layers. Each query token's attention map corresponds to row in the attention matrix, averaged over different heads. Images are from ImageNet (Deng et al., 2009).
27


Reviews:
[{"review_id": "ZBRBFY0I7w", "reviewer": "Reviewer_ctU1", "summary": "This paper focuses on the pre-trained attention patterns for semantic segmentation in the infrared domain. It first concludes from the performance of different pre-training methods in infrared segmentation that the pretraining task is the key factor. Then, it takes deeper research on the attention pattern and states the importance of the hybrid attention pattern. Based on this insight, this paper used NMI to indicate the attention pattern and proposed a new pre-training framework named NMI-HAD to distill the layers with hybrid attention patterns. It also proposes LL-FPN and has improved fine-tuning performance", "questions": "Since the proposed NMI-HAD is layer-wise, I’m curious about the performance when the distillation is more fine-grained. For example, why not apply the distillation on different heads? I may suggest the author to conduct an experiment comparing the performance between layer-wise approach and head-wise approach, which can have deeper insight on the choice of distillation strategy.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe writing is easy to follow.\n2.\tThe ideas proposed by the author are well supported by the detailed experimental results.\n3.\tThe pre-training method is important in semantic segmentation.", "weaknesses": "1. Although the analysis of attention patterns is persuasive, more comparison with the attention patterns in other RGB semantic segmentation datasets, for example, ADE20k, is necessary. Since the idea of enhancing the segmentation performance with the help of both local and global attention (e.g. Swin Transformer) is not new, it is hard to explain the reason why the difference in attention patterns is more important in infrared semantic segmentation. The difference on attention pattern and their impact between RGB dataset and infrared dataset needs to be clarified.\n2. As it is mentioned in line 475 “Excessive constraints on features may intensify the distillation shift and hinder the generalization of distilled models“, an ablation for distillation that combines the features and attention is needed. The comparison between its performance against each method individually can support the claim about excessive constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVpkSJTyga", "reviewer": "Reviewer_FdMz", "summary": "This paper leverages strong pre-trained models to boost the performance of infrared semantic segmentation. The authors identify the best attention patterns and propose the hybrid-attention distillation to improve the performance and efficiency at the same time. A large-scale mixed dataset is also presented for pre-training. The proposed methods achieve the new state-of-the-art results.", "questions": "For NMI-HAD, is only the attention map of a single layer used for distillation?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The benchmark of vision foundation models on infrared segmentation datasets and the introduction of NMI to identity attention patterns for distillation contribute to the community. \n2. Extensive and comprehensive experiments are conducted and the proposed approach significantly surpasses state-of-the-art infrared or RGB segmentation methods.", "weaknesses": "1. The underlying motivation is not very clear. In my opinion, the authors sometimes aim to emphasize the performance in infrared semantic segmentation and sometimes aim to emphasize the versatility of the proposed method. However, on the one hand, the proposed approach is not specific to infrared semantic segmentation tasks. On the other hand, from the viewpoint of visual pre-training and downstream transfer learning, there is a lack of experiments on more general datasets and performance comparisons with related works. Compared to previous works in knowledge distillation, the biggest contribution may be to provide a method to choose which layer to distill. This is the main concern for me. \n\n2. It is strange to distinguish between the last two attention patterns by calling them “local-global” and “global”. The “global” attention patterns are just attention collapse as the authors stated. Therefore, if only considering a single distillation target, the last layer may not be the best choice. I believe previous works have suggested this. If using multiple hierarchical features which is more common, the conclusion may be different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAPuvTWH0y", "reviewer": "Reviewer_nH3J", "summary": "This paper presents a deep analysis on why RGB pre-trained models performs worse on thermal images. Based on the analysis, the authors present a NMI-guided distillation for improved segmentation. Experiments on different datasets are performed.", "questions": "Please answer the title, claim, and experiment questions in weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper presents a deep analysis on benchmarking the infrared semantic segmentation performance of various pre-training models.\n- The experimental results are extensive, and the writings are good.", "weaknesses": "- The title of the paper is inappropriate as it misleadingly suggests a general semantic segmentation task. In fact, this paper mainly focuses on infrared segmentation of small models by adapting pre-trained models.\n- According to the claims of the authors, the MIM (Masked Image Modeling) strategy focuses more on texture information and is not suitable for segmenting infrared images. Why using an MAE-based model as the teacher for distillation can achieve better results?\n- This paper seems no comparison with existing distillation strategies used in domain-adaptive methods. If possible, I think it is better to give a comparison.\n- More experiments on built datasets can be given to demonstrate the effectiveness of proposed method. For example, we first train on RGB images and then fine-tune on infrared image, or jointly train on both dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YJLJAx35qq", "reviewer": "Reviewer_gRqm", "summary": "This work first deeply analyzes the transfer performance on infrared tasks of popular pre-training regimes, including supervised training, contrastive learning and masked image modeling. Based on three valuable observations, this work proposes a pre-training and finetuning framework for infrared image segmentation, dubbed UNIP—specifically, an NMI-HAD distillation algorithm, a mixed large-scale InfMix dataset, and an LL-FPN architecture. Complete experiments and analysis demonstrate the effectiveness of the proposed algorithm.", "questions": "Please refer to the WEAKNESSES part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work is well-motivated, clearly written and very convincing. All observations and hypotheses are proved by empirical or numerical evidence, which makes the whole work complete and sound.", "weaknesses": "Overall, I feel pretty OK with this work, and there are no **MAJOR** weaknesses. Some minor suggestions are as follows for the author's reference.\n\n- As this work focuses only on infrared images, the word infrared should appear in the title to make it sound and avoid misunderstanding, *e.g.*, for infrared semantic segmentation.\n- Some abbreviations should be given their corresponding specific names and appropriate citations if necessary when they first appear, *e.g.*, SSL (semi-supervised learning) in Sec. 2.1.\n- Unalighed letters of ViT in Fig. 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)