InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective
Authors: yuanhong zhang
Muyao Yuan
Weizhan Zhang
Tieliang Gong
Wen Wen
Jiangyong Ying
Weijie Shi
Venue: ICML 2025 spotlightposter (2025)
Keywords: Segment Anything Model
Parameter-efficient fine-tuning
Mutual information
Knowledge distillation
Primary area: applications->computer_vision
TL;DR: InfoSAM improves SAM's performance on specialized tasks by using an information-theoretic approach to distill and preserve domain-invariant knowledge during fine-tuning.

Abstract: The Segment Anything Model (SAM), a vision foundation model, exhibits impressive zero-shot capabilities in general tasks but struggles in specialized domains. Parameter-efficient fine-tuning (PEFT) is a promising approach to unleash the potential of SAM in novel scenarios. However, existing PEFT methods for SAM neglect the domain-invariant relations encoded in the pre-trained model. To bridge this gap, we propose InfoSAM, an information-theoretic approach that enhances SAM fine-tuning by distilling and preserving its pre-trained segmentation knowledge. Specifically, we formulate the knowledge transfer process as two novel mutual information-based objectives: (i) to compress the domain-invariant relation extracted from pre-trained SAM, excluding pseudo-invariant information as possible, and (ii) to maximize mutual information between the relational knowledge learned by the teacher (pre-trained SAM) and the student (fine-tuned model).  The proposed InfoSAM establishes a robust distillation framework for PEFT of SAM. Extensive experiments across diverse benchmarks validate InfoSAM's effectiveness in improving SAM family's performance on real-world tasks, demonstrating its adaptability and superiority in handling specialized scenarios. The code and models are available at https://muyaoyuan.github.io/InfoSAM_Page.

InfoSAM: Fine-Tuning the Segment Anything Model from An Information -hhooreti Perspective
Yuanhong Zhang" 12
Muyao Yuan 12
Weizhan Zhang" 12
Tieliang Gong 13
Wen Wen
Jiangyong Ying" 4
Weijie Shi
Abstract
Tradtional PFET for
Tédillation
The Segment Anything Model (SAM), a vision foundation model, exhibits impressive zero-shot capabilities in general tasks but struggles in spe- cialized domains. Parameter-efficient fine-tuning (PEFT) is a promising approach to unleash the potential of SAM novel scenarios. However, existing PEFT methods for SAM neglect the domain-invariant relations encoded in the pre- trained model. To bridge this gap, we propose InfoSAM, an information-then theoretic approach that enhances SAM fine-tuning by distilling and pre- serving its pre-trained segmentation knowledge. Specifically, we formulate the knowledge transfer process as two novel mutual information based objectives: (i) to compress the domain-invariann relation extracted from pre-trained SAM, exclud- ing ascudo-invariant information as possible, and (ii) to maximize mutual information between the relational knowledge Iearned by the teacher (pre- trained SAM) and the student (fine-tuned tuned model). The proposed InfoSAM establishes robust dis- tillation framework for PEFT of SAM. Extensive experiments across diverse benchmarks validate InfoSAM's effectiveness in improving SAM fam- ily's performance on real-world tasks, demonstrat- ing its adaptability and superiority in handling specialized scenarios. The code and models are available InfoSAM project page.
Not Consider Relation
Relation Transfer
Image
Malk
Luugo
Przzeler
Deraker
Fronder
Teacher
Rdiioo
Acpord
Toernyl
Information
Infermdion- Thraredi: Destiiation
@d Teadifoonl Desilladion Mithoo for SAM
No Relation Guidence
Preneeer
Derodeer
Decodor
Desilldia
Lozadic Adptic
Pull Thasia
Prenedrr
Derocler
Pitned
/entereettttrr Petture
Figure 1: Comparing traditional PEFT and distillation paradigms with our proposed InfoSAM. (a) Existing PEFT methods for SAM directly adjust the trainable parameters of each module individually, often overlooking the cross- module relationships.
(b) Traditional SAM distillation meth- ods focus on model compression via paired feature align- ment but lack relational guidance during projection training, (c) In contrast, our InfoSAM method enhances PEFT by in- corporating information- -heorreti distillation, enabling the transfer of domain-invarte relations from the pre-trained SAM to the fine-tuning student.
1. Introduction
Recently, the Segment Anything Model (SAM) (Kirillov et al., 2023; Ravi et al., 2022) emerged a prominent foundation model for image segmentation. While SAM demonstrates exceptional zero-shot performance on generic object segmentation, it often struggles with domain-specifte real-world segmentation tasks (Chen et al., 2023; Zhong et al., 2024). Therefore, Carameter-Efficiee Fine-Tuning (PEFT) for SAM (Song al., 2024; Zhang et al., 2024a; Peng et al., 2024b) has gained attention as a promising solution, significantly reducing the fine-tun fine-tuning costs associ- ated with SAM's large pre-trained parameter set. Existing PEFT methods for SAM primarily focus on fine-tuning the heavy image encoder (Song et al., 2024; Peng et al., 2024b) or aligning domain- -spucffic features between the mask de- coder and image encoder (Xiao al., 2025). However, a
Equal contribution 'School of Computer Science and Technol- ogy, Xi' an Jiaotong University, Xi' an, China Ministry of Educa- tion Key Laboratory Intelligent Networks and Network Security, an'an Jiaotong University. Xi' an. China "Shaanxi Province Key Laboratory Big Data Knowledge Engineering. Xi' an Jiaotong University, Xi' an. China *China Telecom E-surfing Vision Tech- nology Co., Ltd, Hangzhou, China School of Electrical Engineer- ing, Xi' an Jiaotong University, Xi'an, China. Correspondence to: "Weizhan Zhang <zhangwzh @ xjtu.edu.en>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoredie Perspective
promising improvement avenue is overlooked: preserving the beneficial information in pre-trained models.
SAM by maximizing the mutual information between their extracted relations. This ensures faithful propagation of compressed semantic dependencies, thereby facilitating more effective fine-tuning process. Our experiments on SAM and SAM2, evaluated across diverse domains and 8 datasets, show that InfoSAM achieves superior adaptation and segmentation performance.
Notably, SAM follows an encoder-decoder architecture, where the mask decoder refines the image embeddings ex- tracted by the image encoder to localize objects. Unified training or fine-tuning methods (Shu et al., 2025; Xiao et al., 2025) have demonstrated effectiveness within this frame- work. This suggests that preserving the implicit relationship between the encoder and decoder could be beneficial for model fine-tuning. This relationship may stem from exten- sive pre-training and be embedded in the feature distribu- tions, making it delicate and easily disrupted by unrefined PEFT methods (Wang et al., 2024). We argue that this is because task-specific tuning tends to override or suppress the universal visual features learned during pre-training.
Overall, our contribution can be summarized as follows:
We present InfoSAM, the first -heoreicion-theoreiicc framework for SAM adaptation, introducing an inno- vative distillation approach tailored for SAM PEFT to enhance performance in new scenarios.
InfoSAM proposes novel dual complementary mecha- nisms for SAM adaptation: relational bottleneck that strategically compresses task-irrelevant dependencies while preserving domain-invariant semantics, coupled with adaptive cross- moosl mutual information maxi- mization ensuring provable preservation of essential structural knowledge.
To enhance PEFT by leveraging implicit relationships, nat- ural approach is to extract these relationships from founda- tion models and inject them into fine-tuned models tailored for specific domains. However, not all implicit relationships are beneficial for downstream tasks-oni the key domain- invariant relationships learned from across domains (Hoff- man et al., 2018; Xu et al., 2022) contribute positively to every fine-tuned model. While knowledge distillation serves as a flexible bridge for transferring information between models (Gou et al., 2021), we propose to adopt a distilla- tion approach between the pre-trained model and fine-tuned model to retain -nnariinnn relationships.
We conduct comprehensive benchmark across diverse domains, including natural images, medical imaging, agriculture, and remote sensing. InfoSAM AM consistently demonstrates superior performance compared to other PEFT and distillation techniques across various down- stream tasks.
Therefore, this brings us to two key challenges: 1) How can we extract the domain-invariint relationship from pre- trained foundation models? 2) How can we effectively trans- fer the extracted information to fine-tuned models?
2. Related Work
2.1. Parameter Efficient Fine-Tuning for SAM
To address these challenges, we propose InfoSAM, a novel cerrmation-thoorettcn distillation method specifically de- signed for SAM PEFT. In order for the teacher to provide a good amount of the cemainninvarrian have to find out how this information could be quantified. To this aim, we introduce robust and efficient Rényi's entropy- based quantification from information theory (Ahn et al., 2019) to measure such a relation. However, not all the rela- tions in the pre-trained SAM are domain-invariant, there ex- ists some aseudo-invariane information (e.g., color), which may negatively impact the generalization ability during the fine-tuning process (Li et al., 2022a). Therefore, to address the first challenge, we propose an iideetiiondriveer relation module specifically designed to extract critical structural patterns from the pre-trained SAM. By minimizing mutual information between the module's ' outputs and both encoder- decoder embeddings of SAM, constructs an effective bot- tleneck that forces the module to maintain compressed yet domain-invariant representations. Furthermore, to tackle the second challenge, we effectively distill the valuable re- lational knowledge from pre-trained SAM to the fine-tuned
Parameter-Efficciet Fine-Tuning (PEFT) alleviates the chal- lenges of task-specific deployment in large foundation mod- els by fine-tuning only a minimal subset of parameters while keeping the majority frozen. Several prior works explore fine-tuning SAM for downstream tasks. SAM- Adapter (Chen et al., 2023) is one of the pioneering works applying the PEFT method to SAM, incorporating task- specific prompts for each adapter. SU-SAM (Song al., 2024) presents simple framework to efficiently fine-tune the SAM with Adapter or LoRA. SAM-COBOT (Peng et al., 2024b) boosts existing PEFT techniques for fine-tuning SAM through cross-block orchestration. BLO-SAM (Zhang et al., 2024a) finetunes SAM based on bi-level optimiza- tion, eliminating the need for manual prompts by a learn- able prompt embedding. Conv-LoRA (Zhong et al., 2024) integrates altra-lightweight convolutional parameters into LoRA, injecting image-related inductive biases into the plain ViT encoder.
information, first we
However, the above methods overlook preserving pre- trained information in foundation models during fine-tuning, Our work explores enhancing fine-tuning methods for SAM
2

---

InfoSAM: Fine- Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
Optimization Objective
Image Eacoder
Prompr Encuder
max 1,(r
of zm
rT
Lr
Mask Decoder
X-D
Imagee
Relation Modile f
Relation Information
la (z. zhir
Compross Domain-invariant invariant Information
Bottleneck
Distillation
Relation Module /
Image
golten dic
Relation Information
r?
r
La
Mask Decoder
Pm
Full Tmining
Distill Domain-invariant
Prompt
A
la
Image Frcoder
Enoocee
Frozen Parameters
Information
Figure 2: The Flowchart of InfoSAM. To leverage the domain-invarian relationships within modules from a well-trained foundation model (i.e., SAM) for enhancing PEFT. InfoSAM pioneers an information -hheoreticc framework for parameter- efficient SAM adaptation through two synergistic components: 1) Strategic compression of task- insk-irrelevant dependencies while preserving domain-invariant feature relationships through optimized interaction between image embeddings z!" and mask tokens = (Eq.5), and 2) Cross-model mutual information maximization to ensure faithful knowledge transfer (Eq.6). The right Venn diagrams illustrate the information constraint from the optimization problem.
from a novel perspective: leveraging information- based dis- tillation maintain domain-invariant relationships.
methods like feature disentanglement (Chang et al., 2019) or analysis (Xu et al., 2022) decompose images into domain- invariant (e.g., shapes, edges) and domain-specifts (e.g.. textures, colors) components, aiming to enhance the for- mer while suppressing the latter; (2) GAN-based domain- invariant feature generation, which employs adversarial training to align domains at different levels: image (Li et al., 2022b), feature (Ma al., 2024), and output (Huang et al., 2022). For example, GLGAN (Ma al., 2024) in- tegrates multi-scale global and local features improve cross-domain transferability in remote sensing.
2.2. Knowledge Distillation for SAM
Knowledge distillation (KD) (Gou et al., 2021) effec- tively transfers knowledge from large, well-trained model (teacher) to smaller or simpler one (student). When ap- plying KD to SAM, most efforts focus on compressing and transferring representations for downstream tasks. Mobile- SAM (Zhang et al., 2023a) distills SAM's VIT encoder into a Tiny ViT, while TAM (Shu et al., 2025) uses full-stage KD. Other approaches distill SAM's semantic priors for tasks like medical segmentation (Dong et al., 2024; Shen et al., 2024) and image restoration (Zhang al., 2024b). However, these methods focus paired feature maps, glecting the mnter-module relationships within the teacher SAM. To address this, our approach utilizes information- theoretic principles to extract and transfer compact inter- module relationships the student model.
SAM's large-scale pretraining encodes domain-invariant patterns for strong zero-shot generalization. Recent works leverage these universal visual patterns for downstream tasks (Peng al., 2024a). However, these methods rely complex designs or external data to learn representations. In contrast, focus preserving the domain-invariant information in pre-trained SAM for fine-tuning.
3. Preliminaries
2.3. Nomain-invariant Information in SAM
3.1. Rényi's c-entropy and Mutual Information
The concept of fomain-invarian information was first in- troduced in prior works on domain adaptive segmentation (DAS), which explored cross-domain invariant features such edge and structural information (Hoffman al., 2018). DAS aims to learn domain-invariant invariant representations across multiple domains and follows two main approaches: (i) ex- traction and refinement of domain-invarian features,
In information theory, matrix- Rényi's a-entropy pro- vides novel way quantify single-variable information or interactions across variables directly from samples. Un- like Shannon entropy, leverages the eigenspectrum of Gram matrix in reproducing kernel Hilbert space (RKHS). avoiding costly distribution evaluations (Gong et al., 2022).
3

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Informatton--heerrti Perspective
Definition 1. Let X' X' R be infinitely divisible positive kernel (Bhatia, 2006). Given {ir } C X,, each r; being real-valued scalar or vector, and the Gram matrix K obtained from Kij = x(z;,z)) to Rényi's entropy can be defined as:
To enhance the PEFT process using the frozen pre-trained teacher SAM, the loss can be formulated as:
w arg min L(X, Y T),
(4)
maurix-basedd analog
where T represents intermediate features extracted by the teacher model, capturing the relational information within SAM modules, and Y is the full dense label map. Following prior work, the task-specific loss function L()) is chosen as the structure loss (Zhong et al., 2024).
(A)
log2
Σ (A)
(I)
1 a
where the kernel matrix Aij 1
K3y
is the normalized
n. v K0
In this paper, rather than directly aligning paired representa- tions between teacher and student (Zhang et al., 2023a; Shu et al., 2025), we leverage robust prior relational information from the pre-trained SAM to guide the PEFT process.
version of K and Er(A) = 1. The (A) denotes the ith eigenvalue of A.
Definition 2. Given n pairs of samples {2; (IIa; 35)) jimi and two positive definite kernels K1 X x X + R and K2: : ) x y ~ R. After computing the Gram matrix A and B. joint Rényi's entropy can be defined as:
4.2. An Information View of SAM Distillation
Problem Formulation. Intuitively, the relationships be- tween different modules in well-trained foundation model are invaluable, as they are learned from extensive datasets. However, traditional PEFT methods for SAM, when fine- tuned to downstream tasks, risk disrupting these relation- ships. In this way, we need to address two key questions: how to capture critical relations from the pre-trained SAM and how to effectively transfer it to the fine-tuned model.
Ao B ir(A o B)
S. SS.
S.
where (A B) denotes the Hadamard product between the matrices A and B. The mutual information I, (A(A; B) can be computed as:
= (A) + SS, B)
Firstly, by treating the teacher model as a mapping function, we argue that the critical relation information resides in multivariate mutual information Ia (2] 2T''''' r""), where 27- zy and nf represent the image embedding, mask token and relational interactions of the teacher, respectively. As such, Ia (2,, _T )f") quantifies how much information of can tell about (z,"" =m7). Crucially, this mutual information con- stitutes a learnable bottleneck that fundamentally constrains the knowledge transfer process. The bottleneck mechanism enforces selective attention by restricting the information flow to compressed representation, where only the most salient teacher-student interactions can be preserved. This is particularly vital as not all relations in the pre-trained SAM are universally transferable. For example, invariant features (e.g., geometric outlines) that exhibit cross-domain consistency are effective, while pseudo-ing pseudo-invariant features (e.g., color distributions) that carry domain-speciff biases need to be suppressed (Li et al., 2022a).
The matrix-barn based Rényi's 's mutual information eliminates the need for high-dimensional probability density estimation of Shannon entropy, offering a more accurate and computa- tionally efficient solution (Dong et al., 2023).
4. Methodology
4.1. Background and Notions
The overview of InfoSAM is illustrated in Fig. 2. Given a teacher model and a student model, we denote the pre- trained SAM as of and the fine-tuned SAM as Φ5, which are parameterized by w. Let X ~ D be an input sampled from the downstream dataset D. The representations pro- duced by o² dx( and of (X) are defined as follows: The output features of the image encoder are denoted as z! and =fS, where zf' = E BBXHXWXD, Here, B is the batch size, H and W represent the height and width, respec- tively, and D is the dimension of the image embeddings. Similarly, the output tokens from the two-way transformer in the mask decoder are denoted as = and = where 2TT =m E BBXNDD These tokens encode the target mask information in more abstract manner. Here, N represents the number of masks, and the output token shares the same dimension D as the image embeddings.
To prioritize Nomain-invariaae relations, we constrain the information flow via an upper bound I,:
In =Ts r7) S Ie
(5)
where T. fT :):0) represents the teacher's rela- tional mapping between image embeddings and mask to- kens. The relation module is defined as [T( 2h: 0) R xx xW xD x R BxXxD + xBxNx(H-W)) x(H-W) The 0 repre-
The goal of PEFT is to fine-tune x0((X (X; for adaptation to a new downstream task under the supervision of the teacher model o", where w denotes the trainable PEFT parameters.
sents the parameters of the relation module. This compres- sion forces the module to retain only essential information for distillation.
4

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Informatiio-Theorellt Perspective
After that, we employ a distillation approach transfer the
core relationships by maximizing their mutual information:
Coor, M,
7 Image festure
max Ia [a(r";r
Zm Mask tokan
(Asz, N
(6)
subject to La 2m: ( se
Scale
(baz, NNA X()
where S = (2 =m:00 0) denotes the student's relation in fine-tuned SAM, with f sharing the same parame- ters f². The information bottleneck principle operates through two coupled mechanisms: (i) compression via mini- mizing Lo (2,", > 24"::" r²') to extract minimal sufficient statis- ties r" from (2) 2m), and (ii) distillation via maximizing [=(r""; to preserve maximal predictive information. The Lagrangian formulation explicitly implements this trade-off:
Relation Modile
MatMul
(hos,,
N.D)
Key Proj.
Query Proj.
(ben, D)
N.D)
Laye Norm
Layer Norm
MatMul
(boz, Z1
N..,,N.)
Zm
21
2m
Figure 3: The architecture of iitention-basedd relation mod- ule. It is designed to capture the relationship between image encoder and mask decoder, facilitating effective interaction between these components in SAM.
max La (r 3I. (2g 2To r²)
(7)
where 3 is a hyper-parameter for trade-off.
Compressing Intra-SAM Relations. To efficiently capture the relationship within pre-trained SAM, we propose an attention-based module designed for extraction, illustrated in Fig. 3. Given z!", the output of the image encoder of the input
output of Tf with a polynomial kernel of degree 1. No- tably, the teacher entropy term in this loss is excluded, as the teacher's weights remain fixed during PEFT.
SAM, and =m the mask token embedding, of relation module fr. It mainly uses a combination of attention mechanisms and residual connections.
According to (1), computing eigenvalues of large ma- trices is computationally intensive (Kerr et al., 2009; Yu et al., 2019). To mitigate this, we set a = 2, allowing us to compute matrix- based Rényi's a-entropy via the Frobenius norm: A II = (rA #)= = Σ (A). Consequently, Ly can be reformulated as:
First, both zf and z are passed through Layer Normal- ization step stabilize the features. After that, 2if and 2! are linearly projected into query vector Q E Bx and key vector K E (B respectively:
C, = - logg IIG, + log, |GT
(11)
Q = Wq- LayerNNom(z., (2.))
K WK LayerNorm(
where GT = GT GT o G.. The o is Hadamard prod- uct. The first term in L, acts as a spectral compression regularizer that constrains the relation module and encour- ages it to learn more compact and refined representations. The second term minimizes the joint entropy of the fea- ture interactions across the image encoder, mask decoder, and relation module, effectively filtering spurious relation- ships and preserving comainninvariant interactions critical for cross-domain adaptation.
where Wo, WK E DxD are learnable projection matri- ces. The attention scores are computed by combining two components: the scaled dot product of Q and K and the residuals from the dot product of = and =?- The scores are summed to produce the final attention map:
QK VD
S.
24
Maximizing Inter-SAM Relations. After extracting the essential relationships between the image encoder and the mask decoder, we transfer the relationships by minimizing their distance. A natural choice to accomplish this is by maximizing the mutual information between the two rep- resentations. While most cxisting works (Ahn et al., 2019; Kuang et al., 2023) focus on minimizing a lower bound of mutual information, we directly maximize the matrix-based Rényi's mutual information itself to avoid the expensive evaluation of underlying distribution for distillation loss:
where S. is the attention score. To ensure consistency and comparability, a is flattened and normalized using fg- normalization, resulting in the final output of fT, denoted as r". To encourage the relation encoding process to focus on domain-invariant information, the first loss for relation compression can be expressed as:
C. ; rT
(10)
-Sa(CT (G.) - S. GT G.))
where GT, GT Gf E NNN are the Gram matrices in- duced by a batch of normalized features 2ff an, and the
La = - Ir - - (G,) + (G,
(12)
5

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
Similarly, G and GP denote the Gram matrices correspond- ing to the student and teacher relations, respectively. We denote the G G G. then the distillation loss can be expressed as:
utilizing a CosaneAnnealing scheduler that decays final learning rate of 2 x 10-5. All the methods are trained for 10 epochs with structure loss (i.e., the combination of weighted IoU loss and binary cross entropy loss) unless otherwise specified. During training, prompts are randomly selected from noised ground truth boxes and points at a 1:1 ratio. During evaluation, ground truth boxes are used as the default geometric input prompts to ensure a fair comparison and minimize randomness. More implementation details are provided in Appendix B.
La = logg I 1|: + logg | log2 ||G I1? (13)
Consistent with fr, the La also sets the entropy order a to 2 and utilizes the Frobenius norm for equivalent transfor- mation. From this perspective, the components of La can be viewed as regularization terms (i.e., the first two terms) and a relation alignment (i.e., the third term) between two models, while log, improving robustness to relations.
Datasets. In the natural image domain, we focus on cam- ouflaged object segmentation (Skurowski et al., 2018; Le et al., 2019; Fan et al., 2020a). For medical imaging, we in- vestigate polyp segmentation (Bernal et al., 2015; Jha et al., 2020) and skin lesion segmentation (Codella al., 2018). In agriculture and remote sensing, we use leaf disease segmen- tation (Rath, 2023) and road segmentation datasets (Mnih, 2013) as representative examples, respectively. For further details on the tasks and datasets, please refer to Appendix C.
Overall, combining (11) and (13), the final objective of relation compression and transfer can be defined as:
Linfo = A1 * fr + X2 * La
(14)
where a and '2 are hyyer-parameete to trade-off between sufficiency (domain-invariant invariant information transmitted from rT to r and minimality (the complexity of r²)) Further details and PyTorch-style pseudocode for InfoSAM are pro- vided in Appendix A.2 and A.3.
To verify the effectiveness of our approach, we compare it with two categories of methods: PEFT methods and distilla- tion methods.
PEFT Baselines. The PEFT baselines encompass three types of methods: the direct application of SAM, PEFT methods from the NLP or CV domain, and PEFT meth- ods designed for SAM. These are as follows: 1) The zero- shot performance of the original SAM. 2) Fine-tune SAM's mask decoder only. 3) BitFit (Ben Zaken et al., 2022), which only fine-tunes bias terms in the pre-trained model. 4) AdaptFormer (Chen et al., 2022), which inserts the train- able bottleneck layers into the MLP block of the transformer. 5) LoRA (Hu et al., 2022) inserts trainable bottleneck lay- ers parallel to the frozen linear weight. 6) HQSAM (Ke et al., 2024), which introduces a learnable high-quality output token and enhances mask details by fusing mask decoder features with both early and final ViT features. 7) SU-SAM (Song et al., 2024) presents simple frame- work that efficiently fine-tunes the SAM using Adapter or LoRA. 8) ConvLORA-SAM (Zhong et al., 2024) injects image-related inductive biases into the image encoder of SAM by integrating litht-iighteiight convolutional param- eters into LoRA.
4.3. Applying information theory to SAM
Overall Loss Function. Following previous works (Zhong et al., 2024), we incorporate the proposed information- theoretic distillation loss Lanfo with a structure loss fee (Fan et al., 2020b), which combines the weighted IoU loss and binary cross-entropy loss. The overall loss function is derived as:
L= Loe + Linfo
(15)
Finally, we employ this new loss function for fine-tuning SAM. During fine-tuning, we first learn robust relations and then transfer this knowledge. The Linfo regulates infor- mation flow between SAM's hierarchical representations, avoiding over-retention of low-level details while enhancing geometrically critical features. This aligns with the rate- distortion tradeoff in information bottleneck theory (Tishby & Zaslavsky, 2015), where information is compressed and then generalized.
5. Experiments
Distillation Baselines. In this study, we compare our method with the following baselines: 1) Logit-based dis- tillation (Zhu et al., 2018). 2) single-layer paired feature distillation (i.e., PKD (Cao et al., 2022), PKT (Passalis et al., 2020)), which uses one-stage feature to distill knowledge, with MobileSAM (Zhang et al., 2023a) belonging to this category. 3) maltiple-layers paired feature distillation (i.e., VID (Ahn et al., 2019), IBD (Kuang et al., 2023)), which utilizes multi-staee information to transfer knowledge, with each layer aligned separately. Similarly, TinySAM (Shu
Settings. We conduct experiments using SAM (Kirillov et al., 2023) (with a ViT-B backbone) and SAM2 (Ravi et al., 2022) (with Hiera-B+ backbone) with Adapter (Chen et al., 2022; Song et al., 2024), and LoRA (Hu et al., 2022) across four real-world domains: medical imaging, natural images, agriculture, and remote sensing. We fine-tune SAM's image encoder by adding adapters or LoRA, while fully training the decoder directly. We use a batch size of 4 and the Adam optimizer with an initial learning rate of 2 x 10-4
6

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoreli Perspective
Table I: Comparison of PEFT methods for SAM across various downstream segmentation tasks, All results are based on the ViT-B backbone. "SAM": without adaptation. "decoder-only". directly fine-tuning the mask decoder of SAM.
NATURAL IMAGES
MEDICAL
AGRICULTURE
REMOTE SENSING
METHOD
CAMO
ISIC 2017 Dice f
Kvasir
Leaf
Road
S. f
Es 1
Fg 1
Jac
So 1
Es f
IoU f
Dice f
JoU f
Dice f
SAM
79.7
88.8 92.7 0.34
61.0 nm 85.9 0.4
71.7
71.4 0:16
47.0g 07.61 017
7.2 47.6g 047
12.9
dccodes-only
84.9 0.38
81.8g 0.33
92.2 0:20
90.9 008
95.2 0.18
55.6 112
68.8g 1.17
64.1 047
BirFit
87.5 0
94.5 ons
85.3 n.s
87.7g n1
93.24 now
92.5 0:12
9m0
69. (9.2.1.0
80.3 o/s
58. 1 om
7 ooe
AdaptFommrr
87.9 0.30
94.8+
86.22 a.
87.6 0.4
93.2+ 0:15
93.3 0.68
031
0.11
84.8 one
61.12
75.5
LORA
87.71
1 0.46
87.8
93.3
93.01
96.62
71.4100
59.0 02
74.01 a.1
Adapter
88.2 0.44
94.8 0.34
86.7 n3e
87.7 02s
93.2 0.16
93.4 0.12
97.
74.4
84.3g 0.28
60.5 n.
75.1 0.00
HQ-SAM
85.1 0.10
92.6 0.10
81.0 nsi
86.3 0.33
92.4- a.19
91.1 0.50
66.2 0:44
77.8g 0.5
54.92 0.
70.6 a.11
SU-SAM
88.3 021
95.01 0.21
86.22 0.9
87.8 0
93.24 0:99
93.8 0.00
97.5 000
74.7
84,5 0.56
60.24 026
74.8
Comel aRA-SAM
87,5+ 0.30
94.5 0.17
85.4+ 041
87.7 0.2
93.24
92.94 0.13
96.6 0.25
71 0.44
82.24 0.37
59.6+ 0.2
74.44 0.20
LoRA+Ours
95.2
85.8 am
88.1 aie
93.5
93.4
96.8
72.2
82.8
59.9 a.0
74.6
Odupter=Ours
88.6 0.19
95.1 0,55
87.1 0.37
88.0 aus
93.4 0:00
94.4 0.12
97.9 oum
75.6 9.27
85.24 0.23
61.4 0.30
75.8. 0.27
Table 2: Comparison of distillation methods for SAM fine-tuning across various domains. "Teacher": SAM without adaptation. "Student": fine-tune SAM's image encoder by adding adapters, while fully training the decoder directly. All compared methods utilize student models with the same adapter-based structure.
NATURAL IMAGES
MEDICAL
AGRICULTURE
REMOTF SENSING
METHOD
CAMO
ISIC 2017
Kvasir
Leaf
Road
S,
Jac 1
Dice
S, f
Eg f
1oU
Dice f
IoU f
Dice f
Teacher
79.7+
88.8 94.8 0.54
79.64
61.0+ 87.74 azs
71.7=004
83.0 0.10
88.8+
37,6; an
47.0+ aus
7.2-
12.9
Student
88.2 0.44
86.7 082
93.2g
93.4 012
97.1 n
74.4g n.16
84.3 ozs
60.5: a10
75.1 0.0
Logit
88.44
94.9 ons
87.1 ozz
87.2+
92.9==020
93.2 0.19
96.5+
73.0 ass
83.3 oze
50.9+ oos
67.2=
PKD
87.0+
94.1 oz3
84.34 os7
86.5+
0.7
92.2 025
96.0+
70,2
81.1 10s
56.94 861
72.2-
PKT
87.8+ 0.40
94.5+ 0.5
86.2+ 046
87.4+ a.12
93.0--mm
93.7+ 0.1
97.3 0.5
74,2+ asi
84.2+ o.2
60.7+ 920
75.2- 0.16
IBD
85.24 0,7
92.6 0.s
82.44 o.1
85.1 0.74
91.5 91.
95.3 no0
72.2 a12
82.7 o07
44.9+ h.s
61.5
VID
87.9 0.22
94.8 0.54
86.3 aaz
87.64 0.44
93.1-0029
93.7 0.16
97.4+ 0.07
75. aue
84.94 0.17
60.7+ 0.19
75.4- 0.19
SemCKD
86.2+ a.s
93.5 nz1
82.84 1.44 84.6+
85.44 o:7 85.5+
91.8=
92.4 om
96.2+ nos 96.4+
72.0= uoe
82.8 0.10
53.5 D17
69.4= 0.17
ReviewKD
86.7+ 007
94.00 a.m
919--015
92.4+ 0.30
72.6+ 0s4
83,1+
57.3+ all
72.6- 0.11
TinySAM
83.7+
93
a3.
79.44 1.12
8815 87.8=
93.5 n.24
48.6 1.14
61.04 0os
25.7+ 119
39.6 1.71
MobileSAM
87.1
86.7
9m
96.3 01
82.61 71.92
59.24 ace
74.1_
InlrSAM(Ours)
88.6 6:8
95.1 uus 87.1 ea7
88.04 8:85
93.4_ aue
94.4 0.12
97.9 0.80
75.6 037
85.24 6:23
61.4 6.3
75.8_ 0.77
et al., 2025) employs full-stage distillation. 4) cross-layer feature distillation (i.e., SemCKD (Chen et al., 2021a), Re- viewKD (Chen et al., 2021b)), which utilizes knowledge from multiple layers of the teacher model to supervise the student, by leveraging diverse information extracted from these layers. Currently, no work in SAM has explored cross- layer fusion distillation for PEFT, InfoSAM is the first to address this.
5.1. Segment Anything Across Diverse Domains
We compare InfoSAM with two categories of methods: PEFT methods and distillation methods. The results are presented in Table I and Table 2, respectively. In Table 1, all PEFT methods outperform both the zero-shot performance and decoder-only fine-tuning, highlighting the importance of unified fine-tuning for SAM. Additionally, InfoSAM out- performs other PEFT techniques across various datasets from different domains. Compared to other PEFT meth- ods, InfoSAM preserves the pre-trained, domain-invariant knowledge through information-basee distillation, which proves effective in enhancing segmentation performance.
We report the main experimental results on representative datasets from different domains. Additional experimental results are provided in Appendix D, and visualization results are available in Appendix G. All experiments are conducted three times to mitigate randomness, with both average values and standard errors reported.
In Table 2, it is noteworthy that most distillation methods are detrimental during PEFT, leading to worse performance compared to fine-tuning without distillation. Specifically, TinySAM employs full-stage distillation, requiring the stu-
7

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoreti Perspective
Table 3: Comparison of PEFT methods and distillation methods with SAM2 across various domains. All results are based on the Hiera-B+ B++ backbone. "SAM2": without adaptation.
(a) PEFT Methods Comparison
(b) Distillation Methods Comparison
METHOD
MEDICAL
AGRICULTURE
REMOTR SENSING
METHOD
MEDICAL
AGRICULTURE
REMOTR SENSING
S. (Kvasir)
IoU (Leaf)
IoU (Road)
S. (Kvasir)
IoU (Leaf)
IoU (Road)
SAM2
4 012
42.7+ 0.2 71.8+ ass
6.9
Teacher
87.1 + 0.12
42.7+
6.94
docoder-only
93.2. 007
48.5+ 0.47
Student
94.4+ 0.66
76.8+ 0.56
60.9+ 0.14
93.8- 0.99
75.4+ 029
59.24 0.36
94.0 0.25
748 0.14
57.34 0.07
AdaptFormer
93.7+
73.6-
59.9+
VID
94.1+
71.2+
61.1
LoRA
93.7.t0.0 0.19
75.9 040
60.8: 0.12
ReviewKD
93.44 0.10
72.7
55.9g (.50
Adapter
94.41 0.6
76.8 0.56
60.91 0:14
TinySMM
89.4+ 0.50
45.2+
23.94
LoRA+Ours
94.0.; 0.09
76.1 0.38
60.9 wa6
MobileS
93.1 0.15
74.12 0.5
52.34 0.66
Adapter+Ours
94.5.
77.3.
61.3+
InfoSAM2(Onn)
94.5+
77.3+ 0.4
61.3+
Table 4: Ablation study results of two losses: relation compression loss L, and distillation loss La
VIT-B
VIFL
VITH
75
MEDICAL
AGRICULTURE
REMOTE SENSING
Teacher
TTepSAM
Mebitisoof
Infossal
Raccwer
ThyySaM
MobildiA4
Teater
TheySAM
Moblesddf
Infosse
S. (Kvasir)
IoU (Leaf)
Toll (Road)
74.4
60.5
(a) Performance on the Leaf dataset.
93.6 (-0.2)
75.2 (+0.8)
(1.0()
94.4 (+1.0)
75.6 (+1.2)
61.4 (+0.9)
60.
Table 5: Effects of the Relation Module (RM). Enhancing TinySAM and MobileSA.M using RM.
20
Teacher
MobiesSad
infssat
Raacher
SAM
MobissAd
Teacher
MobilsSaM
InfeSAM
MODEL
METHOD
AGRICULTURE
REMOTE SENSING
1oU (Leaf)
IoU (Road)
(b) Performance on the Road dataset.
48.61 1.44
28.7 1.0
TinySAM
w RM
50.3
33.9
Figure 4: Performance of InfoSAM with larger teacher models (i.e.. ViT-L, ViT-H), while the student remains based on ViT-B backbone. Each subfigure shows IoU metrics for different distillation methods on a specific dataset.
MobileSAM
w/o RM
71.94.m
59.21
w RM
73.8 9.22
61.3g 0.36
5.3. Distillation Across Models of Different Sizes
dent's features to fully mimic the teacher at every stage. However, this becomes catastrophic when the teacher per- forms poorly (e.g., achieving only 7.2% IoU on the Road dataset). contrast, InfoSAM further enhances PEFT per- formance in this challenging scenario, likely due the relation compression process during distillation, which en- sures the student model learns only the essential information from the teacher model.
We also verify the effectiveness of InfoSAM when scal- ing the teacher model to larger sizes. As shown in Fig. 4, InfoSAM shows comparable improvements to distillation methods specifically designed for SAM compression. indicates that InfoSAM is better suited for PFET, even in traditional large-to-small knowledge distillation scenarios.
5.4. Ablation Study
Ablation of Main Components. We conduct an ablation study evaluate the impact of InfoSAM's components: relation compression loss C, and relation distillation loss Ca on three datasets: Kvasir, Leaf, and Road. Using SAM with an adapter as the bascline (Table 4, row 1), row 2 introduces a fixed-dot relation module with mutual information- distillation loss. Results show that incorporating simple relations improves performance, e.g., 0.8% on Leaf, and
5.2. Extended Experiment with SAM2
Note that method is orthogonal to model development, making it easily transferable to SAM2 backbones. As shown in Table 3, InfoSAM demonstrates consistent effectiveness with SAM2. This transferability attributed to InfoSAM's foundation in information theooretcccc derivation, which is structure-indent indeueedenn...........
8

---

InfoSAM: Fine- Fine-Tuning the Segment Anything Model from An Information- Perspective
Table 6: Transterability of the Relation Module (RM), InfoSAM-T represents the model trained with pre-trained relation module from a different domain.
Acknowledgments
This work was supported in part by the National Natu- ral Science Foundation of China under Grants 62192781, 62172326, and 62137002; part by the Project of China Knowledge Centre for Engineering Science and Technol- ogy; and supported by the Fundamental Research Funds for the Central Universities No. xxj032025002.
METHOD
FROZEN
MEDICAL
AGRICULTURE
RM FROM
S,, (Kvasir)
IoU (Laa))
InfoSAM
94.4+..11 93.7.20
InfoSAM-T
Leaf
InfoSAM
75.64 0.30
InfoSAM-T
Kvasir
75.4+ 045
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
mutual information effectively transfers relational features. Combining both losses yields further gains, e.g., 1.4% on
Leaf.
Effects of Relation Module (RM). To investigate the im- pact of the proposed relation module, we first conduct ex- periments verify its effectiveness in enhancing various distillation methods. Specifically, we evaluate and compare the performance of two compact models, MobileSAM and TinySAM, with and without integrating the relation module, as shown in Table 5. We can observe a significant improve- ment with RM, e.g., 1.9% IoU on the Leaf dataset. These results suggest that the relation module can effectively cap- ture and leverage high-level semantic information, thereby providing complementary benefits to existing distillation strategies during the fine-tuning stage.
References
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., and Dai, Z. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9163-9171, 2019.
Ben Zaken, E., Goldberg, Y., and Ravfogel, S. BitFit: Sim- ple parameter-efficient fine-tuning for transformer-basee masked language-models.. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 1-9, 2022.
Furthermore, Table 6 illustrates the effectiveness of domain- invariance dependencies the relation module, we directly apply the module trained on one specific domain to another domain with entirely different knowledge. The results show that it still maintains satisfying results. Moreover, we con- duct experiments explore the nature of Nomain-invariann information. We use the Boundary F1 Score (Zhang et al., 2023b) to evaluate such universal patterns. The results show that our methods employing the relation module perform better in preserving structural edge features. More results and analysis are available in Appendix F.
Bernal, J., Sánchez, F. J., Femandezz-Epparrchh, G., Gil, D., Rodríguez, C., and Vilariño, F. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015
Bhatia, R. Infinitely divisible matrices. The American Mathematical Monthly, 1 113(3):221-235, 2006.
Boyd, S. Convex optimization. Cambridge UP, 2004.
Cao, W., Zhang, Y., Gao, J., Cheng, A., Cheng, K., and
Cheng, J. Pkd: General distillation framework for object detectors via pearson correlation coefficient. Advances Neural Information Processing Systems, 35:14394- 15406, 2022.
6. Conclusion
We introduce InfoSAM, an information-theoreii tuning framework designed for SAM adaptation. From an infor- mation bottleneck perspective, we extract domain-invariant knowledge from the pre-trained SAM and inject into the fine-tuned SAM to enhance adaptation efficiency. Specifi- cally, we first propose an attention-based module capture structural relations while minimizing mutual information to retain the most essential ones. These relations are then trans- ferred by maximizing their mutual information. Extensive evaluations across eight segmentation datasets spanning di- verse domains and tasks strongly validate the effectiveness of IMM.SAM.
Chang, W.-L., Wang, H.-P., Peng, W.-H., and Chiu, W.- C. All about structure: Adapting structural information across domains for boosting semantic segmentation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 1900-1909, 2019.
Chen, D.,. Mei, J.-P., Zhang, Y., Wang, C., Wang, Z., Feng, Y., and Chen, C. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7028-7036, 2021a.

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theorede Perspective
Chen, P., Liu, S., Zhao, H., and Jia, J. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 500855000000 5017,
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., Chen, W., et al. Lora: Low-rank adaptation of large language models. Proceedings of the Interna- tional Conference on Learning Representations, pp. 1-20, 2022.
Chen, S., Ge, C., Tong, Z., Wang, J.. Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:1 15:16664-16661 2022.
Huang, J., Guan, D., Xiao, A., and Lu, S. Multi-level adversarial network for domain adaptive semantic seg- mentation. Pattern Recognition, 123:108384, 2022.
Chen, T., Zhu, L.. Deng, C.,, Cao, R., Wang, Y., Zhang, S., Li, 7.,, Sun, I.., Zang, Y., and Mao, P. Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3367-3375, 2023.
Jha, D., Smedsrud, P. H., Riegler, M. A., Halvorsen, P., De Lange, T., Johansen, D., and Johansen, H. D. Kvasir- seg: segmented polyp dataset. In MultiMedia model- ing: 26th international conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, proceedings, part II 26, pp. 451-462. Springer, 2020.
Codella, N. C., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris. K., Mishra, N., Kittler, H., et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 interna- tional symposium biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168-172. IEEE, 2018.
Ji. W. Li, J., Bi, Q., Liu, T., Li. W., and Cheng, L. Segment anything is not always perfect: An investigation of sam on different real-world applications, 2024.
Ke, L., Ye, M.. Danelljan, M., Tai, Y.-W., Tang, C.-K., Yu, F., et al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36, 2024.
Dong, G., Wang, Z., Chen, Y., Sun, Y., Song, H., Liu, L.., and Cui, H. An efficient segment anything model for the segmentation of medical images. Scientific Reports, 14 (1):19425, 2024.
Kerr, A., Campbell, D., and Richards, M. Qr decomposition on gpus. Proceedings of 2nd Workshop on General Purpose Processing on Graphics Processing Units, pp. 71-78, 2009.
Dong, Y., Gong. T., Yu, S., and Li, C. Optimal randomized approximations for matrix- based rényi's entropy. IEEE Transactions on Information Theory, 69(7):4218 4234, 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. A.C., Lo, W.-Y., al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Fan, D.PP., Ji, G.-P., Sun, G., Cheng, M.-M.,, Shen, J., and Shao, L. Camouflaged object detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2777-2787, 2020a.
Kuang, H., Liu, H., Wu, Y., Satoh, S., and Ji, R. Improving adversarial robustness via information bottleneck distilla- tion. Advances in Neural Information Processing Systems, 36: 10799961008,,,, 2023.
Fan, D.PP., Ji, G.-P., Zhou, T., Chen, G., Fu, H., Shen, J., and Shao, L. Pranet: Parallel reverse attention network for polyp segmentation. In International conference on medi- cal image computing and computer-assisted intervention, pp. 263-273. Springer, 2020b.
Le, T.-N., Nguyen, T. V., Nie, Z., Tran, M.-T., and Sug- imoto, A. Anabranch network for camouflaged object segmentation. Computer vision and image understanding. 88:45-566, 2019.
Gong, T., Dong, Y., Yu, S., and Dong, B. Computationally efficient approximations for matrix-based rényi's entropy. IEEE Transactions on Signal Processing, 70:6170-6184, 2022.
Li, B., Shen, Y., Wang, Y., Zhu, W., Li. D., Keutzer, K., and Zhao, H. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7403 2022a.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. Knowledge distillation: A survey. International Journal Computer Vision, 17896):1789-188 1819, 2021.
Li, J., Zi, S., Song, R., Li, Y., Hu, Y., and Du, Q. A stepwise domain adaptive segmentation network with covariate shift alleviation for remote sensing imagery. JEEE Trans- actions on Geoscience and Remote Sensing, 60:1-15, 2022b.
Hoffman, J.. Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A., and Darrell, T. Cycada: Cycle-consistent adversarial domain adaptation. In International confer- ence on machine learning, pp. 1989-1998. Pmlr, 2018
10

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information- Perspective
Ma, X., Zhang. X., Ding, X., Pun, M.-O., and Ma, S. Decomposition- -hased unsupervised domain adaptation for remote sensing image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Skurowski, P., Abdulameer, H., Blaszezyk, J., Depta, T. Kornacki, A., and Koziet, P. Animal camouflage analysis: Chameleon database. Unpublished manuscript, 2(6):7, 2018.
Miles, R., Yucel, M. K., Manganelli, B., and Saa- aarriari,,,, A. Mobilevos: Real-time video object segmentation con- trastive learning meets knowledge distillation. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10480-10490, 2023.
Song, Y., Zhou, Q., Lu, X., Shao, Z.,, and Ma, L. Simada: A simple unified framework for adapting segment any- thing model in underperformed scenes. arXiv preprint arXiv:2401. 17803, 2024.
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015.
Mnih, V. Machine learning for aerial image labeling. Uni- versity of Toronto (Canada), 2013.
Passalis, N., Tzelepi, M., and Tefas, A. Probabilistic knowl- edge transfer for lightweight deep representation learning- IEEE Transactions on Neural Networks and Learning Systems, 52(5):2030-20333 2020.
Wang. Z., Ji, K., Wang, D., and Cheng, Samel: Empow- ering sam continually learn from dynamic domains. arXiv preprint arXiv:2412.05011 2024.
Wu, J., Wang, Z., Hong. M., Ji, W., Fu, H., Xu, Y., Xu, M., and Jin, Y. Medical sam adapter: Adapting segment anything model for medical image segmentation. Medical image analysis, 102:103547, 2025.
Peng. X., Chen, R., Qiao, F., Kong, L., Liu, Y., Sun, Y., Wang, T., Zhu, X., and Ma, Y. Learning to adapt sam for segmenting cross-domain point clouds. In European Conference Computer Vision, pp. 54-71. Springer, 2024a.
Xiao, A., Xuan, W., Qi, H., Xing, Y., Ren, R., Zhang, X., Shao, L., and Lu, S. Cat-sam: Conditional tuning for few- shot adaptation of segment anything model. In European Conference on Computer Vision, pp. 189-206. Springer, 2025.
Peng, Z., Xu, Z., Zeng, Z., Xie, L., Tian, Q., and Shen, W. Parameter efficient fine-tuning via cross block orchestra- tion for segment anything model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3743-3752, 2024b.
Xu, Q., Yao, L., Jiang, Z., Jiang, G., Chu, W., Han, W., Zhang, W., Wang. C., and Tai, Y. Dirl: Domain invariant representation learning for generalizable semantic seg- mentation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp.
Rath, R. Leaf disease segmentation dataset, 2023. URL https://wwit kaggle wwt/c//t//////eec.oooo /atasets///s sovitrath/leff-diestt attpraa-diassseeeeeeeneeeeaaaaaa with-trainvalid splitt Accessed: January 18, 2025.
2022.
Yu, S., Giraldo, L. G. S., Jenssen, R., and Principe, J. C. Multivariate extension of matrix- rényi's a-order en- tropy functional. IEEE Transactions on Pattern Analysis 2019.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L.,, al. Sam 2: Segment anything in images and videos. Pro- ceedings of the International Conference on Learning Representations, pp. 1-20, 2022.
and Machine Intelligence,
Zhang, C., Han, D., Qiao, Y., Kim, J. U., Bae, S.-H., Lee, S., and Hong, C. S. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306 230..14686499 2023a.
Shen, Y., Li, J., Shao, X., Inigo Romillo, B., Jindal, A., Dreizin, D., and Unberath, M. Fastsam3d: An efficient segment anything model for 3d volumetric medical im- ages. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 542- 552. Springer, 2024.
Zhang. L.,, Liang, Y., Zhang. R., Javadi, A., and Xie, P. Blo-sam: Bi-level optimization based finetuning of the segment anything model for tic segmentation. In Forty-first International Conference on Machine Learning. 2024a.
seman-
Shu, H., Li, W., Tang, Y., Zhang, Y., Chen, Y., Li, H., Wang, Y., and Chen, X. Tinysam: Pushing the envelope for efficient segment anything model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20470-20478. 2025.
Zhang, Q., Liu, X., Li, W., Chen, H., Liu, J., Hu, J., Xiong, Z., Yuan, C., and Wang. Y. Distilling semantic priors from sam to efficient image restoration models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25409-25418 2024b.

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information- Thhoreetcc Perspective
Zhang, Y., Tian, S., Liao, M., Hua, Zou, W., and Xu, C. Learning shape-invariant representation for generalizable semantic segmentation. IEEE Transactions on Image Processing, 32:5031-5044 2023b.
Zhong, Z., Tang, Z., He. T., Fang, H., and Yuan, C. Convolution meets loRA: Parameter efficient finetun- ing for segment anything model. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL http:////ppeeeeeeeeeeeeeeeeeeeee id=ezscMer8LO.
Zhu, X., Gong, S., et al. Knowledge distillation by on- the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
12

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
A. Derivation of information-theorrtie Losses
A.1. Multivariate Entropy and Mutual Information
Following Definition 2 and Eq.(2), we consider the matrix-based Rényi's a-order joint entropy for multiple variables (Yu al., 2019).
Definition 3. Given collection of n samples {s; (2 2xk ) where the superscript i denotes the sample index, each sample contains & (k 2 2) measurements Iı E 22 22 E X2, . Ih E N obtained from the same realization, and the positive definite kernels #1 Xi x Xi IR, X2 X2 + R, .., Nu Xx. + R, a matrix-based analogue to Rényi's a-order joint-entropy among & variables can be defined as:
A A20..... A tr(A1 A o ..... AA)
(A1, A2. , AA) = Sa
(16)
where (Ai);j (x) (A2) (2 xn) (Ak);s (x . and denotes the Hadamard product.
Following Definition 2 and Definition 3, mutual information be extended to measure interactions among multiple variables by grouping them into sets and treating each set as single variable. It can be defined as:
I, (A,,
A = S, (A,. Ad) + S.) -S S,, (A,, Ag. B)
(17)
where A ... A and B denote the normalized Gram matrices.
A.2. Derivation of Relation Compression Loss C, and Relation Distillation Loss La
Relation Compression Loss L,, In this paper, according to Eq.(16) and Eq.(17), the mutual information of the compression process (k = 2) can be defined as:
I. (A), A2; B) = S., (A1, A2) + (B) S., A41,A2 B)
(18)
In the relation compression loss Ly. the matrices A1, A2, and B are normalized Gram matrices constructed from the image embeddings 2/", the mask embeddings 2T1 and the relation module outputs r", respectively. To maintain consistency with the previously defined relation compression loss (refer to (10)) we replace A,, A2, and B with G. Gf and Gf E NxN respectively, where all Gram matrices are computed using the polynomial kernel function defined as:
<(I, v) = x
(19)
where x, RVV are input vectors. For instance, G. is computed the normalized Gram matrix:
n(2F,27 GT tr(n(2] 22))
(20)
where tr(-) denotes the trace of the matrix. Furthermore, Rényi's a-order entropy is reformulated using its eigenvalue expansion (refer to Eq.(1)), leading
C. -a Amm
= S Gff G Sa
S. Gm Gf))
(21)
log2
1 (G
loga
Xa (G
where Gf is also normalized to have trace of one. The teacher entropy term is excluded from this loss because the teacher's weights remain fixed throughout the training process. Substituting the marginal and joint entropy definitions from Definition I and Definition 3, G G o G G.¹ The o is Hadamard product.
Since computing the eigenvalues of large matrices is typically computationally expensive during training (Kerr et al., 2009). we restrict the value of 2. This choice allows us to use the Frobenius norm as a proxy objective function. Notably, the
13

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
Frobenius norm has connection with the eigenspectrum. Specifically, for symmetric matrix A, its Frobenius norm be expressed as:
| 12 = tr(AA)) = Σ A((A)) = Σ (A)
(22)
where (r(+) denotes the trace operation. Since A is symmetric matrix, Σi= (A is equivalent to Σi= (( (Dong et al., 2023). Through this formulation, the Frobenius norm not only simplifies the computation but also retains an intrinsic connection to the matrix eigenspectrum, which significant for both model training and theoretical analysis. Given G/. and G positive semi-definite, L, can be reformulated as:
n
C,
log2
x (G!
log2
(G
2
(23)
= - log2 | || + log2 || G 1|?
The two terms in L, serve distinct puposes. First, the term - loga ||G 1 imposes spectral compression on the relation module. This ensures that the relation module focuses on the most discriminative features while suppressing irrelevant variations. Second, the term log2 f achieves joint entropy minimization across the image encoder, mask decoder, and relation module. It filters out spurious relationships induced by domain-specinn noise while preserving Nomain-invariant interactions encoded in G This is critical for cross-domain adaptation, as it maintains consistency in feature interactions across different data distributions.
Relation Distillation Loss C. Similar to L,, the goal of distillation to maximize the mutual information of teacher-student relations. The mutual information of the distillation process can be defined as:
La (r = (G+) + (. Sa (G.. G.))
(24)
loga
x
(G+))
loga
X2 (G+)
log2
X
(GTS)
1
1-
==1
where G,,, G. and GTS = G.. G is the normalized Gram matrices. Then, applying (2.) and set a = 2, the distillation loss is:
La 1 (rT:r5) :)
1
log,
Σ AS (G+) - 1 i=1
log,
A
(G-.) +
log2
A (G!S)
(25)
a
1 1 a
1=1
i=1
log2 ||G + log2 || 11 - loga ||
Overall, the mutual information loss balances feature constraints by regularizing the student's relational feature complexity to prevent overfitting while aligning the student's and teacher's relational distributions for effective knowledge transfer. The three terms in the distillation loss serve distinct purposes: log2 || captures the complexity of the teacher's relational features, guiding the teacher learn rich and detailed representations rather than overly simplified ones; log2 |C, || regu- larizes the student's 's feature complexity, preventing overfitting by keeping the representations manageable; - logg2 |G 1|
Aligns the relational distributions between the teacher and student, ensuring the student learns the relationships between features as captured by the teacher.
A.3. Pseudocode of InfoSAM
Here, we summarize the core fine-tuning process for SAM using the proposed information-theorete loss in Algorithm. 1. Relationships between features across different network components are modeled and normalized to capture meaningful interactions. These interactions drive two key loss functions: a compression loss that strengthens the student network's s ability to represent complex features and a distillation loss that aligns the student's ss relational understanding with the teacher's. This method emphasizes relational and structural learning to achieve better generalization and accuracy.
14

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Eiformattion-Thoredie Perspective
Algorithm -yyorchhstyle pseudocode for InfoSAM
FLt, FL9: Pre-trained SAM (teacher) and fine-tuned SAM (student)
z_t_i, 2_s_i: The output of the teacher and student image encoders z_t_m, z_s_m: The output tokens in the mask decoder of the teacher and student f_t, fLs: Teacher and student relation modules
y_t, y_s: Teacher and student outputs
y: Ground-truth labels
Frod: Function for computing the square of the Frobenius norm
for X, y in 1cader: Forward pass =_t_,, z_t_,, m, y_t (x)
2_s s_, y_s (x)
Compute structure loss
loss_cc = struct_loss (y_s,
Compute relations between image encoder and mask decoder f_s f_t
r_t f_t(z_t_i, z_t_m)
r_s - [_s(____), 2_s_m)
Normalize the representations
t nererm F n.normalize (z t_i, p=2) 2_t_n_nerr E.normalize (2_t_m, p-2)
Compute nermalized Gram matrices for compression loss_r t_ matmul (z_t_i_norm, st_i_norm.I] ftt_m m- matmul (z_t_m_nomm) 2_t_m_nerm.7
G ttt = matmul (r_t, r_t.T) G G_t_r_no c_t_r / (rac
imr_norm G_t_i G_t_m E_t_r trace (G_t_i G_t_m G_t_r)
Compute normal ized Gram matrices for distillation loss_d 6_s_r matmul (r_s, r_s.T) 6_s_r_nore 6_s_r trace (6_s_r) ts norm G_s G_t trace G_t_z)
Compute relation compression loss r and distillation loss_d loss_r leg2 (Frob (G_t___norm) log2 (Frob ((_t_imr_norm)
loss_d log2 (Frob (6 (G_t_r_norm)) + log2 (Frob (G_o_r_norm) log2 (Frob (6_ta_r_norm)
loss_info 1 loss_r lamda_2 loss_d
The overall loss loss = lcss_ce info_info
Optimization step loss, Loss.backen backward())
zer. potimizer.stepll
B. Implementation Details
B.1. Architectures of Segment Anything Model (SAM)
SAM (Kirillov et al., 2023) consists of three key modules, i.e., image encoder, prompt encoder, and mask decoder. The image encoder is heavy ViT-based network for image feature extraction. The prompt encoder is designed capture positional information from geometric prompts (i.c., points, boxes, or masks) to generate prompt embeddings. The mask decoder, a two-way transformer module, combines image embeddings and prompt tokens to generate the final mask. The released model, trained on 11 million images and billion high-quality masks, demonstrates impressive zero-shot capability in handling various conventional natural images. The latest SAM2 (Ravi al., 2022) introduces a significant evolution over
15

---

InfoSAM: Fine- Fine-Tuning the Segment Anything Model from An Information-Theorele Perspective
Add & Norm
Transformer
Ad & Nerm
Transformer Block
Feed Forward
A ddapter
Add & Norm
Frod Forward
Transformer Block
Transformer Block
Attention
Add & Norm
Transformer Block
Adapter
Transformer Block
Hg
Wx
Self-A ttention
Hicken States
dapter InfoSAM
LoRA of InfoSAM
(a) Adapter
(b) LoRA
Figure 5: The architecture of InfoSAM with Adapter and LoRA
its predecessor by extending its capabilities to the domain of video segmentation. SAM2 replaces the backbone of SAM with Hiera backbones. And introduces an additional streaming memory component designed for processing video frames. However, recent studies(Wu et al., 2025; al., 2024) have revealed that SAM performs poorly in real-world segmentation tasks across domains such as medicine, agriculture, and remote sensing. Fine-tuning SAM for downstream tasks has been widely recommended.
B.2. Architectures of Adapter and LoRA
We implement InfoSAM with two of the most widely adopted PEFT methods: Adapter (Chen al., 2022; Song et al., 2024) and LoRA (Hu al., 2022). We fine-tune the image encoder of SAM and SAM2 by incorporating adapters LoRA, while fully training the mask decoder. Notably, all training hyperparameters used for SAM and SAM2 remain the same.
The Adapter method introduces lightweight modules into the Transformer architecture by adding small projection layers between the original layers. Specifically, the adapter module consists of two linear transformations: down-projection and an up-projection, with a non-linear activation layer (GELU in our implementation) applied between them. Let the input feature be X BxtxD, where B is the batch size, L the sequence length, and D is the feature dimension. The adapter performs the following operations:
z XW down
(26)
Zact = GELU
(27)
Zact W up
(28)
where W down E Dxr and Wnp E R are learnable weight matrices, and r denotes the bottleneck dimension, typically a small fraction of D. In our experiments, we set the bottleneck ratio r/D to 0.25. The final output of the adapter is computed as:
X X+hh
(29)
with a residual connection preserving the original information. As illustrated in Fig. 5(a), two adapter modules are sequentially inserted between the attention and FFN layers.
LoRA (Low-Rank Adaptation) is another cerrammtte--fffcciee method that integrates lightweight modules into the Transformer architecture. Similar to the adapter method, LoRA introduces trainable down-projection and up-projection matrices. However, as shown in Fig. 5(b), instead of being applied between layers, LoRA is added directly to the query (q), key (k), and value (v) projections within the self-attention mechanism.
Given the input feature X E Lx the multi-head self-attention mechanism with heads computes the query, key, and value follows:
Q = XWQ. K, XW k.s.
Vn XW v.k
1,..., H
(30)
16

---

InfoSAM: Fine- Fine-Tuning the Segment Anything Model from An Information-Theoreii Perspective
where WQ.. W K.A: V. DD and Dj: D/HH is the dimensionality of each attention head.
LoRA introduces low-rank updates to each head's query, key, and value projections. Specifically;
Q Qa + AQu:
Ky Kn + AKa, V =V= AV
(31)
where the low-rank updates are defined as:
AQ XW ddwwdown W hups
AK,
XW K. down W K kpps
AV, XW down W V.h.up
(32)
Here: W W. W K,h,down W down E RR Dxr W WQ.h.up K.A.upp W V.h.up E R Pa, Note that the low-rank projec- tion matrices are head-specifie, and is the bottleneck dimension shared across all heads, determining the rank of the adaptation. In our experiments, set 4 for all the LoRA-based methods.
C. Tasks and Datasets
We benchmark InfoSAM across eight diverse benchmarks spanning four different domains, following Conv-LoRA (Zhong et al., 2024). The specific segmentation tasks and corresponding datasets utilized for benchmarking are elaborated below.
C.1. Natural Image
Camouflaged Object Segmentation. The task aims to detect objects hidden within complex or visually cluttered back- grounds, posing greater challenges compared to traditional object segmentation. We use three camouflaged object detection datasets: COD10K (Fan et al., 2020a), CHAMELEON (Skurowski et al., 2018), and CAMO (Le et al., 2019). CODIOK contains 3,040 training and 2,026 testing samples, CHAMELEON provides 76 testing images, and CAMO includes 1,000 training and 250 testing images. The combined dataset of CODIOK and CAMO training images is used, with 10% randomly split for validation, and testing is performed all three datasets.
C.2. Medical Image
Polyp Segmentation. The task of polyp segmentation in gastrointestinal endoscopic images is critical for early colorectal cancer diagnosis and treatment planning, posing significant challenges due the considerable variability in polyp shapes and sizes. We selecte two polyp segmentation datasets: Kvasir (Jha et al., 2020) and CVC-ClinieDB (also known as CVC-612) (Bernal et al., 2015). The Kvasir dataset consists of 1,000 images, while CVC-ClinicDB contains 612 publicly accessible images. (Fan et al., 2020b) splits the images into a 9:1 ratio for training and testing. Additionally, 20% of the training set is randomly selected as a validation set for use during training.
Skin Lesion Segmentation. The task involves identifying different types of skin lesions in medical images, playing a vital role in the early diagnosis and treatment of skin conditions, particularly skin cancer. However, it remains challenging due to ambiguous boundaries and color variations. We select the ISIC 2017 dataset (Codella al., 2018) for skin lesion segmentation, which contains 2,000 images for training, 150 images for validation, and 600 images for testing.
C.3. Agriculture
Leaf Segmentation. The task focuses on identifying individual plant leaves in agricultural images, supporting automation in plant disease control and high-quality food production. We use the Leaf Disease Segmentation dataset (Rath, 2023), which includes 498 images for training and 90 for testing, with 20% of the training set randomly split for validation.
C.4. Remote Sensing
Road Segmentation. This task involves detecting road regions in images or video frames, which is essential for autonomous driving, traffic analysis, and urban planning. We use the Massachusetts Roads Dataset (Mnih, 2013), containing 107 images for training, 13 for validation, and 48 for testing. All the methods are trained for 20 epochs. During validation and testing, we use 5-point prompts instead of noisy ground-truth -rruh box prompts, as the complexity of road structures in remote sensing images is challenging to capture with box- box-dased prompts.
17

---

InfoSAM: Fine- Fine-Tuning the Segment Anything Model from An Information- Perspective
D. Additional Experiment Results
Due to space constraints, experimental results that could not be included the main body are provided here. These include additional results on more polyp segmentation and camouflaged object segmentation datasets using SAM, as well as the complete experimental results conducted on SAM2.
D.1. Additional Results with SAM
Polyp Segmentation: As shown in Table 7, InfoSAM exhibits superior performance in polyp segmentation on the Kvasir and CVC-612 datasets, outperforming both PEFT and distillation-basee methods.
Table Additional results of polyp segmentation.
Kvasir g
CVC-612
METHOD
S,,
Fi
S.,
F
SAM
85.0+; 0.10
88.8 88.8+ 0.29
79.7 0.66
8
0.19
94.5 0.19
85.8 0.34
docoder-only
90.91
95.2
89.1
92.9
97.41
89.5
Biffi
92.5 012
9..3
91.0 037
94.0 0:3
98.5
91.8 081
AdaptFormer
93.3 ac8
97.0+ 6.11
92.8 0.4
95.2 0.15
99.04
93.8
LoRA
93.0+ 01s
96.6+
91.8++057
94.3
98.7
92.3 0.2
Adapter
03.42:12
c.11
92.9
98.8
94.22 6.31
HQ-SAM
91.11
95.5
89.9 068
93.2
98.01
90.9
SU-SAM
93.8 012
97.5 eus
94.1 0.4
95.3
98.9 009
94.42 0.65
SOA
92.94
96.61
92.2 0:46
94.4
98.81
92.6g.
Logit
93.2. 9.19
96.5 6.09
0.1
95.1 0:30
99.0 0034
93.8 0.99
PKD
92.21
96.0
91.9
94.2
98.5
92.3
93.7. 041
97.3 6:53
93.8 0:3
0.19
99.1 0.15
94.3 633
IBD
91.54
95.1
89.9
93. 1
97.4+
90.0
VID
93.7 a.6
97.4 6/
93.4
95.0
98.7+ aus
93.8 c.20
SemCKD
92.41
96.2.cc
91
98.11
91.51
ReviewKD KD
92.4+
96.44
91.6.:
94.0
98.44
92.2+
MobileSAM
92.5 a12
96.31...4
014
98.61
92.41
TaySAM
88.5 031
93.5
07.0
92.1 0
96.4
88.1
InfoSAM(Ours)
94.4 012
97.9 6,89
93.9 0.09
98.9 + 0:89
94.3 8.15
Camouflaged Object Segmentation: As illustrated in Table 8, InfoSAM achieves state-of-the-art performance in cam- ouflaged object segmentation across the CHAMELEON, CAMO, and CODIOK datasets, surpassing both PEFT and distillation-basee approaches.
Table 8: Additional results of camouflaged object segmentation.
CHAMELEON
CAMO Ea
COD10K Es
METHOD
S..
Ed
Jone
S,,
S,,
F
SAM
80.4 a1s
88 48.0 0.30
74,5 0.13
79.7
88.8 0ze
79.6 om
83.5
92.5 a.
79.2+ on1
decoder-only
87.0
93.5
80.0
84.9
92.7 a.14
81.8
87.1
94.4
79.9
BilFit
89.6
96.11
84.2
87.5
94.5 no8
85.3 0.5
89 2
95.9 021
83.61 0.59
AdaptFormer
92.2
97.64 0.30
87.9
94.8 a21
86.2 0.19
90.1
96.5 0.10
85.8
LoRA
90.44
96.5 eas
85.5
87.7
94.6., a.50
85. 85.14
89 8
96.3
84.9
Adapter
92.5 mm
9719
90.1 0.9
94.8.1 a.14
86.7
90.2 0:2
96.51 a.19
86.0
HQ-SAM
87.0
93.3
79.7.
85.
92.6-,
81.04
87.3
94,5
80.0
SU-SAM
92.3
97.9. 02)
90.0 a.3
88.3
95.0 0.12
86.2 0.59
90.2
96.5 a.00
86.0
CORVLORA-SAM
90.6
96.61 a.re
86.0g
87.5
94.5 0.17
85.4
89.8
96.2 0:1
84.9
Logit
91.94 1u47
eu4s
89.2 ass
aus 94.1
87.1 0:2
0.12 89.7
0.10 96.3
86.1
PKD
90.7
96.2 ami
87.5
87.0
84.3
85.4
92.2 n.m
97.7 0.5
90.0 073
87.8 n
94.5 0.55
86.2 0:46
90.3 0.16
96.6 a.10
06.24
IBD
86.64
93.0.4
79.5
85.2
92.644
82.4
87.9
94.84
81 81
VID
92.2 a.s
97.9
89.8 0.0
87.9
94.8 0.34
86.3 0.32
90.3
96.5 a.16
86.2++
SemCKD
88.8
95.1
83.4 am
86.2
93.5 0.1
82.8
88.5
95.64 a.18
83.1
ReviowKD
90.3
96.5.
85.6. an2
86.74
94.0
84.6+
89 4-4
96.0
84.3 0.2
MobileSAM
96.7
88.6
87.1 n36
0.7
85.11:0m
90.1
96.6.; auk
86.1 0.29
Tay SAM
84.1
90.3
76.6
83.7
91.611
81. 81.1
86.3
94,0
80.0 0.4
92.6 014
98.0 6:5
89.6
aus
95.1 685
87.1 au
90.5 eu6
96.7 0.9
86.2
18

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
D.2. Additional Results with SAM2
The complete experimental results of InfoSAM with the SAM2 backbone are provided in Table and Table 10. Maintaining the strong performance of SAM, InfoSAM achieves superior results with SAM2.
Table 9: Complete comparison results of PEFT methods with SAM2 across different domains.
MEDICAL
AGRICULTURE
REMOTE
METHOD
Kvasir Fgg
Leaf
Road
S
F
IoU
Dice
ToU
Dice
SAM2
87.1 a12
90.2- o.e
85.2+ 0:20 92.1
42.7+ 032
53.3+ 6.31
6.1.
12.4= 0.57
decoder-anly
93.21
96.6
71.81
82.24 o.s
48.5
64.7
93.8. acee
97.0- 0.0
93.2+ o.17
75.4+ 0.29
85.2+ cas
59.2+ oas
74.0=033 025
AdaptPormer
93.74
97.2
93.3
73.64
83.7
59.94
74.6=
LoRA
93.7+ 0.10
97.0- cmm
93.2+ 0.37
75.9+ 0.4
85.5+ cas
60.8 ca2
75.3- 0.29
Adapter
94.4
97.5
93.8
76.8 0.56
86.2
60.94
75.4=
LoRA+Ours
94.0 ase
97.0-- 0.08
93.4+
76.1 0.8
85.7+ 6.31
60.9 eas
75.5
Adapter+Ours
94.5
97.4
94.0
77.3 0.14
86.6 sas
61.3
75.8
Table 10: Complete comparison results of distillation methods with SAM2 across various domains.
MEDICAL
AGRICULTURE
REMOTE SENSING
METHOD
Kvasir Es
Leaf
Road
S
FE
Diee
IoU
Dice
Teacher Student
87.1
90.2 nm 97.5 no
85.2.t 0.0
42.7
53.3
6,9
12.4g
94.4 om
93.8 om
76.8 0.16
86.2g 0.5
60.9 n
75.44 n.1
PKT
94.0 o2s
97.2+ n.mn
93.7+ 0.40
74.8 0.14
0..7
57.3 om
72.5 n.s
VID
94.1
97.24 a4s
93.5+
77.2+ 0.37
86.44 026
61.14
75.64 0.30
ReviowKD
03.4
97.0 n.n
92.7
72.7
83.0
55.9
71.3
MohileSAM
93.3++0.1
96.7+ 0.06
92.5
74.1
52.3 84.
68.3
TinySAM
0.14
93.3 0.2
86.3 oze
45.2. on6
23.9 26.
36,5 1.22
(nfsSA
9..5 a.17
97.4 816
94.0
77.3
86.6 au
61.3
75.8
E. Hyper-parameter Sensitivity Analysis
E.1. Analysis of the A and X2 in Linfo
In Fig. 6, we conduct key hyper-parameed sensitivity study of X and X2 in balancing relation compression loss L, and relation distillation loss La across three typical domains. Each sub-figure shows the heat map under different hyper-parameter settings, reflecting the change of loss function. It is recommended to view it in color display for best results. According to the accuracy heat map, we set A = 1, = 0.5.
Accuracy Heatmap for A vi
Accuracy Heatmap for 4 vs h.
Accuracy Heatmap for à
Ma2
16:00
40.
15.25
13.30
A44
14,5
21.6
05
1/3
03
05
1.0
A
AL
A
(a) Kvasir
(b) Leaf
(c) Road
Figure 6: flyper-parameter sensitivity study A and X2 in balancing I,p and Lgg, with Kvasir, Leaf, and Road datasets (Best viewed in color).
19

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information- Perspective
E.2. Analysis of the a Parameter in Matrix-based Rényi's Entropy
In this paper, we set a = 2 to compute matrix-based based Rényi's a-entropy via the Frobenius norm. The core reasons for choosing a 2 in matrix-based Rényi's o-enttrpy are as follows: (i) The primary practical motivations are computational efficiency and alignment with prior works. By setting a through Frobenius norm operations (see Eq.(1 eliminating the necessity for eigenvalue decomposition. This optimization reduces time complexity from (nn²) ) to (n²)) (n represents the sample numbers) (Dong et al., 2023), substantially reducing computational costs while maintaining theoretical rigor, particularly advantageous for high-dimensional data analysis (Yu et al., 2019). Additionally, prior research has successfully applied Rényi entropy with a = 2 in segmentation tasks (Miles et al., 2023), to align with the established practices in this field, we adopt a = 2. (ii) For theoretical reasons, the application requires emphasis on tails of the distribution (rare events) or multiple modalities (distributions with multiple peaks), should be less than 2 and possibly approach most probable region), should be greater than 2 to emphasize central tendencies. et al., 2019). Moreover, the Frobenius norm's 's differentiable and strongly convex properties guarantee rapid convergence in gradient-based optimization algorithms (Boyd, 2004).
enable direct computation of matrix-based Rényi entropy
from above. If the goal is to highlight the dominant mode (the
2 provides neutral weighting (Yu
Furthermore, in Table 11, we conduct an analysis to evaluate the performance of different a values (a = 1.01, 2, 3). Following with prior work (Yu et al., 2019), we set a = 1.01 to asymptotically approach Shannon entropy. The results indicate that a 2 achieves the highest verification accuracy while reducing computational overhead by an order of magnitude. This computational gain stems from its exclusive reliance Frobenius norm operations, whereas 1.01 or require eigenvalue decompositions, which are computationally more expensive.
Table 11: Experiments of different a values in matrix-based Rényi's entropy.
AGRICULTURE
REMOTE SENSING
COMPUTATION TIME
METHOD
ToU (Leaf)
(Road)
a 1.01
75.3 0:31
60.6 012
32.1 30.7
75.6+ 627
61.4+ 0.30
1.2+ 03
75.2
61.2;
35.44
F. Deep Dive into the Relation Model
F1. Understanding the Domain-invarian Information Encoded by the Relation Model
Many recent studies leverage SAM's pre-trained capabilities for downstream tasks by fine-tuning. However, when the fine-tuning data distribution is narrow, the model tends to overfit task-specific local features (Wang al., 2024). We argue that this is mainly because task-specific optimizations will cover or suppress domain-invariant features learned during pre-training.
To substantiate this assumption, we have conducted experiments in Section 5.4 to illustrate that the extracted relation works (see Table 4) and is TinySAM) by 1.7%% loU, indicating the preserved information's effectiveness. Table 6, applying the RM trained on one domain a completely different domain still preserves its effectiveness, suggesting that these transferable relations are domain-invariant and beneficial for fine-tuning.
(see Table 6). In Table 4, the extracted relations boost other distillation methods (e.g.,
We further explore the nature of Nomain-invariant information. We employ relations to represent domain-invariant informa- tion, which serves as an implicit yet generalizable characterization that may inherently encode various domain-agnostte properties. Here, we showcase and evaluate structural edge information using the Boundary F1 Score (BFS) (Zhang et al., 2023b). As shown in Fig. 7, InfoSAM with the relation module outperforms other fine-tuning baselines in boundary preservation, demonstrating that this implicit relational encoding effectively extracts richer structural edge features.
20

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theorele Perspective
Boundary F1 Score at Different Thresholds
80
SAM
SU SAM
70
InfoSAM (Durs)
Boundary F1 Scom
50
40
20
Thresholds
Figure 7: Boundary quality measured by the Fl-score at various thresholds (0, 1, 2, and 3 pixels) on the Leaf dataset. The thresholds represent the allowable pixel distance: if the predicted boundary is within the threshold distance from the ground truth boundary, it regarded as correct. A larger threshold provides a more tolerant evaluation.
Additionally, we visualize the relation maps extracted from the teacher and student models at various stages of training InfoSAM, ranging from the early to late epochs.
Early Epoch
Middle Epoch
Late Epoch
Image
Early Epoch
Middle Epoch
Late Epoch
Image
Teacher
GT
GT
Student
Figure 8: Relation maps evolve from early late epochs. During training, the relation module gradually captures key information from the pre-trained teacher model, leading improved performance of the student model.
F.2 Effectiveness of the Propused Loss for Relation Model Learning
This section investigates the effectiveness the proposed loss Litto in guiding the relation model to learn generalizable features while avoiding trivial solutions. shown in Eq. (11) and Eq. (12), Linto includes regularization terms such as logg G 12 log, 1 and log, ||G [|2 which promote feature diversity and prevent from converging trivial solutions.
To further verify the effectiveness of these regularization terms, we conduct ablation study to assess their impact both qualitatively (through the visualization of relation maps) and quantitatively (through performance on downstream tasks) in Fig. 9 and Table. 12, respectively: (i) For visualization performance, we visualize the relation maps and their corresponding statistical distributions evolving from early to late epochs. As shown in Fig. 9, without the regularization terms, the distribution of the relation maps becomes increasingly narrow during training, and the domain-invarian information
21

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Thooree Perspective
Table 12: Ablation study of regularization terms (RT) in Linfo-
AGRICULTURE
REMOTE SENSING
METHOD
IoU (Leaf)
IoU (Road)
w/O RT
74.6+ a1z 75.6;
59.6 0m
RT
61.4
Table 13: Experiments of different Relation Model architectures. "Attn-n" represents the number of attention layers for RMM
METHOD
AGRICULTURE
REMOTE SENSING
ToU (Leaf)
ToUl (Road)
Det Product
75.2g
Lincar
74.9 0.t
59.3 0.5
Alin-5
75.4+
61.4-
Aitn-3
75.4 0.40
61.7= 0.6
Attn-I (ours)
75.6+ 0.77
61.4-- 0.30
captured by the relation maps becomes less distinct. In contrast, the RM trained with regularization terms maintains a broad relation distribution and a more representative relation map. (ii) For downstream performance shown in Table. 12, the regularization terms benefit our method by improving performance, as demonstrated by 1.0% and 1.8% increase IoU on the Leaf and Road datasets, respectively. Both results indicate that the proposed loss with regularization terms effectively extracts domain invariiatt features, rather than domain-spechi noise, thereby enhancing downstream performance and alleviating the problem of trivial solutions.
Early Epoch
Middle Epoch
Late Epoch
Early Epoch
Middle Epoch
Late Epoch
Relation Maps
w/o Regularization Terms
Statistical Distributions
Relation
Maps
with Regularization Terms
Statistical Distributions
Figure 9: Evolution of relation maps and their statistical distributions over epochs, without and with the regularization term.
E33. Exploring the Relation Model architectures
We conduct analysis compare different model architectures and explore the number of attention layers for relation module (RM). We compare direct dot product, linear layer, multiple attention layers, and our proposed RM across multiple experiments on two distinct domains.
The experimental results show that: (i) attention based RM outperforms other other architectures designs. This indicates that attention mechanism effectively assess the correlations between the input features (i.e., image and mask features), thereby adaptively filtering and enhancing the useful information (e.g., edge details) while reducing redundancy. (ii) If stack an
22

---

InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoree Perspective
appropriate number of attention layers (e.g., layers) in the RM be beneficial for capturing key information. However, stacking too many (e-g., five layers) increases training difficulty and risks overfitting In nutshell, the eurrent RM design is trade-off between performance and computational overhead, and it effectively captures the relationships between image and mask features.
G. Visualization Results
We present visualization results of mask predictions across various datasets using different PEFT methods for SAM (SAM, HQSAM, SU-SAM, ConYLoRA-SAM, and InfoSAM). These results further demonstrate the superiority of proposed InfoSAM.
RGB Image
GT
SAM
HQSAM
SU-SAM
COnVLORA-SAM
InfoSA M
Figure 10: Visualization results on camouflaged object segmentation.
RGB Image
GT
SAM
HQSAM
SU-SAM
ConvLORA-SAM
InfoSAM
Figure 11: Visualization results on leaf disease segmentation.
RGB Image
GT
SAM
HQSAM
SU-SAM
ConvLORA-SAM
InfoSAM
Figure 12: Visualization results on remote sensing road segmentation.
23


Reviews:
[{"review_id": "JecT32wnFE", "reviewer": "Reviewer_B5XL", "summary": "This paper proposes InfoSAM, a new SAM fine-tuning framework that (1) compresses the domain pseudo-invariant information and (2) maximizes mutual information between a pre-trained teacher and a fine-tuned student model. Experiments across diverse datasets demonstrate that InfoSAM significantly enhances segmentation performance compared to traditional parameter-efficient fine-tuning and distillation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed and valuable review on our paper. We will address each of your comments thoroughly in the following sections.\n\n---\n\n**Q1:Insufficient justification for $\\alpha=2$.**\n\nThe core reasons for choosing $\\alpha=2$ in matrix-based Rényi's $\\alpha$-entropy are as follows: \n\n(1) The primary **practical motivations are computational efficiency and alignment with prior works**. By setting $\\alpha=2$, we enable direct computation of matrix-based Rényi entropy through Frobenius norm operations (Eq.11), eliminating the necessity for eigenvalue decomposition. This optimization **reduces time complexity from $O(n^3)$ to $O(n^2)$** ($n$ represents the sample numbers)  (Dong et al., 2023), substantially reducing computational costs while maintaining theoretical rigor, particularly advantageous for high-dimensional data analysis (Yu et al., 2019). Additionally, **prior research (Miles et al., 2023) has successfully applied Rényi entropy with $\\alpha=2$ in segmentation tasks**, to align with the established practices in this field, we adopt $\\alpha=2$. \n\n(2) For theoretical reasons, if the application requires emphasis on tails of the distribution (rare events) or multiple modalities (distributions with multiple peaks), $\\alpha$ should be less than 2 and possibly approach to 1 from above. If the goal is to highlight the dominant mode (the most probable region), $\\alpha$ should be greater than 2 to emphasize central tendencies. **$\\alpha=2$ provides neutral weighting** (Yu et al., 2019).  Moreover, the **Frobenius norm's differentiable and strongly convex properties guarantee rapid convergence in gradient-based optimization algorithms** (Boyd, 2004).\n\n(3) Furthermore, we conducted an analysis to **evaluate the performance of different $\\alpha$ values ($\\alpha=1.01, 2, 3$)**. Following with prior work (Yu et al., 2019), we set $\\alpha=1.01$ to asymptotically approach Shannon entropy. The results indicate that **$\\alpha=2$ achieves the highest verification accuracy while reducing computational overhead by an order of magnitude**. This computational gain stems from its exclusive reliance on Frobenius norm operations (Eq. 11), whereas $\\alpha=1.01$ or $3$ require eigenvalue decompositions, which are computationally more expensive.\n\n*Experiments of different $\\alpha$ values in $L_{info}$ :*\n\n| **Method**    | **Agriculture** | **Remote Sensing** | **Computation Time** |\n| ------------- | --------------- | ------------------ | ------------- |\n|               | IoU (Leaf)      | IoU (Road)         | ms            |\n| $\\alpha=1.01$ | 75.3 ± 0.31     | 60.6 ± 0.12        | 32.1 ± 30.7   |\n| $\\alpha=2$    | **75.6 ± 0.27** | **61.4 ± 0.30**    | **1.2 ± 0.3** |\n| $\\alpha=3$    | 75.2 ± 0.30     | 61.2 ± 0.06        | 35.4 ± 31.2   |\n\n-------\n\n**Q2:Unclear definition of  $L_{info}$**\n\nThank you for your careful review. Equation (14) should indeed be **$L_{info}=\\lambda_1 L_r+\\lambda_2  L_d$**. We will correct this typographical error in the revised paper.\n\n-------------------------------------\n\nReferences:\n\n1. Dong et al. Optimal Randomized Approximations for Matrix-based Renyi’s Entropy. TIT, 2023.\n2. Yu et al. Multivariate Extension of Matrix-Based Rényi's α-Order Entropy Functional. TPAMI, 2019.\n3. Miles et al. MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation. CVPR, 2023.\n4. Boyd S. Convex optimization[J]. Cambridge UP, 2004.", "claims_and_evidence": "Yes. The paper proposes two key mutual information losses: Relation compression loss Lr and Distillation loss Ld. The effectiveness of these two combinations is demonstrated by the performance of SAM and SAM2 in downstream tasks and tasks in different domains. The effectiveness of each is also verified by ablation study.", "methods_and_evaluation_criteria": "The proposed methods make sense for the task, the evaluation also is thorough.", "theoretical_claims": "Yes, I have the following question for author:\n1. In lines 248–250, Equation (14) is L_info=lambda1*Lce+lambda2*Linfo. However, Lce has not been explicitly defined earlier, and Linfo appears on both sides of the equation. I think it should be L_r and L_d?\n2. In lines 258-260, the choice of α = 2 because it simplifies the computation (using the Frobenius norm), are there other theoretical or practical reasons for selecting this value?", "experimental_designs_or_analyses": "Yes, the experimentals design is sound.", "relation_to_broader_scientific_literature": "The paper’s contributions are twofold. \n1. Different from other parameter-efficient fine-tuning (PEFT) methods (SAMAdapter (Chen et al., 2023), Conv-LoRA (Zhong et al., 2024)), it introduces a novel information-theoretic framework that leverages mutual information to preserve critical domain-invariant features. \n2. Different from traditional knowledge distillation techniques (TinySAM (Shu et al., 2023), MobileSAM (Zhang et al., 2023)), this paper considers the inter-module relationships within the teacher SAM, thereby enabling a more effective transfer of structural knowledge.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n- The paper creatively combines parameter-efficient fine-tuning with an information-theoretic framework, introducing mutual information-based losses that focus on preserving domain-invariant inter-module relationships within SAM. This represents an innovative twist on traditional knowledge distillation approaches.\n\nWeaknesses:\n- Insufficient Justification for alpha=2\n- Unclear Definition of L_info", "other_comments_or_suggestions": "None", "questions_for_authors": "1. In lines 248–250, Equation (14) is L_info=lambda1*Lce+lambda2*Linfo. However, Lce has not been explicitly defined earlier, and Linfo appears on both sides of the equation. I think it should be L_r and L_d?\n2. In lines 258-260, the choice of α = 2 because it simplifies the computation (using the Frobenius norm), are there other theoretical or practical reasons for selecting this value?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5aeU9ZRiQ", "reviewer": "Reviewer_uSPM", "summary": "In this paper, the authors focus on parameter-efficient fine-tuning for segment anything (SAM) network from information theory aspect, and propose InfoSAM. Specifically, InfoSAM aims to mine the domain-invariant relations encoded in the pretrained model, and design a new knowledge distillation framework with two new training objectives, i.e., intra-SAM relation loss and inter-SAM relation loss. by preserving domain-invariant relations in the pretrained model and maximizing mutual information between teacher and student models, InfoSAM achieves better segmentation abilities on various downstream segmentation tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review of our paper and the valuable insights you provided. In the following sections, we will provide a detailed response to each of your comments.\n\n--------\n\n**Q1:Risk of trivial solutions in relation module (RM).** \n\nWe clarify that the **teacher's RM and student's RM share identical parameters,** as described in the Problem Formulation in Sec. 4.2 (lines 185–187).\n\nMoreover, our proposed loss function ($L_{info}$) includes **several regularization terms** ($\\log\\_2 \\|| G^{T}\\_{imr} \\||\\_F^2, \\log\\_2 \\|| G^{T}\\_{r} \\||\\_F^2 , \\log\\_2 \\|| G^{S}\\_{r} \\||\\_F^2$), which are elaborated in Sec. 4.2 after Eq. (11) and Eq. (13). These terms explicitly **promote diversity in the feature distribution** and **prevent it from converging to trivial solutions.** \n\nTo further verify the effectiveness of these regularization terms, we conducted an **ablation study to assess their impact both qualitatively (through the visualization of relation maps) and quantitatively (through performance on downstream tasks).** **Both results indicate that the proposed loss with regularization terms effectively extracts domain-invariant features, rather than domain-specific noise, thereby enhancing downstream performance and alleviating the problem of trivial solutions.**\n\n- **Visualization:** We visualize the relation maps and their corresponding statistical distributions evolving from early to late epochs. As shown in Fig. 1 (https://anonymous.4open.science/r/InfoSAM-7D61/README.md), without the regularization terms, the distribution of relation maps becomes increasingly narrow during training, and the domain-invariant information captured by the relation maps becomes less distinct. In contrast, the RM trained with regularization terms maintains a broad relation distribution and a more representative relation map.\n\n- **Performance:** The regularization terms benefit our method by improving performance, as demonstrated by a 1.0% and 1.8% increase in IoU on the Leaf and Road datasets, respectively.\n  | **Method** | **Agriculture** | **Remote Sensing** |\n  | ---------- | --------------- | ------------------ |\n  |            | IoU (Leaf)      | IoU (Road)         |\n  | w/o RT     | 74.6 ± 0.12     | 59.6 ± 0.69        |\n  | w RT       | **75.6 ± 0.27** | **61.4 ± 0.30**    |\n\n---\n\n**Q2:Typographical error of $L_{info}$.**\n\nThank you for your careful review. Eq. (14) should indeed be **$L_{info}=\\lambda_1 L_r+\\lambda_2 L_d$.** We will correct this typographical error in the revised paper.\n\n---\n\n**Q3:Ablation study of various relation modules.**\n\nWe conducted an analysis to compare different model architectures and explore the number of attention layers for relation module (RM). We compare **direct dot product, a linear layer, multiple attention layers, and our proposed RM** across multiple experiments on two distinct domains. \n\nThe experimental results show that: (1)  **attention-based RM outperforms other other architectures designs.** This indicates that attention mechanism effectively assess the correlations between the input features (i.e., image and mask features), thereby adaptively filtering and enhancing the useful information (e.g., edge details) while reducing redundancy. (2) If we stack an appropriate number of attention layers (e.g., 3 layers) in the RM can be beneficial for capturing key information. However, **stacking too many (e.g., five layers) increases training difficulty and risks overfitting.** In a nutshell, **the current RM design is a trade-off between performance and computational overhead**, and it effectively captures the relationships between image and mask features.\n\n*Ablation study of various RM:*\n\n| **Method**    | **Agriculture** | **Remote Sensing** |\n|---------------|-----------------|--------------------|\n|               |   IoU (Leaf)    |   IoU (Road)       |\n| Dot Product   |   75.2 ± 0.35   |   61.0 ± 0.04      |\n| Linear        |   74.9 ± 0.51   |   59.3 ± 0.58      |\n| Attn-5        |   75.4 ± 0.22   |   61.4 ± 0.12      |\n| Attn-3        |   75.4 ± 0.40   |   **61.7 ± 0.06**  |\n| Attn-1 (ours) | **75.6 ± 0.27** |   61.4 ± 0.30      |", "claims_and_evidence": "The formulation regarding intra-SAM and inter-SAM relations is correct. The theoretical analysis is sufficient and convincing.", "methods_and_evaluation_criteria": "The proposed method is intuitive and effectiveness. The evaluation criteria is widely used in segmentation tasks.", "theoretical_claims": "The formulation regarding intra-SAM and inter-SAM relations is correct.", "experimental_designs_or_analyses": "The validity of experimental design and analysis is enough. The experimental results are promising.", "relation_to_broader_scientific_literature": "The proposed method may insight future knowledge distillation works and other improved parameter-efficient fine-tuning works for better visual models.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "weakness: I noticed that both relation modules in the teacher and student SAM are optimized, how to ensure that the proposed relation modules will not fall into trivial solution.", "other_comments_or_suggestions": "Eqn 14 may include typo, I think it should be lambda_1 * L_r + lambda_2 * L_d.", "questions_for_authors": "The ablation study of parameter numbers of relation module, e.g., the number of attention layers and different architecture. A thought analysis can support the claim from information theory aspect.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ngMTkWZTEc", "reviewer": "Reviewer_NDpU", "summary": "The work identifies a problem with PEFT for SAM - the breaking down of domain-invariant relations encoded during pre-training. It proposes InfoSAM, a model that minimizes a lower bound on the mutual information between the encoder and the decoder during PEFT. It does it in a Rényi entropy sense, without having to run similarity determination on the two distributions. The work is evaluated via a number of standard experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback. In the following sections, we will provide a detailed response to each of your comments.\n\n---\n\n**Q1:Literature reviews for domain-invariant information.** \n\n**The concept of domain-invariant information was first introduced in prior works on domain adaptive segmentation (DAS), which explored cross-domain invariant features such as edge and structural information** (Hoffman et al., 2018). DAS aims to learn domain-invariant representations across multiple domains and follows two main approaches: (1) **extraction and refinement of domain-invariant features,** where methods like feature disentanglement (Chang et al., 2019) or analysis (Xu et al., 2022) decompose images into domain-invariant (e.g., shapes, edges) and domain-specific (e.g., textures, colors) components, aiming to enhance the former while suppressing the latter; (2) **GAN-based domain-invariant feature generation**, which employs adversarial training to align domains at different levels: image (Li et al., 2022), feature (Ma et al., 2024), and output (Huang et al., 2022). For example, GLGAN (Ma et al., 2024) integrates multi-scale global and local features to improve cross-domain transferability in remote sensing.\n\nWith the introduction of SAM, this domain-invariant concept has gained further attention. **SAM's large-scale segmentation pretraining on 11 million images inherently encodes cross-domain commonalities,** enabling strong zero-shot generalization. Recent works leverage these universal visual patterns for downstream tasks (Li et al., 2024; Peng et al., 2024). **However, these methods rely on complex designs or external data to learn representations. In contrast, we focus on preserving the domain-invariant information in pre-trained SAM for fine-tuning.**\n\nThanks the thoughtful review again, we will further add this discussion in the related work in the revised paper.\n\n---\n\n**Q2:Loss and preservation of information**\n\nMany recent studies leverage SAM's pretrained capabilities for downstream tasks by fine-tuning. However, when the fine-tuning data distribution is narrow, the model tends to overfit task-specific local features (Wang et al., 2024). **We argue that this is mainly because task-specific optimizations will cover or suppress domain-invariant features learned during pre-training.**\n\nTo substantiate this assumption, **we have conducted experiments in Sec. 5.4 to illustrate that the extracted relation works (see Tab. 5) and is domain-invariant (see Tab. 6)** . \n\n- Tab. 5: Extracted relations boost other distillation methods (e.g., TinySAM) by 1.7%–5.2% IoU, indicating the preserved information's effectiveness.\n\n- Tab. 6: Applying RM trained on one domain to a completely different domain preserves its effectiveness, suggesting that these transferable relations are domain-invariant and beneficial for fine-tuning.\n\nWe further **explore the nature of domain-invariant information**. We employ relations to represent domain-invariant information, which serves as an implicit yet generalizable characterization that may inherently encode various domain-agnostic properties. Here, we **showcase and evaluate structural edge information using the Boundary F1 Score** (BFS) (Peng et al., 2023). As shown in Fig.2 (https://anonymous.4open.science/r/InfoSAM-7D61/README.md), InfoSAM with the relation module outperforms other fine-tuning baselines in boundary preservation, demonstrating that this implicit relational encoding effectively extracts richer structural edge features.\n\n*Boundary F1 Score comparisons on leaf dataset (threshold=3):*\n\n| Method| BFS (↑)|\n| -| - |\n| SAM |39.0 ± 0.16 |\n| HQSAM|63.7 ± 0.65 |\n| SU-SAM|75.1 ± 0.69|\n| ConvLoRA-SAM|71.5 ± 0.56|\n| InfoSAM (Ours)|**76.4 ± 0.29**|\n-------------------------------------\n\nReferences:\n\n1. Hoffman et al. Cycada: Cycle-consistent adversarial domain adaptation. ICML, 2018.\n2. Chang et al. All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation. CVPR, 2019.\n3. Xu et al. DIRL: Domain-Invariant Representation Learning for Generalizable Semantic Segmentation. AAAI, 2022.\n4. Li et al. A stepwise domain adaptive segmentation network with covariate shift alleviation for remote sensing imagery. TGRS, 2022.\n5. Ma et al. Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation. TGRS, 2024.\n6. Huang et al. MLAN: Multi-level adver sarial network for domain adaptive semantic segmentation. PR, 2022.\n7. Li et al. Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classification. Arxiv, 2024.\n8. Peng et al. Learning to Adapt SAM for Segmenting Cross-domain Point Clouds. ECCV, 2024.\n9. Zhang et al. Learning Shape-Invariant Representation for Generalizable Semantic Segmentation. TIP, 2023\n10. Wang et al. SAMCL: Empowering SAM to Continually Learn from Dynamic Domains. Arxiv, 2024.", "claims_and_evidence": "The idea that there exist domain-invariant relations e.g.  edge information, between pre-training data )the teacher model)  and the downstream task (the student model) is rather imporant. I could not find literature in reading the work that substantiates it, nor a similar mention at the beginnings of the benchmark works. I may be willing to put it to my ignorance but any science paper will make an effort to make its central  premise traceable.", "methods_and_evaluation_criteria": "Both do. The entropy hack is sublime, and appropriate going by the literature. Evaluation is very out-of-the-box.", "theoretical_claims": "Proofs are left to backing literature.", "experimental_designs_or_analyses": "No surprises.", "relation_to_broader_scientific_literature": "Read bundled with the following question.", "essential_references_not_discussed": "The claim that a loss of information happens needs to be butressed.", "other_strengths_and_weaknesses": "\\", "other_comments_or_suggestions": "\\", "questions_for_authors": "\\", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally agree that InfoSAM is a novel and effective approach for fine-tuning SAM models. It leverages information-theoretic losses to preserve domain-invariant knowledge, showing significant performance improvements. The paper is recommended for acceptance.
Decision: Accept (spotlight poster)