Title: Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation
Venue: NeurIPS 2024
Accepted: no
Keywords: Open-vocabulary Semantic Segmentation

Hyperspherical Energy

Partial Orthogonal Fine-tuning

Dual Cross Relation Communication
Abstract: Open-vocabulary semantic segmentation seeks to label each pixel in an image with arbitrary text descriptions. Vision-language foundation models, especially CLIP, have recently emerged as powerful tools for acquiring open-vocabulary capabilities. 
However, fine-tuning CLIP to equip it with pixel-level prediction ability often suffers three issues: 1) high computational cost, 2) misalignment between the two inherent modalities of CLIP, and 3) degraded generalization ability on unseen categories. To address these issues, we propose \alg, a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both of the two CLIP modalities. Specifically, the PEFT strategy is achieved by a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices. Since the PEFT strategy is conducted symmetrically to the two CLIP modalities, the misalignment between them is mitigated. Furthermore, we apply an additional constraint to PEFT on the CLIP text encoder according to the hyperspherical energy principle, i.e., minimizing hyperspherical energy during fine-tuning preserves the intrinsic structure of the original parameter space, to prevent the destruction of the generalization ability offered by the CLIP text encoder. Extensive evaluations across various benchmarks show that H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4\% of the total parameters of CLIP.

Abstract

Open-vocabulary semantic segmentation seeks to label each pixel in an image with arbitrary text descriptions. Vision-language foundation models, especially CLIP, have recently emerged as powerful tools for acquiring open-vocabulary capabilities. However, fine-tuning CLIP to equip it with pixel-level prediction ability often suffers three issues: 1) high computational cost, 2) misalignment between the two inherent modalities of CLIP, and 3) degraded generalization ability on unseen categories. To address these issues, we propose H-CLIP, a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both of the two CLIP modalities. Specifically, the PEFT strategy is achieved by a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices. Since the PEFT strategy is conducted symmetrically to the two CLIP modalities, the misalignment between them is mitigated. Furthermore, we apply an additional constraint to PEFT on the CLIP text encoder according to the hyperspherical energy principle, i.e., minimizing hyperspherical energy during fine-tuning preserves the intrinsic structure of the original parameter space, to prevent the destruction of the generalization ability offered by the CLIP text encoder. Extensive evaluations across various benchmarks show that H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4% of the total parameters of CLIP.

1 Introduction

The aim of open-vocabulary semantic segmentation is to create a segmentation model capable of labeling each pixel in an image with categories that are not limited to a specific closed set according to text descriptions. Vision-language foundation models [43, 5, 34, 39, 11, 17, 26, 21, 27, 13, 18, 29, 10, 28, 45], especially CLIP [39], are often utilized to endow open-vocabulary recognition capabilities. Consequently, open-vocabulary semantic segmentation essentially boil down to transferring these vision-language foundation models, originally trained with image-level supervision, to perform pixel-level predictions.

To this end, current methods [52, 48, 7, 50] typically fine-tune CLIP on a benchmark dataset with 29 segmentation annotations, i.e., COCO [2], to equip it with the segmentation ability. However, this 30 often leads to three main issues. First, fine-tuning CLIP on limited categories would affect its 31 generalization ability, resulting in significant performance degradation on unseen categories. Second, 32 current fine-tuning strategies are usually asymmetrical, which inevitably causes a misalignment 33 between the two inherent modalities of CLIP, i.e., image and text [52], which may lead to sub34 optimal performance. Third, although remarkable performance gains, these approaches often rely on 35 computationally extensive full fine-tuning, which raises concerns about scalability and affordability. 36

To address these issues, we propose a symmetric parameter-efficient fine-tuning (PEFT) strategy 37

for CLIP, dubbed H-CLIP. Specifically, we implement this PEFT through a partial orthogonal fine38

tuning (POF) strategy, which introduces a series of efficient block-diagonal learnable transformation 39

matrices into the hyperspherical space. Then, to preserve CLIP's generalization ability, we leverage 40

the hyperspherical energy principle [32, 38], which suggests that maintaining the same hyperspherical 41

energy during fine-tuning preserves the intrinsic structure, i.e., generalization ability. In light of this, 42

we upgrade our POF by incorporating orthogonal constraints in the learnable matrices for updating 43

CLIP's text encoder, as orthogonal transformations keep the hyperspherical energy unchanged during 44

fine-tuning. Subsequently, we introduce a dual cross-relation communication (DCRC) module to 45

explicitly encourage cross-modal and cross-layer communications within all learnable matrices. 46

47

This communication not only preserves the hyperspherical energy but also further mitigating the

48

misalignment problem.

Extensive results demonstrate that H-CLIP achieves new state-of-the-art open-vocabulary semantic 49 segmentation results across three benchmarks by fine-tuning CLIP with approximately 4% of the 50 total parameters of CLIP. 51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

2 Related Work

2.1 Open-vocabulary Semantic Segmentation

Prior open-vocabulary semantic segmentation works typically perform this task through leveraging CLIP [39]. initial efforts like [56] directly fine-tune CLIP on mainstream segmentation datasets, e.g., COCO [2]. However, they claim that fine-tuning CLIP's encoder significantly reduces its ability to generalize to unseen classes. To address this issue, some methods [15, 8, 51, 49] swing to the opposite extreme, fine-tuning an additional mask generator [6] for segmentation while keeping CLIP frozen to maintain generalization-oriented recognition. However, this frozen parameter space lacks segmentation awareness, resulting in a misalignment between regions and text descriptions [30]. Other studies [52, 50, 7] propose an advanced solution that fine-tunes only selected parameters, e.g., certain layers of CLIP, to enable pixel-level predictions while keeping most of CLIP's parameters fixed, thus minimizing losing of generalization. Although the advantages are remarkable, these methods often work with a very small learning rate, implicitly encouraging a small deviation from the pre-trained CLIP, limiting the segmentation performance. In a nutshell, the trade-off between preserving CLIP's generalization and learning segmentation knowledge persists, hindering the final performance. Based on the paradigm of existing fine-tuning-based methods, our method explores a better trade-off from a fresh viewpoint: hyperspherical space.

2.2 Large-scale Model Fine-tuning

Along with the improvement of large-scale foundation models [26, 34, 28, 23, 53, 42, 41, 40, 60], 70 e.g., segment anything model [23], numerous fine-tuning works [37, 36, 4, 57, 58, 14, 54, 47, 31, 61] 71 are proposed to adapt these models to various downstream scenarios. The core of these approaches 72 lies in updating only limited parameters to capture the specific characteristics of different scenarios, 73 while keeping most parameters fixed to maintain generalization. In contrast, fine-tuning CLIP 74 for open-vocabulary semantic segmentation often meets a dilemma. On the one hand, limited 75 parameters typically fall short in facilitating the transition from a classification model, i.e., CLIP, to a 76 segmentation task. On the other hand, directly increasing the number of trainable parameters risks 77 undermining CLIP's ability to generalize to unseen classes, as experimented in CAT-Seg [7]. Most 78 methods [52, 48] solve this issue by simply freezing CLIP's text encoder and fine-tuning its image 79 encoder, inevitably causing misalignment between the two modalities of CLIP. In this paper, we shed 80 light on how to preserve generalization in a symmetric parameter-efficient fine-tuning manner and 81 strive to explore an appropriate fine-tuning method for open-vocabulary semantic segmentation. 82

83

84

3 Preliminaries

3.1 Hyperspherical Energy

Existing fine-tuning methods implicitly assume that a smaller Euclidean distance between the fine85 tuned model and the pre-trained model indicates better preservation of the pre-trained ability. However, 86

118

119

4 Methodology

4.1 Overview of H-CLIP

Fig. 2 illustrates the proposed H-CLIP framework, which is based on two core components: (1) POF 120 updates the pre-trained parameter space of CLIP using a series of block-diagonal transformation 121 matrices. According to analysis in Sec. 1, each parameter matrix in CLIP's text encoder is orthogonal 122 to preserve generalization. (2) DCRC incorporates cross-modal and cross-layer communication 123 within all tunable matrices, facilitating alignment between different modalities. 124

̸

the Euclidean difference is unable to fully capture the degree of semantic preservation. According to 87 the inspiration from Thomson problem[44] which is to determine the minimum electrostatic potential 88 energy configuration of N mutually-repelling electrons on the surface of a unit sphere, we adopt the 89 Hyperspherical Energy to characterize the diversity of the model. The hyperspherical energy function 90 of a fully connected layer W is defined as HE ( W ):= ∑ i = j ∥ ˆ w i -ˆ w j ∥ -1 , where ˆ w i := w i / ∥ w i ∥ 91 denotes the i -th normalized neuron. The power of the model representation can be characterized by the 92 hyperspherical energy of its neurons. Higher energy implies higher redundancy, while lower energy 93 indicates that these neurons of the model are more diverse. For the original semantic information not 94 to be destroyed in the case of fine-tuning, we hypothesize that a good fine-tuning model should have 95 a minimal difference in hyperspherical energy compared to the pre-trained model: 96

̸

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

̸

One can easily observe that the attainable minimum is zero for Eq. (1). In this case, the hyperspherical energy should satisfy an invariance property (the application of the same orthogonal transformation for all neurons demonstrates the pairwise hyperspherical similarity). Based on the hyperspherical energy invariance property, the minimum of zero can be achieved as long as W and W 0 differ only up to a rotation or reflection, i.e., W = RW 0 in which R ∈ R d × d is an orthogonal matrix (The determinant 1 or -1 means rotation or reflection, respectively).

3.2 Notation of Tensor Product

In this section, we introduce the fundamental concept underlying our DCRC (Sec. 4.3): tensor product. A p-order tensor is indexed by p indices and can be represented as a multidimensional array of data. Formally, a p-order tensor A can be written as A = ( a i 1 ,i 2 , ··· ,i p ) ∈ R n 1 × n 2 ×··· n p . Slices of a tensor are matrices defined from the tensor by holding all but two indices constant. For a 3 -order tensor, A (: , : , k ) corresponds the k th frontal slice. For p -order tensors, matrix slices of p -order tensors can be referenced using linear indexing by reshaping the tensor into an n 1 × n 2 × n 3 n 4 · · · n p 3 -order tensor and referring to the k th frontal slice as A (: , : , k ) . A i ∈ R n 1 × n 2 ×··· n p -1 for i = 1 , · · · , n p denotes the ( p -1) -order tensor created by holding the p th index of A fixed at i . It is possible to create a tensor in a block circulant pattern, where each block is a tensor of ( p -1) -order:

where circ ( · ) creates a block circulant tensor and the size of circ ( A ) is ( n 1 n p × n 2 n p ×···× n p -2 n p × 113 n p -1 ) . define unfold ( · ) to take an n 1 ×··· × n p tensor A and return an n 1 n p × n 2 ×··· n p -1 block 114 tensor in the following way: 115

The operation that takes unfold back to tensor form is the 'fold' command. Specially, fold ( · , n p ) 116 takes an n 1 n p × n 2 ×··· × n p -1 block tensor and returns an n 1 ×··· × n p tensor, defined as: 117

125

126

127

128

129

130

131

132

133

134

135

…

Dual Cross Relation

Communication

Decoder

Image

Text

Parameter Space

Fine-tuning

Orthogonal

Matrix

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dfcf)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:d835)

…

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:dfcf)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc97)GLYPH(cmap:d835)

…

…

q

q

q

q

d

d

4.2 Partial Orthogonal Fine-tuning

The core idea of partial orthogonal fine-tuning (POF) is to introduce the concept of hyperspherical space for fine-tuning CLIP. In this hyperspherical space, we fine-tune CLIP's text encoder under an orthogonality design principle from OFT [38] to preserve the hyperspherical energy of the pre-trained parameter space. Similarly, we use Cayley parameterization [3] to ensure a tunable matrix R is strictly orthogonal, formally as:

where Q is skew-symmetric. Here, for R in CLIP's image encoder, we remove the orthogonality constraint, defined as:

where I is an identity matrix. Considering the relatively large dimension d of the pre-trained matrix, for better efficiency, we introduce a block-diagonal structure by parameterizing R with b blocks, formally as:

where R i ∈ R d/b × d/b . Specifically, denote R V = { R v 1 , · · · , R vℓ , · · · , R vL } and R E = 136 { R e 1 , · · · , R eℓ , · · · , R eL } as the sets of block-diagonal matrices in CLIP's image encoder and 137 text encoder, respectively, where L is its number of Transformer layers, R vℓ ∈ R d v × d v , and 138 R eℓ ∈ R d e × d e . For simplicity, we set d v = d e = d . Overall, we develop a H-CLIP framework, 139 and for an input feature map M ℓ in the ℓ th Transformer layer of CLIP, the right branch produces the 140 adjusted feature map via H-CLIP, ˜ M ℓ , formally via: 141

where W ℓ is a pre-trained weight matrix in ℓ th layer of CLIP's encoder, and F ℓ represents ℓ th layer of 142 CLIP's encoder. During the fine-tuning phase, H-CLIP is fine-tuned in conjunction with the original 143 parameter space of CLIP, which is loaded from the pre-trained checkpoint and remains frozen. 144

4.3 Dual Cross Relation Communication 145

Although in POF, we relax the orthogonal constraint for CLIP's image encoder to learn segmentation 146 knowledge, each layer of the image encoder still incorporates a limited number of parameters, which 147 largely restricts the flexibility of the projection adjustment due to the limitation of Hidden Markov 148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

Chain along layers [24, 46, 36]. To address this limitation, one might consider fully fine-tuning instead of using a small number of parameters. However, this approach can cause a misalignment between image and text features in CLIP, resulting in sub-optimal performance [52]. Based on the above analysis, we introduce Dual Cross-Relation Communication (DCRC), which facilitates interaction among different layers and modalities (i.e., text and image). DCRC explicitly enhances the flexibility of fine-tuned projection adjustments and prevents misalignment issues.

DCRC introduces cross-layer and cross-modality communication among different block-diagonal matrices, achieved through two relation projections. To do this, we first treat all blocks in ℓ th layer as an individual slice in this 3 -order tensor T ℓ , which is derived as follows:

Where q = d/b . Then, we treat the tensor T ℓ as an individual slice within a 4 -order tensor T , defined as follows:

Initially, according to the characteristics of gradient propagation in deep learning theory, i.e., chain rule, each frontal slice R · ℓi ∈ { R q × q } ( b + b ) × L is updated sequentially in CLIP's encoder. As a result, updating the T lacks cross-frontal-slice communication, limiting the flexibility of adjusting fine-tuned projection. To address this, we introduce two special tensor products, i.e., 3 -order T-product and Higher-order T-product .

Definition 4.1(3-order T-product) For A ∈ R n 1 × n 2 × n 3 and B ∈ R n 2 × l × n 3 , the 3 -order T-product C ∈ R n 1 × l ×× n 3 = A∗B is defined as:

where ' · ' represents standard matrix product.

Definition 4.2(Higher-order T-product) For A ∈ R n 1 × n 2 × n 3 ···× n p and B ∈ R n 2 × l × n 3 ×···× n p , the High-order T-product C ∈ R n 1 × l × n 3 ···× n p = A∗B is defined as:

If A ∈ R n 1 × n 2 × n 3 , according to the 3-order T-product , there is an invertible transform S 3 ( · ) : R n 1 × n 2 × n 3 → R n 1 × n 2 × n 3 in third dimension and it transform the Eq. (8) as:

where ¯ C = ¯ A⊙ ¯ B denotes the frontal-slice-wise product (Definition 2.1 refers to [19]) ¯ C (; , ; , i ) = ¯ A (; , ; , i ) · ¯ B (; , ; , i ) , i = 1 , 2 , · · · , n 3 and S -1 3 ( · ) is the inverse transform of S 3 ( · ) . According to the definition of the frontal-slice-wise product, the invertible transform S 3 ( · ) is formulated as:

where ' × 3 ' denotes the mode-3 product and S 3 ∈ R n 3 × n 3 is an arbitrary invertible matrix. Similarly, the inverse transform of Eq. (11) is derived as:

Similarly, if A ∈ R n 1 × n 2 ×···× n p , according to the Higer-order T-product , there are invertible transform S i ( · ) : R n 1 × n 2 ×···× n p → R n 1 × n 2 ×···× n p , i = 3 , 4 , · · · , p in i th dimension and they transform the Eq. (9) as:

where ˜ S ( A ) = S p ( S p -1 ( · · · S 3 ( A ) · · · )) , ¯ C = ¯ A ⊙ ¯ B denotes the frontal-slice-wise product ¯ C (; , ; , i ) = ¯ A (; , ; , i ) · ¯ B (; , ; , i ) , i = 1 , 2 , · · · , n 3 n 4 · · · n p and ˜ S -1 ( · ) is the inverse transform of ˜ S ( · ) . Similarly, the inverse transform ˜ S ( · ) is formulated as:

and its inverse transform is derived as: 183

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

Derivation. please refer to supplementary material. 184

■

According to Eqs. (29), (30) and (31), we adopt its idea and design arbitrary invertible relation matrix 185 S 3 ∈ R ( b + b ) × ( b + b ) and S 4 ∈ R L × L to capture the cross-modality and cross-layer information in T . 186 Then the updated tensor T w is formulated as: 187

where the relation matrix S 3 and S 4 are learnable. To better capture the nonlinear interactions inside the whole parameter space, we further adopt k layers deep neural network (DNN) f 3 ( · ) and f 4 ( · ) to replace the transform × 3 S 3 and × 4 S 4 , respectively, and the DNN f 3 ( · ) is formulated as:

where σ ( · ) is a nonlinear scalar function and matrices { W j ∈ R ( b + b ) } k j =1 . The DNN f 4 ( · ) is similar. Finally, the T is updated by T = T + α T w , where α ∈ R ( b + b ) × L is a learnable parameter.

5 Experiments

5.1 Experimental Setup

Datasets. Following previous studies [7, 48], we utilizes the COCO-Stuff dataset [2] as our training set. This dataset comprises approximately 118,000 densely annotated images across 171 distinct semantic categories. During inference, we carry out comparisons with state-of-the-art methods across several semantic segmentation datasets, including ADE20K [55], PASCAL VOC [12], and PASCALContext [35]. ADE20K [55] is a classical semantic segmentation dataset comprising around 20,000 training images and 2,000 validation images. Besides, it includes two different test sets: A-150 and A-847 . The test set A-150 has 150 common categories, while the test set A-847 has 847 categories. PASCAL VOC [12] is a small dataset for semantic segmentation, which includes 1464 training images and 1449 validation images. The dataset contains 20 different foreground categories. We name it as PAS-20 . In line with [7], we also report a score on PAS-20 b , which involves 'background' as the 21st category. PASCAL-Context [35] is upgraded from the original PASCAL VOC dataset. It includes two different test sets: PC-59 and PC-459 for evaluation. The test set PC-59 has 59 categories, while the test set PC-459 has 459 categories.

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

Methods

OVSeg [30]

CAT-Seg [7]

SAN [50]

Ours

Param. (M)

147.2

25.6

8.4

5.6

Table 2: Efficiency comparison in terms of learnable parameters.

Image

Ground truth

CAT-Seg [7]

Ours

Evaluation metric. Following prior works [7, 48], we adopt mean Intersection over Union (mIoU) to evaluate the semantic segmentation performance on the three benchmarks.

Implementation Details. We implement our method using the Transformer-based CLIP model. Following the protocol established in [7], we evaluate our results on two versions of the CLIP model: ViT-B/16 and ViT-L/14. For training, we use the Adam optimizer [22] with an initial learning rate of 5 × 10 -6 for CLIP, and a weight decay of 10 -4 . Training is conducted with one image per mini-batch. We set q = 128 for balancing efficiency and performance. The function f 3 ( · ) and f 4 ( · ) are implemented using two 2-layer MLPs. We act the cost-based approach provided in [7] as our decoder. All models are trained over 80,000 iterations on 4 NVIDIA RTX 3090 GPUs.

5.2 Main Results

Comparing to SOTAs. Here, we compare our proposed H-CLIP with several state-ofthe-art methods, as shown in Table 1, using six test sets across three benchmarks. Overall, we achieve the best results. Most existing open-vocabulary semantic segmentation methods employ traditional fine-tuning approaches, i.e., full or partial fine-tuning (tuning certain layers of CLIP). While these methods offer sufficient flexibility for learning new knowledge, they often result in a significant performance drop on unseen classes, as observed with OVSeg [30]. Among these methods, CAT-Seg [7] achieves performance comparable to ours. However, its fine-tuning scheme is manually controlled through different layer combinations, necessitating a careful design to balance generalization and flexibility, while ours does not suffer from such an issue. Then, compared to SAN [50], another parameter-efficient fine-tuning method that introduces only a limited number of tunable parameters, our approach significantly outperforms it, achieving improvements of 6.6% on the PC-459 dataset and 3.9% on the PC-59 dataset with ViT-B/16 as the base model. These results demonstrate the effectiveness of our method in preserving generalization while learning segmentation knowledge.

Qualitative results. Here, we visualize our method's representative example segmentation results 235 against prevailing methods, e.g., CAT-Seg [7] in the PC-459 dataset. As shown in Figs. 2 4, we 236 observe that our approach is able to generalize on diverse scenarios and produce more accurate 237 results. 238

Image

Ground truth

CAT-Seg [7]

Ours

PACIFIC

ONH-

PACIFIC

PACIFIC

PACIFIC

punoub

punout

punoub

window

bedclothe

floor

late

plate

water

water

water

Image

Ground truth

CAT-Seg [7]

Ours

Sugadies ad

Shuttes

Efficiency comparison. We compare the efficiency of our method with other approaches, including 239 OVSeg [30], CAT-Seg [7], and SAN [50], all of which utilize CLIP ViT models. The comparison, 240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

summarized in Table 2, shows that our method employs the fewest trainable parameters while balancing the generalization of the pre-trained model and the flexibility for learning new knowledge. Additionally, since we introduce a lightweight architecture for calculating relations, specifically two relation matrices, the inference overhead is negligible during the inference phase.

5.3 Ablative Studies

Ablation of Main Components. Here, we conduct an ablation study to demonstrate the benefits of each component of our proposed H-CLIP: partial orthogonal fine-tuning (POF) and dual crossrelation communication (DCRC). We use the ViT-B/16 [9] version of CLIP as the baseline, shown in row 1 of Table 3. Additionally, we implement a mainstream parameter-efficient fine-tuning (PEFT) method, LoRA [16], for comparison with a similar number of learnable parameters, as shown in row 2. Note that LoRA can improve performance compared to the baseline, demonstrating that PEFT is a viable approach for this task. Then, comparing row 5 to row 2, we observe significant performance gains, indicating that our results are driven by our targeted solution rather than merely the number of parameters. Moreover, row 3 shows that using only POF preserves generalization on unseen classes, particularly in the A-847 dataset. Meanwhile, solely adapting DCRC shows limited improvement, as it only enhances communication among frozen weight matrices. Finally, integrating DCRC with POF yields clear performance gains, e.g., a 12.6% improvement on the PC-459 dataset.

Different Design of POF. Table 4 presents experiments introducing different designs into POF. The design of POF is related to (1) block dimension, i.e., q , and (2) how orthogonality constraints are applied. In (a), the results show that larger In (a), the results show that larger q generally performs better than smaller q . However, we find a good trade-off between performance and parameter efficiency, with q = 128 working well across datasets and tasks. Therefore, we maintain this setting in other experiments. In (b), we show that both blindly applying orthogonality constraints to the learnable matrices of all layers and not using any constraints at all can degrade performance on most test sets, demonstrating the value of our analysis with the hyperspherical energy principle.

6 Conclusion

In this paper, we propose a H-CLIP framework to address three issues: 1) high computational cost, 2) misalignment between the two inherent modalities of CLIP, and 3) degraded generalization ability on unseen categories when equipping CLIP with pixel-level prediction ability for open-vocabulary semantic segmentation. Specifically, we propose a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both of the two CLIP modalities. Specifically, the PEFT strategy is achieved by a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices to mitigate misalignment between different modalities. Furthermore, we apply an additional constraint to PEFT on the CLIP text encoder according to the hyperspherical energy principle, i.e., minimizing hyperspherical energy during fine-tuning preserves the intrinsic structure of the original parameter space, to prevent the destruction of the generalization ability offered by the CLIP text encoder. Extensive experiments demonstrate that the proposed H-CLIP framework generalized improves segmentation performance across several benchmarks while introducing approximately 4% of CLIP's total parameters. We hope our approach will provide a new direction and inspire future research in this field.

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

References

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Arthur Cayley. Sur quelques propriétés des déterminants gauches. 1846.

Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails to segment anything? - sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more, 2023.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104-120. Springer, 2020.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1290-1299, 2022.

Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation, 2024.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11583-11592, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. Advances in neural information processing systems , 35:32942-32956, 2022.

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18166-18176, 2022.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems , 33:6616-6628, 2020.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 132(2):581-595, 2024.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision , pages 540-557. Springer, 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12976-12985, 2021.

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

Eric Kernfeld, Misha Kilmer, and Shuchin Aeron. Tensor-tensor products with invertible linear transforms. Linear Algebra and its Applications , 485:545-570, 2015.

Misha E Kilmer and Carla D Martin. Factorization strategies for third-order tensors. Linear Algebra and its Applications , 435(3):641-658, 2011.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning , pages 5583-5594. PMLR, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Jungbeom Lee, Jooyoung Choi, Jisoo Mok, and Sungroh Yoon. Reducing information bottleneck for weakly supervised semantic segmentation. Advances in Neural Information Processing Systems , 34:27408-27421, 2021.

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Languagedriven semantic segmentation. In International Conference on Learning Representations , 2022.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10965-10975, 2022.

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7061-7070, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. Advances in neural information processing systems , 31, 2018.

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

Canyi Lu, Xi Peng, and Yunchao Wei. Low-rank tensor completion with a new tensor nuclear norm induced by invertible linear transforms. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5996-6004, 2019.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

Zelin Peng, Zhengqin Xu, Zhilin Zeng, Lingxi Xie, Qi Tian, and Wei Shen. Parameter efficient fine-tuning via cross block orchestration for segment anything model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Zelin Peng, Zhengqin Xu, Zhilin Zeng, Xiaokang Yang, and Wei Shen. Sam-parser: Fine-tuning sam efficiently by parameter space reconstruction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4515-4523, 2024.

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems , 36:79320-79362, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 , 2019.

Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019.

JJ Tomson. On the structure of the atom: an investigation of the stability and periods of osciletion of a number of corpuscles arranged at equal intervals around the circumference of a circle; with application of the results to the theory atomic structure. Philos. Mag. Series 6 , 7(39):237, 1904.

Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprint arXiv:2111.10023 , 2021.

Zifeng Wang, Xi Chen, Rui Wen, Shao-Lun Huang, Ercan Kuruoglu, and Yefeng Zheng. Information theoretic counterfactual learning from missing-not-at-random feedback. Advances in Neural Information Processing Systems , 33:1854-1864, 2020.

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454 , 2023.

Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoderdecoder for open-vocabulary semantic segmentation, 2023.

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2955-2966, 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2945-2954, 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision , pages 736-753. Springer, 2022.

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. In NeurIPS , 2023.

Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems , 35:36067-36080, 2022.

Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision , pages 696-712. Springer, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV) , 2022.

Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2605-2615, 2023.

466

467

468

469

470

471

472

Appendix of H-CLIP

A Derivation of the Definition

In this section, we provide derivations of definitions in the main paper.

Definition 4.1(3-order T-product) For A ∈ R n 1 × n 2 × n 3 and B ∈ R n 2 × l × n 3 , the 3 -order T-product C ∈ R n 1 × l ×× n 3 = A∗B is defined as:

where ' · ' represents standard matrix product.

Definition 4.2(Higher-order T-product) For A ∈ R n 1 × n 2 × n 3 ···× n p and B ∈ R n 2 × l × n 3 ×···× n p , the High-order T-product C ∈ R n 1 × l × n 3 ···× n p = A∗B is defined as:

Derivation. According to [20], if A is n 1 × n 2 × n 3 , A can be block diagonalized by using Discrete 473 Fourier Transformer (DFT) matrix F n 3 ∈ R n 3 × n 3 as: 474

where ' ⊗ ' denotes the Kernecker product, ' F ∗ n 3 ' denotes the conjugate transpose of F n 3 , ' · ' means 475 standard matrix product and D is a block diagonal matrix. In fact, the i -th block matrix D i of D can 476 be computed by applying DFT of A along 3-rd dimension. The 3-order T-product in Eq. (18) can 477 be computed as: 478

479

480

481

482

483

484

485

486

487

488

489

It is readily shown that ( F n 3 ⊗ I n 2 ) unfold can be computed by applying DFT of B along 3-rd dimension: the result called ¯ B . Thus, Eq. (21) remains to multiply each block matrix D i of D with each block matrix B i of ¯ B , then take an inverse DFT along the 3-rd dimension of the result. Hence, the 3-order T-product in Eq. (18) can be re-formulated as:

where DFT 3 ( · ) is DFT along 3-rd dimension and DFT -1 3 ( · ) is the inverse DFT along 3-rd dimension. In mathematics, the DFT of A along 3-rd dimension is formulated as:

Similarly, the inverse DFT of ¯ A along 3-rd dimension is derived as:

By the detailed theoretical analysis in [33], the DFT has been extended to a general invertible transform S with an invertible transform matrix S . In mathematics, the invertible transform of A along 3-rd dimension is formulated as:

Similarly, the inverse transform of ¯ A along 3-rd dimension is derived as:

Similarly, if A ∈ R n 1 × n 2 ×···× n p , A can be block diagonalized by using a sequence of DFT matrices 490 F n i ∈ R n i × n i , i = 3 , 4 , · , p as: 491

where ˜ A = circ ( unfold ( A )) ∈ R n 1 n 3 n 4 ··· n p × n 2 n 3 ··· n p . Since the matrix D is block diagonal with 492 n 3 n 4 · · · n p blocks each of size n 1 × n 2 , the Higher-order T-product in Eq. (19) can be computed 493 as: 494

495

496

497

498

499

500

501

502

503

where ˜ F = F n p ⊗ F n p -1 ⊗··· ⊗ F n 3 . Using the DEF, it is straightforward to show that the block diagonal matrix D in Eq. (27) can be obtained by repeated DFTs of A along each dimension expect for 1 -st and 2 -nd dimension. Similarly, by using a sequence invertible transform S j ( · ) , i = 3 , 4 , · , p with invertible transform matrix S i , the Higher-order T-product in Eq. (19) can be re-formulated as:

where ˜ S ( A ) = S p ( S p -1 ( · · · S 3 ( A ) · · · )) , ¯ C = ¯ A ⊙ ¯ B denotes the frontal-slice-wise product ¯ C (; , ; , i ) = ¯ A (; , ; , i ) · ¯ B (; , ; , i ) , i = 1 , 2 , · · · , n 3 n 4 · · · n p and ˜ S -1 ( · ) is the inverse transform of ˜ S ( · ) . The inverse transform ˜ S ( · ) is formulated as:

and its inverse transform is derived as:

■

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract clearly states the following claims about the paper's contributions and scope. We aim to address a significant challenge in open-vocabulary semantic segmentation: fine-tuning CLIP to achieve per-pixel predictions without compromising its generalization capabilities. To this end, we propose a novel H-CLIP, which introduces a partial orthogonal fine-tuning strategy that prevents a drop in hyperspherical energy, thereby preserving generalization. Subsequently, H-CLIP employs dual cross-relation communication to increase projection flexibility, facilitating the acquisition of segmentation knowledge.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: Those are not discussed in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are proved clearly in the main paper or the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The detailed experimental settings and information are provided in Sec. 5, and this information is sufficient to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code will be made public after acceptance.

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the experimental settings are clarified in Sec. 5. (and Appendix).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: error bars are not reported because it would be too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computing requirements are provided in Sec. 5.

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We ensure our research adheres to the guidelines.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All the assets used are properly credited and are the license and terms of use explicitly mentioned and properly respected.

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents H-CLIP, a novel framework for open-vocabulary semantic segmentation using the CLIP model. The framework addresses three key challenges: high computational cost, misalignment between CLIP's image and text modalities, and degraded generalization ability on unseen categories when fine-tuning for pixel-level predictions. H-CLIP employs a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both CLIP modalities. This strategy uses efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module to mitigate misalignment issues. Additionally, an orthogonality constraint based on the hyperspherical energy principle is applied to the text encoder to preserve the generalization ability of the pre-trained model.\n\nSTRENGTHS:\nThe introduction of the H-CLIP framework for open-vocabulary semantic segmentation represents a significant innovation. The use of a symmetrical parameter-efficient fine-tuning (PEFT) strategy in hyperspherical space is a unique approach to addressing the challenges associated with fine-tuning vision-language models.\n\nThe paper provides extensive experimental results across multiple benchmarks, including ADE20K, PASCAL VOC, and PASCAL-Context. These experiments validate the effectiveness of H-CLIP, showing its superior performance compared to state-of-the-art methods.\n\nWEAKNESSES:\n1. Consider that the expression in formula 5 does not specify how to interact with the \\boldsymbol{R} matrix.\n\n2. The paper states that current fine-tuning strategies are usually asymmetrical, but it does not provide enough evidence or references to support this claim. The authors should provide empirical evidence or references to support the claim of asymmetry.\n\n3. While the paper extensively discusses the orthogonality constraint in the CLIP image encoder, it lacks an in-depth analysis of how the misalignment problem impacts segmentation performance. The authors should discuss the specific effects of misalignment on segmentation.\n\n4. The paper should mention SAM (Segment Anything) and how the current work is still significant\n\nQUESTIONS:\n1. This problem exists in other areas such as Object Detection or any dense detection tasks, does the proposed method is generalizable enough for other tasks as well?\n2. Some relevant recent works are not discussed and compared such as [1] [2].\n\n[1] Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning (CVPR23)\n[2] Controlling Text-to-Image Diffusion by Orthogonal Finetuning (NeurIPS 2023)\n\nLIMITATIONS:\nThe work is on semantic segmentation and there is no qualitative comparison shown in the main paper. There are some visuals in the supplementary, but most of those are from test set and there is no comparison shown with the baselines and existing methods, so it is not clear where the improvement is coming from. It will be good to see how the results improve with and without alignment.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper presents H-CLIP, a novel approach for parameter-efficient fine-tuning of the CLIP model in hyperspherical space, specifically for open-vocabulary semantic segmentation. H-CLIP includes the introduction of a symmetrical parameter-efficient fine-tuning strategy, leveraging hyperspherical energy principles. And a dual cross-relation communication module is utilized to enhance cross-modal and cross-layer alignment.

STRENGTHS:
- This paper is well-motivated. The proposed H-CLIP effectively addresses common issues in fine-tuning CLIP.
- The paper effectively argues that maintaining the hyperspherical energy helps preserve the model's generalization ability, a critical factor in multi-modal tasks.
- The ablation experiments are thorough and effectively support the arguments.

WEAKNESSES:
- The writing needs improvement. The introduction lacks transitions from existing problems to the approach of this paper, such as introducing the advantages of Hyperspherical Space. 
- Some formula descriptions can be optimized, for example, explaining the meaning of * in Formula 9. 
- Details about comparison methods are needed. In Table 1, the compared method SAN includes an additional backbone.

QUESTIONS:
This work focuses more on cross-modal alignment and the issues related to hyperspherical space, rather than on custom designs for pixel-level predictions. It appears to be more of a generalizable cross-modal fine-tuning paradigm. Have the authors attempted to validate it on tasks beyond open-vocabulary semantic segmentation?

LIMITATIONS:
The authors provide no analysis of the limitations and broader impact. The author can analyze the limitations of this fine-tuning strategy in the field of OVS.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes H-CLIP, a symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in hyperspherical space for both of the two CLIP modalities. The PEFT strategy is achieved by a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices. Extensive evaluations across various benchmarks show that H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4% of the total parameters of CLIP.\n\nSTRENGTHS:\nThis paper achieves SOTA performance.\nThe Parameter-efficient Fine-tuning is explained by tensor computation.\n\nWEAKNESSES:\n1.The novelty is limited.  Partial orthogonal fine-tuning (POF) doesn't directly address the challenges of OVSS but rather offers a generic PEFT approach, so what is the difference between POF and OFT[1]? In Equ.5, which module's weights are used by the pre-trained weight matrix, Q(K or V)’s projection layer or FFN? Method details need detailed explanation.\n2.Some concerns about DCRC. In this section, the author discusses the use of two k layers deep neural network to update the fourth-order tensor in Equ. 7, and provides some mathematical proof. However, these proofs only show that reversible transformations S(·) can be replaced by reversible matrices S (as shown in Equ. 11,12,14,15), and the authors use k layers deep neural network to replace such reversible matrices S, which cannot explain the meaning of reversible transformations. In other words, why adopting reversible transformations to update the fourth-order tensor in Equ. 7, and what is the role of reversible transformations? Is this approach also work in other fields other than semantic segmentation tasks? In addition, If the block diagonal structure is not adopted, Equ. 16 seems to require only one reversible matrices S4 for mapping. Does this reduce the number of parameters?\n3.Insufficient experimental analysis. \n1)The decoder of HCLIP seems to be learnable as well. Does the param in Table 2 calculate the decoder part? And, Is the proposed PEFT method applicable to various decoders? If it is replaced with linear probe, is the proposed method still effective? Need further exploration.\n2)If a different VFM is adopted (not CLIP), is the proposed method still valid?\n3)The proposed method should be compared with more PEFT methods such as VPT, Adapter, LST, SSF [1-5] .\n\n[1] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320–79362, 2023.\n[2] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022.\n[3] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019.\n[4] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991–13005, 2022.\n[5] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109–123, 2022.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nThe limitation of the proposed method should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method called Parameter-Efficient Fine-Tuning in Hyperspherical Space for efficiently solving the open-vocabulary semantic segmentation problem. The method introduces a series of efficient block-diagonal learnable transformation matrices and a dual cross-relation communication module among all learnable matrices. To maintain the generalization ability offered by the CLIP text encoder, the authors designed a constraint to PEFT based on Hyperspherical Energy. Comprehensive results on open-vocabulary semantic segmentation benchmarks demonstrate the strong performance of this PEFT method by training only 4% of the total parameters of CLIP.\n\nSTRENGTHS:\nThe idea of introducing hyperspherical space to achieve parameter-efficient training is interesting. This approach attains state-of-the-art performance on current open-vocabulary semantic segmentation benchmarks with fewer learnable parameters. Additionally, it demonstrates better parameter efficiency than LORA on open-vocabulary semantic segmentation tasks, as shown in Table 3.\n\nWEAKNESSES:\nI do not see any clear weaknesses. However, I acknowledge that I am not familiar with hyperspherical theorems.\n\nQUESTIONS:\n* Although I understand this is a parameter-efficient training strategy, is it possible to provide the training time for this method? I am interested in the training time efficiency perspective. It would be better to also provide a comparison of the time with other methods. (No need for complete training during rebuttal, just ensure the number of iterations is the same and compare the time and results.\n\n* From the segmentation results in Table 3, this method outperforms LoRA. However, since LoRA is widely validated on other tasks, demonstrating the effectiveness of this method on other CLIP-based tasks would provide strong evidence to support its effectiveness and potential broad impact.\n\nLIMITATIONS:\nSee in questions. No other clear limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}