WaveDN: A Wavelet-based Training-free Zero-shot Enhancement for Vision-Language Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:15:22

Abstract: Vision-Language Models (VLMs) built on contrastive learning, such as CLIP, demonstrate great transferability and excel in downstream tasks like zero-shot classification and retrieval. To further enhance the performance of VLMs, existing methods have introduced additional parameter modules or fine-tuned VLMs on downstream datasets. However, these methods often fall short in scenarios where labeled data for downstream tasks is either unavailable or insufficient for fine-tuning, and the training of additional parameter modules may considerably impair the existing transferability of VLMs. To alleviate this issue, we introduce WaveDN, a wavelet-based distribution normalization method that can boost the VLMs' performance on downstream tasks without parametric modules or labeled data. Initially, wavelet distributions are extracted from the embeddings of the sampled, unlabeled test samples. Subsequently, WaveDN conducts a hierarchical normalization across the wavelet coefficients of all embeddings, thereby incorporating the distributional characteristics of the test data. Finally, the normalized embeddings are reconstructed via inverse wavelet transformation, facilitating the computation of similarity metrics between the samples. Through extensive experiments on two downstream tasks, using a total of 14 datasets covering text-image and text-audio modal data, WaveDN has demonstrated superiority compared to state-of-the-art methods.

Main paper: 
 introduction: 
 Large-scale self-supervised pre-training has been extensively validated as highly effective and reliable through notable contributions such as GPT-4 [1] , ERNIE [2] , Sora [3] , and CLIP [4] . The use of pre-trained models on open-set tasks has shown promising results, with transferability that rivals supervised methods designed for closed-set tasks. In the field of computer vision, Contrastive Language-Image Pre-training (CLIP) [4] has demonstrated its efficient and accurate open-set recognition capability across various Figure 1 : Training-free zero-shot enhancement using WaveDN . We propose WaveDN, a method for enhancing the zero-shot transfer abilities of VLMs, without training. By performing wavelet transformation and coefficient feature extraction on a small amount of unlabeled test data embeddings, the estimated wavelet distribution coefficients are obtained. Before calculating similarity, utilize the estimation coefficients from the previous stage to transform each embedding using hierarchical normalization, thereby enhancing performance during testing.visual tasks such as image classification, instance segmentation, object detection, and more downstream tasks. CLIP is trained on an extensive image-text corpus utilizing the InfoNCE Loss [5, 6] , which enhances the similarity among positive sample pairs while diminishing the similarity among negative sample pairs, thereby establishing a meaningful alignment between the visual and linguistic modalities. During model evaluation, solely the dot product between image and text features is requisite for computing their similarity.Presently, a multitude of advancements have been implemented to refine CLIP, including methods rooted in few-shot learning [7] [8] [9] and model fine-tuning [10] [11] [12] , all aimed at augmenting CLIP's efficacy across downstream tasks. Nevertheless, these strategies, which often necessitate supplementary parameter modules or rely on annotated test data, pose significant obstacles to the inherent efficiency and generalization capacity of CLIP. To circumvent the introduction of extra parameters and labeled data, testing enhancement techniques like CALIP [13] and Distribution Normalization(DN) [14] have been introduced. These methods focus on enhancing the zeroshot capability of the model by refining and optimizing the features extracted by the text and image encoders.The research of DN [14] indicates that, during training, the model takes into account information from both positive and negative sample pairs. However, during testing, using only the dot product between two modals' embeddings is a zeroth-order approximation of the InfoNCE loss. The absence of distribution data from the test dataset in similarity calculations results in information loss. This dot product-based similarity calculation method has not fully utilized the model's capabilities. The DN method introduces the mean of test samples during the similarity calculation phase to align the loss functions between testing and training. However, while this method partially addresses the issue of misaligned testing loss functions, performing operations on features with high semantic information, such as subtracting the mean, will lead to additional information loss. The features of the test sample will be compromised due to additional processing. The results in Figure . 2 can better illustrate this point. After implementing the DN, the proportion of classes exhibiting positive effects surpasses those with negative effects, yielding an enhancement in overall recognition accuracy at the dataset level. Nonetheless, at the individual class accuracy level, a considerable fraction of categories exhibit reduced recognition accuracy, with reductions reaching up to 30%. This decline is attributed to substantial degradation in the representation quality of the test data.To tackle this issue, we introduce WaveDN, which employs wavelet transformation to decompose the test samples' embeddings, facilitating a more detailed hierarchical normalization process that maximally preserves the features of the test samples when introducing the test data distribution. The method's brief process is shown in Figure .1. Initially, a wavelet decomposition based wavelet coefficient mean estimation is performed. Similar to DN, WaveDN's mean estimation only requires sampling a small portion of unlabeled test data to obtain a convergent estimation result, as confirmed by experiments. Subsequently, during the inference stage, the previous stage's wavelet coefficients are used to hierarchically normalize the embeddings of the text or images to be predicted. In this process, each embedding is transformed into multiple wavelet coefficients, providing a more detailed reflection of its information. A weighted normalization using wavelet coefficient similarity is then conducted, aiming to preserve the information content of the test data while introducing data distribution information. As shown in Figure .2, WaveDN has a lesser negative impact on fewer classes and a positive impact on more classes, thereby to some extent avoiding representation degradation of test samples and achieving better utilization of test data.The overall process of the WaveDN framework is zero-shot, parameter-free, and does not require additional training, which is consistent with CALIP [13] and DN [14] . Wavelet transformation can effectively handle encoding results with high information density. By considering embedding generated by vision-language models as a time series signal and performing wavelet decomposition, information at different frequencies and times can be obtained, providing a more precise and reliable reference for subsequent transformations. Through our meticulous calculations, WaveDN has demonstrated the capability to strike a delicate equilibrium between aligning the loss function through the incorporation of negative sample information and mitigating information loss in normalized test samples, consequently mitigating the adverse effects of sample mean introduction. As depicted in Figure 2 , our methodology demonstrates enhanced efficacy relative to DN, achieving favorable results devoid of significant adverse effects, such as a marked reduction in recognition accuracy across specific categories. WaveDN not only augments the overall accuracy of category recognition but also preserves the generalization capabilities of VLMs without undermining their robust generalization capacity.To substantiate the efficacy and supremacy of our approach, we employed WaveDN in the context of two downstream tasks: Zeroshot classification and Cross-modal Retrieval, juxtaposed against cutting-edge methodologies. Our experimental framework encompassed the utilization of diverse CLIP visual encoder architectures, with a focus on augmenting modal alignment across text, image, and audio pre-training encoders. Findings derived from evaluations conducted on 12 image datasets and 2 audio datasets unequivocally underscore the overarching superiority of WaveDN. Notably, for some benchmarks, it surpassed certain existing approaches that necessitate few-shot fine-tuning. Furthermore, comprehensive ablation analyses corroborate the computational efficiency and the relatively straightforward and robust implementability of our methodology.The main contributions of WaveDN:• WaveDN creatively employs wavelet transformation to decompose the information-rich embeddings extracted by the VLM model, supporting more detailed analysis and computations for distribution normalization.  
related work 2.1 vision-language models' downstream augmentation: 
 This zero-shot transfer capability positions CLIP [4] on par with supervised models tailored for specific tasks. Efforts to augment the prowess of Visual Language Models have spurred the development of innovative methodologies. Prompt-based approaches [7, 9, 15] , for instance, strive to refine input data, bridging the gap between the distribution of test data and that of the training data for VLMs. Fine-tuning techniques [10, 11, 16] have also emerged as a means to enhance performance on downstream tasks while preserving the original generalization capacities of VLMs. In a bid to enhance the accessibility and versatility of models, researchers have proposed parameter-free and training-free methods. Noteworthy examples include CALIP [13] , which introduces a parameter-free attention mechanism, and SuS-X [17] , which adopts a name-only strategy to construct image support sets. There are also some works [18, 19] that leverage the power of large language models to enrich text labels, maximizing information utilization during testing. Moreover, Distribution normalization [14] introduces a novel approach aligning InfoNCE loss [20] during testing, effectively harnessing distribution information from test data without the need for labeled data. In this paper, we explore a distribution estimation approach utilizing wavelet decomposition, accompanied by a hierarchical normalization method that exerts minimal impact on the features of test data relative to the DN method.Currently, many studies utilize Discrete Wavelet Transform (DWT) to enhance the efficiency of deep learning models. The Grid-based neural radiation field wavelet transformation method [21] diminishes the computational demands of neural rendering while preserving the benefits of supplementary data structures. Discriminative Wavelet Sub-bands [22] utilize unsampled 2D discrete wavelet transformation for the recognition of deformed facial images. HifaFace [23] employs wavelet transformation to convert images into multiple frequency domains, wherein the high-frequency components facilitate the recovery of detailed features. NEGAN, a noise prior learner [24] , predominantly aligns the high-frequency segments of noisy images using discrete wavelet transformation (DWT). DaReN-eRF [25] implements Inverse Dual-Tree Complex Wavelet Transformation (DTCWT) to reconstruct plane-based information, with features for each spatiotemporal point derived by amalgamating vectors from these reconstituted planes. The DWT-FFC frequency branch [26] leverages DWT to extract complex high-frequency features. In our research, we treat the embeddings produced by VLM encoders as one-dimensional discrete time series signals. We apply discrete wavelet transformation to these signals, extracting the wavelet distribution coefficients. This process decomposes embeddings, which are rich in semantic information, into finer scales, thereby facilitating more precise calculations and hierarchical normalization operations. 
method 3.1 preliminary: 
 3.1.1 CLIP. CLIP is a large-scale visual language model pre-trained on 400 million text-image pairs. To achieve cross-modal feature extraction and prediction, CLIP consists of a text encoder and a visual encoder, donated as 𝐸 𝑇 and 𝐸 𝐼 . The text segment employs a transformer encoder [27] for text encoding, whereas the image segment utilizes ResNet [28] or Vision-transformer [29] structures for image encoding, enabling it to excel in cross-domain image and text processing. For an image 𝑖 and a text 𝑡, the image embedding 𝐼 ∈ 𝑅 1×𝐶 and text embedding 𝑇 ∈ 𝑅 1×𝐶 can be obtained as follows:EQUATIONEQUATIONAs shown in Eq. 3, the similarity between text embedding and image embedding is calculated through dot product. In downstream tasks, the similarity used for prediction is obtained through the dot product of text embeddings and image embeddings after L2 normalization.EQUATIONFor the image recognition task, the goal is to determine which of the 𝐾 categories the test image belongs to. The text prompt contains the names of the 𝐾 categories as tokens. The encoded 𝐹 𝑇 = {𝑇 1 ,𝑇 2 , . . . ,𝑇 𝐾 } contains 𝐾 embeddings, where the j-th vector in the set 𝐹 𝑇 ∈ 𝑅 𝐾 ×𝐶 represents the features of prompts containing the name of the j-th category. For a test image embedding 𝐼 1 ∈ 𝑅 1×𝐶 . Calculating similarity using Eq.3 results in a similarity matrix 𝑀 ∈ 𝑅 1×𝐶 , where the text prompt corresponding to the highest similarity value in the matrix is the classification result for this image.During training, CLIP utilizes the Information Noise Contrastive Estimation(InfoNCE) [5, 6] loss function, which considers both positive and negative sample pairs. However, during testing, only the dot product calculation using Eq. 3 is used as a similarity calculation method, which has been proven in DN [14] to be a zeroth-order approximation of InfoNCE loss. This research also highlights that using zeroth-order approximation to calculate similarity leads to the loss of information regarding the distribution of negative samples, which is crucial for predictions. To alleviate this issue, during the prediction phase, the DN method replaces the approach of Eq. 3 with the similarity calculation method from Eq. 4 to introduce distribution information of negative sample pairs, which has been proven to be a reliable first-order approximation of InfoNCE loss. 𝜇 𝐼 and 𝜇 𝑇 are estimated mean values obtained through sampling a small number of test samples' embeddings. Once the wavelet basis is selected, the low-pass filter coefficients ℎ 𝑘 and high-pass filter coefficients 𝑔 𝑘 can be determined. The first-level decomposition in the iterative process of wavelet decomposition is as follows: For each subsequent level 𝑢 (2 ≤ 𝑢 ≤ 𝑙), the same decomposition process is applied to the previous approximation coefficient 𝑐𝐴 𝑢 -1 to obtain new approximation coefficient 𝑐𝐴 𝑢 and detail coefficient 𝑐𝐷 𝑢 :EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONThe highest level wavelet approximation coefficient 𝐴 𝑙 and all wavelet detail coefficients {𝐷 𝑙 , 𝐷 𝑙 -1 , . . . , 𝐷 1 } can represent all the information of the original signal. In the following text, we can represent the process of discrete wavelet transformation(DWT) as Eq. 9, where the embedding 𝜈 output by the encoder is decomposed through DWT into 𝑙 + 1 wavelet coefficients E = {𝑒 1 , 𝑒 2 , . . . , 𝑒 𝑙+1 }.EQUATIONThe inverse discrete wavelet transform(IDWT) can be performed by combining the scaling function 𝜃 𝑗,𝑘 (𝑡) and wavelet function 𝜑 𝑗,𝑘 (𝑡) of the selected wavelet basis as shown in Eq. 10. In the following text, we can abbreviate this process as shown in the form of Eq. 11EQUATIONEQUATIONIntroducing a distribution with negative sample information is crucial during the testing phase to align with InfoNCE loss. However, it is essential to minimize the disruption of the rich information content in the embeddings. Based on this concept, we propose the WaveDN method, as illustrated in Figure . 3. It involves two main steps: extracting wavelet distributions and hierarchical normalization. The former aims to estimate a distribution with less information loss compared to directly taking the mean. The latter incorporates this distribution estimation into the similarity calculation process during the testing phase to utilize information from the negative samples while also ensuring the maximum preservation of the test sample' embeddings. EQUATIONEQUATIONDecomposing the embedding into multiple wavelet coefficients, each representing partial information of the embedding at different scales and positions. Through wavelet decomposition, the embedding with high information density is to some extent unfolded. The process of extracting the mean distribution is to take the average of wavelet coefficients at the same level. By performing operations as shown in Eq. 14 and Eq. 15, wavelet average distributions of images A 𝐴𝑣𝑔 and texts B 𝐴𝑣𝑔 are obtained for the hierarchical normalization in the next step. This feature extraction method, first performing DWT and then taking the mean, can maximize the preservation of local information for each sample. Compared to directly averaging the embedding, which can only provide overall average information, our method can achieve more accurate distribution feature extraction. A 𝐴𝑣𝑔 = { 𝑁 𝑖=1 𝛼 𝑖,After performing hierarchical normalization computation as shown in Algorithm 1, the normalized coefficient sets A ′ and B ′ are obtained. Specifically, the hierarchical normalization operation involves applying a weighted normalization to each wavelet coefficient sequence in the wavelet coefficient set, calculating the similarity between the wavelet coefficients of each layer of the test sample and the mean wavelet coefficients extracted earlier for the corresponding layer as the normalization weight. Also, there is a fixed constant 𝜆 that will be used in the normalization process, whose value is determined through experiments.By performing weighted normalization of wavelet coefficients at different levels, our method can execute normalization operations of different intensities at different levels. For a certain level of wavelet coefficients, if its similarity with the mean wavelet coefficients of the corresponding level is lower, it indicates that this level contains more information representing the sample itself, which should be preserved to a greater extent. Through this hierarchical normalization with different intensities, our method can introduce distribution information while retaining more specific information about the test samples.Using IDWT, transform the variables from A ′ 𝑘 and B ′ 𝑘 obtained above to the same dimensions as the original embeddings. Obtain normalized sets of image embeddings 𝐹 ′ 𝐼 and text embeddings 𝐹 ′ 𝑇 , then calculate the similarity matrix of the two using the Eq.20 for the final result prediction.EQUATIONEQUATIONEQUATION 
experiments: 
 In the experimental section, we conducted experiments on 14 datasets with various VLM backbones for two downstream tasks. We aim to discuss and validate the following point through experimentation. ′ = HierarchicalNormalization(A, A Avg , A ′ ). B ′ = HierarchicalNormalization(B, B Avg , B ′ ). Output: A ′ , B ′ .structures? 2) Compared to current advanced VLM testing enhancement methods, does our method have superiority? Does WaveDN mitigate the deficiencies of DN? 3) Will our method be affected by parameter settings? How much impact will the selection of sampling quantity and related wavelet parameters have on the effectiveness of WaveDN? Does the method possess robustness? 4) During the testing phase, is the computational cost significant for applying wavelet transform and hierarchical normalization to all embeddings? Does WaveDN result in a substantial computational overhead?4.1.1 Zero-shot classification. This task involves using a pre-trained VLM model directly to predict the category to which the test samples belong. Our experiments include zero-shot recognition of both image and audio data. For image data, each category is represented by a fixed text template input to the text encoder to obtain the corresponding feature representation. The image encoder encodes the image to get the image feature representation, then calculates the similarity with the text embedding to determine the category with the highest similarity as the prediction result. The image dataset used includes: ImageNet1K [30] , Cifar100 [31] , SUN397 [33] , Standford Cars [32] , Flowers102 [37] , Food101 [36] , DTD [34] , OxfordPets [35] , EuroSAT [38] and FGVCAircraft [39] . For audio data, the only difference from images is the use of an audio encoder for feature extraction. The audio dataset used includes UrbanSound8K [41] and ESC-50 [40] . To ensure the correctness of the test phase, we maintain a non-overlapping folds partition consistent with the one used Table 1 : Zero-shot Top1 Accuracy on 10 Image Datasets ImageNet [30] Cifar100 [31] Cars [32] SUN397 [33] DTD [34] Pets [35] Foods [36] Flowers [37] EuroSAT [38] Aircraft [39] Average CLIP(Vit-B32) [4] 59. by the original authors for both audio datasets. For all the zero-shot experiments, we report Top1 accuracy(Acc@1) on each dataset.4.1.2 Cross-modal Retrieval. Cross-modal querying task involves using a sample from one modality to query samples from another modality. It can be said that the classification task is a subtask of the cross-modal querying task. In our experiments, we conducted three cross-modal zero-shot querying experiments: text-to-image, imageto-text, and text-to-audio. For image-related tasks, we utilized the Flicker30k [43] and the COCO [44] , while for audio-related tasks, we used the UrbanSound8K [41] and ESC-50 [40] datasets. For retrieval experiments, the setup details for image-related datasets align with DN [14] , while for audio-related datasets, our setup aligns with AudioCLIP [42] . We report the results of each experiment in terms of recall@1(R@1), recall@5(R@5) and recall@10(R@10).During the experiment, we extensively tested various image and audio encoders. For image-related tasks, we utilized two different CLIP encoders, VIT-B32 and RN50, with implementation, image data preprocessing, and pretraining weights sourced from CLIP [4] 's open-source content. For audio-related tasks, we employed the model implementation, training weights, and audio data sampling and preprocessing methods publicly available in AudioCLIP [42] , conducting experiments with both full-training and partial-training audio encoders. Across each dataset, in addition to testing with WaveDN, we also conducted experiments based on DN [14] 's opensource implementation and hyperparameter settings to test the DN on all datasets for deeply comparing the differences between the two methods. Due to the need for both WaveDN and DN to sample test samples and then estimate distributions, and since random sampling results are inherently random, all our experimental results are obtained by conducting experiments on 5 randomly selected random seeds and averaging the results, aiming to minimize the randomness of the experimental results. During the distribution feature estimation, we sampled only 100 test samples for both, consistent with the approach used in DN [14] . For both methods, the samples used for distribution estimation are sourced from the dataset's test set to avoid introducing additional information and to simulate a realistic testing process, rather than from the validation set.WaveDN uses Daubechies Wavelet as the wavelet basis for wavelet decomposition. Due to its hierarchical nature, it is well-suited for discrete wavelet decomposition. In the experiments, WaveDN consistently utilizes the "db6" wavelet basis with a decomposition level of 5. The impact of wavelet basis selection and decomposition levels on the effectiveness of WaveDN is extensively discussed in the R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 CLIP(Vit-B32) [4] 51 ablation experiments section. The choice of wavelet basis and decomposition levels may have a slight impact on the experimental results. However, all combinations have a positive impact on the experimental results.3.1 Zero-shot classification results. The results of the zero-shot experiment for images are shown in Table . 1. We conducted a performance comparison on two types of CLIP image encoders, DN and WaveDN. WaveDN demonstrated significant improvements over the baseline across all datasets and outperformed DN. In the 14 datasets tested, WaveDN increased the performance by an average of 1.31% on Vit-B32 and 1.86% on RN50. The comparative effectiveness is illustrated in Figure . 4 , where WaveDN shows superior overall performance, especially notable on the EuroSAT dataset with improvements of 5.39% and 7.53% on two different architectures. However, the results on Pets and Flowers indicate that the DN-related methods have negative optimization, possibly because CLIP (Vit-B32) itself has already adapted well to the data distribution of the test datasets. Additional operations on the already well-represented data may lead to information loss. Nonetheless, WaveDN reduces this negative impact compared to DN. Moreover, experimental results on a large number of datasets also show that this situation of negative optimization on the final results occurs less frequently.The experimental results for zero-shot audio classification are shown in Table . 2. WaveDN demonstrates significant improvements on both Full-training and Partial-training audio encoders, with increases of 5.21% and 2.82% respectively compared to the baseline. Notably, on UrbanSound8k, the accuracy of full-training audio recognition increased significantly by 5.34%. WaveDN demonstrates good adaptability for audio-related classification problems.. We conducted image-to-text and text-to-image retrieval experiments on MSCOCO and Flicker30k, with the experimental results shown in Table . 3. For all image encoder structures, using DN or WaveDN had a positive impact on all retrieval tasks. Moreover, WaveDN showed improvement over DN in most experimental results, reflecting WaveDN's support for various downstream tasks to some extent. We also observed that in some cases where the results were not ideal, such as the image-to-text retrieval of CLIP(RN50)+WaveDN on Flicker30k, DN performed better. This could be due to the unique data distribution Text → Audio R@1 R@5 R@10 R@1 R@5 R@10 AudioCLIP(Partial-training) [42] Experimental text-to-audio retrieval was conducted on ESC-50 and UrbanSound8k datasets, with the experimental results shown in Table . 4. Since both datasets only had a single text label for each audio, the audio-to-text retrieval experiment essentially overlapped with the zero-shot classification experiment. Therefore, we focused solely on text-to-audio retrieval. WaveDN demonstrated significant improvements across all results and managed to avoid the negative impact of DN methods on cross-modal retrieval. For instance, in the ESC-50 experiment with AudioCLIP (Partial-training), while DN improved the R@1 performance, there were decreases in R@5 and R@10. This was attributed to DN disrupting the intrinsic feature representation of samples. WaveDN, on the other hand, circumvented this issue and exhibited better performance across all metrics.4.4.1 Convergence of Distribution Extraction. Due to the need for WaveDN to sample multiple unlabeled test samples for estimating test data wavelet distributions, the quantity of sampled samples will have a significant impact on the distribution estimation results, thereby affecting the method's effectiveness. Therefore, we conducted sampling quantity experiments using CLIP(Vit-B32) to extract text and image features on Cifar100.The experimental results, as shown in Figure . 5, indicate the similarity between the wavelet distribution estimated using the corresponding sample quantity and the true distribution. The true distribution is obtained by performing wavelet transform and feature extraction on all samples. The results show that when the sampled samples approach 10, the similarity between the sampled estimated distribution and the true distribution reaches 97.5%, and when the sample quantity exceeds 40, the estimated distribution has already converged to the true distribution. These results indicate that only a small amount of unlabeled samples is needed to achieve effective distribution feature estimation. In our experiment, we uniformly used a sample quantity of 100 to ensure consistency with DN's sampling quantity. The experimental results in Table . 5 show the influence of different selection schemes on the experimental results. All the results are averaged by experiments on 10 random seeds. Compared with the 64.1% top1 accuracy of CLIP(Vit-B32) on Cifar100, all the selection schemes of wavelet bases and layers after using WaveDN can improve the recognition accuracy, which proves the universality of our method. Different combinations of wavelet bases have a weak influence on the results, and we finally completed all our experiments with the combination of "db6" and "level 5", which is a combination with good adaptability to the characteristics of embedding. When performing zero-shot image classification on the Cifar100 dataset, the computational overhead introduced by WaveDN averages only 0.0001 GFlops per sample. Compared to the original calculation method without WaveDN, the computational overhead for CLIP (Vit-B32) increased by only 0.0022%, and for CLIP (RN50) it increased by just 0.0017%. This further demonstrates the efficiency of our proposed method, as it only requires a small amount of computational overhead to enhance the performance of VLM on multiple downstream tasks.  
conclusion: 
 In this paper, we introduce WaveDN, a parameter-free, training-free method for enhancing the zero-shot capability of VLMs. WaveDN innovatively interprets the embeddings as a time series signal, subsequently applying discrete wavelet transform for feature extraction. By estimating feature distribution and employing hierarchical normalization, WaveDN aligns with the InfoNCE loss during the testing phase, without compromising the original sample feature representation. WaveDN requires only a small number of unlabeled test samples, with computational costs negligible compared to the entire inference phase. Experiments and results in multimodal settings also demonstrate WaveDN's beneficial effects on multiple downstream tasks with various VLMs. WaveDN is influenced by sample distribution in specific scenarios, indicating that future research could beneficially focus on this aspect. Improved methods for sampling and distribution estimation may exist and warrant exploration. 
 
 Appendix:  


Decision: This paper introduces WaveDN, a method to enhance the zero-shot transfer capability of the Visual Language Model (CLIP) using wavelet transform. WaveDN aligns the InfoNCE loss with the test data feature distribution by hierarchically normalizing the mean wavelet distribution of a few test samples directly during the test phase, without adding new model parameters or requiring additional training. Extensive experiments across various benchmarks demonstrate the effectiveness of the design. Although Reviewer ufbn questioned the novelty of this work, the authors have addressed most of the concerns during the rebuttal. Besides, most reviewers acknowledge the motivation, presentation, and extensive validation of the proposed method. Consequently, the AC recommends to accept this submission.

Reviews: review: paper_summary: paper_summary: This paper proposes a parameter-free, training-free, and computationally efficient VLM zero-shot enhancement method. It achieves similarity calibration by extracting different levels of wavelet coefficients from an unlabeled test set and performing hierarchical normalization on the test set data features.
strength_weakness: strength_weakness: Advantages: 
1.	The method is straightforward and achieves relatively good results by normalizing the distribution of the dataset from the perspective of wavelet coefficients. 
2.	The experimental content is relatively complete, with comparative experiments conducted on 14 datasets, and ablation studies on various components of the method are also quite comprehensive and thorough.
limitations: limitations: Disadvantages:
1.	There are concerns regarding the experimental setup. The number of comparative methods is too few; the paper should compare its method with other approaches mentioned in related work, such as those in references [13], [17], [18], and [19]. 
2.	Generally, zero-shot learning allows access to only one test sample at a time, whereas this paper’s setup allows access to most of the test samples at once.
paper_summary: paper_summary: This paper proposed a method, WaveDN, to enhance the zero-shot transfer capability of Visual Language Model (CLIP) using the wavelet transform. WaveDN achieves the alignment of InfoNCE loss and test data feature distribution by hierarchically normalizing the mean wavelet distribution of a few test data samples directly in the test phase, without introducing new model parameters and without training. Experiments on 14 image and audio datasets demonstrate the effectiveness and superiority of the approach.
strength_weakness: strength_weakness: 1. WaveDN applies wavelet transform before the similarity calculation process of VLM (CLIP), which can realize more accurate distribution feature extraction compared to DN, and has a certain novelty.
2. WaveDN applies wavelet-based feature extraction, which maximizes the preservation of local information about each sample, and hierarchical normalization, which, by controlling the strength of each level of normalization, introduces distributional information while better preserving more specific information about the test sample.
3. WaveDN is used directly in the testing phase without adding new model parameters or training, which does not affect the zero-shot capability of the VLM (CLIP) itself, and the computational overhead of WaveDN is very low.
limitations: limitations: 1. In line 491“Also, there is a fixed constant 𝜆 that will be used in the normalization process, whose value is determined through experiments.”, during the hierarchical normalization stage of WaveDN, please specify the method of obtaining the constant λ. Will WaveDN be sensitive to this constant λ? Is it possible to do a set of experiments to demonstrate the effect of different λ on the retention effect of distribution information and test sample-specific information?
2. While Table 1 in the main text demonstrates that WaveDN has a performance improvement in the DTD texture dataset compared to DN and Baseline, in Figure 1 in the Appendix, from a category perspective, WaveDN shows a large additional negative impact in the -0.06~0.00 interval compared to DN in the DTD texture dataset (as shown in the red box) and a large additional positive impact in the 0.12~0.18 interval (as shown in the green box), does this suggest that in some cases WaveDN will have a more extreme impact on some of the categories tested? Is this related to the randomness of the sampling?
paper_summary: paper_summary: This paper introduces a wavelet-based distribution normalization method for adapting current VLMs, which is a training-free approach. The method is based on previous work DN and further takes wavelet transform to minimize information loss during the feature normalization. Experiments on two tasks, namely zero-shot classification and cross-modal retrieval demonstrate promising results of the proposed method.
strength_weakness: strength_weakness: 1.	The paper is well-organized and easy to follow.
2.	The motivation to improving DN is reasonable and clearly presented.
3.	The idea of incorporating DWT into feature normalization is interesting.
limitations: limitations: Based on the statements in the paper, my main concern is that the experiments did not thoroughly verify the author's motivation.
1.	The motivation of using DWT is to minimize the information loss in feature normalization. Why and how can DWT achieve this purpose? Please elaborate on this.
2.	Based on 1., Could you provide visualization analysis to validate whether normalizing after DWT can reduce information loss compared to DN?
Some additional concerns are about the performance:
3.	Please make a performance comparison to CALIP.
4.	In Table 1, the performance increase with ViT-B/32 backbone compared to DN is limited on most datasets (less than 1%). 
5.	I am curious about the performance comparison between your method and the prompt tuning approach (e.g., CoOp). If possible, could you provide a simple comparison?
Based on the above strengths and concerns, my initial rating is borderline accept.
paper_summary: paper_summary: This paper proposes a plug-and-play module between the CLIP model and the final similarity metric. It applies the wavelet transform to the clip embeddings and then adopts a hierarchical normalization. The idea follows previous DN work in that it only changes the metric calculation so it just needs to sample from the test dataset instead of fine-tuning and retraining the CLIP model. The work conducted plenty of experiments and achieved somehow potential results.
strength_weakness: strength_weakness: 1. The paper is well presented.
2. The formulation of the method is formal and complete, with enough ablation studies to verify the components.
3. Lots of experiments demonstrate their efficacy (but not significant).
limitations: limitations: Though the paper is complete and the experiments are fair enough. I still have several concerns.

1. The novelty of this work sounds quite incremental to me, compared to the previous work [1]. Basically, this work just substitutes the mean with a wavelet distribution from a small portion of the test samples. They normalize the text and image distribution in the wavelet domain rather than direct multiplication in [1].
2. Further, the slight enhancement on many datasets strengthens my concern regarding the incremental idea. In Tab1, their improvements are less than 2%, mostly less than 1.5%. Further for large-scale datasets in Tab3, their effects compromise compared to DN.
3. Introducing wavelet transform as well as the necessary inverse transform will bring additional computation costs, I do not see any ablations regarding such computation costs analysis. Please add more ablations here. 
4. There is a presentation deficiency on L128 Page 2. I can not see any proof that the degeneration contributed to the representation quality. What I can see from Fig2 is that after using your method, it does somehow enhance the accuracy for some categories. But nothing to do with representation. So I am concerned about your design out of this motivation that is to use wavelet to enforce representation ability. And this motivation is further diminished by the slight performance.
5. Some of the arguments are not convincible. 1) In the abstract, why the training of an additional parameter module would impair the existing transferability of VLMs? From my perspective, those adapters are quite flexible to downstream tasks. 2) On L120 of Page 2, how to prove the additional information loss, any proof about these, citations and ablations and experiments. and what are the compromised features, more explanation is required.
6. Related works should be enhanced. 1) the distinction between your work and previous works is not clear, further, you should better present your novelty. 2) Further, which wavelet method do you choose, and why do they have more semantic modeling capability? What is the purpose you choosing it? Compared to others, why is it the one? More explanation here.

[1]Test-Time Distribution Normalization for Contrastively Learned Vision-language Models