AVHash: Joint Audio-Visual Hashing for Video Retrieval

Authors: yuxiang  zhou
zhe  sun
rui  liu
yong  chen
dell  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:41:21

Abstract: Video hashing is a technique of encoding videos into binary vectors, facilitating efficient video storage and high-speed computation. Current approaches to video hashing predominantly utilize sequential frame images to produce semantic binary codes. However, videos encompass not only visual but also audio signals. Therefore, we propose a tri-level Transformer-based audio-visual hashing technique for video retrieval, named AVHash. It first processes audio and visual signals separately using pre-trained AST and ViT large models, and then projects temporal audio and keyframes into a shared latent semantic space using a Transformer encoder. Subsequently, a gated attention mechanism is designed to fuse the paired audio-visual signals in the video, followed by another Transformer encoder leading to the final video representation. The training of this AVHash model is directed by a video-based contrastive loss as well as a semantic alignment regularization term for audio-visual signals. Experimental results show that AVHash significantly outperforms existing video hashing methods in video retrieval tasks. Furthermore, ablation studies reveal that while video hashing based solely on visual signals achieves commendable mAP scores, the incorporation of audio signals can further boost its performance for video retrieval.

Main paper: 
 introduction: 
 In this digital age, short videos 1 have evolved into a prominent content form, emerging as a new narrative style, information source, communication mode, self-expression avenue, and interaction space, significantly influencing modern cognitive psychology, aesthetic experiences, and value preferences [27] . Therefore, the management and processing of videos are crucial yet challenging due to their vast scale and continuous expansion. Video hashing [28] , the technique that encodes a video into a binary vector, can not only reduce memory requirements but also facilitate further indexing and computation, making it a potential tool for video analysis and understanding [1, 45] .Current video hashing methods begin by summarizing a given video through a series of keyframes, which are then modeled using non-temporal or temporal networks such as MLP [29, 36] , CNN [21, 22] , LSTM [7, 16] , and Transformer [39] . To name just a few, MCMSH [13] models video feature contexts at three granularities, integrating them into an MLP-Mixer for a comprehensive representation; SRH [12] employs pre-trained CNN features of video keyframes as inputs to an LSTM network for binary code generation; DSVH [2] utilizes 3D convolutions on spatio-temporal keyframes via a 3DCNN to create {0, 1}-embeddings; ConMH [42] leverages a Transformer encoder-decoder on sequential pre-trained CNN features to learn binary codes of videos in a self-supervised fashion. It is evident that the underlying assumption of these approaches is as follows. The current computing capabilities face challenges, or are relatively insufficient, for processing all frames of a video clip, including short videos lasting just a few minutes. However, due to the high redundancy in video frames, summarizing a video's semantic content through keyframe extraction is an effective strategy.Furthermore, we also observe that existing approaches to video hashing tend to view videos as Silent Films, neglecting a crucial element -audio signals -which are prevalent today and vitally important for full video comprehension. Neuroscience research [8] suggests that the combined effect of multiple sensory stimuli can produce an effect where "the whole is greater than the sum of its parts". For example, the concurrent stimulation of sight and sound during the viewing of TV shows or movies can evoke deep and holistic emotions, as well as enhance understanding of the content. In the context of machine learning, the principle that multi-modal learning works better than single-modal has also been mathematically analyzed and empirically validated by Huang et al. [17] . This inspires our investigation into video hashing that integrates both audio and visual signals, instead of solely relying on frame images.Given a video with synchronized audio-visual signals, we first extract N (e.g., 25) keyframe images and divide its audio into N sound waveform segments; then, we map them to a shared latent space through separate visual-Transformer and audio-Transformer networks respectively; in the end, we combine the audio-visual embeddings into the ultimate video representation using a crossattention mechanism and an av-Transformer. This completes our method for translating a video from its original signals to its final semantic representation. Given its novel integration of audio and visual signals, we designate this method as "AVHash", whose primary contributions are highlighted below:• It distinguishes itself by learning to hash videos with both audio and visual components, not merely frame images, and introduces a three-tier Transformer network to map its separate original signals to a shared latent space and finally to the video representation. To the best of our knowledge, this is the first work that fully leverages both audio and visual elements for video hashing. • It engages in video-oriented contrastive learning within the final video space and also ensures semantic alignment between audio and visual signals in the common latent space. This approach stems from the understanding that a video's audio-visual signals can be considered not only separately but also collectively as a whole. • It is validated through comprehensive experiments on two public datasets, ActivityNet and FCVID, under various conditions against a number of state-of-the-art competitors. Notably, AVHash demonstrates significant improvements over those existing methods, showing that while visual-based video hashing already achieves high mAP scores, incorporating audio signals further enhances these metrics. 
related work: 
 Video hashing [26, 34, 35, 43, 44, 46] aims to generate a semantic binary code for each video, and therefore significantly reduce storage space and enhance computation speed. This process would be of great benefit to downstream applications including video retrieval, analysis, and understanding. Current video hashing methods first extract a video's representative image frames and then apply non-temporal or temporal models to create {0, 1} embeddings. The early approaches typically leverage a pre-trained CNN to extract features from video keyframes first, and then average them to obtain an overall representations. Subsequently, these methods employ image hashing techniques such as LSH [9] , ITQ [11] , SDH [24, 33] , COSDISH [19] , SCDH [4] , CSQ [47] , LTH [3] , and MDSH [40] to produce a binary vector, which serves as the video's hash code. More recent approaches predominantly use deep neural networks. For example, DVH [28] utilizes temporal pooling across sequential MLP outputs from video keyframes; SRH [12] captures keyframe CNN features and feeds them into an LSTM network for supervised learning; DSVH [2] extends 2DCNN to 3DCNN for modeling spatio-temporal keyframes; ConMH [42] inputs pre-trained CNN features of sequential keyframes into a Transformer encoder-decoder network, enabling self-supervised learning to hash.To sum up, existing learning-to-hash methods for video retrieval only utilize visual signals (i.e., frame images) of videos, overlooking a crucial component: audio waveforms. By contrast, our proposed AVHash technique learns to project videos into Hamming space using both visual and audio signals jointly.In addition, while AVHash utilizes multi-modal signals, it is fundamentally different from existing multi-modal hashing methods such as CLIP4Hashing [49] , S 2 BIN [31] , EDMH [5, 6] , and SDAH [30] . Those multi-modal hashing techniques aim to learn hash codes for each individual modality (e.g., text, image, or video) for cross-modal retrieval tasks (e.g., image-to-text, text-to-image, video-to-text, text-to-video searches). However, AVHash just learns one-modality's embeddings by fully utilizing its audio-visual components and only performs video-to-video retrieval task. Conversely, AVHash focuses on learning binary embeddings for a single modality, video, while leveraging its audio-visual components, for the purpose of performing video-to-video retrieval better. 
the proposed method 3.1 problem statement: 
 Given a set of V videos, denoted as S = {v i } V i=1 , our AVHash aims to learn a hash function that maps each video to a compact binary code.Specifically, each video is represented by its N keyframes {I j } N j=1 and N audio segments {A j } N j=1 . We extract ViT features [32] {f I j } N j=1for keyframes and AST features [10] {f A j } N j=1 for audio segments, where I and A correspond to the image and audio modalities respectively. These modal features are then fed into their respective branches of the AVHash architecture (see Fig. 1 ), where the features sequentially pass through separate Transformer encoders to generate image/audio embeddings, a cross-attention for inter-modal interactions, a gated-attention for feature fusion, a multimodal Transformer encoder, and conclude at a hash layer to produce the video's embedding. This embedding is then binarized into the final hash code b i ∈ {-1, +1} q using a Sign(•) function for video v i , where q represents the code length.Our AVHash consists of the following modules: "Input", " Modality-Specific Encoder", "Cross Attention", "Gated Attention", "Multi-Modal Encoder", "Hash Layer", and "Loss Function" (see Fig. 1 ). Image:(L S =1, H S =768, A S =12); Audio: (L S =1, H S =768, A S =12). 3: Cross-Attention I → A: (L I →A =1, H I →A =768, A I →A =12); A → I: (L A→I =1, H A→I =768, A A→I =12).FC layer (1536 × 2);Tanh(•); Softmax(•). 5: Multi-Modal Encoder Transformer Blocks (L M =1, H M =768, A M =12).6: Hash Layer (Binary-like) FC layer(768 × q); Tanh(•).Sign(•).Input. To enable the model to process data from these modalities, we use the AST [10] and CLIP [32] models to extract features from audios and images, respectively, resulting in {f A j } N j=1 ∈ R N ×768 and {f I j } N j=1 ∈ R N ×768 . Before feeding them into the subsequent modules, a CLS token needs to be added in front of the feature sequence, i.e.,EQUATIONEQUATIONwhere E A pos and E I pos correspond to the position encodings of timeseries audios and temporal keyframes, respectively.Modality-Specific Encoder. AVHash comprises two modalityspecific encoders, designed to capture the unique characteristics inherent to each modality. We employ standard Transformer as audio and visual encoders. Each encoder consists of L S Transformer layers, and each Transformer layer is mainly composed of Multi-Head Self-Attention (MSA) (with the number of multi-heads as A S ) and an MLP layer. The output of (ℓ -1)-th MLP layer is fed to the (ℓ)-th Multi-Head Self-Attention (MSA) layer. Layer Normalization (LN) is applied before each layer, and a residual connection is incorporated after each layer. The MLP layer contains two linear fully connected (FC) sub-layers with a non-linear GELU activation function:EQUATIONEQUATIONwhere ℓ = 1, . . . , L S , m ∈ {A, I } denotes the audio or visual modality. By following the above procedure, we could get the latent representation z L S m of video v i 's audio and keyframe features. Notice that the output z L S m of each encoder contains (1 + N ) parts corresponding to the preset token CLS and N audio/visual features. Then, we use the first part's embedding, dubbed z L S m [0], as the holistic representation and the second part's embedding, dubbed z L S m [1:], as the representation for each feature. Cross-Modality Attention. After unimodal feature encoding, we employ a cross attention to capture the semantic interactions EQUATIONwhere m 1 denotes the target modality and m 2 is the source modality, and the MCA is given as follow:EQUATIONEQUATIONandEQUATIONEQUATIONEQUATIONwhere W, W Q i , W K i , and W V i are learnable matrice, and A c equals to A A→I or A I →A . Thus, we obtain representations enriched with cross-modal information z A→I , z I →A , by the following formula:EQUATIONEQUATION)Gated-Attention Fusion Module. To fuse semantics of different modalities, we introduce a gated attention to determine the importance of each modality. For simplicity, we implement this module via a linear layer followed by a Tanh activation function. This module accepts two modalities' features as input and concatenates the features of each modality by rows. The output of the module is fed into the Softmax function to ensure that the sum of weights across two modalities equals to 1:EQUATIONthe final fused features are then formulated as: 63M BTH [25] 2M SRH [12] 4.5M DKPH [23] 4M AVH [41] 22M SSTH [48] 5M AVHash (Ours) 28.4M ConMH [42] 11.3M --EQUATION)Multi-Modal Encoder. After fusing the two modality features, to obtain the final representation of the video, we introduce a multimodal Transformer encoder. This encoder consist of L M Transformer layers, with each composed of Multi-head Self-Attention (MSA) (the number of multi-heads is A M ) and an MLP layer. Before feeding z AI , a cls token is added at the beginning of z AI , and its corresponding output serves as the video's final representation:EQUATIONEQUATIONEQUATIONwhere ℓ = 1, . . . , L M . By following the above procedure, we can get the latent representation z L M V of video v i . Note that the output z L M V of each encoder also contains (1 + N ) parts corresponding to the preset token CLS and N multimodal features. We employ the first part's embedding, i.e., z L M V [0], as its final representation. Hash Layer. To obtain the final hash codes of video v i , we add a hash layer, i.e., a FC network with the Tanh(•) function, on top of the Multi-Modal Encoder. Formally, given video v i , we transform its Multi-Modal Encoder's output z L M V [0] into a q-dimensional binarylike real-valued vector g i via:EQUATIONwhich is then forwarded to the Sign(•) function for the final binary code b i , i.e.,EQUATION) Loss Function. We leverage the InfoNCE loss [14, 37] to train our network. InfoNCE loss consists of three types samples: an anchor sample a, a positive sample p, and n o negative samples {n i } n o i=1 . The anchor and positive samples belong to the same class, while the anchor and negative samples belong to different classes. This InfoNCE loss maximizes the agreement between positive pairs and the dis-agreement between positive and negative samples, which is formulated as:EQUATIONwhere sim(•, •) denotes the similarity function (e.g., cosine similarity), τ and n o are two pre-set hyper-parameters. In our experiments, we build 5 types of positive and negative sample pairs:EQUATIONandEQUATIONwhere L 1 is the semantic alignment between audio and visual signals in the common latent semantic space, and L 2 represents the video-based contrastive loss in the video space. Then, the overall objective can be constructed as:EQUATION)where α is hyperparameter. By minimizing the InfoNCE loss, AVHash can learn discriminative hash codes, thereby realizing efficient largescale video retrievals. The specific configurations of AVHash's architecture is provided in Table 1 , and AVHash is implemented and trained (from Layer-0 to Layer-6 in Table 1 ) via pytorch.Given a new video v oos , we can use the trained AVHash model to compute its real-valued binary-like vector:EQUATIONfirst and then binarize it into the final hash code:EQUATION 
experiments 4.1 datasets: 
 We assess AVHash using two public video datasets: ActivityNet 2 [15] and FCVID 3 [18] . ActivityNet comprises We evaluate the retrieval performance of different approaches using the most widely-adopted metric, mean Average Precision at top-K results (mAP@K) [5, 12, 41, 42] . For our study, we set K to 100.When conducting video retrieval experiments, we designate the testing samples as the query set, with the validation and training sets serving as the video database. Additionally, we order the retrieved results by their Hamming distances from the queries and record the performances of different methods with code lengths of 32 as well as 64 bits.Regarding baseline approaches, we include SSTH [48] , BTH [25] , DKPH [23] , MCMSH [13] , ConMH [42] , SRH [12] , AVH [41] , and DSVH [2] , chosen for their competitive retrieval performances. The first five methods are unsupervised, while the last three are supervised.Image processing. Consistent with SSTH [48] , BTH [25] , and ConMH [42] , we uniformly sample 25 frames from each video in the ActivityNet and FCVID datasets. Each image is resized to 336×336×3 and segmented into 14×14 patches. These images are then processed using the visual branch of CLIP, which features a depth of 24 layers, 16 multi-head attentions, and a hidden size of 1024, resulting in high-level image representations within a 1024-dimensional feature space. These features are subsequently transformed to 768-dimensional vectors via a fully connected layer. Audio processing. We also evenly divide each audio track from the ActivityNet and FCVID datasets into 25 segments, corresponding to the 25 frames. These audio segments are then processed using AST [10] , which is characterized by an embedding dimension of 768, 12 layers, and 12 heads, generating high-level audio representations within a 768-dimensional feature space.Network Architecture. Our architecture comprises three primary components: the Audio Encoder, the Image Encoder, and the Multi-modal Encoder. Each component employs a Transformer encoder architecture, configured with a single layer, 12 multi-head attentions, and a hidden layer size of 768. Notably, we limit each component to one layer to ensure that our AVHash network, which has 28.4M learnable parameters, remains comparable in size to competitors such as AVH [41] with 22M parameters and DSVH [2] with 63M parameters, as illustrated in Table 3 .Model training. During the training phase, we employ the Adam optimizer [20] for gradient descent, setting the learning rate at 10 -4 and the batch size at 128. In accordance with Eq. ( 20), we use four negative samples (n o =4) and a temperature parameter (τ ) of 0.1. Our AVHash model is implemented using PyTorch on an Nvidia Tesla V100 GPU (32GB). Additional details is available in our code release on Github website: https://github.com/iFamilyi/AVHash. For comparative purposes, the source codes and parameters of competing models are accessible on Github or provided courtesy of their authors. Table 4 presents the mAP@K (K=100) values of all the methods in comparison on the ActivityNet (-20, -50, and -all) and FCVID (-20, -50, and -all) datasets. Our AVHash method consistently outperforms the others, including AVH [41] , SRH [12] , and ConMH [42] , across all configurations. Specifically, AVHash achieves impressively high mAP scores -generally exceeding 0.9 -while the scores of competing methods seldom reach 0.5. For instance, at a code length of 32 bits, the mAP@100 scores for AVHash (0.9559 and 0.9731) are approximately 2.88 and 2.32 times higher than those of AVH (0.3320 and 0.4203) on the ActNet-20 and FCVID-20 datasets, respectively. These results strongly suggest the superior efficacy of our proposed approach to video hashing. One may question why the performance gap between 32-bit and 64-bit configurations is marked for the baseline methods but negligible for our AVHash. This difference can be attributed to the architecture of AVHash (Fig. 1 ), where the only variation between the 32-bit and 64-bit models is in their final fully connected layer (768×32 vs. 768×64), with both configurations having a similar scale of learnable parameters.We would like to explore the impact of visual and audio signals on the performance of AVHash in video retrieval tasks. This curiosity leads us to conduct comparative experiments: AVHash (with both signals) vs. AVHash with visual input only vs. AVHash with audio input only, denoted as "audio+visual", "visual", and "audio" respectively, as detailed in Table 5 . The findings reveal a hierarchy: "audio+visual" outperforms the others, followed by "visual", and "audio" ranks last. It is noteworthy that while visual signals alone could enable AVHash to achieve commendably high mAP@100 scores, the addition of audio signals would further improve AVHash's performance, which affirmatively highlights the positive contribution of audio.Moreover, we selectively analyze subsets of ActNet-all and FCVIDall, illustrating the mAP@100 values for AVHash under both "au-dio+visual" and "visual" settings in Fig. 2&3 . These figures demonstrate that audio signals generally enhance video comprehension, as indicated by the predominance of red bars over white ones. However, there are instances where audio waveforms might confuse the interpretation of frame images, suggesting an opportunity for the development of more sophisticated models.Two critical parameters, the learning rate lr and regularization factor α (Eq. ( 23)), significantly influence our model's performance. Fig. 4 compiles and illustrates their impacts upon mAP@100 scores. Insights from Fig. 4 (a)&4(b) reveal that an lr of 1e-4 enables AVHash to attain high performance on both the ActivityNet and FCVID datasets. Likewise, analysis from Fig. 4 (c)&4(d) suggests setting α to 50 for optimal results, regardless of whether the dataset is ActivityNet or FCVID. Consequently, we adopt lr =1e-4 and α=50 as the default settings for our AVHash method.Fig. 5 displays the mAP@100 scores and loss curves for AVHash on both datasets, utilizing 32-bit and 64-bit codes. Each subplot indicates the iteration number on the x-axis, while the left and right y-axes, in blue and red respectively, denote the mAP@100 scores and normalized values of the objective function 4 . The plots clearly demonstrate that AVHash achieves convergence within 300 iterations, thereby confirming the efficacy of AVHash's training process.We selected 10 classes at random, each with 50 samples from the ActNet-all and FCVID-all datasets, and projected them into a 2D space using t-SNE [38] , as illustrated in Fig. 6&7 . Notably, our AVHash method clusters these videos more distinctly than both the competitive supervised methods AVH/DSVH and the unsupervised approach MCMSH. This demonstrates AVHash's superior capabilities in representation learning. 
conclusions: 
 This paper sets itself apart from previous studies on video hashing by utilizing both audio and visual components of videos in a trilevel Transformer architecture, named AVHash, for creating binary embeddings of videos. Specifically, AVHash first maps separate audio and video signals to a shared latent semantic space, before  
 
 Appendix: The loss for each iteration is normalized via dividing it by the loss of the first iteration. 


Decision: Two reviewers simultaneously agreed on the limited novelty issue within the submission, the responses may not be convincing to address such major concern.

***TPC Addendum***
The paper has received a mixed verdict. Its limitations in terms of novelty are noted. So are its strengths in terms of jointly learning the visual and audio representation and good presentation. The paper is innately multimodal which is a priority for this conference, hence the TPC is recommending a poster accept to allow for the conversation to continue at the conference.

Reviews: review: paper_summary: paper_summary: This work proposes a video hashing technique that encodes videos into binary vectors using both audio and visual signals. The technique employs a tri-level Transformer-based model that separately processes audio and visual signals using pre-trained AST and ViT large models, then projects these into a shared latent semantic space with a Transformer encoder. A gated attention mechanism fuses the paired audio-visual signals, leading to the final video representation. The model is trained using a video-based contrastive loss and a semantic alignment regularization term for audio-visual signals. Experimental results demonstrate that AVHash significantly outperforms existing video hashing methods in video retrieval tasks.
strength_weakness: strength_weakness: Well written and well structured paper, with clear model architecture explanation. The proposed method demonstrates impressive effectiveness and improvement over baseline models. This approach has many practical applications for efficiency video indexing and retrieval. The t-SNE clustering results imply that the hashes closely cluster similar categories and provides clear separation between groups.
limitations: limitations: - There appears to be a correlation between number of parameters and mAP@100 scores, so perhaps larger models are inherent better. However, 28.4 million parameters may be difficult to deploy on resource-constrained environments. 

- Although the the model outperforms all other model on small datasets, its performance diminishes on larger datasets while other models continue to improve. I suspect scalability issues to larger and more diverse datasets. 

- According to Table 5, video alone is able to achieve a very impressive mAP@100 score, higher than all other methods. This raises questions about why existing hashing methods that solely use frame images cannot achieve similar performance levels.
paper_summary: paper_summary: Video hashing constitutes a sophisticated method for converting video data into compact binary sequences, enabling optimized storage solutions and rapid computational tasks. Departing from standard practices that mainly leverage sequential frames for the derivation of semantic binary fingerprints, this paper introduces an innovative methodology through AVHash. This approach is underpinned by a pioneering tri-level Transformer design that harmoniously synthesizes both the auditory and visual facets of videos into binary representations.
strength_weakness: strength_weakness: The paper's key advantages are summarized as follows:
1.Joint Audio-Visual Hashing: Pioneering a novel concept where both audio and visual signals are harnessed together for video hashing, breaking from the norm of relying solely on visual frames.
2.Gated Attention Fusion: Introducing a mechanism for audio-visual signal fusion that selectively weighs and integrates information, enhancing the interaction between modalities.
3.Tri-Level Transformer Network: Implementing a distinctive three-stage Transformer architecture that initially separates, aligns, and then fuses signals in a shared semantic space, enriching representations.
4.Contrastive and Regularized Training: Employing a dual strategy with contrastive loss for video-centric alignment and a regularization term for audio-visual coherence, guiding the model towards optimal retrieval performance.
limitations: limitations: While the AVHash methodology marks a substantial advancement in video retrieval by integrating audio-visual information effectively, there are several shortcomings should be addressed:
1.In section 4.3, although the comparison of 32-bit and 64-bit code length is mentioned, the influence of different code lengths on retrieval accuracy and computational cost, and the basis for selecting the optimal code length were not analyzed in depth.
2.Audio Quality Variation: Real-world videos have variable audio conditions, such as noise, accents, and background sounds. The paper doesn't extensively evaluate robustness under diverse audio variations, potentially affecting practical performance.
3.In figure 3&4, there are some cases that the performance of image + audio is reduced, and the authors do not seem to perform an in-depth analysis.
paper_summary: paper_summary: This paper proposes to integrate visual and audio signals to learn hash representations for video retrieval. Experimental results on two datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The basic idea, jointly learning the visual and audio representation is good.
limitations: limitations: 1. The training resources will be a major challenge for jointly learning visual and audio information. I noticed that the authors only conduct the experiments on two datasets, FCVID and ActivityNet. However, there is another commonly used dataset, YFCC, which is far larger than these two. I wonder if the computation resource is enough to train the model on YFCC and how long it will take to train the model? 
2. In L364, it seems that the target modality and source modality are not explained. Do they mean visual and audio modalities?
3. Figure 3 and Figure 4 shows the performance with/without audio. It turns out that two categories in each dataset show worse performance when adding audio. I think it would be better to visualize the audio signals of them to analyze when the audio will help and when will not. 
4. L729-L732 is not convincing enough. As far as I know, most video hash methods use the final fully connected layer to control the code length. So I think there should be another reason for the phenomenon. 
5. Just a minor suggestion, Figure 1 can be improved as it tells little information.
6. Some relevant hashing-based video retrieval works are not included. 
e.g. Li et al., Structure-adaptive neighborhood preserving hashing for scalable video search, TCSVT, 2021.
paper_summary: paper_summary: This paper introduces AVHash, a three-level Transformer-driven audiovisual hashing system for video search. It takes pre-trained AST and ViT models for separate audio and visual analysis and aligns them into a shared semantic space through Transformer encoding. Training involves audio-visual simultaneous contrastive loss and semantic alignment, allowing AVHash to achieve superior performance in retrieval tasks compared to previous methods.
strength_weakness: strength_weakness: 1. The writing of this paper is clear and readers can easily understand the designed modules.
2. The experiment results are very good, much better than existing methods.
limitations: limitations: 1. Novelty is limited:

   a) Merging audio signals into video representations is a common practice [1][2][3].You can even integrate more video information, such as subtitles, optical flow and other information.

   b) Network architectures and loss functions for retrieval tasks are common [4].

   c) The authors should focus on innovating the hashing function itself rather than employing well-known engineering techniques.

2. Prior methods are typically tested on datasets like YFCC, which is absent in this paper. This omission might stem from the substantial computational cost associated with model training?
3. Given that the backbone networks employed by the authors are highly advanced, comparisons may be skewed. Would it be feasible to use an identical visual backbone for a fairer comparison?

[1]Audio-based near-duplicate video retrieval with audio similarity learning

[2]Audio-Visual Mismatch-Aware Video Retrieval via Association and Adjustment

[3]Multi-modal Transformer for Video Retrieval

[4]Audio-enhanced text-to-video retrieval using text-conditioned feature alignment
paper_summary: paper_summary: This paper focuses on the video retrieval using hashing. This paper proposes a tri-level Transformer-based audio-visual hashing technique for video retrieval, named AVHash, which leverages audio signals to enhance the learning of video hashing functions. Experimental results show that AVHash outperforms existing video hashing methods in video retrieval tasks.
strength_weakness: strength_weakness: S1:The topic of video hashing is interesting and the use of audio signals for multimodal understanding of video is reasonable.
S2:The overall architecture of the proposed AVHash is feasible.
S3:The introduction of the proposed AVHash is clear.
limitations: limitations: L1:The proposed AVHash is not novel. The architecture (encoding with modality encoder and then fusing with cross-attention/gated attention) is universally filed in multi-modal understanding. This work is just a crude application of an existing method to the hash field.
L2:As far as I am concerned, the introduction of audio signals is not always helpful. For example, in the ActivityNet dataset, much of the audio in the videos does not contribute to action understanding, and much of it is even background noise that can interfere with learning.
L3:The experiment is not comprehensive enough. The authors use only two datasets to verify the effectiveness of the proposed method, which is not convincing. 
L4:There seems to be some errors in the experimental settings. It is recommended to use the same settings (dataset splitting and evaluation metrics) with existing baselines such as ConMH and CHAIN. Also, the authors should check the results in Table 4, which I find confusing.
L5:The overall presentation should be improved. In particular, Figures 3 and 4 take up too much space.
L6:The authors should also check the t-SNE figures. It seems that the authors make some mistakes in drawing the t-SNE visualisations (it's a far cry from the results I've seen so often).
L7:The experimental section should be reorganised, e.g. it is suggested to add more analysis to the ablation study and to delete the convergence curves.