# Discriminative Codebook Hashing for Supervised Video Retrieval

Authors: Xiaoman Bian; Rushi Lan; Xiaoqin Wang; Chen Chen; Zhenbing Liu; Xiaonan Luo; Kuei-Kuei Lai
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

In recent years, hashing learning has received increasing attention in supervised video retrieval. However, most existing supervised video hashing approaches design hash functions based on pairwise similarity or triple relationships and focus on local information, which results in low retrieval accuracy. In this work, we propose a novel supervised framework called discriminative codebook hashing (DCH) for large-scale video retrieval. The proposed DCH encourages samples within the same category to converge to the same code word and maximizes the mutual distances among different categories. Specifically, we first propose the discriminative codebook via a predefined distance among intercode words and Bernoulli distributions to handle each hash bit. Then, we use the composite Kullback–Leibler (KL) divergence to align the neighborhood structures between the high-dimensional space and the Hamming space. The proposed DCH is optimized via the gradient descent algorithm. Experimental results on three widely used video datasets verify that our proposed DCH performs better than several state-of-the-art methods.


---

## Body

## 1. Introduction

Under the condition of the increase in smartphones, the amount of video data has shown an explosive growth trend [1–3]. For example, TikTok has over 400 million daily active users who upload approximately 2,000 videos every minute. YouTube receives a total of 100 hours of videos per minute [4–6]. Due to the economic storage and efficiency of binary codes, hash-based methods have been widely applied to visual retrieval tasks [7–13].Previous hash-related work [14] mainly focused on image hashing and can be divided into data-independent and data-dependent methods. Data-independent approaches learn binary codes without data information but through random space projection. The most representative algorithm is local sensitive hashing (LSH) [15], which generates huge redundant information using random mapping and obtains satisfactory performance with long hash codes. Data-dependent hash methods [16–18], which can also be divided into unsupervised hashing and supervised hashing, are proposed to generate more efficient hash codes by maintaining the neighborhood structure between data. For example, Gong et al. [19] proposed iterative quantization hashing (ITQ), which minimizes quantization error by rotating principal component analysis (PCA) projection data. Spectral hashing (SH) [20] assumes that data obey a uniform distribution and divides the data according to the main direction of the data stream. Density sensitive hashing (DSH) [21] extends LSH by studying structural information. Zhang et al. [22] developed a convergence-preserving parametric learning algorithm, called latent factor hashing (LFH), to learn similarity-preserving binary codes based on latent factor models. Liu et al. [23] proposed kernel supervised hashing (KSH) by applying kernel-based formulas to accommodate linearly inseparable data and designed a greedy algorithm to solve the hash function optimization problem.In recent years, hashing methods proposed for video retrieval have also received extensive attention [24–31] and are composed of two categories: machine learning methods and deep hashing. Machine learning methods, resembling image hashing approaches, learn binary codes of video keyframes based on the low-level manual features and then calculate video hashing codes via averaging. Wu et al. [4] employed video hashing via using color histograms to obtain global features. This is the first application of hash learning in the video field. Multiple-feature hashing (MFH) [32] adopts the weight-based method to combine different features. Ye et al. [33] used video structural information in the supervised learning paradigm to obtain the optimal binary codes. Stochastic multiview hashing (SMVH) [34] attempts to separately calculate the probability similarity matrices of video frames in the feature space and the Hamming space, and then, the difference between the above two probability matrices is minimized using the KL divergence. Nie et al. [35] defined joint multiview hashing (JMVH) by maximizing the interclass distance and minimizing the innerclass distance to preserve the global structure and local structure with multiple features. Boosting temporal video hashing (BTVH) [36] studies the multitable learning problem to boost the performance and captures the inherent similarity of video from both visual and temporal perspectives. In addition, some researchers in recent years have used deep networks to obtain the temporal and spatial information between keyframes. For instance, central similarity quantization (CSQ) [37] learns the temporal information by using 3D convolutional neural networks and proposes a view point called hash center to enhance the central similarity.However, most existing video hashing approaches may lead to the following problems. (1) Low discriminability among different categories: functions based on pairwise similarity or triple relationships only consider local information, which results in good maintenance of the information of similar samples but shows poor performance in distinguishing samples from different categories. (2) Poor performance in real-world scenarios: in real application scenarios, similar data often accounts for only a small proportion, and most samples are not similar, which leads to low efficiency when the data are imbalanced [37]. (3) Greater time costs on deep learning: deep learning frameworks are time-consuming when training models and have no significant performance based on the spatiotemporal information extracted by the network. Hence, these video hashing functions cannot learn discriminative hash codes to enhance the performance.To solve the above problems, in this work, we propose a novel framework for supervised video retrieval, called discriminative codebook hashing, which considers the global structure to construct the hash function. DCH encourages samples within the same category to converge to the identical codeword and maximizes the mutual distances between different categories. Specifically, the discriminative codebook is first generated based on two characters: the predefined distance between intercode words and Bernoulli distributions for ensuring that each hash bit stores more information. Then, to keep the similarity matrix between the feature space and the Hamming space, the composite KL divergence is proposed to solve this problem. Finally, the gradient descent algorithm is utilized to optimize the algorithm. In this way, we can obtain discriminative binary codes for video retrieval. Figure1 shows the framework of DCH, and the method we proposed has the following innovations:(i)
We proposed the discriminative codebook based on the predefined distance between intercode words and Bernoulli distributions for ensuring each hash bit to store more information(ii)
The DCH method, which can maximize the distance of the intercode words generated by the predefined codebook to learn discriminative binary codes for supervised video retrieval, is proposed(iii)
We verify our proposed method by experimenting on three widely used datasets, which shows that DCH has a significant improvement in contrast with several state-of-the-art methodsFigure 1
The framework of DCH. We divide the entire experiment into two steps, namely, offline learning and online retrieval. In the offline phase, we join keyframe features and predefined codebook to learn hash functions. In the online phrase, we map the query video into a set of binary codes through hash functions. Next, we use the exclusive or (XOR) operation to obtain the Hamming distance between the query video and samples in the database. Finally, we take videos with the shortest Hamming distance as the video retrieval results.The other sections are organized as follows. Section2 introduces some preliminary works. Section 3 introduces the proposed discriminative codebook hashing in detail. The experimental work is presented in Section 4, and the conclusion of DCH is shown in Section 5.


## 2. Preliminary Work

In this section, we briefly introduce the preliminary work, namely, stochastic multiview hashing [34]. It is a supervised video retrieval method that aims to preserve the similarity structure from the original space to the Hamming space.LetV=vii=1nv be the video set, where vi indicates the ith video of V and nv is the number of videos. H=hii=1nv is hash code of the video set, where hi∈0,1 is l-bit length binary codes transformed by vi. The video features are extracted based on the set of keyframe features X=xii=1n, where xi∈ℝ1×d, n is the number of keyframes, and d is the dimension of each keyframe. Z=zii=1n represents the corresponding binary codes of the keyframes, where zi∈ℝ1×l. The conversion relationships between the above variables are formulated as(1)Z˜=XW+b,(2)Z=sigmoidZ˜,(3)hi=T1Indi∑j∈Indizj⋅,where Z˜∈ℝn×l is the temporal result of linear projection, b∈ℝl is a bias parameter, W∈ℝd×l is the projection matrix, Indi is the set of frames, and Indi is the sum of samples in the set. The high-dimensional keyframe feature matrix X is first projected into the lower matrix Z˜. Then, the sigmoid function is used to map the variable between 0 and 1. Finally, a thresholding function is used to change the data into a binary code with Ty=0 if y<0.5 and Ty=1, otherwise.SMVH keeps the similarity matrix between the feature space and the Hamming space using a composite KL divergence measure. In particular, it separately calculated the similarity probability matrixP in the original space and the pairwise similarity matrix Q among samples in the Hamming space. Then, the KL divergence is used to examine how well the above two probability matrices P and Q match. Therefore, the objective function of SMVH is defined as follows:(4)minW,bSKLW,b+μ2WF2,where μ>0 controls the weight of the regular term to prevent overfitting and SKLW,b is the composite KL divergence. The latter can be represented as(5)SKLW,b=λKLPQ+1−λKLQP,where 0≤λ≤1 controls the influence of the composite KL divergence, P=pii=1n∈ℝn×n is the similarity structure based on X, and Q=qii=1n∈ℝn×n is another probability matrix preserving the similarity information of Z in the Hamming space. In addition, the KL divergence is defined as follows:(6)KLPQ=∑i=1n∑j≠ipj|ilogpj|iqj|i,where pj|i is a conditional probability that reflects the similarity between xi and xj, and another conditional probability qj|i represents the probability of returning zj given the query zi.


## 3. Discriminative Codebook Hashing

In this section, we present the proposed DCH in detail through four parts, including the proposed discriminative codebook, the objective function, algorithmic optimization, and complexity analysis.


### 3.1. Discriminative Codebook

Motivated by CSQ [37], we propose a novel and discriminative codebook C=cii=1m for supervised video retrieval, where ci∈0,11×l is the code word of the ith category. The proposed codebook is defined according to two characters. The first is that the value in the same bit of different code words obeys a Bernoulli distribution. Specifically, the proportions of 0 and 1 of the same bit in different categories are both 50%, that is, c⋅i has a 50% probability of being 0 or 1, which will maximize the entropy and store more information in each bit. The other is that the mutual distances among intercode words are defined as follows:(7)DHci,cj≥l2−f,where DH is the Hamming distance between code words ci and cj, l is the length of binary codes, and f represents the fault tolerance. The mutual distance between intercode words will be the largest constrained by equation (7).Overall, the proposed codebook encourages samples within the same category to converge to the same codeword and maximizes the mutual distance between different categories. Therefore, the proposed codebook can preserve global structures and help generate discriminative binary codes for video retrieval. The scheme of the proposed discriminative codebook is presented in Algorithm1.Algorithm 1:Discriminative codebook.
Input: the number of categories m; the number of samples per category ni; code length l; maximum number of iterations Tc; fault tolerance rate f.Output: codebook C∈Rm×l(1)
for iteration tc=1:Tc(2)
for category i=1:m(3)
c⋅i[random half coordinate]=1(4)
c⋅i[the rest coordinate]=0(5)
end(6)
if any two rows of C satisfy equation (7)(7)
break(8)
end(9)
end


### 3.2. Objective Function

According to the proposed discriminative codebookC, we expand each row of the codebook matrix C into R=rii=1n according to the number of samples, where ri∈ℝ1×l. The detailed generation process of R is shown in Algorithm 2. We minimize the error between the binary codes and the predefined codebook as(8)minW,bZ−RF2.Algorithm 2:Discriminative codebook hashing.
Input: training data X∈ℝn×d; codebook C∈ℝm×l; maximum number of iterations T; code length l; parameters λ, μ, γ; learning rate α;Output: hash codes H∈0,1nv×l.(1)
Initialization: initialize the projection matrix W and bias matrix bas a random matrix and vector.(2)
GeneratingRaccording to the number of samples:(3)
for category i=1 : m(4)
R=R;repmatCi,:,ni,1(5)
end(6)
Gradient descent:(7)
for iteration i=1 : T(8)
W-Step: Wi+1=Wi+αdW(9)
b-Step: bi+1=bi+αdb(10)
end(11)
Video binary code computation: video hash codes are obtained by equations (1)–(3).Specifically, for eachzi∈Z, we take ri as the codebook of zi∈Z to make samples in the same category share the same codebook and samples in different categories have discriminative binary codes.To keep the similarity matrix between the feature space and the Hamming space, we join the composite KL divergence and our proposed codebook to construct the overall objective function of DCH as follows:(9)minW,bSKLW,b+γ2Z−RF2+μ2WF2,where γ controls the weight of the error loss between the codebook and the learned hash codes, and the second term of equation (9) aligns values between binary codes and their corresponding code word.In this way, our proposed DCH can solve the problem that other algorithms only consider the pairwise relationships and ensure that samples in the same category share the same code word. Furthermore, DCH maximizes the mutual distances between different categories and then obtains discriminative binary codes.


### 3.3. Algorithmic Optimization

The optimization problem has two main variables:W and b. Our solution is to use the gradient descent algorithm to find good solutions. To facilitate the writing, we split the objective function equation (9) into three parts:(10)Φ1W,b=SKLW,b,Φ2W,b=γ2Z−RF2,Φ3W=μ2WF2.The detailed optimization procedure is presented as follows.W-Step: the corresponding problem is to minimize the following loss function:(11)minWSKLW,b+γ2Z−RF2+μ2WF2.To compute the optimalW, the relevant deviation formula can be expressed as(12)dW=∂Φ1W,b∂W+∂Φ2W,b∂W+∂Φ3W∂W.The derivative of∂Φ1W,b w.r.t. W can be computed as follows:(13)∂Φ1W,b∂W=∂Φ1W,b∂zik∂zik∂wkjd×l,where ∂Φ1W,b/∂zik and ∂zik/∂wkj are represented as(14)∂Φ1W,b∂zik=2λpi|t−qi|t+pt|i−qt|i+1−λ∗qt|i∑g≠iqg|ilogqg|ipg|i+qi|t∑g≠tqg|tlogqg|tpg|t−logqt|ipt|i−logqi|tpi|tzik−ztk,∂zik∂wkj=zik1−zikxji.Following the norm derivation law,∂Φ2W,b/∂W can be optimized as follows:(15)∂Φ2W,b∂W=∂Φ2W,b∂Z∂Z∂W=XTZ−R⊙Z⊙1−Z,where ⊙ indicates that the elements in the same position of two matrices are multiplied.For∂Φ3W/∂W, we have the derivative that(16)∂Φ3W∂W=μW.b-Step: the subproblem of b is given by(17)minbSKLW,b+γ2Z−RF2.The deviation w.r.t.b can be expressed as(18)db=∂Φ1W,b∂b+∂Φ2W,b∂b.The derivative of∂Φ1W,b/∂b is described as follows:(19)∂Φ1W,b∂b=∂Φ1W,b∂zik∂zik∂bk1×l,where(20)∂zik∂bk=zik1−zik.The second term of equation (18) is described as follows:(21)∂Φ2W,b∂b=∂Φ2W,b∂Z∂Z∂b=Z−R⊙Z⊙1−Z.Algorithm2 describes the overall algorithm optimization process of the proposed DCH.


### 3.4. Complexity Analysis

The time complexity of the entire training process of SMVH [34] is approximately OTn3+n2, and the proposed DCH algorithm adds two parts time-consuming on this basis. The first part is the learning process of C, and the time complexity is OTcl. The second part is that the time complexity of optimizing equations (15) and (21) together is Odnl in each iteration. Therefore, the overall time complexity of DCH is On2+Tcl+Tn3+dnl. In this work, time complexities OTcl and Odnl can be ignored due to Tc,l,d≪n so that our complexity is nearly OTn3+n2. Additionally, the calculation of the hash codes is a linear projection with a time complexity of approximately O1, and the online search can be performed by XOR operations. Although the algorithm proposed in this paper adds a constraint on SMVH, the maximum number of iterations T directly affects the time complexity of the algorithm. It can be proven in subsequent experiments that DCH can converge in fewer iterations. Thus, the time complexity of DCH is in a reasonable range.


## 3.1. Discriminative Codebook

Motivated by CSQ [37], we propose a novel and discriminative codebook C=cii=1m for supervised video retrieval, where ci∈0,11×l is the code word of the ith category. The proposed codebook is defined according to two characters. The first is that the value in the same bit of different code words obeys a Bernoulli distribution. Specifically, the proportions of 0 and 1 of the same bit in different categories are both 50%, that is, c⋅i has a 50% probability of being 0 or 1, which will maximize the entropy and store more information in each bit. The other is that the mutual distances among intercode words are defined as follows:(7)DHci,cj≥l2−f,where DH is the Hamming distance between code words ci and cj, l is the length of binary codes, and f represents the fault tolerance. The mutual distance between intercode words will be the largest constrained by equation (7).Overall, the proposed codebook encourages samples within the same category to converge to the same codeword and maximizes the mutual distance between different categories. Therefore, the proposed codebook can preserve global structures and help generate discriminative binary codes for video retrieval. The scheme of the proposed discriminative codebook is presented in Algorithm1.Algorithm 1:Discriminative codebook.
Input: the number of categories m; the number of samples per category ni; code length l; maximum number of iterations Tc; fault tolerance rate f.Output: codebook C∈Rm×l(1)
for iteration tc=1:Tc(2)
for category i=1:m(3)
c⋅i[random half coordinate]=1(4)
c⋅i[the rest coordinate]=0(5)
end(6)
if any two rows of C satisfy equation (7)(7)
break(8)
end(9)
end


## 3.2. Objective Function

According to the proposed discriminative codebookC, we expand each row of the codebook matrix C into R=rii=1n according to the number of samples, where ri∈ℝ1×l. The detailed generation process of R is shown in Algorithm 2. We minimize the error between the binary codes and the predefined codebook as(8)minW,bZ−RF2.Algorithm 2:Discriminative codebook hashing.
Input: training data X∈ℝn×d; codebook C∈ℝm×l; maximum number of iterations T; code length l; parameters λ, μ, γ; learning rate α;Output: hash codes H∈0,1nv×l.(1)
Initialization: initialize the projection matrix W and bias matrix bas a random matrix and vector.(2)
GeneratingRaccording to the number of samples:(3)
for category i=1 : m(4)
R=R;repmatCi,:,ni,1(5)
end(6)
Gradient descent:(7)
for iteration i=1 : T(8)
W-Step: Wi+1=Wi+αdW(9)
b-Step: bi+1=bi+αdb(10)
end(11)
Video binary code computation: video hash codes are obtained by equations (1)–(3).Specifically, for eachzi∈Z, we take ri as the codebook of zi∈Z to make samples in the same category share the same codebook and samples in different categories have discriminative binary codes.To keep the similarity matrix between the feature space and the Hamming space, we join the composite KL divergence and our proposed codebook to construct the overall objective function of DCH as follows:(9)minW,bSKLW,b+γ2Z−RF2+μ2WF2,where γ controls the weight of the error loss between the codebook and the learned hash codes, and the second term of equation (9) aligns values between binary codes and their corresponding code word.In this way, our proposed DCH can solve the problem that other algorithms only consider the pairwise relationships and ensure that samples in the same category share the same code word. Furthermore, DCH maximizes the mutual distances between different categories and then obtains discriminative binary codes.


## 3.3. Algorithmic Optimization

The optimization problem has two main variables:W and b. Our solution is to use the gradient descent algorithm to find good solutions. To facilitate the writing, we split the objective function equation (9) into three parts:(10)Φ1W,b=SKLW,b,Φ2W,b=γ2Z−RF2,Φ3W=μ2WF2.The detailed optimization procedure is presented as follows.W-Step: the corresponding problem is to minimize the following loss function:(11)minWSKLW,b+γ2Z−RF2+μ2WF2.To compute the optimalW, the relevant deviation formula can be expressed as(12)dW=∂Φ1W,b∂W+∂Φ2W,b∂W+∂Φ3W∂W.The derivative of∂Φ1W,b w.r.t. W can be computed as follows:(13)∂Φ1W,b∂W=∂Φ1W,b∂zik∂zik∂wkjd×l,where ∂Φ1W,b/∂zik and ∂zik/∂wkj are represented as(14)∂Φ1W,b∂zik=2λpi|t−qi|t+pt|i−qt|i+1−λ∗qt|i∑g≠iqg|ilogqg|ipg|i+qi|t∑g≠tqg|tlogqg|tpg|t−logqt|ipt|i−logqi|tpi|tzik−ztk,∂zik∂wkj=zik1−zikxji.Following the norm derivation law,∂Φ2W,b/∂W can be optimized as follows:(15)∂Φ2W,b∂W=∂Φ2W,b∂Z∂Z∂W=XTZ−R⊙Z⊙1−Z,where ⊙ indicates that the elements in the same position of two matrices are multiplied.For∂Φ3W/∂W, we have the derivative that(16)∂Φ3W∂W=μW.b-Step: the subproblem of b is given by(17)minbSKLW,b+γ2Z−RF2.The deviation w.r.t.b can be expressed as(18)db=∂Φ1W,b∂b+∂Φ2W,b∂b.The derivative of∂Φ1W,b/∂b is described as follows:(19)∂Φ1W,b∂b=∂Φ1W,b∂zik∂zik∂bk1×l,where(20)∂zik∂bk=zik1−zik.The second term of equation (18) is described as follows:(21)∂Φ2W,b∂b=∂Φ2W,b∂Z∂Z∂b=Z−R⊙Z⊙1−Z.Algorithm2 describes the overall algorithm optimization process of the proposed DCH.


## 3.4. Complexity Analysis

The time complexity of the entire training process of SMVH [34] is approximately OTn3+n2, and the proposed DCH algorithm adds two parts time-consuming on this basis. The first part is the learning process of C, and the time complexity is OTcl. The second part is that the time complexity of optimizing equations (15) and (21) together is Odnl in each iteration. Therefore, the overall time complexity of DCH is On2+Tcl+Tn3+dnl. In this work, time complexities OTcl and Odnl can be ignored due to Tc,l,d≪n so that our complexity is nearly OTn3+n2. Additionally, the calculation of the hash codes is a linear projection with a time complexity of approximately O1, and the online search can be performed by XOR operations. Although the algorithm proposed in this paper adds a constraint on SMVH, the maximum number of iterations T directly affects the time complexity of the algorithm. It can be proven in subsequent experiments that DCH can converge in fewer iterations. Thus, the time complexity of DCH is in a reasonable range.


## 4. Experiments

In this section, we first introduce the datasets used in this paper, and then, the baselines and some experimental details will be introduced. Finally, we present the experimental results.


### 4.1. Datasets

CC_WEB_VIDEO [4] is the most useful dataset in near-duplicate video retrieval (NDVR) research, which contains data from YouTube, Google, and Yahoo. There are 12,877 videos that are divided into 24 sets, and keyframes are extracted by a uniform sampling method to represent the video. Since some videos do not have label information, we take 3,482 videos with labels as the experimental dataset. In each category, we select 70% of the video data as the training set and the remainder as the testing set. We extract 10 keyframes for each video uniformly and extract 4096-dimensional features to represent keyframes by using the pretrained VGG-19 network.HMDB51 [38] contains 6,766 human action videos selected from movies and some other public sources such as YouTube. The dataset is divided into 51 categories, and each of them includes approximately 100 clips. In each category, we randomly select 45 video samples. Of these, 25 videos are added to the training set and the rest are select to the testing set. We uniformly extract 10 keyframes for each video, and the VGG-19 pretraining network is used to extract the 4096-dimensional deep features.UCF101 [39] contains 13,320 videos which has been divided into 101 human behavior categories, such as sports, instruments, character interactions, and others used for action recognition. We randomly select 70 videos in each category to join the training set, and 30 videos to join the testing set. For each video, 10 keyframes are uniformly selected to represent the video. We use VGG-19 to extract the 4096-dimensional features for each keyframe.


### 4.2. Experimental Setting

#### 4.2.1. Baselines

Several state-of-the-art hash functions, including ITQ [19], SH [20], DSH [21], LFH [22], KSH [23], JMVH [35], and SMVH [34], are used for comparison. Among these methods, ITQ, SH, and DSH are unsupervised hashing methods, while LFH, KSH, JMVH, and SMVH are supervised hashing methods. For the comparative test, we use the source codes published to conduct the experiment. JMVH and SMVH can also be used for multiview video retrieval, but in this paper, we only test these methods as a single view method. It is worth noting that all the experimental results are obtained in MATLAB R2016a on the same computer with an Intel Core i7-6700 CPU @ 3.40 GHz, 72 GB RAM and the 64 bit Windows 10 operating system.


#### 4.2.2. Evaluation Metrics

We use four popular evaluation metrics to comprehensively evaluate experimental results. The mean average precision (mAP) is widely used in the retrieval field. The higher the mAP score is, the better the retrieval performance of the method is. The precision@K curve represents the precision accuracy versus the firstK retrieved samples, where precision represents the proportion of the number of retrieved correct videos to the total number of retrieved videos. The recall@K curve represents the average recall rate versus the first K retrieved samples, where recall represents the proportion of the correct video volume retrieved in all near-duplicate video samples. The precision-recall (PR) curve is an index used to evaluate reliability and is widely used in the fields of medicine and machine learning.


#### 4.2.3. Parameter Selection

We have three model parameters, includingλ, μ, and γ, and the number of iterations T. According to SMVH [34], we set λ=0.9 and μ=0.01. As shown in Figure 2(a), when γ is in the range of 0.05 to 1, the results are stable across three different datasets. Therefore, we empirically choose γ=1 in our proposed model. The maximum number of iterations T determines the training time cost and the performance, so it is worth discussing. Figure 2(b) shows the effect of the maximum iterations T in the range of 100 to 1400 on mAP performance. For HMDB51, it can be seen that the best mAP is generated with T=800 before decreasing. However, in the other two datasets, T=800 is not an optimal experimental result. Therefore, after comprehensive consideration, T=1000 is set as the final parameter setting.Figure 2
Parameter analysis on the CC_WEB_VIDEO, HMDB51, and UCF101 datasets. (a) mAP vs.γ (weight parameter γ) and (b) mAP vs. T (iteration parameter T).
(a)(b)


### 4.3. Results and Discussion

Table1 shows the mAP results for different lengths of hash codes on the three datasets, and the results of other evaluation metrics are shown in Figures 3–5. We will give the detailed analysis of all results of the three datasets in the following parts.Table 1
The mAP of different hash code lengths on three datasets, where the best experimental results are given in bold.
MethodCC_WEB_VIDEOHMDB51UCF10132 bits48 bits64 bits96 bits32 bits48 bits64 bits96 bits32 bits48 bits64 bits96 bitsITQ [19]0.68770.77250.80990.77000.06970.07490.07930.08850.13830.16200.18010.2119SH [20]0.67290.70260.69940.67080.06620.06570.06420.06530.10330.11380.12440.1395DSH [21]0.65100.70600.69290.81580.05050.06280.06710.07500.07200.06670.08150.1082LFH [22]0.83270.80880.98540.99120.01410.02080.01480.02250.00320.00380.00780.0113KSH [23]0.93680.90300.94770.87610.24700.28110.30540.31440.32220.35980.39720.4075JMVH [35]0.78420.55760.43350.37450.28070.30150.24180.12950.39410.51660.60070.6875SMVH [34]0.93460.94110.95430.74900.12120.13990.13740.0319——0.00940.0304DCH0.95310.97630.98860.98580.25680.38190.36000.41500.53100.61370.66090.6458Figure 3
Precision@K (a–d), recall@K (e–h), and PR (i–l) curves on the CC_WEB_VIDEO dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 4
Precision@K (a–d), recall@K (e–h), and PR (i)–(l) curves on the HMDB51 dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 5
Precision@K (a–d), recall@K (e–h), and PR (i–l) curves on the UCF101 dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)According to Table1, for the CC_WEB_VIDEO dataset, the mAPs are very high because the dataset is movie clips, and videos of the same category are near-duplicate videos. As shown in Table 1, the performance of the proposed DCH is at least 1.85% better than that of the other methods from 32 to 64 bits. When the code length is 96 bits, the mAP of DCH is slightly lower than that of LFH. As shown in Figure 3, the experimental results of our method in precision@K and recall@K are equal to or slightly higher than those of most other methods. Besides, as the code length increases, the performance of our proposed DCH gradually surpasses that of other methods. Figures 3(i)–3(l) show that the area surrounded by DCH is gradually increasing.Table1 shows that our proposed DCH performs better than other hash methods in most cases in the HMDB51 dataset. Although the mAP performance of the JMVH method surpasses 2.39% over that of DCH with 32 bits, the mAPs of our proposed DCH are better than those of the other comparison methods in the subsequent experiments. Figure 4 shows that when the length of hash codes is larger than 32 bits, regardless of whether precision@K curve, recall@K curve, or PR curve is used, DCH has excellent performance compared with other methods in all metrics for the precision@K curve, recall@K curve, and PR curve.For the UCF101 dataset, DCH obtained the optimal experimental results in the range of [32, 48, 64] bits. It is worth noting that the size of the UCF101 dataset is relatively large, and SMVH cannot obtain discriminative video hash when the hash code length is very small. Therefore, SMVH has no experimental results available forl=32 and l=48. As shown in Figure 5, the performance of DCH is much higher than those of some of the methods except JMVH. We can see that the recall rate of DCH for positive samples is slightly lower than that of JMVH based on Figures 5(e)–5(h). Figures 5(i)–5(k) show that the performance of DCH for 32 to 48 bits is better than those of all other methods for the PR curve.


## 4.1. Datasets

CC_WEB_VIDEO [4] is the most useful dataset in near-duplicate video retrieval (NDVR) research, which contains data from YouTube, Google, and Yahoo. There are 12,877 videos that are divided into 24 sets, and keyframes are extracted by a uniform sampling method to represent the video. Since some videos do not have label information, we take 3,482 videos with labels as the experimental dataset. In each category, we select 70% of the video data as the training set and the remainder as the testing set. We extract 10 keyframes for each video uniformly and extract 4096-dimensional features to represent keyframes by using the pretrained VGG-19 network.HMDB51 [38] contains 6,766 human action videos selected from movies and some other public sources such as YouTube. The dataset is divided into 51 categories, and each of them includes approximately 100 clips. In each category, we randomly select 45 video samples. Of these, 25 videos are added to the training set and the rest are select to the testing set. We uniformly extract 10 keyframes for each video, and the VGG-19 pretraining network is used to extract the 4096-dimensional deep features.UCF101 [39] contains 13,320 videos which has been divided into 101 human behavior categories, such as sports, instruments, character interactions, and others used for action recognition. We randomly select 70 videos in each category to join the training set, and 30 videos to join the testing set. For each video, 10 keyframes are uniformly selected to represent the video. We use VGG-19 to extract the 4096-dimensional features for each keyframe.


## 4.2. Experimental Setting

### 4.2.1. Baselines

Several state-of-the-art hash functions, including ITQ [19], SH [20], DSH [21], LFH [22], KSH [23], JMVH [35], and SMVH [34], are used for comparison. Among these methods, ITQ, SH, and DSH are unsupervised hashing methods, while LFH, KSH, JMVH, and SMVH are supervised hashing methods. For the comparative test, we use the source codes published to conduct the experiment. JMVH and SMVH can also be used for multiview video retrieval, but in this paper, we only test these methods as a single view method. It is worth noting that all the experimental results are obtained in MATLAB R2016a on the same computer with an Intel Core i7-6700 CPU @ 3.40 GHz, 72 GB RAM and the 64 bit Windows 10 operating system.


### 4.2.2. Evaluation Metrics

We use four popular evaluation metrics to comprehensively evaluate experimental results. The mean average precision (mAP) is widely used in the retrieval field. The higher the mAP score is, the better the retrieval performance of the method is. The precision@K curve represents the precision accuracy versus the firstK retrieved samples, where precision represents the proportion of the number of retrieved correct videos to the total number of retrieved videos. The recall@K curve represents the average recall rate versus the first K retrieved samples, where recall represents the proportion of the correct video volume retrieved in all near-duplicate video samples. The precision-recall (PR) curve is an index used to evaluate reliability and is widely used in the fields of medicine and machine learning.


### 4.2.3. Parameter Selection

We have three model parameters, includingλ, μ, and γ, and the number of iterations T. According to SMVH [34], we set λ=0.9 and μ=0.01. As shown in Figure 2(a), when γ is in the range of 0.05 to 1, the results are stable across three different datasets. Therefore, we empirically choose γ=1 in our proposed model. The maximum number of iterations T determines the training time cost and the performance, so it is worth discussing. Figure 2(b) shows the effect of the maximum iterations T in the range of 100 to 1400 on mAP performance. For HMDB51, it can be seen that the best mAP is generated with T=800 before decreasing. However, in the other two datasets, T=800 is not an optimal experimental result. Therefore, after comprehensive consideration, T=1000 is set as the final parameter setting.Figure 2
Parameter analysis on the CC_WEB_VIDEO, HMDB51, and UCF101 datasets. (a) mAP vs.γ (weight parameter γ) and (b) mAP vs. T (iteration parameter T).
(a)(b)


## 4.2.1. Baselines

Several state-of-the-art hash functions, including ITQ [19], SH [20], DSH [21], LFH [22], KSH [23], JMVH [35], and SMVH [34], are used for comparison. Among these methods, ITQ, SH, and DSH are unsupervised hashing methods, while LFH, KSH, JMVH, and SMVH are supervised hashing methods. For the comparative test, we use the source codes published to conduct the experiment. JMVH and SMVH can also be used for multiview video retrieval, but in this paper, we only test these methods as a single view method. It is worth noting that all the experimental results are obtained in MATLAB R2016a on the same computer with an Intel Core i7-6700 CPU @ 3.40 GHz, 72 GB RAM and the 64 bit Windows 10 operating system.


## 4.2.2. Evaluation Metrics

We use four popular evaluation metrics to comprehensively evaluate experimental results. The mean average precision (mAP) is widely used in the retrieval field. The higher the mAP score is, the better the retrieval performance of the method is. The precision@K curve represents the precision accuracy versus the firstK retrieved samples, where precision represents the proportion of the number of retrieved correct videos to the total number of retrieved videos. The recall@K curve represents the average recall rate versus the first K retrieved samples, where recall represents the proportion of the correct video volume retrieved in all near-duplicate video samples. The precision-recall (PR) curve is an index used to evaluate reliability and is widely used in the fields of medicine and machine learning.


## 4.2.3. Parameter Selection

We have three model parameters, includingλ, μ, and γ, and the number of iterations T. According to SMVH [34], we set λ=0.9 and μ=0.01. As shown in Figure 2(a), when γ is in the range of 0.05 to 1, the results are stable across three different datasets. Therefore, we empirically choose γ=1 in our proposed model. The maximum number of iterations T determines the training time cost and the performance, so it is worth discussing. Figure 2(b) shows the effect of the maximum iterations T in the range of 100 to 1400 on mAP performance. For HMDB51, it can be seen that the best mAP is generated with T=800 before decreasing. However, in the other two datasets, T=800 is not an optimal experimental result. Therefore, after comprehensive consideration, T=1000 is set as the final parameter setting.Figure 2
Parameter analysis on the CC_WEB_VIDEO, HMDB51, and UCF101 datasets. (a) mAP vs.γ (weight parameter γ) and (b) mAP vs. T (iteration parameter T).
(a)(b)


## 4.3. Results and Discussion

Table1 shows the mAP results for different lengths of hash codes on the three datasets, and the results of other evaluation metrics are shown in Figures 3–5. We will give the detailed analysis of all results of the three datasets in the following parts.Table 1
The mAP of different hash code lengths on three datasets, where the best experimental results are given in bold.
MethodCC_WEB_VIDEOHMDB51UCF10132 bits48 bits64 bits96 bits32 bits48 bits64 bits96 bits32 bits48 bits64 bits96 bitsITQ [19]0.68770.77250.80990.77000.06970.07490.07930.08850.13830.16200.18010.2119SH [20]0.67290.70260.69940.67080.06620.06570.06420.06530.10330.11380.12440.1395DSH [21]0.65100.70600.69290.81580.05050.06280.06710.07500.07200.06670.08150.1082LFH [22]0.83270.80880.98540.99120.01410.02080.01480.02250.00320.00380.00780.0113KSH [23]0.93680.90300.94770.87610.24700.28110.30540.31440.32220.35980.39720.4075JMVH [35]0.78420.55760.43350.37450.28070.30150.24180.12950.39410.51660.60070.6875SMVH [34]0.93460.94110.95430.74900.12120.13990.13740.0319——0.00940.0304DCH0.95310.97630.98860.98580.25680.38190.36000.41500.53100.61370.66090.6458Figure 3
Precision@K (a–d), recall@K (e–h), and PR (i–l) curves on the CC_WEB_VIDEO dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 4
Precision@K (a–d), recall@K (e–h), and PR (i)–(l) curves on the HMDB51 dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 5
Precision@K (a–d), recall@K (e–h), and PR (i–l) curves on the UCF101 dataset.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)According to Table1, for the CC_WEB_VIDEO dataset, the mAPs are very high because the dataset is movie clips, and videos of the same category are near-duplicate videos. As shown in Table 1, the performance of the proposed DCH is at least 1.85% better than that of the other methods from 32 to 64 bits. When the code length is 96 bits, the mAP of DCH is slightly lower than that of LFH. As shown in Figure 3, the experimental results of our method in precision@K and recall@K are equal to or slightly higher than those of most other methods. Besides, as the code length increases, the performance of our proposed DCH gradually surpasses that of other methods. Figures 3(i)–3(l) show that the area surrounded by DCH is gradually increasing.Table1 shows that our proposed DCH performs better than other hash methods in most cases in the HMDB51 dataset. Although the mAP performance of the JMVH method surpasses 2.39% over that of DCH with 32 bits, the mAPs of our proposed DCH are better than those of the other comparison methods in the subsequent experiments. Figure 4 shows that when the length of hash codes is larger than 32 bits, regardless of whether precision@K curve, recall@K curve, or PR curve is used, DCH has excellent performance compared with other methods in all metrics for the precision@K curve, recall@K curve, and PR curve.For the UCF101 dataset, DCH obtained the optimal experimental results in the range of [32, 48, 64] bits. It is worth noting that the size of the UCF101 dataset is relatively large, and SMVH cannot obtain discriminative video hash when the hash code length is very small. Therefore, SMVH has no experimental results available forl=32 and l=48. As shown in Figure 5, the performance of DCH is much higher than those of some of the methods except JMVH. We can see that the recall rate of DCH for positive samples is slightly lower than that of JMVH based on Figures 5(e)–5(h). Figures 5(i)–5(k) show that the performance of DCH for 32 to 48 bits is better than those of all other methods for the PR curve.


## 5. Conclusion

In this paper, we propose a novel supervised video hashing framework, termed discriminative codebook hashing, which can generate discriminative binary codes for video retrieval. The proposed DCH encourages samples within the same category to converge to the same code word and maximizes the mutual distances between different categories. Specifically, we generate a discriminative codebook to distinguish between samples of different categories more accurately. Extensive experimental results prove that the performance of DCH is significantly improved compared to several state-of-the-art methods. In future work, we will use a smaller matrix storing the similarity information between samples to avoid consuming considerable training time and space when the amount of data is large. This will improve the performance of the model while reducing the time complexity.



---

*Source: 5845094-2021-08-25.xml*