Deep Reinforcement Learning for Image Hashing

arXiv:1802.02904v1 [] 7 Feb 2018

JIAN ZHANG, YUXIN PENG∗ , and ZHAODA YE, Peking University, China.
Deep hashing methods have received much attention recently, which achieve promising results by taking
advantage of the strong representation power of deep networks. However, most existing deep hashing methods
learn a whole set of hashing functions independently and directly, while ignore the correlation between
different hashing functions that can promote the retrieval accuracy greatly. Inspired by the sequential decision
ability of deep reinforcement learning, in this paper, we propose a new Deep Reinforcement Learning approach
for Image Hashing (DRLIH). Our proposed DRLIH models the hashing learning problem as a Markov Decision
Process (MDP), which learns each hashing function by correcting the errors imposed by previous ones and
promotes retrieval accuracy. To the best of our knowledge, this is the first work that tries to address hashing
problem from deep reinforcement learning perspective. The main contributions of our proposed DRLIH
approach can be summarized as follows: (1) We propose a deep reinforcement learning hashing network.
In our proposed DRLIH approach, we utilize recurrent neural network (RNN) as agents to model the hashing
functions, which take actions of projecting images into binary codes sequentially, so that current hashing
function learning can take previous hashing functions’ error into account. (2) We propose a sequential
learning strategy based on proposed DRLIH. We define the state as a tuple of internal features of RNN’s
hidden layers and image features, which can well reflect history decisions made by the agents. We also propose
an action group method to enhance the correlation of the hash functions in the same group. Experiments on
three widely-used datasets demonstrate the effectiveness of our proposed DRLIH approach.
CCS Concepts: • Computing methodologies → Visual content-based indexing and retrieval; • Information systems → Image search;
Additional Key Words and Phrases: Deep reinforcement learning, image hashing, image retrieval.
ACM Reference Format:
Jian Zhang, Yuxin Peng, and Zhaoda Ye. 2018. Deep Reinforcement Learning for Image Hashing. ACM Trans.
Multimedia Comput. Commun. Appl. 1, 1 (February 2018), 18 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTION

With rapid growth of images on the web, the large scale image retrieval has attracted much
attention. Many hashing methods have been proposed for the fast similarity measurement in image
retrieval [4, 7, 13, 17, 18, 35, 40, 46]. Generally speaking, hashing methods learn several mapping
functions for the image data, so that the similar images are mapped into similar binary codes.
Traditional hashing methods use hand-crafted features (e.g. GIST [26], Bag-of-Visual-Words [6]) as
image representations. However, hand-crafted features can not well represent the image content,
which limit the performance of image retrieval. Inspired by the successful applications of deep
networks on many computer vision problems such as image classification and object detection [15],
∗ Corresponding

author

Authors’ address: Jian Zhang; Yuxin Peng; Zhaoda Ye, Peking University, Insititute of Computer Science and Technology,
Beijing, 100871, China. pengyuxin@pku.edu.cn.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the
full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2018 Copyright held by the owner/author(s). Publication rights licensed to the Association for Computing Machinery.
1551-6857/2018/2-ART $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:2

J. Zhang et al.

deep hashing methods [17, 19, 42, 43, 45, 48, 49] have been proposed to take advantage of deep
network’s feature representation power.
Existing deep hashing methods [2, 17, 21, 42] have achieved promising results on image retrieval.
However, they learn the whole set of hashing functions independently and directly, which ignore
the correlation between different hashing functions that can promote the retrieval accuracy greatly.
There exist several traditional hashing methods [35, 36] that learn the hash code sequentially.
However, these methods require complicated optimization that cannot be directly adopted into
deep networks.
Recently, deep reinforcement learning has achieved some breakthroughs. For example, deep
reinforcement learning achieves human-level performance in Atari games and GO [25, 33]. A
standard reinforcement learning model includes an agent and an environment. The agent receives
the information from the environment and chooses the actions from the action set that tend
to maximize the sum of the reward function. It is noted that the hashing methods have native
relationship to the reinforcement learning, which is the motivation of this paper. 1) In traditional
hashing methods, the discrete property of the hash codes leads to a hard optimization problem,
while we can regard discrete hash code as the action of the agent and solve the optimization
problem in the reinforcement learning framework. 2) The agent in the reinforcement learning
framework chooses actions to maximize the sum of reward in a task where the decisions should not
be made independently. This property enlightens us that if we regard the hash code generation as a
sequential task, it’s possible to generate the hash code dependently within reinforcement learning
framework.
Based on the analysis, in this paper, we propose a Deep Reinforcement Learning approach for
Image Hashing (DRLIH). Instead of learning a whole set of hashing functions independently, our
proposed DRLIH approach models the hashing learning process as a sequential decision making
process by the designed deep reinforcement learning network. The main contributions of our
proposed approach can be summarized as follows:
• A Deep reinforcement learning hashing network is proposed to learn hashing functions
sequentially and progressively. The proposed network consists of a feature representation
network and a policy network. The policy network is composed by RNN network, and it
serves as the agents to sequentially project images into binary codes. We design the policy
network to output the probability of projecting images into hash code 1, and calculate the
probability of hash code 0. We also propose two hierarchical rewards functions to drive the
training of our proposed DRLIH network.
• A sequential learning strategy is proposed to capture the ranking errors caused by previous generated hash codes. We define the states as tuples of the image feature and internal
features of RNN, which reflect the history decisions, thus the agent can obtain the history
information and capture the previous ranking errors to make next decision. We also propose
the action group as the minimal step to the agent to enhance the relevance of the hash bits in
the same group and promote the retrieval accuracy.
Experiments on three widely used datasets demonstrate the effectiveness of our proposed DRLIH
approach. The rest of this paper is organized as follows. Section 2 reviews some representative
related work. Section 3 presents our proposed deep reinforcement learning approach for image
hashing, section 4 shows the experiments on three widely-used image datasets, and section 5
concludes this paper.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing
2

:3

RELATED WORKS

In this section, we briefly introduce some related works from two aspects: image hashing methods
and deep reinforcement learning.
2.1

Image Hashing Methods

Generally speaking, image hashing methods project images into a common Hamming space while
similar data will be mapped into similar binary codes for efficient image retrieval. The image
hashing methods can be categorized into traditional methods and deep hashing methods. The
traditional methods can be further divided into unsupervised methods and supervised methods.
We will give a brief overview to some representative image hashing methods.
2.1.1 Unsupervised Hashing Methods. The unsupervised methods design hash functions without
using image labels and can be divided into data independent methods and data dependent methods.
Locality Sensitive Hashing (LSH) [7] is a representative data independent method which maps data
into binary codes by random linear projection. There are several works extend the LSH, such as
Multi-probe LSH [24] and SIKH [27]. Data dependent methods analyze the data properties, such as
data distributions and manifold structures, for better learning hash functions. For example, Spectral
Hashing (SH) [40] designs the hash codes to be balanced and uncorrelated. Anchor Graph Hashing
(AGH) [23] discovers neighborhood structures by anchor graphs. Topology Preserving Hashing
(TPH) [44] tries to learn a Hamming space to preserve the consistent neighborhood rankings of data
points. Gong et al. propose Iterative Quantization (ITQ) [8] to learn hash functions by minimizing
the quantization error of mapping data to the vertices of a binary hypercube. Irie et al. propose
Locally Linear Hashing (LLH) [11] to capture the local linear structures with locality-sensitive
sparse coding and then recover these structures in Hamming space.
2.1.2 Supervised Hashing Methods. Unsupervised hashing methods search images by nearest
neighbors under a certain kind of distance metric, but neighbors may not be semantically similar.
Thus supervised hashing methods leverage the semantic information to generate hash codes to
address this problem. Binary Reconstruction Embedding (BRE) [16] obtains the binary codes by
minimizing the reconstruction error between the original distances and the reconstructed distances.
Wang et al. propose Semi-supervised Hashing (SSH) [35] to obtain the binary codes by minimizing
the empirical error over the labeled data and the information entropy of the binary codes over
both labeled and unlabeled data. Liu et al. propose Supervised Hashing with Kernels (KSH) [22]
to preserve the pairwise relationship which is provided by labels in the Hamming space. Order
Preserving Hashing (OPH) [38] and Ranking Preserving Hashing (RPH) [39] obtain the binary codes
by preserving the ranking information. Supervised Discrete Hashing (SDH) [32] integrates hash code
generation and classifier training and improve the retrieval accuracy. Supervised hashing methods
utilize the semantic information of the images and usually perform better than unsupervised
hashing methods.
2.1.3 Deep Hashing Methods. Inspired by the successful of deep neural networks in many computer vision problems, such as image classification and object detection [15], deep hashing methods [17, 19, 42, 43, 45, 48, 49] are proposed. Convolutional Neural Network Hashing (CNNH) [42]
is a representative work of the deep hashing methods, which first applies the convolutional neural
networks (CNNs) to the hashing methods. There are two stages in CNNH: hash code learning stage
and hash function learning stage. Given a training image set I = {I 1 , I 2 , . . . , In },in the first learning
stage (Stage 1), CNNH learns the approximate hash codes for training images by optimizing the
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:4

J. Zhang et al.

following loss function:
1
min ∥S − HH T ∥F2
(1)
H
q
where ∥ · ∥F denotes the Frobenius norm; S denotes the semantic similarity of image pairs in I, in
which Si j = 1 when image Ii and I j are semantically similar, otherwise Si j = −1; H ∈ {−1, 1}n×q
denotes the approximate hash codes preserving the pairwise similarities in S. However, Equation (1)
can not be easily optimized, CNNH relaxes the integer constraints on H and randomly initializes
H ∈ [−1, 1]n×q , then utilizes the coordinate descent algorithm to optimizes equation (1). In the
second learning stage (Stage 2), CNNH adopts the deep networks to learn image features and
hash functions. CNNH utilizes the deep framework in [9] and adds an output layer with sigmoid
activation to generate q-bit hash codes. And the approximate hash codes learned in Stage 1 are used
as the ground-truth to train the deep network. But CNNH is a two-stage framework, the separation
between the two stages limits the performance of hash learning.
Thus several works propose some frameworks to jointly learn the image representation and
generate hash codes. For example, Network in Network Hashing (NINH) [17] constructs the
framework based on the Network in Network architecture [20], which uses a sub-network stacked
by several convolutional layers to extract image features and adopts a divide-and-encode module
encouraged by sigmoid activation function and a piece-wise threshold function to generate binary
hash codes. In the training process, NINH utilizes a triplet ranking loss [3, 31, 37] function to
directly guide hash learning:
lt r ipl et (I , I + , I − ) =
max(0, 1 − (∥b(I ) − b(I − )∥ H − ∥b(I ) − b(I + )∥ H ))
+

s.t . b(I ), b(I ), b(I ) ∈ {−1, 1}
−

(2)

q

where I , I + and I − specify the triplet constraint that image I is more similar to image I + than
to image I − based on image labels, which exploit the relative similarity of training images; b(·)
denotes binary hash code, and ∥ · ∥ H denotes the Hamming distance. For easier optimization, NINH
relaxes the integer constraints on binary hash codes and replaces Hamming distance with Euclidean
distance.
There are several works extend the NINH, such as Bit-scalable Deep Hashing method [45]
further manipulates hash code length by weighing each bit of hash codes. Deep Hashing Network
(DHN) [49] both minimizes the quantization errors and triplet ranking loss for better retrieval
performance. Deep Semantic Preserving and Ranking-based Hashing (DSRH) [43] introduces the
orthogonal constraints into triplet ranking loss to make hash bits independent. There are also some
pairwise similarity based deep hashing methods [19, 49] that preserve the semantic similarities of
the pairwise labels.
2.2

Deep Reinforcement Learning

The reinforcement learning is the problem faced by an agent that must learn behavior through
trial-and-error interactions with a dynamic environment [12]. In the standard deep reinforcement
learning model, the agent receives the current state of the environment as input, then the agent will
choose an action as output. The action changes the state of the environment, and the environment
communicates to the agent through a scalar reinforcement signal named reward which reflects the
quality of the taken action. The goal of the reinforcement learning is to train the agent to choose
actions that maximize the sum of reward. Recently, deep reinforcement learning methods have
achieved some progresses in many domains. Mnih et al. [25] utilize deep neural networks to learn
an action-value function to play Atari games, which reaches human-level performance. Silver et
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing

:5

al. [33] use policy network and value network to play Go and beat the world-level professional
player. Deep reinforcement learning has also been applied in variety computer vision problems.
Caicedo et al. [1] propose a deep reinforcement learning method for active object localization that
the agent is trained to deform the bounding box using sample transformation actions. Zhou et
al. [29] propose a deep reinforcement learning based image caption model, which utilize a "policy
network" and a "value network" to collaboratively generate captions. Zhao et al. [47] utilize the
information entropy to guide a reinforcement learning agent to select the key part of an image for
better image classification. Rao et al. [28] proposed attention-aware deep reinforcement learning
(ADRL) to discard the misleading and confounding frames and focus on the attentions in face videos
for better person recognition. Inspired by the recent advances of deep reinforcement learning,
in this paper we intend to address hashing problem by the sequential decision ability of deep
reinforcement learning.

Environment
Images

Agent

Representation learning layers

Image feature

History

state
·

Policy
Network

Label

Reward
Fucntion

Action

π(a|s)

Reward

Fig. 1. Overview of our proposed deep reinforcement learning hashing network, which consists of a representation learning network, a policy network and a reward function. The policy network serves as agents and
makes decisions to project images into binary codes, while the reward function drives the sequential training
of the whole network.

3

THE PROPOSED APPROACH

Different from existing deep hashing methods, we model the hashing problem as a Markov Decision
Process (MDP), which provides a formal framework to model an agent that makes a sequence of
decisions. In our proposed DRLIH, we consider a batch of images as the environment, and define the
states as image features combined with images’ internal features of the policy network. The agent
projects images into binary codes based on the environment and its current states. In the following
part, we first give the formal notations of hashing problems, then we introduce the definition
of state, action and reward in our proposed deep reinforcement learning hashing network, then
we introduce the network structure of our proposed DRLIH in detail, finally we introduce the
sequential learning strategy of DRLIH network.
3.1

Notations

Given a set of n images X ∈ RD . The goal of hashing methods is to learn a set of q hashing functions
H = [h 1 , h 2 , · · · , hq ], which encode an image x ∈ X into a q-dimensional binary code H (x) in
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:6

J. Zhang et al.

the Hamming space, while preserving the semantic similarity of images. Most deep hashing
methods learn H independently and directly, while ignore the correlations between different
hashing functions. In this paper we intend to learn the hashing functions H sequentially by deep
reinforcement learning.
3.2

Definition of the Reinforcement

3.2.1 State. In our DRLIH approach, the current hash code is supposed to be generated in
serialization which can capture the ranking errors caused by the previous generated hash codes.
So the state has to reflect the history information of previous hash codes. We define the state as
a tuple (h, i), where h is a history action vector of the generated hash codes, and i is the image
feature vector. The image feature vector is extracted from the original images using a pre-trained
CNN model, and the history action vectors can be obtained from the policy network.
3.2.2 Action. Given the state tuple s = (h, i), the agent will predict the probability of the actions
for current state. It is noted that the hashing problem have only two possible actions (1 or 0) and
the sum of the action probability equals to 1. Different from most of the reinforcement methods
which predict the probability distribution for every possible action, we just take the probability of
the hash code 1 as the policy network output. The overall probability distribution is formulated as
follows:

1 − policy(s, θ ) a=0
P(a|s, θ ) =
(3)
policy(s, θ ) a=1
where the policy(s, θ ) denotes the output of the policy network with input state s and parameters θ .
Considering that only one bit of hash code does not have enough ability to correct the history
ranking errors, we propose the action group to address this problem. An action group is composed
by k adjacent hash functions. Every action in the group share the same reward which is designed
to enhance the ability to correct the ranking errors. The action group is the minimal step for the
agent which is the actual action definition in the framework. The action probability πθ (si , Ai ) is
formulated as follows:
Ai = [ati +1 , ati +2 , ..ati +k ]
πθ (si , Ai ) =

k
Ö

P(ati +j |ŝi, j , θ )

(4)

j=1

where Ai is the i − th action group, ati +1..k is the element of the action group, ŝi, j is the input state
of the ati +1 and will be given more details in next sub section.
3.2.3 Reward. After we obtain the generated hash codes, we use a triplet ranking loss as the
reward function to measure the quality of generated codes. It is noted that the probability of the
hash code 1 can be regarded as relaxed hash code, which is widely used in the many hashing
methods [17, 45, 49]. The higher probability of the action to project images into hash code 1, the
value of the probability is closer to 1 and vice versa. So we adopt the probability sequence of
projection images into hash code 1 as the relaxed hash code to calculate the reward.
t ,
For training images (X, Y), we sample a set of triplet tuples based on the labels, T = {(x i , x i+ , x i− )}i=1
+
−
in which x i and x i are two similar images with the same labels, while x i and x i are two dissimilar
images with different labels, and t is the number of sampled triplet tuples. For the triplet tuples
(x i , x i+ , x i− ), i = 1 · · · t, the reward function is defined as:
J (h(x i ), h(x i+ ), h(x i− )) =
max(0, mt + ∥h(x i ) − h(x i+ )∥ 2 − ∥h(x iL ) − h(x i− )∥ 2 )

(5)

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing

:7

where ∥ · ∥ 2 denotes the Euclidean distance, h(·) denotes the probability sequence to the correspondence image, and the constant parameter mt defines the margins between the relative similarity of
the two pairs (x i , x i+ ) and (x i , x i− ), that is to say, we expect the distance of dissimilar pair (x i , x i− ) to
be larger than the distance of similar pair (x i , x i+ ) by a margin of at least mt . Triplet ranking loss
based reward can well reflect the semantic ranking information, which well evaluates the quality
of previous generated hash codes.
There are two hierarchical rewards to encourage the agent to find the correct hash bits. The first
reward is the action group reward which mainly focuses on the hash code quality in the group
level. The second reward is the global action reward which focus on the qulity of the whole hash
code. The rewards can be formulated as:
д

Ri, j = −J (h j (x i ), h j (x i+ ), h j (x i− ))
+
−
RG
i = −J (h(x i ), h(x i ), h(x i ))

(6)

д

where Ri, j denotes the group action reward of the i − th image in j − th action group, h j denotes
the probability sequence of the j − th action group, and RG
i denotes the global action reward of the
i − th image.
3.3

Deep Reinforcement Learning Hashing Network

The overall framework of our proposed deep reinforcement learning network is shown in Figure 1,
which consists of two parts. The first part is the environment including a representation network
and a reward function which provides the reward and state for the agent. The second part of the
framework is a policy network as the agent, which obtains the state from the environment and
generates the hash codes.
3.3.1 The representation network. The representation network serves as a feature extractor, it is
a deep convolutional network composed of several convolutional layers and fully connected layers.
The representation network provides the image features which are denoted as i in the state tuple
(h, i). We adopt the VGG-19 network [34] as the representation network. The first 18 layers follow
the exactly same settings in the VGG-19 network. We mainly focus on the design of policy network
and the reward function.
3.3.2 Policy network. The policy network is composed of a RNN layer and a policy layer. RNN
layer transforms the image features into an internal state, while the policy layer further maps the
internal state into a policy probability. The main idea behind the RNN model is a built-in memory
cell, which stores information over previous steps and implies the history actions taken in the
past. The Figure 2 shows the details of the policy network, which maps the state tuple (h, i) to the
probability of an action group. The memory cell aggregates information from two sources: the
previous cell memory unit, and the input vector in current step. Formally, for a state tuple (h, i),
the RNN layer maps the input to an output sequence by computing activations of the units in the
network with the following equation recursively:
c t = tanh(Wx i x t + bx i + Whi c t −1 + bhi )

(7)

where x t , c t are the input and hidden vectors, t denotes the t-th step, Wxi and Whi are the weight
matrix from the input x t and hidden vectors to the new hidden state, bxi and bhi are the bias terms.
We initialize the RNN with c 0 = h and x 0 = i. But in the following step, we set the x i = c i−1
to emphasis the history action information in one action group. Although the image feature is
only the input of the first step, the information of the image feature remains in the hidden state
with the changing history action information. Thus the hash codes are influenced by the history
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:8

J. Zhang et al.

information and adjusted to correct the errors caused by previous actions. The last hidden state c k
will be regarded as the history information h to synthesis the next step state.
The policy layer is a fully connect layer, and is defined as:
ht (x) = siдmoid(WhT c t + v)

(8)

where c t is the output extracted from RNN layer in step t, Wh denotes the weights in the policy layer,
and v is the bias parameter. Through the policy layer, the output of RNN at step t is mapped into
[0, 1]. We apply a threshold function to obtain the final binary codes from the policy probability:
bk (x) = д(h(x)) = sдn(hk (x) − 0.5),
3.4

k = 1, · · · , q

(9)

Agent Training strategy

Image
feature

History

Global reward

Image
feature

π(a|s)

RNN
Cell
History

Action group
Image
feature

π(a|s)

π(a|s)

Action
group
π(a|s)

Image
feature

Action group
reward

History

Fig. 2. The unrolled details of the policy network, during the training stage, global reward ensures the
generated codes to preserve the semantic ranking information, while the action group reward drives the
network to correct the errors caused by previous generated hash codes.

Finally we introduce the agent training strategy. Firstly we unroll the policy network to better
show the details, as shown in Figure 2, we use two reward functions to drive the training of policy
network which are defined in equation 6.
We use the Monte-Carlo Policy Gradient [41] to update the parameters to maximize the expected
total reward of the action group.:
Õ д
Lд (θ ) = E A∼π (A |s;θ ) ( Ri )
i

д

≈

Õ

=

ÕÕ

i
i

loд[π (Ai |si ; θ )]Ri

(10)
д

loд[P(ai,k |ŝi,k ; θ )]Ri

k

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing

:9

where Lд (θ ) is the expected total reward of the action group.
The global action reward mainly focuses on the quality of the whole set of hash codes, we adopt
gradient decent method to optimize the global action reward. According to equation (5), we can
compute the sub-gradient for each triplet tuple (x i , x i+ , x i− ), with respect to h(x i ), h(x i+ ) and h(x i− )
as:
∂RG
i

+

= 2(h(x iL ) − h(x iL )) × Ic
−

∂h(x iL )
∂RG
i

+
∂h(x iL )
∂RG
i
−
∂h(x iL )

+

= 2(h(x iL ) − h(x iL )) × Ic

(11)

= 2(h(x iL ) − h(x iL )) × Ic
−

Ic = Imt + ∥h(x L )−h(x L+ ) ∥ 2 − ∥h(x L )−h(x L− ) ∥ 2 >0
i

i

i

i

where Ic is an indicator function, Ic = 1 if c is true, otherwise Ic = 0. Thus the global reward can
ensure the generated codes to preserve the semantic ranking information.
At last, we explain how the policy network has the ability to correct the history ranking errors
through the details of the RNN gradient. The gradient of the RNN is formulated as:
δht = θ ′(c t )(δkt w h + δht +1w hi )
∂JL
∂c t
∂J
△
L
δkt =
∂ht
δht =
△

(12)

where c t and ht are the hidden state and output of RNN respectively, θ (c t ) denotes the function
which maps the c t to ht . The gradient of hidden layer in the step t consists of two parts: the
gradient from the output in step t, and the gradient from the hidden layer in t + 1 step. The latter is
considered as the sequential reward which can correct the errors caused by previous generated
hash codes. In the training stage, the previous hash functions will receive the gradient information
from current hash function and update the parameters to improve the retrieval accuracy.
4

EXPERIMENT

In this section, we will introduce our experiments conducted on 3 widely used datasets (CIFAR10, NUS-WIDE and MIRFlickr) with 8 state-of-the-art methods, including unsupervised methods
LSH [7], SH [40], ITQ [8] and supervised methods SDH [32], CNNH [42] NINH [17], DSH [21]
and HashNet [2]. LSH, SH, SDH, and ITQ are traditional hashing methods without deep networks,
while CNNH, NINH, DSH, HashNet and our proposed DRLIH are deep hashing methods which
take the raw image pixels as input to conduct hashing function learning.
4.1

Datasets

We conduct experiments on three widely used image retrieval datasets:
• The CIFAR10 [14] dataset consists of 60000 32 × 32 color images from 10 classes, each
of which has 6000 images. For the fairly comparison, following [17, 42], 1000 images are
randomly selected as the query set (100 images per class). For the unsupervised methods, all
the rest of the images are used as the training set. For the supervised methods, 5000 images
(500 images per class) are further randomly selected to form the training set.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:10

J. Zhang et al.
• The NUS-WIDE [5] dataset contains nearly 270000 images, each image is associated with
one or multiple labels from 81 semantic concepts. Following [17], only the 21 most frequent
concepts are used, where each concept has at least 5000 images. Following [17, 42], 2100
images are randomly selected as the query set (100 images per concept). For the unsupervised
methods, all the rest of images are used as the training set. For the supervised methods, 500
images from each of the 21 concepts are randomly selected to form the training set of total
10500 images.
• The MIRFlickr [10] dataset consists of 25,000 images collected from Flickr, and each image
is associated with one or multiple labels of 38 semantic concepts. 1000 images are randomly
selected as the query set. For the unsupervised methods, all the rest images are used as the
training set. For the supervised methods, 5000 images are randomly selected from the rest of
images to form the training set.

Table 1. MAP scores with different length of hash codes on CIFAR10, NUS-WIDE and MIRFlickr datasets.
MAP scores are calculated based on top 5000 returned images for NUS-WIDE dataset. The best results of
each code length are shown in boldface.

12bit
0.816
0.765
0.708
0.792
0.683
0.430
0.339
0.244
0.133
0.255
0.158
0.124
0.116

DRLIH (ours)
HashNet∗
DSH∗
NINH∗
CNNH∗
SDH-VGG19
ITQ-VGG19
SH-VGG19
LSH-VGG19
SDH
ITQ
SH
LSH

CIFAR10
24bit 32bit
0.843 0.855
0.823 0.840
0.712 0.751
0.818 0.832
0.692 0.667
0.652 0.653
0.361 0.368
0.213 0.213
0.171 0.178
0.330 0.344
0.163 0.168
0.125 0.125
0.121 0.124

CIFAR-10

0.9

0.6

Precision @ 48 bits

Precision @ hamm.dist.<=2

0.7

0.5
0.4
0.3
0.2
0.1

NUS-WIDE
24bit 32bit
0.846 0.845
0.833 0.830
0.804 0.815
0.827 0.827
0.784 0.790
0.797 0.819
0.800 0.806
0.697 0.689
0.567 0.618
0.510 0.519
0.478 0.483
0.445 0.443
0.414 0.432

12bit
0.823
0.812
0.793
0.808
0.768
0.730
0.777
0.712
0.518
0.460
0.472
0.452
0.436

CIFAR-10

0.9

0.8

0
10

48bit
0.853
0.843
0.720
0.830
0.623
0.665
0.375
0.209
0.198
0.360
0.169
0.126
0.131

0.8

0.7

0.7

0.6
0.5
0.4

30

Number of bits
(a)

40

50

12bit
0.796
0.777
0.651
0.772
0.763
0.732
0.686
0.618
0.575
0.595
0.576
0.561
0.557

MIRFlickr
24bit 32bit
0.811 0.810
0.782 0.785
0.681 0.684
0.756 0.760
0.757 0.758
0.739 0.737
0.685 0.687
0.604 0.598
0.584 0.604
0.601 0.608
0.579 0.579
0.562 0.563
0.564 0.562

DRLIH(ours)
HashNet*
DSH*
NINH*
CNNH*
SDH-VGG19
ITQ-VGG19
SH-VGG19
LSH-VGG19
SDH
ITQ
SH
LSH

0.5
0.4
0.3

0.2

0.2

48bit
0.814
0.785
0.686
0.778
0.755
0.747
0.689
0.595
0.614
0.605
0.580
0.562
0.569

CIFAR-10

0.6

0.3

0.1
20

48bit
0.853
0.840
0.800
0.827
0.740
0.830
0.817
0.682
0.651
0.525
0.476
0.437
0.442

0.9

0.8

Precision @ 48 bits

Methods

0.1
200

400

600

800

Number of retrieved samples
(b)

1000

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall @ 48 bits
(c)

Fig. 3. The comparison results on CIFAR10. (a) Precision within Hamming radius 2 using hash lookup; (b)
Precision at top k returned results. (c) Precision-Recall curves of Hamming Ranking with 48bit.

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing
NUS-WIDE

0.9

NUS-WIDE

0.9

Precision @ 48 bits

0.5
0.4
0.3

Precision @ 48 bits

0.75

0.6

0.7
0.65
0.6
0.55

0.2

0.5

0.1

DRLIH(ours)
HashNet*
DSH*
NINH*
CNNH*
SDH-VGG19
ITQ-VGG19
SH-VGG19
LSH-VGG19
SDH
ITQ
SH
LSH

0.8

0.8

0.7

Precision @ hamm.dist.<=2

NUS-WIDE

0.9
0.85

0.8

0
10

:11

0.7

0.6

0.5

0.4

0.45
0.4
20

30

40

50

0.3
200

Number of bits
(a)

400

600

800

1000

0

0.1

0.2

0.3

Number of retrieved samples
(b)

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall @ 48 bits
(c)

Fig. 4. The comparison results on NUS-WIDE. (a) Precision within Hamming radius 2 using hash lookup; (b)
Precision at top k returned results. (c) Precision-Recall curves of Hamming Ranking with 48bit.
MIRFLICKR

0.9

MIRFLICKR

0.9

0.8

0.9

0.85

0.7

DRLIH(ours)
HashNet*
DSH*
NINH*
CNNH*
SDH-VGG19
ITQ-VGG19
SH-VGG19
LSH-VGG19
SDH
ITQ
SH
LSH

0.85

0.6
0.5
0.4
0.3

Precision @ 48 bits

0.8

Precision @ 48 bits

Precision @ hamm.dist.<=2

MIRFLICKR

0.95

0.75

0.7

0.8
0.75
0.7
0.65

0.65
0.2

0.6
0.6

0.1
0
10

0.55

0.55
20

30

Number of bits
(a)

40

50

0.5
200

400

600

800

Number of retrieved samples
(b)

1000

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Recall @ 48 bits
(c)

Fig. 5. The comparison results on MIRFLICKR. (a) Precision within Hamming radius 2 using hash lookup;
(b) Precision at top k returned results. (c) Precision-Recall curves of Hamming Ranking with 48bit.

4.2

Experiment Settings and Evaluation Metrics

We implement the proposed approach based on the open-source framework pytorch 1 . The parameters of the first 18 layers in our network are initialized with the VGG-19 network [34], which is
pre-trained on the ImageNet dataset [30]. Similar initialization strategy has been used in other
deep hashing methods [21, 43, 49]. The dimension of RNN’s hidden layer is set to be 4096 in the
policy network. In all experiments, our networks are trained with the initial learning rate of 0.001,
we decrease the learning rate by a factor of 10 every 10000 steps. And the mini-batch size is 16,
the weight decay parameter is 0.0005. For the parameter in our proposed loss function, we simply
set mt = 1 in all the experiments. For the length of action group, we set it as 12 through out the
experiments.
We compare the proposed DRLIH approach with eight state-of-the-art methods, including
unsupervised methods LSH, SH and ITQ, supervised methods SDH, CNNH, NINH, DSH and
HashNet. The brief introductions of these methods are listed as follows:
• LSH [7] is a data independent unsupervised method, which uses randomly generated hash
functions to map image features into binary codes.
• SH [40] is a data dependent unsupervised method, which learns hash functions by making
hash codes balanced and uncorrelated.
1 http://pytorch.org

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:12

J. Zhang et al.
• ITQ [8] is also a data dependent unsupervised method, which learns hash functions by
minimizing the quantization error of mapping data to the vertices of a binary hypercube.
• SDH [32] is a supervised method, which leverages label information to obtain hash codes by
integrating hash code generation and classifier training.
• CNNH [42] is a two-stage deep hashing method, which learns hash codes for training images
in first stage, and trains a deep hashing network in second stage.
• NINH [17] is a one-stage deep hashing method, which learns deep hashing network by a
triplet loss function to measure the ranking information provided by labels.
• DSH [21] is a one-stage deep hashing method, which designs a loss function to maximize
the discriminability of the output space by encoding the supervised information from the
input image pairs, and simultaneously imposing regularization on the real-valued outputs to
approximate the desired discrete values.
• HashNet [2] directly learns the binary hash codes and addresses the ill-posed gradient and
data imbalance problem in an end-to-end framework of deep feature learning and binary
hash encoding.

For our proposed DRLIH, and compared CNNH, NINH, DSH and HashNet methods, we use the
raw image pixels as input. The implementations of CNNH, DSH and HashNet are provided by
their authors, while NINH is our own implementation. Since the representation learning layers of
CNNH, NINH, DSH and HashNet are different from each other, for a fair comparison, we use the
same VGG-19 network as the base structure for all the deep hashing methods. And the network
parameters of all the deep hashing methods are initialized with the same pre-trained VGG-19 model,
thus we can perform a fair comparison between them. The results of CNNH, NINH, DSH and
HashNet are referred as CNNH∗, DSH∗, NINH∗ and HashNet∗ respectively.
For other compared traditional methods without deep networks, we represent each image by
hand-crafted features and deep features respectively. For hand-crafted features, we represent images
in the CIFAR10 and MIRFlickr by 512-dimensional GIST features, and images in the NUS-WIDE
by 500-dimensional bag-of-words features. For a fair comparison between traditional methods
and deep hashing methods, we also conduct experiments on the traditional methods with deep
features, where we extract 4096-dimensional deep feature for each image from the same pretrained VGG-19 network. We denote the results of traditional methods using deep features by
LSH-VGG19, SH-VGG19, ITQ-VGG19 and SDH-VGG19. The results of SDH, SH and ITQ are obtained
from the implementations provided by their authors, while the results of LSH are from our own
implementation.
To objectively and comprehensively evaluate the retrieval accuracy of the proposed approach and
the compared methods, we use four evaluation metrics: Mean Average Precision (MAP), precision
at top k returned results, precision-recall curves and precision within Hamming radius 2 using
hash lookup. These four evaluation metrics are defined as follows:
• The MAP scores are computed as the mean of average precision (AP) for all queries, and AP
is computed as:
n
1Õ k
AP =
× relk
(13)
R
Rk
k =1

where n is the size of database, R is the number of relevant images in database, Rk is the
number of relevant images in the top k returns, and relk = 1 if the image ranked at k-th
position is relevant and 0 otherwise.
• Precision at top k returned results (topK-precision): The precision with respect to different
numbers of retrieved samples from the ranking list.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing

:13

• Precision within Hamming radius 2: Precision curve of returned images with the Hamming
distance smaller than 2 using hash lookup.
4.3

Experiment Results

4.3.1 Experiment results on CIFAR10 dataset. Table 1 shows the MAP scores with different length
of hash codes on CIFAR10 dataset. Overall, the proposed DRLIH achieves the highest average MAP
of 0.842, and consistently outperforms state-of-the-art methods on all hash code lengths. More
specifically, the result tables are partitioned into three groups: deep hashing methods, traditional
methods with deep features and traditional methods with hand-crafted features. Compare with
the highest deep hashing methods HashNet∗, which achieves average MAP of 0.818, the proposed
DRLIH has an absolute improvement of 0.024. Compare with the highest traditional methods using
deep features SDH-VGG19, which achieves an average MAP of 0.600, the proposed method has an
absolute improvement of 0.242. While the highest traditional methods using handcrafted features
SDH achieves average MAP of 0.322, the proposed approach has an absolute improvement of 0.520.
Figure 3(a) shows the precisions within Hamming radius 2 using hash lookup. The precision
of proposed DRLIH consistently outperforms state-of-the-art methods on all hash code lengths.
The precision of most traditional methods decrease when using longer hash codes. This is because
the number of images sharing the same Hamming code decreases exponentially for longer hash
codes (e.g., 48bit), which will cause some queries fail to return images within Hamming radius
2. While the proposed DRLIH achieves the highest precision on 48bit code length, which shows
the robustness of proposed method on longer hash codes. Figure 3(b) shows the precision at top k
returned results, we can also observe that our proposed DRLIH achieves the best precision compared
with state-of-the-art methods. Figure 3(c) demonstrates the precision-recall curves using Hamming
ranking with 48bit codes. DRLIH still achieves the best accuracy on all recall levels, which further
demonstrates the effectiveness of proposed method.
4.3.2 Experiment results on NUS-WIDE dataset. Table 1 shows the MAP scores with different
length of hash codes on NUS-WIDE dataset. Following [42], we calculate the MAP scores based on
top 5000 returned images. Similar results can be observed, our proposed DRLIH still achieves the
best MAP scores (average 0.842). DRLIH achieves an absolute improvement of 0.013 on average
MAP compared to the highest deep methods HashNet∗ (average 0.829). Compare with the highest
traditional method using deep features ITQ-VGG19, which achieves an average MAP of 0.800,
DRLIH has an absolute improvement of 0.042. It is also interesting to observe that with the deep
features extracted from VGG-19 network, the traditional methods such as SDH and ITQ achieve
comparable results with deep hashing methods.
Figures 4(a), (b) and (c) demonstrate the precision within Hamming radius 2 using hash lookup,
precision at top k returned results and the precision-recall curves using Hamming ranking with 48
bits. Similar trends can be observed, on these three evaluation metrics, the proposed DRLIH also
achieves promising results on NUS-WIDE dataset on all hash code lengths, which further shows
the effectiveness of our proposed DRLIH.
4.3.3 Experiment results on MIRFLICKR dataset. Table 1 shows the MAP scores with different
length of hash codes on MIRFlickr dataset. We can also observe that our proposed DRLIH still
achieves the best MAP scores (average 0.808). DRLIH achieves an absolute improvement of 0.026 on
average MAP compared to the highest deep methods HashNet∗ (average 0.782). Compare with the
highest traditional method using deep features SDH-VGG19, which achieves an average MAP of
0.739, DRLIH has an absolute improvement of 0.069. Compare with the highest traditional method
using hand-crafted features SDH, which achieves an average MAP of 0.602, DRLIH has an absolute
improvement of 0.206.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:14

J. Zhang et al.

We can observe from Figure 4(a) that the proposed DRLIH achieves the best precision within
Hamming radius 2 using hash lookup. Figure 4(b) demonstrates the precision at top k returned
results, our proposed DRLIH achieves the best precision compared with state-of-the-art methods.
Figure 4(c) shows the precision-recall curves using Hamming ranking with 48 bits, similar trends
can be observed, the proposed DRLIH also achieves best precision on all recall levels compared
with state-of-the-art methods.
Finally, we demonstrate the top 10 retrieval results of NUS-WIDE using Hamming ranking on
48bit hash codes. As shown in Figure 6, our proposed DRLIH achieves the best results.
query

top10 returned samples

query

top10 returned samples

DRLIH
HashNet*
NINH*

DRLIH
HashNet*
NINH*

Fig. 6. Some retrieval results of NUS-WIDE dataset using Hamming ranking on 48bit hash codes. The blue
rectangles denote the query images. The red rectangles indicate wrong retrieval results. We can observe that
DRLIH achieves the best results.

Table 2. Comparison between baseline method and our proposed approach with different length of hash
codes on CIFAR10, NUS-WIDE and MIRFLICKR dataset.

Methods
DRLIH (ours)
Baseline_noseq
Baseline_ng

12bit
0.816
0.759
0.805

CIFAR10
24bit 32bit
0.843 0.855
0.802 0.811
0.823 0.820

48bit
0.853
0.822
0.826

12bit
0.823
0.803
0.811

NUS-WIDE
24bit 32bit
0.846 0.845
0.811 0.820
0.824 0.829

48bit
0.853
0.824
0.835

12bit
0.796
0.764
0.774

MirFlickr
24bit 32bit
0.811 0.810
0.775 0.776
0.778 0.784

48bit
0.814
0.784
0.789

4.3.4 Baseline experiments. To verify the effectiveness of action group and sequence learning
strategy of deep reinforcement learning, we also conduct two baseline experiments. More specifically, we first set the length of the action group as 1, which implies that we train the network
without action group, we denote this method as Baseline_ng. We also conduct a baseline experiment
without the sequence learning strategy, we replace the agent with a fully-connected layer whose
dimension is the same as the hash code length and train the network by the triplet ranking loss.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing
DRLIH(Ours)

Baseline_noseq

CIFAR-10

0.9

0.8

0.6

0.5

0.4

0.7

0.87

Precision @ 48 bits

0.7

0.86
0.85
0.84

40

Number of bits
(a)

0.4

0.1
200

50

0.5

0.2

0.82
30

0.6

0.3

0.83

20

CIFAR-10

0.9

0.88

Precision @ 48 bits

Precision @ hamm.dist.<=2

Baseline_ng

CIFAR-10

0.89

0.8

0.3
10

:15

400

600

800

1000

0

Number of retrieved samples
(b)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

Recall @ 48 bits
(c)

Fig. 7. The comparison results of baseline methods on CIFAR10. (a) Precision within Hamming radius 2
using hash lookup; (b) Precision at top k returned results; (c) Precision-Recall curves of Hamming Ranking
with 48bit.
DRLIH(Ours)

Baseline_noseq

NUS-WIDE

0.86

0.78
0.76

0.8

Precision @ 48 bits

Precision @ 48 bits

Precision @ hamm.dist.<=2

0.8

0.86

0.85

0.84

0.83

0.74

30

40

0.6

0.5

0.3
200

50

Number of bits
(a)

0.7

0.4

0.82
20

NUS-WIDE

0.9

0.87

0.82

0.72
10

NUS-WIDE

0.88

0.84

Baseline_ng

400

600

800

1000

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

Recall @ 48 bits
(c)

Number of retrieved samples
(b)

Fig. 8. The comparison results of baseline methods on NUS-WIDE. (a) Precision within Hamming radius 2
using hash lookup; (b) Precision at top k returned results; (c) Precision-Recall curves of Hamming Ranking
with 48bit.
DRLIH(Ours)

Baseline_noseq

MIRFLICKR

0.9

Baseline_ng

MIRFLICKR

0.9

MIRFLICKR

0.95
0.9

0.89

0.86

0.84

0.82

Precision @ 48 bits

0.85

Precision @ 48 bits

Precision @ hamm.dist.<=2

0.88

0.88

0.87

0.86

0.8
0.75
0.7
0.65
0.6

0.8

0.85
0.55

0.78
10

0.5

0.84
20

30

Number of bits
(a)

40

50

200

400

600

800

Number of retrieved samples
(b)

1000

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

Recall @ 48 bits
(c)

Fig. 9. The comparison results of baseline methods on MIRFLICKR. (a) Precision within Hamming radius 2
using hash lookup; (b) Precision at top k returned results; (c) Precision-Recall curves of Hamming Ranking
with 48bit.

This layer serves as hashing layer that maps features of representation network into binary codes
directly and independently. We denote this method as Baseline_noseq. Compare the proposed
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:16

J. Zhang et al.

DRLIH approach with Baseline_ng, we can verify the effectiveness of action group. Compare our
DRLIH approach with Baseline_noseq, we can verity the effectiveness of the sequence learning
strategy. The results are shown in table 2, and we can observe that compare with Baseline_ng,
our proposed DRLIH approach improves the average MAP score from 0.819 to 0.842 on CIFAR10
dataset, from 0.825 to 0.842 on NUS-WIDE dataset and from 0.781 to 0.808 on MIRFlickr dataset.
This demonstrates the that the proposed action group method can promote the retrieval accuracy.
Compare with Baseline_noseq, our proposed DRLIH approach improves the average MAP score
from 0.799 to 0.842 on CIFAR10 dataset, from 0.815 to 0.842 on NUS-WIDE dataset and from 0.775 to
0.808 on MIRFlickr dataset. This demonstrates the effectiveness of our proposed sequence learning
strategy of deep reinforcement learning hashing network. We can also observer that Baseline_ng
has better performance than the Baseline_noseq, which demonstrates the effectiveness of the sequential learning strategy. Figure 7, 8 and 9 show the precision within hamming radius 2, precision
at top k returned results and precision recall curves on three datasets, we can observe that the
proposed DRLIH approach outperforms two baseline methods on these three evaluation metrics.
5

CONCLUSION

In this paper, we have proposed a novel Deep Reinforcement Learning approach for Image Hashing
(DRLIH). First, we propose a policy based deep reinforcement learning network for modeling
hashing functions. We utilize Recurrent neural network (RNN) to model hashing functions as
agents, which take actions of projecting images into binary codes sequentially. While we regard
hash codes and the image features as states, which provide history actions taken by agents. The
whole network is trained by optimizing two hierarchical reward functions. Second, we propose a
sequential learning strategy based on proposed DRLIH, which can iteratively optimize the overall
accuracy by correcting the error generated by history actions. Experiments on three widely used
datasets demonstrate the effectiveness of our proposed approach.
The future work lies in two aspects: First, we will try to define the sequential learning process
explicitly, such that we can better model the sequential learning. Second, we intend to exploit more
advanced deep reinforcement learning framework to achieve better retrieval accuracy.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grants 61771025
and 61532005.
REFERENCES
[1] Juan C Caicedo and Svetlana Lazebnik. 2015. Active object localization with deep reinforcement learning. In Computer
Vision (ICCV), 2015 IEEE International Conference on. IEEE, 2488–2496.
[2] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. 2017. HashNet: Deep Learning to Hash by Continuation.
In The IEEE International Conference on Computer Vision (ICCV).
[3] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2010. Large scale online learning of image similarity through
ranking. Journal of Machine Learning Research (JMLR) 11, Mar (2010), 1109–1135.
[4] Zhixiang Chen, Jiwen Lu, Jianjiang Feng, and Jie Zhou. 2017. Nonlinear Discrete Hashing. IEEE Transactions on
Multimedia (TMM) 19, 1 (2017), 123–135.
[5] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2014. NUS-WIDE: a real-world
web image database from National University of Singapore. In ACM international conference on image and video retrieval
(CIVR). 48.
[6] Li Fei-Fei and Pietro Perona. 2005. A bayesian hierarchical model for learning natural scene categories. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). 524–531.
[7] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. 1999. Similarity search in high dimensions via hashing. In
International Conference on Very Large Data Bases (VLDB), Vol. 99. 518–529.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

Deep Reinforcement Learning for Image Hashing

:17

[8] Yunchao Gong and Svetlana Lazebnik. 2011. Iterative quantization: A procrustean approach to learning binary codes.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 817–824.
[9] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving
neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012).
[10] Mark J Huiskes and Michael S Lew. 2008. The MIR flickr retrieval evaluation. In ACM International Conference on
Multimedia Information Retrieval (MIR). 39–43.
[11] Go Irie, Zhenguo Li, Xiao-Ming Wu, and Shih-Fu Chang. 2014. Locally linear hashing for extracting non-linear
manifolds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2115–2122.
[12] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of
artificial intelligence research 4 (1996), 237–285.
[13] Mehran Kafai, Kave Eshghi, and Bir Bhanu. 2014. Discrete cosine transform locality-sensitive hashes for face retrieval.
IEEE Transactions on Multimedia (TMM) 16, 4 (2014), 1090–1103.
[14] Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. (2009).
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural
networks. In Annual Conference on Neural Information Processing Systems (NIPS). 1097–1105.
[16] Brian Kulis and Trevor Darrell. 2009. Learning to hash with binary reconstructive embeddings. In Annual Conference
on Neural Information Processing Systems (NIPS). 1042–1050.
[17] Hanjiang Lai, Yan Pan, Ye Liu, and Shuicheng Yan. 2015. Simultaneous feature learning and hash coding with deep
neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3270–3278.
[18] Peng Li, Meng Wang, Jian Cheng, Changsheng Xu, and Hanqing Lu. 2013. Spectral hashing with semantically consistent
graph for image indexing. IEEE Transactions on Multimedia (TMM) 15, 1 (2013), 141–152.
[19] Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. 2016. Feature learning based deep supervised hashing with pairwise
labels. In International Joint Conference on Artificial Intelligence (IJCAI). 1711–1717.
[20] Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).
[21] Haomiao Liu, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2016. Deep supervised hashing for fast image retrieval.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2064–2072.
[22] Wei Liu, Jun Wang, Rongrong Ji, Yu-Gang Jiang, and Shih-Fu Chang. 2012. Supervised hashing with kernels. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR). 2074–2081.
[23] Wei Liu, Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. 2011. Hashing with graphs. In International Conference on
Machine Learning (ICML). 1–8.
[24] Qin Lv, William Josephson, Zhe Wang, Moses Charikar, and Kai Li. 2007. Multi-probe LSH: efficient indexing for
high-dimensional similarity search. In International conference on Very large data bases (VLDB). 950–961.
[25] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves,
Martin Riedmiller, Andreas K Fidjeland, and Georg Ostrovski. 2015. Human-level control through deep reinforcement
learning. Nature 518, 7540 (2015), 529.
[26] Aude Oliva and Antonio Torralba. 2001. Modeling the shape of the scene: A holistic representation of the spatial
envelope. International Journal of Computer Vision (IJCV) 42, 3 (2001), 145–175.
[27] Maxim Raginsky and Svetlana Lazebnik. 2009. Locality-sensitive binary codes from shift-invariant kernels. In Annual
Conference on Neural Information Processing Systems (NIPS). 1509–1517.
[28] Yongming Rao, Jiwen Lu, and Jie Zhou. 2017. Attention-Aware Deep Reinforcement Learning for Video Face Recognition.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3931–3940.
[29] Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. 2017. Deep Reinforcement Learning-Based Image
Captioning with Embedding Reward. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on.
IEEE, 1151–1159.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Aditya Khosla, Michael Bernstein, et al. 2015. ImageNet Large Scale Visual Recognition Challenge. International
Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252.
[31] Matthew Schultz and Thorsten Joachims. 2003. Learning a Distance Metric from Relative Comparisons. In Advances in
Neural Information Processing Systems (NIPS). 41–48.
[32] Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. 2015. Supervised discrete hashing. In IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). 37–45.
[33] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser,
Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. 2016. Mastering the game of Go with deep neural
networks and tree search. nature 529, 7587 (2016), 484–489.
[34] K. Simonyan and A. Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. In
International Conference on Learning Representations (ICLR).

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

:18

J. Zhang et al.

[35] Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. 2010. Sequential projection learning for hashing with compact codes. In
International Conference on Machine Learning (ICML). 1127–1134.
[36] Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. 2012. Semi-supervised hashing for large-scale search. IEEE Transactions
on Pattern Analysis and Machine Intelligence 34, 12 (2012), 2393–2406.
[37] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu.
2014. Learning fine-grained image similarity with deep ranking. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). 1386–1393.
[38] Jianfeng Wang, Jingdong Wang, Nenghai Yu, and Shipeng Li. 2013. Order preserving hashing for approximate nearest
neighbor search. In ACM International Conference on Multimedia (ACM-MM). 133–142.
[39] Qifan Wang, Zhiwei Zhang, and Luo Si. 2015. Ranking preserving hashing for fast similarity search. In International
Joint Conference on Artificial Intelligence (IJCAI). 3911–3917.
[40] Yair Weiss, Antonio Torralba, and Rob Fergus. 2009. Spectral hashing. In Annual Conference on Neural Information
Processing Systems (NIPS). 1753–1760.
[41] Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Springer, 5–32.
[42] Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. 2014. Supervised Hashing for Image Retrieval via
Image Representation Learning. In AAAI Conference on Artificial Intelligence (AAAI). 2156–2162.
[43] Ting Yao, Fuchen Long, Tao Mei, and Yong Rui. 2016. Deep Semantic-Preserving and Ranking-Based Hashing for
Image Retrieval. In International Joint Conference on Artificial Intelligence (IJCAI). 3931–3937.
[44] Lei Zhang, Yongdong Zhang, Jinhui Tang, Xiaoguang Gu, Jintao Li, and Qi Tian. 2013. Topology preserving hashing
for similarity search. In ACM International Conference on Multimedia (ACM-MM). 123–132.
[45] Ruimao Zhang, Liang Lin, Rui Zhang, Wangmeng Zuo, and Lei Zhang. 2015. Bit-scalable deep hashing with regularized
similarity learning for image retrieval and person re-identification. IEEE Transactions on Image Processing (TIP) 24, 12
(2015), 4766–4779.
[46] Yongdong Zhang, Lei Zhang, and Qi Tian. 2014. A prior-free weighting scheme for binary code ranking. IEEE
Transactions on Multimedia (TMM) 16, 4 (2014), 1127–1139.
[47] Dongbin Zhao, Yaran Chen, and Le Lv. 2016. Deep reinforcement learning with visual attention for vehicle classification.
IEEE Transactions on Cognitive and Developmental Systems (2016).
[48] Fang Zhao, Yongzhen Huang, Liang Wang, and Tieniu Tan. 2015. Deep semantic ranking based hashing for multi-label
image retrieval. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1556–1564.
[49] Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. 2016. Deep Hashing Network for Efficient Similarity Retrieval.
In AAAI Conference on Artificial Intelligence (AAAI). 2415–2421.

ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 1, No. 1, Article . Publication date:
February 2018.

