# Kernel-Based Multiview Joint Sparse Coding for Image Annotation

Authors: Miao Zang; Huimin Xu; Yongmei Zhang
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

It remains a challenging task for automatic image annotation problem due to the semantic gap between visual features and semantic concepts. To reduce the gap, this paper puts forward a kernel-based multiview joint sparse coding (KMVJSC) framework for image annotation. In KMVJSC, different visual features as well as label information are considered as distinct views and are mapped to an implicit kernel space, in which the original nonlinear separable data become linearly separable. Then, all the views are integrated into a multiview joint sparse coding framework aiming to find a set of optimal sparse representations and discriminative dictionaries adaptively, which can effectively employ the complementary information of different views. An optimization algorithm is presented by extending K-singular value decomposition (KSVD) and accelerated proximal gradient (APG) algorithms to the kernel multiview framework. In addition, a label propagation scheme using the sparse reconstruction and weighted greedy label transfer algorithm is also proposed. Comparative experiments on three datasets have demonstrated the competitiveness of proposed approach compared with other related methods.


---

## Body

## 1. Introduction

With the surge of private and online images, automatic image annotation has become of great research interest in computer vision since it is a vital step for image retrieval and management [1, 2]. Image annotation aims to automatically predict a set of semantic labels, such as “sea,” “beach,” and “sand,” for the unannotated images by learning the relevance among images. However, the visual similar images may not be correlated with each other in semantics, which makes the problem still challenging.In the past decade, considerable research efforts [3–12] have been made for automatic annotation. They can be roughly classified into three types of models: discriminative model, generative model, and nearest neighbor model. Discriminative model [3–5] treats image annotation as a multilabel classification problem, in which each label is a single class. Given a test image, a semantic label will be propagated to it only if a corresponding classifier makes a decision that the image belongs to it. But this type of model neglects the correlation between different labels, which is also very important for image annotation. Generative model [6, 7] attempts to infer the correlations or joint probabilities between images and semantic concepts. By learning the statistical models, the test image is annotated based on probability computation. However, there exist many parameters in these models, which leads to heavy computational cost for parameter estimation. The nearest neighbor (NN) based model [8–12] predicts labels by taking a weighted combination of the label absence or presence among neighboring images. Because of its simplicity and efficiency, NN-based methods attract more researching attention.Recently, sparse coding scheme and its variations have been successfully used in image annotation task. It is substantially related to the NN-based model since the test image is often represented by a few of representative samples in a low-dimensional manifold. For example, Wang et al. [13] reconstructed the test image sparsely and transferred labels by the sparse coefficients. Gao et al. [14] considered each (sub)class of images as a (sub)group and employed multilayer group sparsity to classify and annotate single-label images concurrently. Cao et al. [15] utilized the group sparse reconstruction framework to learn the label correlation for dictionary and reconstructed the test image for label prediction under weakly supervised case. Lu et al. [16] presented a more descriptive and robust visual bag-of-words (BOW) representation by semantic sparse recoding method for image annotation and classification. Jing et al. [17] learned a label embedded dictionary as well as a sparse representation for image annotation. In addition, the sparse representation can be further enhanced by exploiting the kernel mapping, since it maps nonlinear separable features into a higher dimensional feature space, in which features with the similar labels are closely grouped together while those without the same labels become linearly separable. Moran and Lavrenko [18] introduced a sparse kernel learning framework for the continuous relevance model, which can adaptively select different kernels for different features, and obtained great improvement.Despite these efforts, most of existing annotation methods combine the information from different image features by concatenating them into a long feature vector, which treats all the features equally and omits their different contributions for the final decision. To solve this problem, Kalayeh et al. [19] introduced a multiview learning technique in image annotation, in which each type of features as well as label matrix is considered as a view, and all the views are adaptively integrated to exploit the complementary information, but the sparsity prior to the training samples used to reconstruct the test image is not considered. Liu et al. [20] introduced a multiview sparse coding framework for semisupervised image annotation. But they assumed the different views share a common sparse pattern, which ignores the diversity between the views. Yuan et al. [21] adopted a multitask joint sparse representation for image classification, allowing different coefficient representations for different task and enforcing the similarity among different tasks by joint sparsity. But image annotation is a multilabel classification problem, which cannot use the framework directly.Motivated by the previous research, we formulate image annotation as a kernel-based multiview joint sparse coding (KMVJSC) learning problem. Figure1 describes the framework of KMVJSC. Particularly, we map the feature views and label views to an implicit high-dimensional space and integrate all the views adaptively by multiview learning to strengthen the power of multiple modalities from an image. We aim to find a set of optimal sparse representations as well as dictionaries for each view simultaneously. Different views in an image should have similar coding coefficients to joint represent the same image on the one hand; on the other hand, these coefficients from different views should have some diversity to reflect the distinctive property of different views. Thus, we adopt different sparse coefficients for different views, allowing each view to be flexibly coded over its associated dictionary; at the same time, we employ the joint sparsity constraint to make the sparse coefficients among different views to be similar. The optimization algorithm and label prediction scheme under the proposed framework are developed. Given a test image, we also map its multiple features into the same kernel space and reconstruct the test image by joint sparse coding using the learned atom representation dictionary. The product of the atom representation dictionary and the corresponding sparse coefficients is considered as scores of the near neighbors for the test image, and a greedy label transfer scheme is used to get the annotation. Experiments on three datasets demonstrate the effectiveness of our proposed method and the competitive performance compared with the related methods.Figure 1
Schematic illustration of the proposed KMVJSC approach for automatic image annotation. First, we extract multiple features from all the training images; each type of feature as well as the label matrix is treated as a view. Then, we map the different views into a kernel space, where joint sparse coding and dictionary learning are conducted. To label a test image, we extract its multiple features and reconstruct the test image using the learned atom representation dictionary in the mapped space by joint sparse coding. Finally, the product of atom representation dictionary and the corresponding sparse coefficients gives score for the near neighbors of the test image, and a greedy label transfer scheme is used to get the annotation.In summary, the major contributions of this paper are as follows: (1) An effective kernel-based multiview joint sparse coding framework is proposed and successfully applied in image annotation; (2) the optimization algorithm is proposed by extending the accelerated proximal gradient (APG) and K-singular value decomposition (KSVD) algorithms into a kernel-based multiview case; (3) a label prediction algorithm is proposed for kernel sparse representation framework based on the sparse reconstruction and weighted greedy label transfer scheme.The rest of this paper is organized as follows: Section2 briefly discusses the related work. Section 3 describes the details of our kernel-based multiview joint sparse representation, optimization algorithm, and label prediction scheme. Experiment results are reported and analyzed in Section 4. Finally, we conclude this paper in Section 5.


## 2. Related Work

In this section, we will review the related work including sparse representation, kernel sparse learning, and multiview learning.


### 2.1. Sparse Representation Based Image Annotation

Sparse coding aims to represent the observed data as a linear combination of dictionary entries (or training samples), with the constraint that each image feature vector is only represented by a subset of all the available dictionary atoms. Denote the feature matrix formed by originalN training samples as X=[X1,X2,…,XN]∈RP×N, where Xi=[xi1,xi2,…,xiP]T (i=1,2,…,N) is the feature vector of the ith sample image and P is the feature dimension. Then, the conventional sparse representation aims to seek a sparse coefficient matrix W=W1,W2,…,WN∈RNd×N and the associated dictionary D=[D1,D2,…,DNd]∈RP×Nd by solving the following optimization problem [22]:(1)argminD,W∑i=1NXi-DWiF2+λWi1,where Wi1 is theL1-norm sparse constraint regularization and λ is a trade-off parameter used to balance the sparsity and the reconstruction error.For label transfer, given a test image, [13] adopts all the training images as dictionary and propagates the labels from the training images to the test image directly by the product of label matrix of training images with the learned sparse vector. Document [14] predicts the test image label based on the reconstruction error in the (sub)groups and assigns labels from the (sub)groups with the minimum reconstruction error to the test image. But the discriminative power of sparse representation in the original space is still limited compared to the kernel space.


### 2.2. Kernel Sparse Representation

The kernel sparse coding improves (1) by introducing a kernel trick on both training images and dictionary and has obtained great success in image classification. Let ϕ(X) denote the matrix whose columns are obtained by embedding the input features X=[X1,X2,…,XN]∈RP×N into a feature space using the mapping ϕ. That is, ϕX=ϕX1,ϕX2,…,ϕXN. Furthermore, let ϕD=ϕD1,ϕD2,…,ϕDNd denote the dictionary in this mapped feature space; the kernel sparse representation [23] problem is defined as follows:(2)argminD,W∑i=1NϕXi-ϕDWiF2+λWi1.Equation (2) can be rewritten as in (3), which depends on a Mercer kernel function κ(·,·), but not the mapping ϕ:(3)argminD,W∑i=1NκXi,Xi+WiTKDDWi-2WiTKDXi+λWi1,where ΚDD∈RNd×Nd is a positive semidefinite matrix with the entries computed from Mercer kernel: {ΚDD}ij={ϕD,ϕD}ij=κ(Di,Dj), and KD(Xi) is a Nd-dimension vector with {ΚD(Xi)}j=κ(Dj,Xi). A Mercer kernel κDi,Dj is a function representing the nonlinear similarity between two vectors Di and Dj. Some commonly used kernels include polynomial kernels, Gaussian kernels, and histogram intersection kernels (HIK).For image classification, given a test image, it is assigned to the class with the minimum reconstruction error obtained by the learned dictionary and sparse coefficients for that class in the mapped space. However, image annotation task is a multilabel classification problem and cannot use this judgment decision for label transfer directly. In addition, the label transfer scheme [13] in original space cannot be moved to the mapped space since the mapping function is unknown. Thus, the application of kernel sparse representation in image annotation is limited.


### 2.3. Multiview Learning

While the previous framework has been proven successful for many tasks, it has only been applied to the single-view case. With multiple types of input modalities, [20] proposed a multiview learning based sparse representation model, which is based on the following general framework:(4)argminDv,W∑i=1N∑v=1VXiv-DvWiF2+λWi1.Here, each image sample is represented byV different features of views, X(v) and D(v) are the feature matrix for all the training images and learned dictionary of the vth view, respectively, and W is the sparse coefficient matrix shared for all views.In [20], labels are treated as an additional (V+1)th view; then, X(V+1) and D(V+1) represent the label view matrix of training samples and the associated label view dictionary, respectively. For a new test sample y={y(1),y(2),…,y(V)}, the corresponding sparse code w can be obtained by solving the convex problem:(5)argminw∑v=1Vyv-Dvw22+λw1and the label view of the test sample y(V+1) is then predicted by y(V+1)=D(V+1)w directly. Although [20] implements the image annotation effectively, it adopts shared sparse coefficients for all views, which is not the case in reality because of the diversity of different views.


## 2.1. Sparse Representation Based Image Annotation

Sparse coding aims to represent the observed data as a linear combination of dictionary entries (or training samples), with the constraint that each image feature vector is only represented by a subset of all the available dictionary atoms. Denote the feature matrix formed by originalN training samples as X=[X1,X2,…,XN]∈RP×N, where Xi=[xi1,xi2,…,xiP]T (i=1,2,…,N) is the feature vector of the ith sample image and P is the feature dimension. Then, the conventional sparse representation aims to seek a sparse coefficient matrix W=W1,W2,…,WN∈RNd×N and the associated dictionary D=[D1,D2,…,DNd]∈RP×Nd by solving the following optimization problem [22]:(1)argminD,W∑i=1NXi-DWiF2+λWi1,where Wi1 is theL1-norm sparse constraint regularization and λ is a trade-off parameter used to balance the sparsity and the reconstruction error.For label transfer, given a test image, [13] adopts all the training images as dictionary and propagates the labels from the training images to the test image directly by the product of label matrix of training images with the learned sparse vector. Document [14] predicts the test image label based on the reconstruction error in the (sub)groups and assigns labels from the (sub)groups with the minimum reconstruction error to the test image. But the discriminative power of sparse representation in the original space is still limited compared to the kernel space.


## 2.2. Kernel Sparse Representation

The kernel sparse coding improves (1) by introducing a kernel trick on both training images and dictionary and has obtained great success in image classification. Let ϕ(X) denote the matrix whose columns are obtained by embedding the input features X=[X1,X2,…,XN]∈RP×N into a feature space using the mapping ϕ. That is, ϕX=ϕX1,ϕX2,…,ϕXN. Furthermore, let ϕD=ϕD1,ϕD2,…,ϕDNd denote the dictionary in this mapped feature space; the kernel sparse representation [23] problem is defined as follows:(2)argminD,W∑i=1NϕXi-ϕDWiF2+λWi1.Equation (2) can be rewritten as in (3), which depends on a Mercer kernel function κ(·,·), but not the mapping ϕ:(3)argminD,W∑i=1NκXi,Xi+WiTKDDWi-2WiTKDXi+λWi1,where ΚDD∈RNd×Nd is a positive semidefinite matrix with the entries computed from Mercer kernel: {ΚDD}ij={ϕD,ϕD}ij=κ(Di,Dj), and KD(Xi) is a Nd-dimension vector with {ΚD(Xi)}j=κ(Dj,Xi). A Mercer kernel κDi,Dj is a function representing the nonlinear similarity between two vectors Di and Dj. Some commonly used kernels include polynomial kernels, Gaussian kernels, and histogram intersection kernels (HIK).For image classification, given a test image, it is assigned to the class with the minimum reconstruction error obtained by the learned dictionary and sparse coefficients for that class in the mapped space. However, image annotation task is a multilabel classification problem and cannot use this judgment decision for label transfer directly. In addition, the label transfer scheme [13] in original space cannot be moved to the mapped space since the mapping function is unknown. Thus, the application of kernel sparse representation in image annotation is limited.


## 2.3. Multiview Learning

While the previous framework has been proven successful for many tasks, it has only been applied to the single-view case. With multiple types of input modalities, [20] proposed a multiview learning based sparse representation model, which is based on the following general framework:(4)argminDv,W∑i=1N∑v=1VXiv-DvWiF2+λWi1.Here, each image sample is represented byV different features of views, X(v) and D(v) are the feature matrix for all the training images and learned dictionary of the vth view, respectively, and W is the sparse coefficient matrix shared for all views.In [20], labels are treated as an additional (V+1)th view; then, X(V+1) and D(V+1) represent the label view matrix of training samples and the associated label view dictionary, respectively. For a new test sample y={y(1),y(2),…,y(V)}, the corresponding sparse code w can be obtained by solving the convex problem:(5)argminw∑v=1Vyv-Dvw22+λw1and the label view of the test sample y(V+1) is then predicted by y(V+1)=D(V+1)w directly. Although [20] implements the image annotation effectively, it adopts shared sparse coefficients for all views, which is not the case in reality because of the diversity of different views.


## 3. The Proposed Method

In this section, we introduce in detail our proposed method, optimization procedure, and label propagation algorithm. Throughout this paper, given a matrixM, we will use the term Mi to denote its ith column vector and Mi,· to denote its ith row vector.


### 3.1. Kernel-Based Multiview Joint Sparse Representation

Motivated by the previous works of [20, 21], we utilize different sparse coefficient representation for different views, allowing the flexibility of sparse coefficients from distinct views; then, we introduce a joint sparsity constraint to our kernel-based multiview sparse learning framework to keep the correlation among multiple views. Specifically, let X1,X2,…,XV be a set of N training image samples obtained from V different feature views; each view contains the sample feature denoted by Xv=X1v,X2v,…,XNv∈RPv×N(v=1,2,…,V), where Xiv∈RPv is the ith sample. The label information can be considered as another view XV+1=[X1V+1,X2V+1,…,XNV+1]∈RPV+1×N, PV+1 is the number of labels, XiV+1∈RPV+1 is the label vector of the ith image, and each entry is either 1 or 0 representing the presence or absence of a given label in the image. Let Dv=[D1v,D2v,…,DNdv]∈RPv×Nd be an overcomplete dictionary (Nd>Pv), and Djv denotes the jth column of Dv. Let ϕ(Xv) and ϕDv denote the corresponding sample matrix and dictionary of the vth view obtained using the mapping ϕ, respectively. Thus, we have ϕXv=ϕX1v,ϕX2v,…,ϕXNv and ϕDv=ϕD1v,ϕD2v,…,ϕDNv. Then, the object function of kernel-based multiview joint sparse representation can be defined as(6)argminDv,Ωi∑i=1N∑v=1V+1ϕXiv-ϕDvWivF2+λΩi1,2,where Wiv∈RNd corresponds to the sparse representation coefficient of Xiv over dictionary Dv in the mapped space, Ωi=Wi1,Wi2,…,WiV+1∈RNd×(V+1), 0≤i≤N, and Ωi1,2 is defined as the sum of the L1-norm of all rows of the matrix Ωi, which encourages Ωi to be sparse in column direction and dense in row direction. It helps to enforce the sparse coefficients of different views to share a similar pattern. λ is the tuning parameter to control the regularization term.The dictionaries from different views must be correlated with each other since they are learned from the same training images. To keep the correlation, we employDv=XvA, since dictionary atoms lie within the subspace spanned by the input data, where A is called the atom representation dictionary and Xv is called base dictionary [24]. Then, (6) can be rewritten as(7)argminA,Ωi∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.Compared with the dictionary obtained by using all the training samples directly, in which each dictionary atom corresponds to a training sample, the learned dictionary in (6) has no explicit physical meaning in their structure; that is, atoms located in the same column of different dictionary views may not originate from the same training sample, and also the correlations among different views are lost. By introducing ϕ(Dv)=ϕ(Xv)A, we can learn a shared implicit dictionary A among multiple views in a similar way as (6), while the explicit nature (atom location information) of the dictionary is also maintained in the base dictionary ϕ(Xv), which is very important for subsequent label transfer in kernel space.


### 3.2. Optimization

To solve the problem in (7), we adopt an iterative optimization procedure by alternately optimizing technique [25] with respect to Ωi(i=1,2,…,N) and A while holding the other one fixed. In the following, we provide a brief description of the alternating optimization for our method. For convenience, Notation lists the important notations used in this paper.First, keepingA fixed, the problem in (7) can be simplified as (8)argminΩi∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.In the next step, keepingΩi fixed, the problem in (7) is simplified as (9)argminA∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2,which can also be represented by the matrix form:(10)argminA∑v=1V+1ϕXv-ϕXvAWvF2.Here, Wv=W1v,W2v,…,WNv∈RNd×N.Such process can be iterated until the solutions ofΩi and A converge to some local minimum. The following is a detailed description for optimization algorithm solving problems (8) and (10).Learning Sparse Codes. Equation (8) can be decoupled into N distinct subproblems; the ith (1≤i≤N) subproblem is formulated as follows:(11)argminΩi∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.The objective function in (11) is a nonsmooth convex function since Ωi1,2 is nondifferentiable, which can be solved by accelerated proximal gradient (APG) method [26, 27] with the extension that signals are now in the multiview kernel space with a very high dimension.Then, the optimization process can be iterated alternately by generalized gradient mapping and aggregation steps [27].In the generalized gradient mapping step, we denoteKXXv=ϕ(Xv)Tϕ(Xv), KXv(Xi)=ϕ(Xv)Tϕ(Xiv), and f(Ωi)=∑v=1V+1ϕ(Xiv)-ϕ(Xv)AWivF2; then, the gradient of f(Ωi) with respect to Ωi is calculated as follows:(12)∇fΩi=∂f∂Wi1∂f∂Wi2⋯∂f∂WiV∂f∂WiV+1=2ATKXX1AWi1-ATKX1Xi,ATKXX2AWi2-ATKX1Xi,…,ATKXXV+1AWiV+1-ATKXV+1Xi.Then, we apply the gradient mapping to (11) to get the following equivalent objective function:(13)argminΩifΩi+λΩi1,2=argminΩifU+∇fU,Ωi-U+η2Ωi-UF2+λΩi1,2.It comprises the regularization termλΩi1,2 and the approximation of f(Ωi) by the first-order Taylor expansion at U regularized as the Euclidean distance between Ωi and U, where η is a parameter controlling the step penalty.In thet+1th iteration, we calculate (Ωi)t+1 as(14)Ωit+1=argminΩifUt+∇fUt,Ωi-Ut+η2Ωi-UtF2+λΩi1,2.Document [26] gives the solution for solving the optimization problem of (14) as(15)Ωij,·t+1=1-ληZj,·2+Zj,·,where Z=Ut-1/η∇f(Ut), [·]+=max{0,·}, and Zj,·, (Ωi)j,· are the jth row of matrixes Z and Ωi, respectively.In the aggregation step,Ut+1 is computed as a linear combination of (Ωi)t+1 and (Ωi)t:(16)Ut+1=Ωit+1+1-atatat+1Ωit+1-Ωit.Here, we set at+1=2/t+3 conventionally.The steps of optimizing algorithm for (8) by the extended APG are elaborated as Algorithm 1.Algorithm 1:Optimizing method for problem (8).
Input:
Training image samplesXv, (v=1,2,…,V+1)
Corresponding dictionaryA,
Regularization parameterλ and step size η.
Output:
Sparse coefficient matrixΩi, (i=1,2,…,N)
(1)For  i=1:N
(2)Initialization: Initialize (Ωi)0∈RNd×(V+1), U0=(Ωi)0 to be zero matrix; a0=1; and set t=0.
(3)While not converged Do
Generalized gradient mapping step:
(4)   CalculateZ=Ut-1/η∇f(Ut) by Eq. (12)
(5)Forj=1:Nd
(6)    Calculatejth row of (Ωi)t+1 by Eq. (15)
(7)End For
Aggregation step:
(8)at+1=2/t+3
(9)Ut+1=(Ωi)t+1+1-at/atat+1(Ωi)t+1-(Ωi)t
(10)t=t+1
(11)End While
(12)End ForDictionary Updating. For problem (10), considering the matrices XM=(X1)T,(X2)T,…,(XV+1)TT∈R∑v=1V+1Pv×N, WM=(W1)T,(W2)T,…,(WV+1)TT∈R(V+1)·Nd×N, and AM=A,A,…,A∈RN×(V+1)·Nd, we obtain(17)AM=argminAMϕXM-ϕXMAMWMF2.This is equivalent to the dictionary update stage in kernel dictionary learning algorithms. We adopt the kernel KSVD dictionary update strategy in [24] to efficiently solve it. Then, A is chosen as the first Nd column of AM.


### 3.3. Label Prediction

The proposed method treats labels as an additional view, so we can infer the label information from the sparse coefficients. In particular, given a test image represented by multiple feature views{y1,y2,…,yV}, visual features of training images Xv, the learned dictionary A from the features, and labels of the training images, we obtain the sparse coefficient vector {ω1,ω2,…,ωV} (ωV∈RNd,v=1,2,…,V) for the test image in terms of learned dictionary by solving the following convex problem, which can be solved similarly as to problem (11).(18)argminωv∑v=1Vϕyv-ϕXvAωvF2+λω1ω2⋯ωV1,2.Since the magnitude of the product termAωv can be considered as the importance of certain training samples in the reconstruction of the test image, the priority of the samples used for label propagation is based on the magnitude of its corresponding product term Aωv. Although ωV+1 is unknown, it must share the similar pattern with ωv(1≤v≤V), since they are used to represent the same image. So, we iterate for each view v, choosing the samples in XV+1 corresponding the top five values in Aωv. Then, the tag propagation for the test image is based on a weighted version of greedy label transfer scheme [8]. The proposed label prediction and propagation scheme is summarized in Algorithm 2.Algorithm 2:Label prediction procedure.
Input:
The feature view of test imageyv∈RPv,
Feature view of training imagesXv, (v=1,2,…,V),
Learned dictionaryA∈RN×Nd,
Tuning parameterλ.
Output:
Predicted labelsyV+1 for the test image.
(1) Calculateωv  (v=1,2,…,V) by Eq. (18)
(2)For  v=1,2,…,V
(3)  Find the five largest values inAωv  and corresponding indices
(4)  Find sample label column vectors inXV+1  corresponding to these indices
(5)End For
(6) Compute and rank the weighted frequency of labels appeared in all the found samples, with weights equaling to
the product coefficients ofAωv
(7) Transfer labels according to their calculated frequency.


## 3.1. Kernel-Based Multiview Joint Sparse Representation

Motivated by the previous works of [20, 21], we utilize different sparse coefficient representation for different views, allowing the flexibility of sparse coefficients from distinct views; then, we introduce a joint sparsity constraint to our kernel-based multiview sparse learning framework to keep the correlation among multiple views. Specifically, let X1,X2,…,XV be a set of N training image samples obtained from V different feature views; each view contains the sample feature denoted by Xv=X1v,X2v,…,XNv∈RPv×N(v=1,2,…,V), where Xiv∈RPv is the ith sample. The label information can be considered as another view XV+1=[X1V+1,X2V+1,…,XNV+1]∈RPV+1×N, PV+1 is the number of labels, XiV+1∈RPV+1 is the label vector of the ith image, and each entry is either 1 or 0 representing the presence or absence of a given label in the image. Let Dv=[D1v,D2v,…,DNdv]∈RPv×Nd be an overcomplete dictionary (Nd>Pv), and Djv denotes the jth column of Dv. Let ϕ(Xv) and ϕDv denote the corresponding sample matrix and dictionary of the vth view obtained using the mapping ϕ, respectively. Thus, we have ϕXv=ϕX1v,ϕX2v,…,ϕXNv and ϕDv=ϕD1v,ϕD2v,…,ϕDNv. Then, the object function of kernel-based multiview joint sparse representation can be defined as(6)argminDv,Ωi∑i=1N∑v=1V+1ϕXiv-ϕDvWivF2+λΩi1,2,where Wiv∈RNd corresponds to the sparse representation coefficient of Xiv over dictionary Dv in the mapped space, Ωi=Wi1,Wi2,…,WiV+1∈RNd×(V+1), 0≤i≤N, and Ωi1,2 is defined as the sum of the L1-norm of all rows of the matrix Ωi, which encourages Ωi to be sparse in column direction and dense in row direction. It helps to enforce the sparse coefficients of different views to share a similar pattern. λ is the tuning parameter to control the regularization term.The dictionaries from different views must be correlated with each other since they are learned from the same training images. To keep the correlation, we employDv=XvA, since dictionary atoms lie within the subspace spanned by the input data, where A is called the atom representation dictionary and Xv is called base dictionary [24]. Then, (6) can be rewritten as(7)argminA,Ωi∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.Compared with the dictionary obtained by using all the training samples directly, in which each dictionary atom corresponds to a training sample, the learned dictionary in (6) has no explicit physical meaning in their structure; that is, atoms located in the same column of different dictionary views may not originate from the same training sample, and also the correlations among different views are lost. By introducing ϕ(Dv)=ϕ(Xv)A, we can learn a shared implicit dictionary A among multiple views in a similar way as (6), while the explicit nature (atom location information) of the dictionary is also maintained in the base dictionary ϕ(Xv), which is very important for subsequent label transfer in kernel space.


## 3.2. Optimization

To solve the problem in (7), we adopt an iterative optimization procedure by alternately optimizing technique [25] with respect to Ωi(i=1,2,…,N) and A while holding the other one fixed. In the following, we provide a brief description of the alternating optimization for our method. For convenience, Notation lists the important notations used in this paper.First, keepingA fixed, the problem in (7) can be simplified as (8)argminΩi∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.In the next step, keepingΩi fixed, the problem in (7) is simplified as (9)argminA∑i=1N∑v=1V+1ϕXiv-ϕXvAWivF2,which can also be represented by the matrix form:(10)argminA∑v=1V+1ϕXv-ϕXvAWvF2.Here, Wv=W1v,W2v,…,WNv∈RNd×N.Such process can be iterated until the solutions ofΩi and A converge to some local minimum. The following is a detailed description for optimization algorithm solving problems (8) and (10).Learning Sparse Codes. Equation (8) can be decoupled into N distinct subproblems; the ith (1≤i≤N) subproblem is formulated as follows:(11)argminΩi∑v=1V+1ϕXiv-ϕXvAWivF2+λΩi1,2.The objective function in (11) is a nonsmooth convex function since Ωi1,2 is nondifferentiable, which can be solved by accelerated proximal gradient (APG) method [26, 27] with the extension that signals are now in the multiview kernel space with a very high dimension.Then, the optimization process can be iterated alternately by generalized gradient mapping and aggregation steps [27].In the generalized gradient mapping step, we denoteKXXv=ϕ(Xv)Tϕ(Xv), KXv(Xi)=ϕ(Xv)Tϕ(Xiv), and f(Ωi)=∑v=1V+1ϕ(Xiv)-ϕ(Xv)AWivF2; then, the gradient of f(Ωi) with respect to Ωi is calculated as follows:(12)∇fΩi=∂f∂Wi1∂f∂Wi2⋯∂f∂WiV∂f∂WiV+1=2ATKXX1AWi1-ATKX1Xi,ATKXX2AWi2-ATKX1Xi,…,ATKXXV+1AWiV+1-ATKXV+1Xi.Then, we apply the gradient mapping to (11) to get the following equivalent objective function:(13)argminΩifΩi+λΩi1,2=argminΩifU+∇fU,Ωi-U+η2Ωi-UF2+λΩi1,2.It comprises the regularization termλΩi1,2 and the approximation of f(Ωi) by the first-order Taylor expansion at U regularized as the Euclidean distance between Ωi and U, where η is a parameter controlling the step penalty.In thet+1th iteration, we calculate (Ωi)t+1 as(14)Ωit+1=argminΩifUt+∇fUt,Ωi-Ut+η2Ωi-UtF2+λΩi1,2.Document [26] gives the solution for solving the optimization problem of (14) as(15)Ωij,·t+1=1-ληZj,·2+Zj,·,where Z=Ut-1/η∇f(Ut), [·]+=max{0,·}, and Zj,·, (Ωi)j,· are the jth row of matrixes Z and Ωi, respectively.In the aggregation step,Ut+1 is computed as a linear combination of (Ωi)t+1 and (Ωi)t:(16)Ut+1=Ωit+1+1-atatat+1Ωit+1-Ωit.Here, we set at+1=2/t+3 conventionally.The steps of optimizing algorithm for (8) by the extended APG are elaborated as Algorithm 1.Algorithm 1:Optimizing method for problem (8).
Input:
Training image samplesXv, (v=1,2,…,V+1)
Corresponding dictionaryA,
Regularization parameterλ and step size η.
Output:
Sparse coefficient matrixΩi, (i=1,2,…,N)
(1)For  i=1:N
(2)Initialization: Initialize (Ωi)0∈RNd×(V+1), U0=(Ωi)0 to be zero matrix; a0=1; and set t=0.
(3)While not converged Do
Generalized gradient mapping step:
(4)   CalculateZ=Ut-1/η∇f(Ut) by Eq. (12)
(5)Forj=1:Nd
(6)    Calculatejth row of (Ωi)t+1 by Eq. (15)
(7)End For
Aggregation step:
(8)at+1=2/t+3
(9)Ut+1=(Ωi)t+1+1-at/atat+1(Ωi)t+1-(Ωi)t
(10)t=t+1
(11)End While
(12)End ForDictionary Updating. For problem (10), considering the matrices XM=(X1)T,(X2)T,…,(XV+1)TT∈R∑v=1V+1Pv×N, WM=(W1)T,(W2)T,…,(WV+1)TT∈R(V+1)·Nd×N, and AM=A,A,…,A∈RN×(V+1)·Nd, we obtain(17)AM=argminAMϕXM-ϕXMAMWMF2.This is equivalent to the dictionary update stage in kernel dictionary learning algorithms. We adopt the kernel KSVD dictionary update strategy in [24] to efficiently solve it. Then, A is chosen as the first Nd column of AM.


## 3.3. Label Prediction

The proposed method treats labels as an additional view, so we can infer the label information from the sparse coefficients. In particular, given a test image represented by multiple feature views{y1,y2,…,yV}, visual features of training images Xv, the learned dictionary A from the features, and labels of the training images, we obtain the sparse coefficient vector {ω1,ω2,…,ωV} (ωV∈RNd,v=1,2,…,V) for the test image in terms of learned dictionary by solving the following convex problem, which can be solved similarly as to problem (11).(18)argminωv∑v=1Vϕyv-ϕXvAωvF2+λω1ω2⋯ωV1,2.Since the magnitude of the product termAωv can be considered as the importance of certain training samples in the reconstruction of the test image, the priority of the samples used for label propagation is based on the magnitude of its corresponding product term Aωv. Although ωV+1 is unknown, it must share the similar pattern with ωv(1≤v≤V), since they are used to represent the same image. So, we iterate for each view v, choosing the samples in XV+1 corresponding the top five values in Aωv. Then, the tag propagation for the test image is based on a weighted version of greedy label transfer scheme [8]. The proposed label prediction and propagation scheme is summarized in Algorithm 2.Algorithm 2:Label prediction procedure.
Input:
The feature view of test imageyv∈RPv,
Feature view of training imagesXv, (v=1,2,…,V),
Learned dictionaryA∈RN×Nd,
Tuning parameterλ.
Output:
Predicted labelsyV+1 for the test image.
(1) Calculateωv  (v=1,2,…,V) by Eq. (18)
(2)For  v=1,2,…,V
(3)  Find the five largest values inAωv  and corresponding indices
(4)  Find sample label column vectors inXV+1  corresponding to these indices
(5)End For
(6) Compute and rank the weighted frequency of labels appeared in all the found samples, with weights equaling to
the product coefficients ofAωv
(7) Transfer labels according to their calculated frequency.


## 4. Experiments

To validate the effectiveness of the proposed KMVJSC for automatic image annotation task, we conduct experiments and compare the results with related algorithms on three popular benchmark databases.


### 4.1. Experimental Settings

The three datasets, we used to evaluate the performance of our approach are as follows.Corel5K dataset is the most popular dataset for annotation evaluation [8–10, 13, 17–19, 28, 29]. It consists of 5000 images from 50 different topics, such as “beach,” “aircraft,” and “horse,” and each topic includes 100 similar images. The training and testing sets contain 4500 and 500 images, respectively. Each image is manually annotated with 1 to 5 labels from a dictionary of 260 labels, and with 3.5 labels on average.IAPRTC12 [30] dataset consists 19627 images of natural scenes dealing with sports and actions and photographs of people, animals, cities, landscapes, and so on. In the dataset, 17665 images are selected for training and the remaining 1962 images are chosen for testing. The number of labels in this dataset is 291, and each image has up to 23 labels with an average of 5.7 labels per image. Besides, each label averagely relates with 347 images.MIRFlickr25K [31] contains 25000 images which are collected from the social photography site https://www.flickr.com and equally split for training and test. It provides 38 labels as the ground-truth annotation, such as “animals”, “baby”, and “baby∗”. The words with “∗” indicate a further strict annotation for labels. Each image is associated with up to 17 labels and 4.7 labels on average, while each label averagely relates with 1560 images. Besides, the dataset provides 1386 tags. Since the Flickr tags are noisy, we kept the tags that appear at least 50 times, resulting in a vocabulary of 457 tags, and use it as another type of feature.We adopt histogram intersection kernel as the kernel function because it is a parameter-free kernel, and it has obtained excellent performance in evaluating the similarity between two histograms [14].Following [32], we adopt 15 visual features but limit the size of large feature vectors. Specifically, in order to reduce the computation complexity of both training and testing procedures, we use color histogram with 10 bins per color channel. For features encoded with the spatial layout information, we quantize the color with 8 bins in each channel and reduce the k-means cluster centers to 500 for SIFT features. Thus, the 15 distinct features include 3 color features (RGB, LAB, HSV, 1000D), 2 SIFT features (DenseSIFT, HarrisSIFT, 1000D), 2 Hue features (DenseHue, HarrisHue, 100D), 7 above features with layout encoding (RGBV3H1, LABV3H1, HSVV3H1, 1536D; DenseSIFTV3H1, HarrisSIFTV3H1, 1500D; DenseHueV3H1, HarrisHueV3H1, 300D), and a GIST feature (512D). In addition, for MIRFlickr25K dataset, we endow each image with a binary vector indicating the absence or presence of each tag, which leads to a Tag feature of length 457D.There are two parameters that need to be tuned, that is, dictionary sizeNd and regularization parameter λ. In this paper, Nd is selected from {2000,2500,3000,3500,4000,4500}, and λ is selected from {0.0001,0.001,0.01,0.1,1,10}. We perform KMVJSC with fivefold cross validation on training set and conduct sensitivity analysis in Section 4.3. Finally, we set the parameters of our KMVJSC algorithm as Nd=2000, λ=0.1 on Corel5K dataset and as Nd=3000, λ=0.001 on the other two datasets. Due to the random entries in initialization, we repeat all the experiments 5 times separately and report the average results.Following most research [10, 13, 17, 19, 32], all the test images are annotated by the top 5 relevant labels. To estimate the performance, we calculate precision (P), recall (R), and F1-measure for each label. For a given label c, the three metrics are defined by R=Nc/Ng, P=Nc/Nt, and F1=2×R×P/(R+P). Here, Ng is the number of images labeled with c in the ground truth, Nt is the number of images labeled with c by our automatic annotation algorithm, and Nc is the number of correct labeled images with c. We present the mean value over all labels for each metric. Besides, the number of labels with nonzero recall (N+) is also used.


### 4.2. The Compared Methods

We compare our KMVJSC with some sparse coding based baselines including multilabel sparse coding (MSC) [13], kernel sparse coding (KSC) [23], multiview Hessian discriminative sparse coding (MVHDSC) [20], multiview joint sparse coding (MVJSC) in original space, and kernel-based multiview sparse coding (KMVSC) without enforcing the joint sparsity across views. For single-view methods (MSC and KSC), we concatenate the multiple features as a long vector. For KSC, we use the same label transfer scheme as our method except using learned dictionary and reconstruct coefficient directly; for MVJSC, we calculate the average sparse coefficient from multiple feature views and use it as the sparse coefficient of label view and transfer labels with the same way as that of MVHDSC; and for KMVSC, we use the same label transfer method as our KMVJSC. Moreover, we take the same parameter selection strategy for these methods, where all the training, validation, and test sets are exactly the same as those used for our method.


### 4.3. Experimental Results

Table1 lists some image examples from three datasets and the predicted labels using our method, together with the ground truths. It contains at least one mismatched label compared with ground-truth labels (perfectly matched annotations are not listed here). The differences in predicted labels are marked in italic font. The results in Table 1 show that, in many cases, some predicted labels missed by the ground-truth annotation can still explain the image well, such as “meadow” for the first image from Corel5K dataset. Besides, some semantically similar words were also treated as errors, such as “cathedral” for the last image from IAPRTC12 dataset, which shows the potential effectiveness of our proposed method for automatic image annotation task.Table 1
Comparison of predicted labels with ground-truth annotations for images from Corel5K, IAPRTC12, and MIRFlickr25K datasets. Italic labels are the difference between predicted and ground-truth labels while some of them are also relevant to the image.
Images from Corel5K
Ground truths
Cow, bulls, antlers, elk
Grass, ruins, stone, column
Coral, anemone, ocean, reefs
Grass, bear, polar, tundra
Cars, tracks, turn, prototype
Predicted labels
Meadow, cow, bulls, elk, antlers
Ruins, stone,palace, column, grass
Coral, anemone, ocean, reefs,flowers
Grass, bear, polar, tundra,tree
Cars, tracks, turn, prototype,birds
Images from IAPRTC12
Ground truths
cloth, front, girl, hat, wall
court, man, player, tennis
beach, boat, fog, man, sea, tower
bush, cloud, group, hill, island, lake, people, sky
church, lamp, night, sky, square, street, tower
Predicted labels
front, girl, hat,jacket, wall
court, man, player, tennis,net
beach, boat, fog, man,people
group, hill, island, lake,photo
cathedral, lamp, night, square, street
Images from MIRFlickr25K
Ground truths
animals, bird,bird∗, clouds, plant_life, sky, structures, sunset, tree
plant_life, structures, tree,tree∗
flower,flower∗, night, plant_life
animals, plant_life, structures, tree, water
car,car∗, plant_life, sky, structures, transport, tree
Predicted labels
clouds, plant_life, sky, structures, tree
plant_life,sky, structures, tree, tree∗
flower,flower∗, plant_life, indoor, night
animals, plant_life, tree,sky, structures
indoor, structures, plant_life, sky, treeThe annotation performance of our proposed algorithms on three datasets along with other related methods is listed in Table2, where the methods with “∗” represent the implementation using our features, and KMVJSC+T refers to the annotation with our method using another Tag feature for MIRFlickr25K besides the 15 feature views and the label view. The maximum values are bold. From the result in Table 2, we can make the following observations.Table 2
Performance comparison using different methods on Corel5K, IAPRTC12, and MIRFlick25K datasets. Methods with “∗” means the implementation using our features. The best scores for each method are highlighted in bold. Using the same features, KMVJSC get the best results on all the three datasets. KMVJSC+T is the KMVJSC method using an additional Tag feature on MIRFlick25K dataset.
Datasets
Corel5K
IAPRTC12
MIRFlick25K
Method
AP
AR
F
1
N
+
AP
AR
F
1
N
+
AP
AR
F
1
N
+
M
S
C
∗ [13]
0.26
0.32
0.287
138
0.35
0.28
0.311
252
0.32
0.24
0.274
36
M
V
H
D
S
C
∗ [20]
0.35
0.37
0.360
160
0.45
0.32
0.374
260
0.40
0.30
0.343
38
K
S
C
∗ [23]
0.29
0.33
0.309
143
0.36
0.30
0.327
256
0.33
0.24
0.278
37
KMVSC
0.35
0.38
0.364
154
0.44
0.33
0.377
262
0.39
0.29
0.333
38
MVJSC
0.34
0.36
0.350
150
0.42
0.32
0.363
261
0.37
0.29
0.325
38
KMVJSC
0.37
0.4
0.384
162
0.46
0.34
0.391
268
0.41
0.32
0.359
38
KMVJSC+T
—
—
—
—
—
—
—
—
0.43
0.33
0.373
38(1) For single-view methods, KSC is slightly better than MSC; for multiview methods, kernel methods including KMVSC and KMVJSC both outperform MVJSC, which demonstrates the power of kernel mapping. We also can see that MVHDSC is still comparable with KMVSC, which is probably because MVHDSC enforces structured sparsity on dictionary additionally.(2) All multiview methods dramatically outperform the single-view methods on all datasets. Although MSC takes multilabel information into consideration, its performances are inferior to those of multiview methods including MVJSC, which shows multiview learning can harness the label view and feature views in a more natural way and capture the relation between multiple labels and that between labels and visual features for discrimination.(3) Compared to KMVSC, which neglects the joint sparsity across multiple views, our KMVJSC clearly outperforms KMVSC, which shows the constraint of joint sparsity across different views is valuable to enforce the robustness in coefficient estimation among different views.(4) Adding the Flickr tags as features helps to improve the annotation performance further, which illustrates that tags are another type of complementary information to visual features and label view and can be effectively used in multiview leaning.


### 4.4. Sensitivity Analysis of Parameters

We investigate the sensitivity of the parametersNd and λ in our approach using the three datasets. The corresponding results are presented in Figures 2 and 3.Figure 2
F
1 measure of our KMVJSC with Nd changing on the training set of the three datasets. The horizontal axis represents the size of the dictionary, from 2000 to 4500 with a step size of 500. The vertical axis represents the performance of F1.Figure 3
F
1 measure of our KMVJSC with λ changing on the training set of the three datasets. The horizontal axis represents the tuning parameter λ, from 1e-4 to 10 with a step size of 10 times. The vertical axis represents the performance of F1.It should be noted that the results in these two figures are obtained by 5-fold cross validation on the training image set of the three datasets, respectively. Figure2 illustrates the values of the F1 measure versus different size of dictionary with fixing λ (0.1 for Corel5K dataset and 0.001 for the other two datasets). From the results, we can observe that although the performance in terms of F1 measure varies on changing the value of Nd, it is not very sensitive to its choice and remains fairly stable. Considering that high dimension corresponds to expensive computing cost, we set Nd=2,000 for Corel5K dataset and Nd=3,000 for IAPRTC12 and MIRFlickr25K datasets to leverage the performance and the cost. Figure 3 shows the values of the F1 measure versus different λ with the selected fixing Nd. It is observed that the L1,2-norm joint sparsity regularization term is effective while it is not too small or too large. If it is too small, the correlations between different views are lost while it limits the flexibility of all the variety of views with large values. We can see that the best result of our approach is reached when λ=0.1 on Corel5K dataset and λ=0.001 on the other two datasets.


## 4.1. Experimental Settings

The three datasets, we used to evaluate the performance of our approach are as follows.Corel5K dataset is the most popular dataset for annotation evaluation [8–10, 13, 17–19, 28, 29]. It consists of 5000 images from 50 different topics, such as “beach,” “aircraft,” and “horse,” and each topic includes 100 similar images. The training and testing sets contain 4500 and 500 images, respectively. Each image is manually annotated with 1 to 5 labels from a dictionary of 260 labels, and with 3.5 labels on average.IAPRTC12 [30] dataset consists 19627 images of natural scenes dealing with sports and actions and photographs of people, animals, cities, landscapes, and so on. In the dataset, 17665 images are selected for training and the remaining 1962 images are chosen for testing. The number of labels in this dataset is 291, and each image has up to 23 labels with an average of 5.7 labels per image. Besides, each label averagely relates with 347 images.MIRFlickr25K [31] contains 25000 images which are collected from the social photography site https://www.flickr.com and equally split for training and test. It provides 38 labels as the ground-truth annotation, such as “animals”, “baby”, and “baby∗”. The words with “∗” indicate a further strict annotation for labels. Each image is associated with up to 17 labels and 4.7 labels on average, while each label averagely relates with 1560 images. Besides, the dataset provides 1386 tags. Since the Flickr tags are noisy, we kept the tags that appear at least 50 times, resulting in a vocabulary of 457 tags, and use it as another type of feature.We adopt histogram intersection kernel as the kernel function because it is a parameter-free kernel, and it has obtained excellent performance in evaluating the similarity between two histograms [14].Following [32], we adopt 15 visual features but limit the size of large feature vectors. Specifically, in order to reduce the computation complexity of both training and testing procedures, we use color histogram with 10 bins per color channel. For features encoded with the spatial layout information, we quantize the color with 8 bins in each channel and reduce the k-means cluster centers to 500 for SIFT features. Thus, the 15 distinct features include 3 color features (RGB, LAB, HSV, 1000D), 2 SIFT features (DenseSIFT, HarrisSIFT, 1000D), 2 Hue features (DenseHue, HarrisHue, 100D), 7 above features with layout encoding (RGBV3H1, LABV3H1, HSVV3H1, 1536D; DenseSIFTV3H1, HarrisSIFTV3H1, 1500D; DenseHueV3H1, HarrisHueV3H1, 300D), and a GIST feature (512D). In addition, for MIRFlickr25K dataset, we endow each image with a binary vector indicating the absence or presence of each tag, which leads to a Tag feature of length 457D.There are two parameters that need to be tuned, that is, dictionary sizeNd and regularization parameter λ. In this paper, Nd is selected from {2000,2500,3000,3500,4000,4500}, and λ is selected from {0.0001,0.001,0.01,0.1,1,10}. We perform KMVJSC with fivefold cross validation on training set and conduct sensitivity analysis in Section 4.3. Finally, we set the parameters of our KMVJSC algorithm as Nd=2000, λ=0.1 on Corel5K dataset and as Nd=3000, λ=0.001 on the other two datasets. Due to the random entries in initialization, we repeat all the experiments 5 times separately and report the average results.Following most research [10, 13, 17, 19, 32], all the test images are annotated by the top 5 relevant labels. To estimate the performance, we calculate precision (P), recall (R), and F1-measure for each label. For a given label c, the three metrics are defined by R=Nc/Ng, P=Nc/Nt, and F1=2×R×P/(R+P). Here, Ng is the number of images labeled with c in the ground truth, Nt is the number of images labeled with c by our automatic annotation algorithm, and Nc is the number of correct labeled images with c. We present the mean value over all labels for each metric. Besides, the number of labels with nonzero recall (N+) is also used.


## 4.2. The Compared Methods

We compare our KMVJSC with some sparse coding based baselines including multilabel sparse coding (MSC) [13], kernel sparse coding (KSC) [23], multiview Hessian discriminative sparse coding (MVHDSC) [20], multiview joint sparse coding (MVJSC) in original space, and kernel-based multiview sparse coding (KMVSC) without enforcing the joint sparsity across views. For single-view methods (MSC and KSC), we concatenate the multiple features as a long vector. For KSC, we use the same label transfer scheme as our method except using learned dictionary and reconstruct coefficient directly; for MVJSC, we calculate the average sparse coefficient from multiple feature views and use it as the sparse coefficient of label view and transfer labels with the same way as that of MVHDSC; and for KMVSC, we use the same label transfer method as our KMVJSC. Moreover, we take the same parameter selection strategy for these methods, where all the training, validation, and test sets are exactly the same as those used for our method.


## 4.3. Experimental Results

Table1 lists some image examples from three datasets and the predicted labels using our method, together with the ground truths. It contains at least one mismatched label compared with ground-truth labels (perfectly matched annotations are not listed here). The differences in predicted labels are marked in italic font. The results in Table 1 show that, in many cases, some predicted labels missed by the ground-truth annotation can still explain the image well, such as “meadow” for the first image from Corel5K dataset. Besides, some semantically similar words were also treated as errors, such as “cathedral” for the last image from IAPRTC12 dataset, which shows the potential effectiveness of our proposed method for automatic image annotation task.Table 1
Comparison of predicted labels with ground-truth annotations for images from Corel5K, IAPRTC12, and MIRFlickr25K datasets. Italic labels are the difference between predicted and ground-truth labels while some of them are also relevant to the image.
Images from Corel5K
Ground truths
Cow, bulls, antlers, elk
Grass, ruins, stone, column
Coral, anemone, ocean, reefs
Grass, bear, polar, tundra
Cars, tracks, turn, prototype
Predicted labels
Meadow, cow, bulls, elk, antlers
Ruins, stone,palace, column, grass
Coral, anemone, ocean, reefs,flowers
Grass, bear, polar, tundra,tree
Cars, tracks, turn, prototype,birds
Images from IAPRTC12
Ground truths
cloth, front, girl, hat, wall
court, man, player, tennis
beach, boat, fog, man, sea, tower
bush, cloud, group, hill, island, lake, people, sky
church, lamp, night, sky, square, street, tower
Predicted labels
front, girl, hat,jacket, wall
court, man, player, tennis,net
beach, boat, fog, man,people
group, hill, island, lake,photo
cathedral, lamp, night, square, street
Images from MIRFlickr25K
Ground truths
animals, bird,bird∗, clouds, plant_life, sky, structures, sunset, tree
plant_life, structures, tree,tree∗
flower,flower∗, night, plant_life
animals, plant_life, structures, tree, water
car,car∗, plant_life, sky, structures, transport, tree
Predicted labels
clouds, plant_life, sky, structures, tree
plant_life,sky, structures, tree, tree∗
flower,flower∗, plant_life, indoor, night
animals, plant_life, tree,sky, structures
indoor, structures, plant_life, sky, treeThe annotation performance of our proposed algorithms on three datasets along with other related methods is listed in Table2, where the methods with “∗” represent the implementation using our features, and KMVJSC+T refers to the annotation with our method using another Tag feature for MIRFlickr25K besides the 15 feature views and the label view. The maximum values are bold. From the result in Table 2, we can make the following observations.Table 2
Performance comparison using different methods on Corel5K, IAPRTC12, and MIRFlick25K datasets. Methods with “∗” means the implementation using our features. The best scores for each method are highlighted in bold. Using the same features, KMVJSC get the best results on all the three datasets. KMVJSC+T is the KMVJSC method using an additional Tag feature on MIRFlick25K dataset.
Datasets
Corel5K
IAPRTC12
MIRFlick25K
Method
AP
AR
F
1
N
+
AP
AR
F
1
N
+
AP
AR
F
1
N
+
M
S
C
∗ [13]
0.26
0.32
0.287
138
0.35
0.28
0.311
252
0.32
0.24
0.274
36
M
V
H
D
S
C
∗ [20]
0.35
0.37
0.360
160
0.45
0.32
0.374
260
0.40
0.30
0.343
38
K
S
C
∗ [23]
0.29
0.33
0.309
143
0.36
0.30
0.327
256
0.33
0.24
0.278
37
KMVSC
0.35
0.38
0.364
154
0.44
0.33
0.377
262
0.39
0.29
0.333
38
MVJSC
0.34
0.36
0.350
150
0.42
0.32
0.363
261
0.37
0.29
0.325
38
KMVJSC
0.37
0.4
0.384
162
0.46
0.34
0.391
268
0.41
0.32
0.359
38
KMVJSC+T
—
—
—
—
—
—
—
—
0.43
0.33
0.373
38(1) For single-view methods, KSC is slightly better than MSC; for multiview methods, kernel methods including KMVSC and KMVJSC both outperform MVJSC, which demonstrates the power of kernel mapping. We also can see that MVHDSC is still comparable with KMVSC, which is probably because MVHDSC enforces structured sparsity on dictionary additionally.(2) All multiview methods dramatically outperform the single-view methods on all datasets. Although MSC takes multilabel information into consideration, its performances are inferior to those of multiview methods including MVJSC, which shows multiview learning can harness the label view and feature views in a more natural way and capture the relation between multiple labels and that between labels and visual features for discrimination.(3) Compared to KMVSC, which neglects the joint sparsity across multiple views, our KMVJSC clearly outperforms KMVSC, which shows the constraint of joint sparsity across different views is valuable to enforce the robustness in coefficient estimation among different views.(4) Adding the Flickr tags as features helps to improve the annotation performance further, which illustrates that tags are another type of complementary information to visual features and label view and can be effectively used in multiview leaning.


## 4.4. Sensitivity Analysis of Parameters

We investigate the sensitivity of the parametersNd and λ in our approach using the three datasets. The corresponding results are presented in Figures 2 and 3.Figure 2
F
1 measure of our KMVJSC with Nd changing on the training set of the three datasets. The horizontal axis represents the size of the dictionary, from 2000 to 4500 with a step size of 500. The vertical axis represents the performance of F1.Figure 3
F
1 measure of our KMVJSC with λ changing on the training set of the three datasets. The horizontal axis represents the tuning parameter λ, from 1e-4 to 10 with a step size of 10 times. The vertical axis represents the performance of F1.It should be noted that the results in these two figures are obtained by 5-fold cross validation on the training image set of the three datasets, respectively. Figure2 illustrates the values of the F1 measure versus different size of dictionary with fixing λ (0.1 for Corel5K dataset and 0.001 for the other two datasets). From the results, we can observe that although the performance in terms of F1 measure varies on changing the value of Nd, it is not very sensitive to its choice and remains fairly stable. Considering that high dimension corresponds to expensive computing cost, we set Nd=2,000 for Corel5K dataset and Nd=3,000 for IAPRTC12 and MIRFlickr25K datasets to leverage the performance and the cost. Figure 3 shows the values of the F1 measure versus different λ with the selected fixing Nd. It is observed that the L1,2-norm joint sparsity regularization term is effective while it is not too small or too large. If it is too small, the correlations between different views are lost while it limits the flexibility of all the variety of views with large values. We can see that the best result of our approach is reached when λ=0.1 on Corel5K dataset and λ=0.001 on the other two datasets.


## 5. Conclusions

In this paper, we present a kernel-based multiview joint sparse coding framework for image annotation problems. It can learn a set of optimized sparse representations as well as dictionaries in a multiview kernel space. We consider label view (or tag view) as an additional view and adaptively utilize the relationship between label views and visual feature views to find the more discriminative sparse representation by incorporating multiview learning and theL1,2-norm joint sparsity. We extend KSVD and AGM to a multiview kernel version to solve our optimization problem. The priority of sparse coefficients in the kernel space is used to propagate labels to the test images by weighted greedy label transfer scheme. Experimental results on three widely used datasets show the effectiveness of our proposed method compared to the related methods for image annotation tasks. Considering that some non-ground-truth annotations can still describe the image well in our experiments, for the future research, we try to further involve human perception [33] to measure the relevancy between the non-ground-truth annotations and the images to show the potential flexibility of our proposed framework. Besides, we are going to apply deep learning features [10, 34] and multiple kernel learning [18] into our framework.



---

*Source: 6727105-2017-03-19.xml*