Fast and Robust Multi-View Multi-Task Learning via Group Sparsity
Lu Sun1 , Canh Hao Nguyen1 , Hiroshi Mamitsuka1,2
1
Bioinformatics Center, Institute for Chemical Research, Kyoto University, Japan2
Department of Computer Science, Aalto University, Finland{sunlu, canhhao, mami}@kuicr.kyoto-u.ac.jp

Task 1

Abstract

Introduction

In recent years, many real-world applications, such as web
page classification, bioinformatics analysis, semantic image annotation and product recommendation, usually exhibit
dual-heterogeneity [He and Lawrence, 2011], i.e., each task
has heterogeneous features from multiple views (feature heterogeneity), and multiple tasks are probably correlated via
one or more common views (task heterogeneity). For example, in web page classification, each web page has at least
two views: text and images, and multiple labels, such as politics, science and sports. In image annotation, each image hasfeatures extracted from multiple sources, such as color histogram, edge direction and wavelet texture, and is probably
annotated with multiple objects, like cat, dog, lion, etc.
In order to handle these heterogeneous datasets, MultiView Learning (MVL) and Multi-Task Learning (MTL) were
proposed for feature heterogeneity and task heterogeneity, respectively. In order to improve the performance of a baselinelearner, MVL aims to combine the information from multiple feature views, while MTL is proposed to learn multiple

Prediction
1

Multi-task

Panda

ParrotCat

Multi-view

View 3+=......

1

View 1
View 2......

Multi-view multi-task learning has recently attracted more and more attentions due to its dualheterogeneity, i.e., each task has hetergeneous features from multiple views, and probably correlateswith other tasks via common views. Existing methods usually suffer from three problems: 1) lackthe ability to eliminate noisy features, 2) hold astrict assumption on view consistency and 3) ignore the possible existence of task-view outliers.
To overcome these limitations, we propose a robust method with joint group-sparsity by decomposing feature parameters into a sum of two components, in which one saves relevant features (forProblem 1) and flexible view consistency (for Problem 2), while the other detects task-view outliers(for Problem 3). With a global convergence property, we develop a fast algorithm to solve the optimization problem in a linear time complexity w.r.t.
the number of features and labeled samples. Extensive experiments on various synthetic and realworld datasets demonstrate its effectiveness.

Gorilla Dog
Task T
View 1
View 2

Prediction
T

View 3Figure 1: Illustration of the framework of AGILE. Its weight matrix
Θ is decomposed into two components, W and H, where W saves
task relationship and view consistency with row-sparsity, while H
captures task outliers and inconsistent views with group-sparsity. In
prediction, the output of each task is calculated by a linear model.

correlated tasks together. Experimental results in a variety
of applications have shown the superiority of their performance against baseline learners. However no MVL method
have considered heterogeneity of MTL and vice versa, limiting their application on the datasets with dual-heterogeneity.
To overcome the limitation of MVL and MTL, recently
Multi-View Multi-Task Learning (MVMTL) has been proposed by considering both view consistency and task relationship [He and Lawrence, 2011]. Compared with MVL
and MTL methods, it achieved much success on various realworld problems with dual-heterogeneity [Zhang and Huan,2012; Jin et al., 2013; Lu et al., 2017]. However, there are still
three challenges in MVMTL. 1) Feature learning should beapplied in the learning phase. Multi-view multi-task datasets
usually emerge in high-dimensionality, where only a subset
of features, i.e, discriminative features, are useful, and thenoisy features might reduce the prediction performance. 2)
Modeling flexible view consistency is recently demanded, as
it is possible that different views provide complementary information, rather than strictly consistent with each other. One
simple example is web page classification, where the visualcontent is complementary to the text content. 3) The existence of task outliers and inconsistent views should be considered. For example, each of texts and images has uniqueinformation in web page classification.

To cope with the aforementioned challenges, we propose
a fAst and robust method with Group sparsIty for multiview multi-task LEarning (AGILE), to capture a more flexible relationship among both tasks and views, and meanwhile discard useless features and views. To handle the dualheterogeneity in a robust manner, the weight matrix Θ of AGILE is decomposed into a sum of two group-sparse components: W and H. Specifically, for Challenges 1 and 2, W
saves task relationship with selected task-common featuresby `2,1 -norm-based regularization, and models flexible view
consistency by regularizing a new norm we call group tracelasso. For Challenge 3, H simultaneously captures task outliers and inconsistent views by penalizing a group lasso regularization. We illustrate the framework of AGILE in Fig. 1.
To optimize the convex objective function of AGILE, we propose a fast optimization algorithm with global convergence,
which has a linear time complexity w.r.t. the number of features and labeled samples. Empirical results on synthetic andreal-world datasets demonstrate the superior performance of
AGILE, compared with cutting-edge methods. The contributions of this work can be summarized into three-folds.
• We propose a robust MVMTL method, enabling to copewith dual-heterogeneity, select relevant common features, and capture task outliers and inconsistent views.
• To promote flexible view consistency, we propose thegroup trace lasso norm, which is regarded as a generalization of trace lasso [Grave et al., 2011].

both task-specific feature map and task-view shared multilinear structures. However, MFM is unable to directly captureinconsistent views and utilize a possible large amount of unlabeled data. In [Zhou et al., 2018], MTMVL is treated as amulti-objective optimization problem by integrating relationships among tasks, views and samples.
Existing MVMTL methods typically build the models on
original features, however, irrelevant features would negatively influence the prediction performance. Besides, theyusually assume that predictive models from multiple views
are consistent, and multiple tasks are correlated, which probably contradicts real-world applications.

3

The AGILE method

3.1

Preliminary

For MVMTL, suppose that we have T tasks, and each task has
V views. For the t-th task, let Xt = [X1t , ..., XVt ] ∈ Rnt ×dand Ut ∈ Rmt ×d be the labeled and unlabeled data matrices,
respectively, where Xvt ∈ Rnt ×dv is the v-th view data inthe t-th task, ∀t, v. Each data matrix Xt is associated witha target vector yt ∈ Rnt , ∀t. Without loss of generality, for
an arbitrary matrix A, kAkF and kAk∗ denote its Frobenius
norm and trace norm, respectively.
Additionally, we definePp`2,1 -norm by kAk2,1 =
ka
k
i·
i=1
2 , where kai· k2 is the`2 -norm of the i-th row ai· of A.
For the t-th task of a MVMTL problem, we consider a linear predictive function defined by• We develop a fast optimization algorithm to solve theconvex objective function, and show that it has a linear
time complexity in problem size.

2

Related works

To address the problems with dual-heterogeneity, IteM2 [Heand Lawrence, 2011] was first proposed for MVMTL. IteM2
constructs a bi-partite graph for each view, and projects
any two tasks to a new reproducing kernel Hilbert space
based on their common views. However, it is a transductive method, which is designed only for problems with nonnegative features, and unable to generate predictive models for future testing samples. To overcome these limitations, inductive learning methods [Zhang and Huan, 2012;
Jin et al., 2013] were later developed based on linear models
and co-regularization [Sindhwani et al., 2005]. In [Zhang and
Huan, 2012], regMVMT was proposed by minimizing thedifference of predictive models for distinct tasks on the sameview. A more general method, CSL-MVMT [Jin et al., 2013]
was developed by assuming that a low-dimensional subspace
is shared among related tasks with common views. But thetwo methods hold a strong assumption on view consistency,
i.e., predictive models from different views should be consistent with each other on the unlabeled data, which probably violates the problem setting of various real-world problems. Inspired by linear discriminant analysis, MAMUDA [Jin et al.,2014] was proposed as a supervised feature extraction method
to handle dual-heterogeneity via shared latent spaces among
multiple views. Based on multilinear factorization machines,
MFM was recently proposed in [Lu et al., 2017] by learning

V

yt ≈

1 X v v
1
X θ = Xt θt> ,V v=1 t tV(1)

where θt = [θt1 ; ...; θtV ] ∈ Rd with θtv ∈ Rdv denoting theweight vector for the v-th view of the t-th task, ∀t, v. Eq. (1)actually averages the prediction results from all the V viewsin the t-th task, ∀t. For simplicity, we omit the intercept in
the linear model by assuming that samples and targets havebeen centered in column-wise.

3.2

Formulation

For robust MVMTL, we decompose the weight matrix Θ =[θ1 , ..., θT ] ∈ Rd×T into two components W and H. It allows to model correlations among tasks and views by W,
and capture tasks/views that are specific to some views/tasks
by H, which cannot be fit into W. Thus, based on (1) andempirical risk minimization with squared `2 -norm loss, theoptimization problem of AGILE is
min

Θ=W+H

T
X
1
t=1

2

2

yt −

1
Xt θt + R(W) + R(H),
V
2

(2)

where R(W) and R(H) denote the regularization terms on
W = [w1 , ..., wT ] and H = [h1 , ..., hT ], respectively.
R(W) aims to cope with dual-heterogeneity, while R(H) is
designed to detect task outliers and inconsistent views.
For dual-heterogeneity, we decompose R(W) into a sumof two parts, so as to capture both task relationship and view
consistency. In terms of task relationship, we consider thatmultiple tasks are correlated based on an identical featuresubset, i.e., task-common features. In this respect, the `2,1 norm [Liu et al., 2009] is imposed on W to promote its rowsparsity (feature-wise group sparsity). Since the `2,1 -norm
regularizer kWk2,1 encourages sparsity among features andnon-sparsity among tasks, discriminative task-common features will be assigned larger values. In this way, irrelevant
features are removed by relying on the complementary information from multiple views, and task relationship is modeled
via the non-zero rows (common features) of W.
In terms of view consistency, the independence assumption [Belkin et al., 2006] of multiple views has been widelyadopted by MVL methods [Blum and Mitchell, 1998; Sindhwani et al., 2005], indicating that predictive models frommultiple views can achieve mutual agreement on the unlabeled data. On some practical applications, however, it ispossible that a few views become useless for certain tasks
due to noise pollution, and the assumption is violated. To
address this challenge, for the t-th task, we impose a novelgroup trace lasso norm kUt Wt k∗ , in which Wt is a special
diagonal matrix whose diagonal elements are vectors {wtv }v ,

 1
wt


d×V
..
,
(3)
Wt = 
∈R.
wtV

with wtv ∈ Rdv denoting the v-th view weight vector of wt .
In group trace lasso, Ut Wt is a matrix containing the predictions from multiple views (each in a column) on the unlabeled data Ut . Hence, its low-rankness imposes the relationships among views in the multi-view setting. If predictivemodels are restrictively consistent among multiple views, therank of Ut Wt equals to 1. In addition, as shown in Sec. 3.3,
the regularizer promotes group-sparsity among views, so thatconsistent views will get larger values while useless views are
discarded by W. Therefore, we formulate R(W) by
R(W) := α kWk2,1 + β

TX

kUt Wt k∗ ,where α and β are positive hyperparameters. In this way,
W models task relationship via common features, promotes
flexible view consistency and discards useless views.
For robustness, we expect to capture both task outliers, thathave specific supporting features, and inconsistent views, thatare inconsistent to other views but provide complementary information. Thus, a group-sparsity inducing regularizer R(H)is imposed on H by treating each task-view pair as a group.
Specifically, for the v-th view, view-specific task outliers arePT
captured by regularizing t=1 khvt k2 , where hvt ∈ Rdv denotes the weight vector of the v-th view of ht . Similarly, forthe t-th task, task-specific inconsistent views are detected byPV
regularizing v=1 khvt k2 . By summing over V views and T
tasks on these two terms, we have
R(H) := γkhvt k2 = γ kHkG1 ,3.3

Remarks on group trace lasso

To model view consistency, some MVMTL methods [Zhang
and Huan, 2012; Jin et al., 2013] adopt co-regularization[Sindhwani et al., 2005]. In contrast, AGILE applies a novel
regularizer kUWk∗ in (6) for saving flexible consistency.
Following lemmas1 present its important properties.
Lemma 1. For the t-th task, if predictions from multiple
views are consistent, the following inequality holds,
kUt Wt k∗ = kUt wt k2 ≤ kUt kop kwt k2 ,(7)

where k·kop denotes operator norm. If prediction from multiple views are orthogonal, the following inequality holds,
kUt Wt k∗ = kUt wt kG1 ≤VXkUvt kop kwtv k2 ,

(8)

v=1

PV
where kUt wt kG1 = v=1 kUvt wtv k2 is Group `1 -norm.
Lemma 2. For the t-th task, the following inequality holds,
kUt wt k2 ≤ kUt Wt k∗ ≤ kUt wt kG1 .

(9)

(4)

t=1

T X
V
X

outlier in the v-th view, or the v-th view is inconsistent in
the t-th task, corresponding hvt would take arbitrary values,
otherwise zero.
Based on (2), (4) and (5), we now have the optimization
problem for the proposed AGILE method:
12ky − Xθk2 +α kWk2,1 +β kUWk∗ +γ kHkG1 ,min
Θ=W+H 2
(6)
P
P
where X ∈ R t nt ×dT , U ∈ R t mt ×dT and W ∈ RdT ×V T
1
are block diagonal matrices with the t-th blocks being P
V Xt ,
Ut and Wt , respectively, and y = [y1 ; ...; yT ] ∈ R t nt .
In (6), θ ∈ RdT denotes a vectorization of the weight matrix
PTΘ. Note that, the equation t=1 kUt Wt k∗ = kUWk∗ holds
based on Lemma 2.3 in [Recht et al., 2010].

(5)

t=1 v=1

where k·kG1 denotes the Group `1 -norm of groups {hvt }t,v ,and γ is a positive hyperparameter. Once the t-th task is a

Lemmas 1 and 2 tell us if predictions from two views arestrongly consistent, the proposed norm behaves like `2 -normon these two predictive models; if predictions from two views
are orthogonal, the norm equals to the group `1 norm, imposing group sparsity on {wtv }Vv=1 . Thus, it enables to equally
penalize the two consistent views, and discard the useless inconsistent views. In this sense, this method lies in between
the usual strict assumption of “all views contain ‘exactly’ thesame information” and feature concatenation, which imposesno relationship among the weights.
Note that the proposed norm is similar with the trace lasso[Grave et al., 2011], since both impose a trace norm regularizer on the product of a data matrix and a diagonal matrix
of model parameters. The difference exists in the diagonal
matrix, where each diagonal element is a parameter in tracelasso, and a group (view) of parameters in the proposed norm.
In other words, the proposed norm can be regarded as group
trace lasso, as trace lasso is its special case once the numberof parameters in each group is restricted to be one.
1
The proofs are provided in supplementary materials: https://www.dropbox.com/s/5koj44hmx4qb504/AGILE sup.pdf?dl=0.

4

Optimization algorithm

Since the objective function (6) is jointly convex w.r.t. W
and H, the global optimum is obtained by alternatively updating W and H. The trace norm in (6) is non-smooth andinvolves the product of U and W, thus it is impossible to directly employ the proximal gradient method. To circumvent
this difficulty, we apply ADMM [Boyd et al., 2011] to optimize (6) by introducing an auxiliary variable P = UW,
leading to the augmented Lagrangian defined as follows,1
2
L(Θ, P, Q) = ky − Xθk2 + α kWk2,1 + β kPk∗
2
1
2
+ kUW − P + QkF + γ kHkG1 , (10)2where Q is the scaled dual variable. The algorithm on solving(10) repeats the following three steps until convergence:22(a) Θ∗ ← minΘ 12 ky − Xθk2 + 12 kUW − P + QkF +α kWk2,1 + γ kHkG1 ,2(b) P∗ ← minP 12 kUW − P + QkF + β kPk∗ ,(c) Q∗ ← Q + UW − P.
For Problem (a), we propose to solve it by proximal
method [Nesterov, 2013], which decomposes its objective
into two components, smooth f (Θ) and non-smooth g(Θ):
1
122
f (Θ) = ky − X(w + h)k2 + kUW − P + Qk2 ,22(11)g(Θ) = α kWk2,1 + γ kHkG1 .
In the k-th iteration, the optimal solution can be calculated by
the follow two proximal operators:
1
2
min kW−( Wk − η∇w f (Θk ) )k2 + α kWk2,1 , (12)
W 2η
1
2
(13)
min kH−( Hk − η∇h f (Θk ) )k2 + γ kHkG1 ,H 2η
where ∇w f and ∇h f denote the derivatives of f (Θ) w.r.t W
and H, respectively, and η is a step size, which satisfies
f (Θk+1 ) ≤ f (Θk ) + h∇w f (Θk ), Wk+1 − Wk i
12
+ h∇h f (Θk ), Hk+1 − Hk i +Θk+1 − Θk 2 . (14)2ηThe optimization problems (12) and (13) can be analytically solved by applying soft thresholding operations as summarized in [Liu et al., 2009] and Lemma 3, respectively.
Lemma 3. If H∗ is the optimal solution of the optimization
problem (12), its t-th column h∗t is given by the proximal operator, Sηγ (rt ) = [Sηγ (r1t ); ...; Sηγ (rVt )], where

( 
ηγ
v
r
1
−
krvt k2 > ηγvt
krt k2Sηγ (rvt ) =(15)
00 ≤ krvt k2 ≤ ηγ,
and rt denotes the t-th column of Hk − η∇h f (Θk ).
For Problem (b), it admits a closed-form solution by applying a matrix soft thresholding operation [Cai et al., 2010].
It is worth noting that, based on Lemma 2.3 in [Recht et al.,2010] and the fact that UW is a diagonal block matrix, Problem (b) can be decoupled for T tasks and solved efficiently.

Table 1: The statistics of used four real-world dual-heterogeneous
datasets. Here V , T and d denote the number of views, tasks andfeatures, respectively, and np and nn are the number of positive andnegative samples in each task, respectively.
DatasetV

FOX
Mirflickr
NUS-Object
NUS-Scene2 4 3,458
178∼635
888∼1,345 Text categorization
2 8 193668∼7,849 3,413∼10,594 Image annotation
5 7 634964∼2,370 8,000∼9,406 Image annotation
5 15 629 1,039∼11,995 4,409∼15,365 Image annotation

4.1

T

d

np

nn

DomainAnalysis on time complexity

In practice, we apply fast-ADMM [Goldstein et al., 2014]
and accelerated proximal method [Nesterov, 2013] to accelerate the optimization algorithm2 , leading to an optimal convergence rate of O( k12 ) with k being the number of iterations.
In time complexity, for updating Θ, gradient calculation(∇w f and ∇Ph f ) and soft thresholding have time complexities of O(d t (nt + V mt )) and O(dT ), respectively. On
updating P, matrix soft thresholding needs to P
conduct SVD
on UW, leading to a time complexity of O(V t m2t ), provided that V  mt , ∀t. P
Therefore, the total
P time complexityof each iteration is O(d( t nt + T ) + V t m2t ), in a linear
complexity w.r.t. the number of features and labeled samples.

5

Experiments

5.1

Experimental setting

Synthetic data
For the generation of synthetic datasets, the number of tasks
and views are set as T = 12 and V = 6, respectively, andeach task has n = 150 labeled and m = 150 unlabeled data.
The dimensionality dv of P
each view is selected from 30 to
60 by step 6, with d =v dv = 300. The data matrix
X1 ∈ Rn×d1 and the weight matrix W1 ∈ Rd1 ×T fromthe 1-st view are randomly sampled from normal distributions N (0, 25) and N (0, 16), respectively. In W1 , top 30%of the features are treated as useful features by setting 70%
of rows to 0. To keep view consistency, Xv ∈ Rdv ×T and
Wv ∈ Rdv ×T (v > 1) are generated by Xv = X1 Pv> andWv = Pv W1 , respectively, with constraints Pv> Pv = Id1
and Pv ∈ Rdv ×d1 , whose columns are left-singular vectors of a matrix randomly sampled from uniform distribution
U(0, 1). Another weight matrix H ∈ Rd×T is randomly sampled from N (0, 16), where a certain set of {hvt }t,v is treatedas outliers by assigning 0 to the rest. Finally, the target yt iscalculated by yt = V1 Xt (wt + ht ) + δt , with δt ∼ N (0, 1)
being stochastic noise.
Real-world data
We conduct experiments on four real-world heterogeneous
datasets: FOX, Mirflickr, NUS-Scene and NUS-Object. The
FOX dataset is extracted from FOX web news [Qian andZhai, 2014], while Mirflickr, NUS-Scene and NUS-Objectrefer to image annotation problem [Huiskes and Lew, 2008;
Chua et al., 2009]. The statistics are summarized in Table 1,
and more details are presented in supplementary materials.
2

The procedure of the algorithm is provided in the supplement.

Comparing methods
We compare AGILE3 with six cutting-edge methods: ElasticNet [Zou and Hastie, 2005], rMTFL [Gong et al., 2012],
coMVL [Sindhwani et al., 2005], IteM2 [He and Lawrence,2011], CSL-MVMT [Jin et al., 2013] and MFM [Lu et al.,2017]. As a generalization of ridge regression [Hoerl and
Kennard, 1970] and lasso [Tibshirani, 1996], Elastic-Net isselected as a baseline method, while rMTFL and coMVL areselected as representative MTL and MVL methods, respectively. As state-of-the-art MVMTL methods, IteM2 , CSLMVMT and MFM4 are introduced in comparison, and thecodes are provided by corresponding authors.
Configuration
For evaluation, in each task, we randomly select a%, a%,
20% and 20% of total samples as labeled training set, unlabeled training set, validation set and testing set, respectively,
and a is selected from {10, 20, 30}. We repeat this process
five times, and report average results with standard deviation.
In parameter setting, the weight balancing `1 and `2 regularizers in Elastic-Net is selected from {0.2, 0.4, 0.6, 0.8, 1}. As
recommended in original papers, the dimensionality of latent
space in CSL-MVMT and MFM is set as 20. Values for otherparameters are selected from {10a |a| ∈ [3]}. For each iterative algorithm, we terminate it once the relative change of its
objective is below 10−5 , and set the maximum number of iterations as 500. The performances of comparing methods areevaluated by Area Under ROC-Curve (AUC) and Accuracy.

5.2

Experiments on synthetic data

Illustration of weight matrix decomposition
Weight matrix decomposition of AGILE on one designed
synthetic dataset is illustrated in Fig 2, where Θ∗ = W∗ +H∗
denotes the design truth model, and Θ = W + H is learned
by AGILE with the setting α = 10, β = 1 and γ = 36.
As shown in Fig. 2, AGILE successfully recovers the groupsparse pattern in Θ∗ by assigning group-sparsity to W and
H. To validate group trace lasso, for each task we select one
view as the useless view by assigning noise and 0 to corresponding Xvt and wtv , respectively, producing a feature-wiseand view-wise group-sparse W∗ in Fig. 2(a). As shown in
Fig. 2(b), W detects not only noise features (row-sparsity)
but also useless views (group-sparsity).
Comparison of MVMTL methods
Next, we analyze the robustness of AGILE. In this experiment, we generate nine synthetic datasets by changing thepercentage of task-view outliers in H from 10% to 90% by
step 10%. Fig. 3 shows the comparison results of AGILE andthree MVMTL methods in AUC and Accuracy, where AGILE
consistently achieves superior performance.
To evaluate MVMTL methods in running time, we generate two sets of synthetic datasets by varying the numberof training samples and features from 100 to 1000 by step
100, respectively. Fig. 4 shows results of AGILE and three3

We provide the MATLAB code of AGILE at: https://www.
dropbox.com/s/f6pvpy6umagwa1b/AGILE.zip?dl=0
4
MFM-F-S in [Lu et al., 2017] is used here for its best performance among the variants of MFM.

(a) Designed model Θ∗(b) Learned model Θ

Figure 2: Illustration of weight matrix decomposition of AGILE on
the synthetic dataset. Left: designed model Θ∗ = W∗ + H∗ ;Right: learned model Θ = W + H by AGILE. White (black) color
indicates non-zero (zero) values in magnitude.

Figure 3: Robustness analysis of AGILE and three MVMTL methods on nine synthetic datasets, which are generated by varying thepercentage of task-view outliers from 10% to 90% by step 10%.

MVMTL methods in running time on the two sets of datasets.
AGILE and IteM2 consume the least running time as increasing the data size. In contrast, the running time of MFM and
CSL-MVMT significantly increase as the number of training
samples and features increases, respectively.

5.3

Experiments on real-world data

Evaluation of comparing methods
To evaluate the performances of comparing methods, an experiment on four real-world heterogeneous datasets is conducted. The ratio n/N of labeled training samples is selected from the set {10%, 20%, 30%}. Experimental results
are reported in Table 2, where the best performance is highlighted in boldface. In Table 2, as n/N increases, the performances of comparing methods increase. AGILE obtains
the best performance in 75% of total cases. This performancesuperiority probably comes from AGILE’s ability on capturing task-view outliers and removing useless features. Another MVMTL method, CSL-MVMT, performs the second
best among MVMTL methods, and competes with comparing methods except AGILE and rMTFL, indicating the importance on modeling dual-heterogeneity in real-world datasets.
By handling only task-heterogeneity, rMTFL performs betteror comparable compared with the baseline Elastic-Net, especially on FOX, indicating that there probably is a strong
task-heterogeneity in the dataset. IteM2 performs worst on
the used datasets, and one possible explanation is that it isoriginally designed for problems with non-negative features.

Table 2: Experimental results on real-world datasets by selecting the percentage n/N of labeled data from {10%, 20%, 30%}.
Datasetn/N Elastic-Net

rMTFL

coMVL

AUC
IteM2 CSL-MVMT

10% .967±.002 .976±.001 .965±.002 .814±.004
20% .969±.002 .978±.001 .967±.003 .808±.009
FOX
30% .971±.001 .980±.002 .970±.001 .816±.006
10% .573±.002 .621±.001 .618±.002 .519±.002
20% .579±.002 .630±.001 .629±.002 .518±.001
Mirflickr
30% .655±.002 .630±.002 .629±.001 .510±.002
10% .851±.003 .853±.005 .877±.001 .565±.004
NUS-Object 20% .857±.001 .862±.004 .853±.002 .564±.003
30% .864±.003 .865±.002 .860±.002 .566±.003
10% .744±.010 .724±.006 .777±.001 .646±.002
NUS-Scene 20% .747±.001 .753±.003 .745±.001 .676±.009
30% .769±.001 .770±.001 .767±.001 .696±.012.972±.005.973±.003.976±.003.633±.003.633±.002.636±.004.858±.001.860±.001.862±.002.756±.001.760±.007.769±.003

MFM

AGILE

.923±.012 .985±.002.934±.011 .988±.002.929±.010 .991±.001.618±.003 .645±.002.615±.002 .635±.008.620±.006 .640±.002.836±.002 .884±.001.848±.001 .871±.006.856±.003 .874±.003.744±.005 .761±.005.745±.001 .761±.010.767±.005 .772±.007

Elastic-Net

rMTFL

coMVL

Accuracy
IteM2 CSL-MVMT.874±.004 .946±.001 .825±.003 .855±.002.892±.003 .951±.001 .850±.004 .856±.006.898±.003 .951±.003 .856±.003 .855±.002.731±.001 .759±.001 .759±.001 .704±.001.741±.006 .761±.001 .761±.001 .705±.000.796±.001 .763±.001 .764±.001 .701±.002.874±.002 .875±.003 .882±.001 .746±.001.878±.001 .880±.001 .877±.001 .745±.000.881±.001 .881±.000 .881±.001 .746±.001.838±.003 .833±.002 .848±.000 .712±.001.841±.001 .843±.000 .840±.000 .739±.006.842±.001 .844±.002 .843±.001 .743±.007.942±.002.944±.002.948±.004.754±.004.755±.004.757±.005.847±.003.850±.002.848±.001.839±.003.842±.007.844±.005

MFM

AGILE

.763±.020 .956±.002.770±.019 .963±.003.736±.013 .969±.001.766±.004 .728±.006.767±.004 .717±.012.770±.003 .710±.027.857±.001 .888±.000.866±.002 .889±.002.871±.001 .890±.001.820±.003 .822±.006.840±.006 .843±.009.844±.008 .845±.005

Figure 6: Sensitivity analysis of α, β and γ in AUC on NUS-Object.
Parameter values are shown in the logarithmic scale.
Figure 4: Comparison of methods in running time on two sets ofsynthetic datasets, which are generated by varying the numbers of
training samples (Left) and features (Right) from 100 to 1000.

{10a |a| ∈ [4]}. Fig. 6 shows the experimental resultsin AUC with n/N = 30%. The subfigure on α and β isshown by fixing γ = 1, and the similar setting is used for theother two subfigures. As shown in Fig. 6, AGILE achieves
its best performance on NUS-Object with α ≤ 101 , β = 100
and 101 ≤ γ ≤ 102 . Generally, it is recommended to assign
smaller values to α and β, and a relatively larger value to γ.

6

Figure 5: Analysis on the effect of regularizers in AGILE on NUSObject. AGILEα=0 , AGILEβ=0 and AGILEγ=0 denote three variants of AGILE by setting α = 0, β = 0 and γ = 0, respectively.

Analysis on the effect of regularizations
To evaluate the effect of regularizations used in AGILE, anexperiment is performed by assigning 0 to each of three parameters, respectively, and evaluation results on NUS-Object
are shown in Fig. 5. As shown in Fig. 5, setting γ = 0 results in the largest performance loss in four out of six cases,
demonstrating the importance of capturing task-view outliers
on improving performance. Although setting β = 0 gives theleast performance loss in half of cases, there is still a significant loss compared with the original AGILE.
Hyperparameter sensitivity analysis
The sensitivity of AGILE in three regularization parameters α, β and γ is investigated on NUS-Object. Specifically, α controls the row-sparsity of W, β measures the degree of view consistency and γ controls the group-sparsityof H. Values of the parameters are selected from the set

Conclusion

In this paper, we propose a fast and robust MVMTL method,
AGILE, by decomposing the weight matrix into two components, and adopting a joint regularization to promote groupsparsity. To select relevant features and model flexible view
consistency, joint group-sparsity is imposed on the first component by `2,1 -norm and group trace lasso. To detect taskoutliers and inconsistent views, the second component isregularized by group `1 -norm to encourage its group sparsity. Thanks to the convexity of the objective function, afast iterative algorithm is developed to solve it with global
convergence. Experiments on both synthetic and real-world
datasets demonstrate its effectiveness. It shows that in real
MVMTL setting, data exhibit all the problems, like existenceof noisy features, useless views and task-view outliers, andour method is flexible enough to handle it.

Acknowledgements
L. S. has been supported in part by JST ACCEL (grant number JPMJAC1503). C. H. N. has been supported in part by
MEXT Kakenhi 18K11434. H. M. has been supported in partby JST ACCEL (grant number JPMJAC1503), MEXT Kakenhi (grant numbers 16H02868 and 19H04169), FiDiPro byTekes (currently Business Finland) and AIPSE program by
Academy of Finland.

References[Belkin et al., 2006] Mikhail Belkin, Partha Niyogi, andVikas Sindhwani. Manifold regularization: A geometricframework for learning from labeled and unlabeled examples. Journal of machine learning research, 7(Nov):2399–
2434, 2006.
[Blum and Mitchell, 1998] Avrim Blum and Tom Mitchell.
Combining labeled and unlabeled data with co-training. In
Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100. ACM, 1998.
[Boyd et al., 2011] S Boyd, N Parikh, B Peleato E Chu, andJ Eckstein. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers.
Foundations and Trends in Machine Learning, 3(1):1–122,
2011.
[Cai et al., 2010] Jian-Feng Cai, Emmanuel J. Candès, andZuowei Shen. A Singular Value Thresholding Algorithm
for Matrix Completion. SIAM Journal on Optimization,
20(4):1956–1982, 2010.
[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang
Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. Nuswide: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image andVideo Retrieval (CIVR’09), Santorini, Greece., 2009.
[Goldstein et al., 2014] Tom
Goldstein,
BrendanO’Donoghue, Simon Setzer, and Richard Baraniuk.
Fast alternating direction optimization methods. SIAM
Journal on Imaging Sciences, 7(3):1588–1623, 2014.
[Gong et al., 2012] Pinghua Gong, Jieping Ye, and Changshui Zhang. Robust multi-task feature learning. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages
895–903. ACM, 2012.
[Grave et al., 2011] Edouard Grave, Guillaume R Obozinski, and Francis R Bach. Trace lasso: a trace norm regularization for correlated designs. In Advances in Neural
Information Processing Systems, pages 2187–2195, 2011.
[He and Lawrence, 2011] Jingrui He and Rick Lawrence. A
graph-based framework for multi-task multi-view learning. Proceedings of the 28th International Conference on
Machine Learning, ICML 2011, pages 25–32, 2011.
[Hoerl and Kennard, 1970] Arthur E Hoerl and Robert W
Kennard.
Ridge regression: Biased estimation for
nonorthogonal problems. Technometrics, 12(1):55–67,
1970.
[Huiskes and Lew, 2008] Mark J. Huiskes and Michael S.
Lew. The mir flickr retrieval evaluation. In MIR ’08: Proceedings of the 2008 ACM International Conference on
Multimedia Information Retrieval, New York, NY, USA,2008. ACM.
[Jin et al., 2013] Xin Jin, Fuzhen Zhuang, Shuhui Wang,
Qing He, and Zhongzhi Shi. Shared Structure Learning for Multiple Tasks with Multiple Views. In Hendrik
Blockeel, Kristian Kersting, Siegfried Nijssen, and FilipŽelezný, editors, Machine Learning and Knowledge Discovery in Databases, pages 353–368, Berlin, Heidelberg,
2013. Springer Berlin Heidelberg.
[Jin et al., 2014] Xin Jin, Fuzhen Zhuang, Hui Xiong,
Changying Du, Ping Luo, and Qing He. Multi-task Multiview Learning for Heterogeneous Tasks. Proc.CIKM,
3:441–450, 2014.
[Liu et al., 2009] Jun Liu, Shuiwang Ji, and Jieping Ye.
Multi-task feature learning via efficient l2, 1-norm minimization. In Proceedings of the Twenty-Fifth Conference
on Uncertainty in Artificial Intelligence, UAI ’09, pages
339–348, Arlington, Virginia, United States, 2009. AUAI
Press.
[Lu et al., 2017] Chun-Ta Lu, Lifang He, Weixiang Shao,
Bokai Cao, and Philip S. Yu. Multilinear Factorization
Machines for Multi-Task Multi-View Learning. In Proceedings of the Tenth ACM International Conference on
Web Search and Data Mining - WSDM ’17, pages 701–
709, 2017.
[Nesterov, 2013] Yurii Nesterov. Introductory lectures on
convex optimization: A basic course, volume 87. Springer
Science & Business Media, 2013.
[Qian and Zhai, 2014] Mingjie Qian and Chengxiang Zhai.
Unsupervised feature selection for multi-view clusteringon text-image web news data. In Proceedings of the
23rd ACM international conference on conference on information and knowledge management, pages 1963–1966.
ACM, 2014.
[Recht et al., 2010] Benjamin Recht, Maryam Fazel, andPablo A Parrilo. Guaranteed minimum-rank solutions oflinear matrix equations via nuclear norm minimization.
SIAM review, 52(3):471–501, 2010.
[Sindhwani et al., 2005] Vikas Sindhwani, Partha Niyogi,
and Mikhail Belkin. A co-regularization approach to semisupervised learning with multiple views. In Proceedings of
ICML workshop on learning with multiple views, volume
2005, pages 74–79. Citeseer, 2005.
[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage
and selection via the lasso. Journal of the Royal StatisticalSociety. Series B (Methodological), pages 267–288, 1996.
[Zhang and Huan, 2012] Jintao Zhang and Jun Huan. Inductive multi-task learning with multiple view data. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’12,
page 543, 2012.
[Zhou et al., 2018] D. Zhou, J. Wang, B. Jiang, H. Guo, and
Y. Li. Multi-task multi-view learning based on cooperative multi-objective optimization. IEEE Access, 6:19465–
19477, 2018.
[Zou and Hastie, 2005] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of
the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301–320, 2005.