Multiplicative Sparse Feature Decomposition for Efficient Multi-View Multi-Task
Learning
Lu Sun1 , Canh Hao Nguyen1 , Hiroshi Mamitsuka1,2
1
Bioinformatics Center, Institute for Chemical Research, Kyoto University, Japan2
Department of Computer Science, Aalto University, Finland{sunlu, canhhao, mami}@kuicr.kyoto-u.ac.jp
Abstract
Multi-view multi-task learning refers to dealing
with dual-heterogeneous data, where each sample
has multi-view features, and multiple tasks are correlated via common views. Existing methods do
not sufficiently address three key challenges: (a)saving task correlation efficiently, (b) building asparse model and (c) learning view-wise weights.
In this paper, we propose a new method to directly handle these challenges based on multiplicative sparse feature decomposition. For (a), theweight matrix is decomposed into two components
via low-rank constraint matrix factorization, whichsaves task correlation by learning a reduced number of model parameters. For (b) and (c), the firstcomponent is further decomposed into two subcomponents, to select topic-specific features andlearn view-wise importance, respectively. Theoretical analysis reveals its equivalence with a generalform of joint regularization, and motivates us to develop a fast optimization algorithm in a linear complexity w.r.t. the data size. Extensive experiments
on both simulated and real-world datasets validate
its efficiency.

1

Introduction

Multi-View Multi-Task Learning (MVMTL) is an importantlearning problem with a variety of interesting real-world applications, such as text categorization [He and Lawrence,2011], bioinformatics analysis [Zhang and Huan, 2012], semantic image annotation [Jin et al., 2013] and web pageclassification [Lu et al., 2017]. The objective of MVMTL
is to improve the prediction of targets by relying on dualheterogeneity [He and Lawrence, 2011]. Dual-heterogeneity
comprises task-heterogeneity and feature-heterogeneity, indicating that multiple tasks possibly related with each otherby the common views of samples, and each sample can havevarious representations from different views. A representative example is classification of protein functions, in which
each protein is possibly associated with multiple functionalclasses, including metabolism, transcription and cellular organization, and has distinct features from multiple views,
such as protein sequences and 3D structures.

The problems with single-heterogeneity have been extensively studied in Multi-Task Learning (MTL) and Multi-View
Learning (MVL). The intuitive idea behind MTL/MVL is
that, learning performance for one single task/view can be improved by leveraging samples from other related tasks/views.
MVL captures feature-heterogeneity by saving view consistency in the way that predictive models from multiple views
achieve mutual agreement on the unlabeled data [Blum and
Mitchell, 1998; Hardoon et al., 2004]. In contrast, MTL saves
task-heterogeneity by modeling task correlation and learning
multiple correlated tasks together, based on joint regularization [Liu et al., 2009b; Gong et al., 2012a] or feature decomposition [Jalali et al., 2010; Han and Zhang, 2015].
Recent research has gradually shifted its emphasis fromthe problems with single-heterogeneity to the ones with dualheterogeneity. Several MVMTL methods have recently been
proposed, based on joint regularization [Zhang and Huan,2012], factorization machines [Lu et al., 2017], and multiobjective optimization [Zhou et al., 2018]. Current MVMTL
methods usually suffer from three problems: (a) saving taskcorrelation in the way that models of each task-pair are similar, which is too strong on many real problems, and results
in a relatively large number of model parameters to learn; (b)building a learning model on original features, however, irrelevant features would harm the generalization ability, and
it is probable that a group of correlated tasks shares groupspecific features; (c) modeling strict view consistency, thatmodels from multiple views should be consistent, and assigning a same weight to different views. But it can be expectedthat multiple views provide supplementary information, anddifferent views contribute distinct importance to the model.
To address all the three problems, in this paper, we propose a novel method via multiplicative Sparse feature decomPosition for muLti-vIew multi-Task learning (SPLIT),
which decomposes the weight matrix Θ into three multiplicative components A, B and H. Under an assumption that tasks
can be reconstructed by a small number of latent topics, Θ isfirst decomposed as Θ = WH by low-rank constraint matrix
factorization, leading to a reduced number of effective parameters of Θ. To learn a topic-specific sparse model and a viewweighting scheme, W is further decomposed as W = A ◦ B
by Hadamard (element-wise) product, where A and B selecttopic-specific relevant features and assign different weights
to distinct topic-view pairs, respectively. Fig. 1 illustrates theFigure 1: Multiplicative feature decomposition used for generating
a simulated dataset with 4 views, 4 topics and 8 tasks. The weightmatrix Θ is decomposed by Θ = (A ◦ B)H. Components A andB store topic-specific features and view-wise weights, respectively,
while W = A ◦ B and H together model task correlation. Darker
blue (yellow) color indicates smaller (larger) values in magnitude.

multiplicative sparse feature decomposition. Our theoreticalanalysis shows two things: 1) the family of methods is equivalent to a joint regularization with a more general form of
regularizers, and 2) feature-sparse component A is directly
related to view-weighting component B, motivating us to derive a fast optimization algorithm in a linear complexity w.r.t.
the data size. Promising empirical results on various datasets
demonstrate the efficiency of SPLIT for MVMTL. We highlight the contributions of this paper as follows:• We focus on the key missing points of current MVMTL
methods, i.e. feature selectivity and view weighting, anddevelop a new method that allows us to efficiently select relevant features, and assign view-wise importance.
This was not realized by any method so far.
• Theoretical analysis reveals that the equivalence between a general family of the proposed framework andthe jointly regularized approach, leading to two novel
MVMTL formulations.
• An efficient optimization algorithm is developed to solve
the optimization problem in a linear time w.r.t. the data
size, which has not been realized by any method so far.
We begin by discussing related works in Section 2. Next,
we introduce the proposed SPLIT in Section 3. We provide
theoretical analysis of a family of SPLIT in Section 4, anddevelop an efficient algorithm in Section 5. In Section 6, we
present experimental results on evaluation. Finally, we conclude this paper in Section 7.

2

Related works

For Multi-task learning, based on the assumption that task
relationship is shared through task-common features, regularization with `p,q -norm (p > 1, q ≥ 1) is imposed on theweight matrix to encourage group sparsity [Liu et al., 2009a;
Gong et al., 2012a]. To capture both task commonality andspecificity, feature decomposition approaches are proposed
via decomposing the weight matrix into multiple components

by summation [Chen et al., 2011; Gong et al., 2012b] or multiplication [Lozano and Swirszcz, 2012; Wang et al., 2016].
Multi-view learning aims to utilize features from different views to improve the performance of a baseline learner.
Two types of methods, co-training [Blum and Mitchell, 1998;
Muslea et al., 2006; Sun and Jin, 2011] and co-regularization[Sindhwani et al., 2005; Xie and Sun, 2015; Kan et al., 2016],
are proposed to make predictive models from multiple views
consistent on the unlabeled data.
For multi-view multi-task learning, IteM2 [He andLawrence, 2011] constructs a bi-partite graph for each view,
and projects any two tasks to a new reproducing kernel
Hilbert space based on their common views. As an inductive learning method, CSL-MTMV [Jin et al., 2013] aims tomine a low-dimensional subspace shared among related tasks
with common views. Based on multilinear factorization machines, MFM was recently proposed in [Lu et al., 2017] bylearning both task-specific feature map and task-view sharedmultilinear structures. In [Li and Huan, 2018], asymmetricbilinear factor analyzers with rank constraints are applied tocapture the interactions among tasks and views.

3

The proposed method

3.1

Preliminary

For a MVMTL problem with V views and T tasks, given thevdata Xt = [X1t , ..., XVt ] ∈ Rnt ×d of t-th
P task, where Xt ∈Rnt ×dv denotes the v-th data with d = v dv , we introduce
a linear model to approximate the t-th target yt ∈ Rnt :Vyt ≈1 X v v
1
Xt θt = Xt θt ,
V v=1
V

∀t.

(1)

In (1), θt = [θt1 ; ...; θtV ] ∈ Rd is the parameter model ofthe t-th task, with θtv ∈ Rdv being its sub-vector of the v-th
view. For T tasks, we have Θ = [θ1 , ..., θT ] ∈ Rd×T . For
convenience, we omit the intercept in (1) by assuming thatthe input data and target have been centered in column-wise.

3.2

Methodology

In order to model task correlation efficiently, we assume thatmultiple tasks are correlated through a subspace constructedby a limited number of latent topics. For instance, in image
annotation, the tags (tasks) “ocean” and “sky” can be treatedas a single latent topic, since they typically emerge together
and share similar color features. Thus, instead of directly
learning Θ ∈ Rd×T , we impose a low-rank constraint on
it, and restrict Θ to learn only a reduced number of effective
parameters, which is much less than d × T . To this end, matrix factorization is applied to decompose Θ by Θ = WH,
where W ∈ Rd×K and H ∈ RK×T with K denoting thenumber of latent topics, K ≤ T . Each column of W corresponds to one latent topic, and H is usually considered as afactor loading matrix, whose t-th column ht stores the coefficients of K latent topics for the t-th task. In this sense, themodel parameter θt belonging to the t-th task enables to bereconstructed by θt = Wht . This low-rank constraint matrix
decomposition not only captures task correlations, but also
controls overfitting by reducing the model size.

In terms of multi-view sparse learning, it is possible that asmall feature subset from one view is crucial to detect a certain topic, even though that view is not discriminative for the
topic. In other words, we expect that each topic is supported
by a specific subset of features, while different views contributes distinct importance. A real example is image annotation, where features from edge view are critical to detect alatent topic comprising tags (tasks) “car”, “bus” and “truck”,
and almost irrelevant for the topic consisting of tags (tasks)“ocean” and “sky”. Thus, to learn a topic-specific sparsemodel and a view-weighting scheme, for the v-th view of thek-th column (topic) of W, we decompose it by
wkv = βkv αvk = αvk ◦ βkv 1dv = αvk ◦ βkv ,(2)v
dv
where αk ∈ R aims at selecting topic-specific features forthe v-th view, βkv ≥ 0 denotes the topic-specific weight of thev-th view, and 1dv is a all-one vector in size of dv . Therefore,
the topic model W can be reformulated by

 
1d1

 
..
W = A ◦ (3)
 B  = A ◦ B,.
1dV
where B ∈ RV ×K with βkv being the element in the v-th
row of the k-th column, ∀t, v. In (3), A ∈ Rd×K and
B ∈ Rd×K store the topic-specific sparse model and theview-weighting scheme, respectively, with the k-th column
being αk = [α1k ; ...; αVk ] and βk = [βk1 ; ...; βkV ], respectively. We then use `1 -norm and Frobenius norm
P toi regularize A and B, respectively, with kAk1,1 =ij |αj | andP
2
kBkF = ij |βji |2 . In this way, topic-specific irrelevant features will be discarded by A, while non-discriminative views
will be assigned with small weights by B.
Inspired by above motivations, we decompose the model
parameter Θ into a product of three component, i.e., Θ =WH = (A ◦ B)H, and propose the multiplicative multi-task
multi-view sparse feature learning model as follows:
min
Θ

T
X

L(yt ,

t=1

1
2
2
Xt θt ) + λ1 kAk1,1 + λ2 kBkF + η kHkF ,V

s.t. Θ = (A ◦ B)H, B ≥ 0.
For the v-th view of the t-th task (∀t, v), we haveKX
θtv = Wv ht = (Av ◦ Bv )ht =hkt (αvk ◦ βkv ),

(4)

(5)

k=1

where hkt is the coefficient of the k-th topic for the t-th task.

4

Theoretical analysis

In this section, we present theoretical analysis of a generalform of (4), which helps us to develop an efficient algorithm
for a family of SPLIT. This general form is formualted by
T
X

K

V

K X
V
X
k=1 v=12|βkv |q + η kHkF ,(1, 1)(1, 2)(2, 1)(2, 2)
2
1
1
2
√
√
3
3
3
3
γ
2 λ1 λ2
γ = 2λ1 λ2
2λq
γ = 2 λ1 λ2
1 λ2
P
P
P p v
P q
2
2
3
3
R(W)
kwkv k2
kwk k1
kwkv k1
kwkv k2
k,v
k,v
q k,v
q k,v
2
2
−1
−1
−1
−1
βkv
λ1 λ2 kαvk k1
λ1 λ2 kαvk k1 λ1 λ2 kαvk k2
λ1 λ2 kαvk k2

qPp
where k·kp denotes `p -norm with kαkp = p
j |αj | . Obviously, (6) becomes (4) with p = 1 and q = 2. The following
theorem1 shows that, under some conditions, (6) is equivalent
to a jointly regularized model.
Theorem 1. Let (Ŵ, Ĥ) be the optimal solution of the following optimization problem,
T
X

K

V

XX q
1
p
2
2s
Xt θt )+γ
kwkv kp +η kHkF ,
Θ=WH
V
t=1
k=1 v=1(7)
where wkv is the v-th view sub-vector of the k-th column ofW. If {Â, B̂, Ĥ} is theq optimal solution of (6), we have Ŵ =min

L(yt ,Â ◦ B̂, given γ = 2

p
2− qs

λ1

p

λ2qs and s = p+q
2q . In addition,

the view-weighting component B̂ is related with the topicspecific sparse component Â by the following formulaqv p∀v, k.
(8)
βkv = q λ1 λ−1
2 kαk kp ,Theorem 1 reveals the family in (6) is equivalent to a jointly
regularized problem in (7) with a more general form of regulariers, while Eq. (8) implies the sparsity of the topic-specificcomponent A is relative to the sparsity of the view-weighting
component B. A direct instantiation of Theorem 1 with p = 1
and q = 2 is exactly related with (4). Table 1 summarizes theconclusions derived from Theorem 1 with p, q ∈ {1, 2}.

5

Optimization algorithm

Despite of the equivalence of (4) and (7), here we aim to solve(4). It is because optimizing A and B separately is mucheasier than optimizing W directly, and B gives some insight
into the relationship between views and topics. The objectivefunction of (4) is convex w.r.t. A, B and H, respectively,
which motivates us to develop an alternating algorithm. For
simplicity, least squared loss is considered here. The algorithm repeats following three steps until convergence.
(i) Update A with fixed B and H:
min

T
X
t=1

yt −

1
Xt (A ◦ B)ht
V2

+ λ1 kAk1,1 . (9)It is a lasso-like problem, which can be solved by many
efficient gradient descent optimization approaches. Let

k=1

+ λ2

(p, q)

A

XX
1
p
min
L(yt , Xt θt ) + λ1
kαvk kpVΘ=(A◦B)H,
t=1
v=1
B≥0

Table 1: A summary of conclusions of Theorem 1 with p, q ∈{1, 2}. Here R(W) is the regularization term w.r.t. W in (7).

(6)

1Proofs are provided in the supplement: https://www.dropbox.
com/s/vwl1jtt0qiso8j8/SPLIT sup.pdf?dl=0.

∇f (A) denote the derivative of the loss function in (9)w.r.t A, whose k-th column equals to


T
X
2
1
∇f (A)|k = βk ◦
X
(A
◦
B)h
−
y
hkt X>
t
t
t .
t
V
V
t=1
(10)
Then, proximal gradient descent method can be applied
to update A according toA∗ ← sof t(A − µ∇f (A), λ1 ),(11)where sof t(a, b) = sign(a) max(|a| − b, 0) is the soft
thresholding operator, and µ is the learning rate determined by line search.
(ii) Update B with fixed A and H: According to Theorem 1
and Table 1, B is solved with a closed-form solution.
(iii) Update H with fixed W = A ◦ B:
min
H

T
X

yt −

t=1

1
Xt WhtV22+ η kHkF .

(12)

Let X̃t = Xt W, above problem has a closed-form solution, whose t-th column is calculated by
1 >X̃ X̃t + ηV IK )−1 X̃>(13)t yt .
V t
Since it typically follows K ≤ T  d, the closed-form
solution (13) can be efficiently computed.
Proposition 1. The proposed iterative optimization algorithm does not increase the objective function of (4) at each
iteration, indicating thatht = (

J(A(i+1) , B(i+1) , H(i+1) ) ≤ J(A(i) , B(i) , H(i) ),(14)

in the (i + 1)-th iteration, with J(A, B, H) denoting the objective function of (4) w.r.t. A, B and H.
Proposition 1 guarantees that the optimization algorithm
does not increase the objective value of (4) in each iteration.
In practice, accelerated proximal method [Nesterov, 2013] isapplied to accelerate the algorithm. In terms of time complexity analysis, updating A comprises two major steps, gradient computation and soft thresholding, resulting in time complexities
P of O(d(N + T K)) and O(dK), respectively, with
N = t nt being the total number of samples from multiple
tasks. Computation of B needs a time complexity of O(dK),and updating H with the closed-form solution has a time costof O(K 2 (N + T K)). Therefore, the total time complexity ofeach iteration is O(d(N + T K) + K 2 (N + T K)), which islinear in number of samples, features and tasks.

6

Experiments

6.1

Datasets and comparing methods

Four real-world datasets are used for performance evaluation,and their statistics are summarized in Table 2. We compare
SPLIT with five methods for performance evaluation. Ridge
regression [Hoerl and Kennard, 1970] and Lasso [Tibshirani, 1996] are selected as baseline methods. As representative MTL methods with multiplicative feature learning, MLL

Table 2: The statistics of used real-world datasets, where V and T
denote the number of views and tasks,
P respectively, dv is the numberof features in the v-th view (d = v dv ), and ntp /ntn is the numberof positive/negative samples in the t-th task (nt = ntp + ntn ).
nt
URL
ntp
ntn
Mirflickr
2 8 43∼150 193
668∼7,849 3,413∼10,594 URL1
Caltech101 6 4 40∼1,984 3,766
123∼798 1,588∼2,263 URL2
NUS-Object 5 7 64∼225 634964∼2,370 8,000∼9,406 URL3
NUS-Scene 5 15 63∼224 629 1,039∼11,995 4,409∼15,365 URL3
Datasets

V

T

dv

d

URL1: https://press.liacs.nl/mirflickr/
URL2: http://www.vision.caltech.edu/Image Datasets/Caltech101/
URL3: http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm[Lozano and Swirszcz, 2012] and MMTFL [Wang et al.,2016] enable to model various patterns of task-specific/taskcommon features, and outperform several MTL methods
on real-world applications. As a state-of-the-art MVMTL
method, MFM [Lu et al., 2017] is selected due to its superior
performance compared with existing MVMTL methods. The
proposed SPLIT2 is implemented in two variants: SPLIT1and SPLIT2 , corresponding to p = 1 and p = 2, respectively,
both with q = 2 in (6). The difference between them is that,for an irrelevant feature, SPLIT1 might shrink corresponding
element in A to zero, and SPLIT2 might shrink it to a small
non-zero value, instead.

6.2

Configuration

In evaluation, for each task we randomly select a%, 20%
and 20% of its total samples as training set, validation set
and testing set, respectively, with a ∈ {10, 20, 30}. We repeat this procedure five times, and report the mean value andstandard deviation of two metrics, Area Under ROC-Curve(AUC) and Accuracy. Grid search is conducted on the evaluation set, and the best parameter setting is used for prediction on the testing set. For grid search, values of regularization coefficients of comparing methods are selected from
{10a |a| ∈ {0, 1, 2, 3, 4}}. The dimensionality of latent
space in MFM is set as 20, as recommended in [Lu et al.,2017]. The number K of latent topics of SPLIT is set according to K
T ∈ {0.3, 0.5, 0.7, 0.9}. For each iterative algorithm,
we terminate it once the relative change of its objective is below 10−5 , and set the maximum number of iterations as 1000.

6.3

Experiments on simulated datasets

Simulated datasets
For simulated datasets, we set the number of tasks and views
are set as T = 8 and V = 4, respectively, and selectthe dimensionality
dv of the v-th view from {25, 42, 58, 75}Pwith d =v dv = 200. Each task has the same number (n = 200) of labeled samples. The weight matrix Θ
is decomposed into three parts in a multiplicative way, i.e.,Θ = (A ◦ B)H, with the latent dimensionality being K = 4.
Elements of A ∈ Rd×K are randomly sampled according tonormal distribution N (0, 16), while elements of B ∈ RV ×T
and H ∈ RK×T are randomly sampled based on uniform distributions U(0, 1) and U(−1, 1), respectively. To make A and2
We provide the MATLAB code of SPLIT at: https://www.
dropbox.com/s/ej7joxq6nv2yoto/SPLIT.zip?dl=0

Figure 4: Performance of SPLIT1 on three simulated datasets byvarying its number K of latent topics from 1 to 8 by step 1. The
datasets are generated by changing the number K ∗ of latent semantics in truth model Θ∗ according to K ∗ ∈ {1, 2, 4}.
Figure 2: Illustration of multiplicative feature decomposition by
SPLIT1 on simulated data with designed model Θ shown in Fig. 1.
b = (Ab ◦ B)b H
b is learned by SPLIT1 . Darker yellow
The model Θ(blue) color indicates larger (smaller) values in magnitude.

Figure 3: Comparison of SPLIT1 , SPLIT2 and two baseline methods on ten simulated datasets, which are generated by varying thepercentage of zero-elements in A from 0% to 90% by step 10%.

B sparse, a% and b% of total elements are assigned with value
0, respectively. Finally, for the t-th task, its target vector yt iscalculated by yt = V1 Xt θt + δt , where Xt is randomly sampled from normal distribution N (0, 25), and δt ∼ N (0, 1)
denotes the stochastic noise.
Case study on feature decomposition and sparse learning
We illustrate multiplicative feature decomposition of SPLIT
b = (Ab◦on one designed synthetic dataset in Fig 23 , where Θ1b Hb is learned by SPLIT with the setting λ1 = 10 , λ2 =B)
103 and η = 104 . The designed model Θ = (A ◦ B)His shown in Fig. 1. As shown in Fig. 1 and Fig. 2, SPLIT1
successfully detects the underlying models by selecting topicspecific features in A, learning view-wise weights in B, andsaving task correlation in W = A ◦ B and H.
Next, to evaluate the performance of SPLIT on handlingthe datasets with sparse underlying models, we conduct anexperiment on ten synthetic datasets by varying the percentage of zero-elements (irrelevant features) in A from 0% to
90% by step 10%. Fig. 3 shows the comparison result of
SPLIT1 , SPLIT2 and two baseline methods, Lasso and Ridge,
in AUC and Accuracy. As the sparsity of A increases,
SPLIT1 and SPLIT2 consistently outperform two baselines
with a significant advantage. The sparse ratio 0.7 in A seems
3

A quantitative comparison is presented in the supplement.

to be a turning point for sparse methods (SPLIT1 and Lasso)
and dense methods (SPLIT2 and Ridge). Sparse methods perform worse than their dense counterparts when the ratio is below 0.7, but outperform them once the ratio exceeds 0.7. It
shows that sparse learning is necessary for MVMTL, when
the underlying model is indeed sparse.
Analysis on task correlation modeling
To model task correlation, SPLIT constructs multiple tasks
based on a limited number of latent topics. We expectthat it would avoid overfitting, and give computational efficiency. To evaluate this effect, we generate three simulated
datasets by changing the number K ∗ of latent topics withK ∗ ∈ {1, 2, 4}, and apply SPLIT1 on each dataset by varying its number K of latent topics from 1 to 8 by step 1. As
shown in Fig. 4, as the value of K increases, SPLIT consumes
more running time, because it needs to learn a larger numberof effective parameters. In terms of AUC, as the value of K ∗increases, SPLIT always achieves the best performance when
K = K ∗ . Therefore, once tasks are modeled by a small
number of latent topics in one dataset, SPLIT has a chance toimprove the performance in less running time.

6.4

Experiments on real-world datasets

Evaluation of comparing methods
For evaluation of comparing methods, we conduct an experiment on four multi-view multi-task datasets, and report empirical results in AUC and Accuracy by Table 34 . In theexperiment, we change the ratio n/N of training samples
from 10% to 30% by step 10%. In Table 3, the best performance is highlighted in boldface. For all methods, we
can see that as the ratio n/N increases, the performance inboth AUC and Accuracy increases as well. Specifically, thetwo variants of SPLIT, SPLIT1 and SPLIT2 , together perform the best in 20 cases out of total 24 cases. Such observation validates the effectiveness of SPLIT on handling
MVMTL problems via multiplicative sparse feature decomposition. Its ability on selecting relevant features, weighting different views and saving task correlation, leads to amore powerful MVMTL learner. As multiplicative multitask learning methods, MLL and MMTFL perform the second best, and compete with SPLIT1 and SPLIT2 on the Caltech101 and NUS-Object datasets. It is probably because
4

Statistical test on the results is provided in the supplement.

Table 3: Experimental results on four real-world datasets by selecting the percentage n/N of labeled data from {10%, 20%, 30%}.
Dataset

n/N

Ridge

Lasso

MLL

AUC
MMTFL

MFM

SPLIT1SPLIT2

Ridge

Lasso

MLL

Accuracy
MMTFL

MFM

SPLIT1SPLIT2

10% .615±.004 .623±.000 .637±.002 .635±.002 .618±.003 .638±.004 .639±.004 .775±.002 .777±.001 .788±.001 .787±.001 .766±.004 .791±.001 .792±.002
Mirflickr
20% .630±.003 .632±.003 .652±.003 .651±.003 .615±.002 .656±.002 .655±.002 .780±.001 .780±.000 .792±.001 .792±.001 .767±.004 .794±.002 .795±.001
30% .650±.002 .652±.002 .663±.002 .664±.002 .620±.006 .667±.002 .666±.002 .784±.001 .785±.001 .794±.001 .794±.001 .770±.003 .798±.001 .797±.001
10% .986±.002 .990±.000 .998±.000 .997±.000 .978±.003 .991±.002 .990±.001 .929±.004 .983±.001 .978±.003 .983±.001 .969±.005 .981±.003 .982±.003
Caltech101 20% .990±.000 .990±.001 .997±.000 .997±.001 .987±.004 .999±.000 .992±.002 .981±.001 .986±.000 .981±.001 .983±.002 .974±.004 .991±.001 .989±.002
30% .990±.000 .990±.000 .999±.000 .999±.000 .990±.001 .999±.000 .992±.001 .986±.000 .987±.001 .993±.001 .990±.001 .984±.003 .991±.003 .990±.001
10% .843±.004 .833±.002 .841±.006 .841±.013 .836±.002 .845±.004 .839±.004 .863±.003 .859±.001 .876±.001 .878±.005 .857±.001 .869±.003 .866±.001
NUS-Object 20% .851±.002 .863±.001 .869±.001 .862±.004 .848±.001 .870±.001 .866±.001 .876±.001 .881±.001 .888±.002 .885±.003 .866±.002 .888±.002 .885±.002
30% .860±.002 .867±.003 .877±.002 .874±.002 .856±.003 .877±.002 .874±.001 .880±.001 .882±.001 .894±.001 .892±.001 .871±.001 .894±.001 .892±.001
10% .744±.006 .734±.012 .748±.003 .753±.003 .744±.005 .780±.001 .782±.002 .835±.001 .832±.012 .842±.001 .847±.000 .820±.003 .864±.001 .864±.001
NUS-Scene 20% .747±.001 .743±.001 .760±.002 .772±.002 .745±.001 .791±.001 .792±.001 .840±.001 .842±.001 .858±.001 .861±.001 .840±.006 .865±.001 .865±.000
30% .767±.001 .771±.001 .775±.004 .793±.001 .767±.005 .800±.001 .800±.001 .845±.001 .846±.001 .863±.000 .866±.001 .844±.008 .867±.001 .868±.001

Figure 5: Analysis on the view-weighting effect of SPLIT on the
NUS-Scene dataset. In this experiment, the training ratio is selectedfrom {10%, 20%, 30%}. SPLITλ2 =0 is a degenerated variant of
SPLIT1 by assigning a same weight to all views.

both of the methods enable to select task-specific features andencourage task correlations via common features. In terms of
two baseline methods, Lasso outperforms Ridge regression inalmost all the cases, indicating the importance on learning asparse model by discarding useless features for MVMTL.
Analysis on the view-weighting effect
To demonstrate the effectiveness of view-weighting component B used in SPLIT, an experiment is performed by comparing SPLIT1 and SPLIT2 with a special variant SPLITλ2 =0 ,which removes the view-weighting component B and decomposes Θ by Θ = AH. The training ratio is selectedfrom {10%, 20%, 30%}, and comparison results are shownin Fig. 5. From Fig. 5, we can see that both SPLIT1 and
SPLIT2 significantly outperform SPLITλ2 =0 . It shows that it
is important to learn view-wise weights for performance improvement on the NUS-Scene dataset.
Sensitivity analysis on hyperparameters
To understand the behavior of SPLIT, sensitivity analysis on
hyperparameters is conducted on NUS-Object with the training ratio being 30%. SPLIT has three regularization parameters λ1 , λ2 and η, controlling topic-specific sparsity, viewwise importance and regression weight, respectively, and one
parameter K on the latent dimensionality, controlling thestrongness of task correlation. Values of λ1 , λ2 and η areselected from {10a |a| ∈ {0, 2, 4, 6, 8}}, while the value of
K is varied from 1 to 10 by step 1. Three experiments are
conducted to evaluate the pairwise correlation between theparameters. The first experiment on λ1 and K is conducted

Figure 6: Sensitivity analysis of λ1 , λ2 , η and K on the NUS-Object
dataset. Values (shown in the logarithmic scale) of λ1 , λ2 and η areselected from {10a |a| ∈ {0, 2, 4, 6, 8}}, while the value of K isvaried from 1 to 10 by step 1.

by fixing λ2 = η = 1, and similar setting is applied for othertwo experiments. Fig. 6 leads to three conclusions: (1) as
the value of K increases, the performance first increases, andthen becomes stable once K ≥ 5; (2) compared to λ2 andη, the performance is more sensitive to the value change of
λ1 ; (3) the best performance on NUS-Object is achieved by
setting λ1 ≤ 102 , and λ2 , η ≤ 104 .

7

Conclusion

In this paper, we propose a novel method, SPLIT, via multiplicative sparse feature decomposition, so as to address threechallenges in MVMTL: saving task correlations efficiently,
selecting relevant features and assigning view-wise weights.
Our theoretical analysis provides an equivalence guarantee of
SPLIT with a general form of joint regularization, according
to which two formulations with specific settings are proposedand solved by an efficient optimization algorithm in a linear
complexity w.r.t. the data size. Extensive experiments on avariety of datasets show that it is necessary for a successful
MVMTL method to model task correlation, select relevant
features and learn view-wise weights, and demonstrate thatthe proposed SPLIT enables to address the challenges in both
simulated and real-world MVMTL applications.

Acknowledgements
L. S. has been supported in part by JST ACCEL (grant number JPMJAC1503). C. H. N. has been supported in part by
MEXT Kakenhi 18K11434. H. M. has been supported in partby JST ACCEL (grant number JPMJAC1503), MEXT Kakenhi (grant numbers 16H02868 and 19H04169), FiDiPro byTekes (currently Business Finland) and AIPSE program by
Academy of Finland.

References[Blum and Mitchell, 1998] Avrim Blum and Tom Mitchell.
Combining labeled and unlabeled data with co-training. In
Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100. ACM, 1998.
[Chen et al., 2011] Jianhui Chen, Jiayu Zhou, and Jieping
Ye. Integrating low-rank and group-sparse structures forrobust multi-task learning. In Proceedings of the 17th
ACM SIGKDD international conference on Knowledge
discovery and data mining, pages 42–50. ACM, 2011.
[Gong et al., 2012a] Pinghua Gong, Jieping Ye, and Changshui Zhang. Multi-stage multi-task feature learning. In
Advances in neural information processing systems, pages
1988–1996, 2012.
[Gong et al., 2012b] Pinghua Gong, Jieping Ye, and Changshui Zhang. Robust multi-task feature learning. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages
895–903. ACM, 2012.
[Han and Zhang, 2015] Lei Han and Yu Zhang. Learning
multi-level task groups in multi-task learning. In AAAI,
volume 15, pages 2638–2644, 2015.
[Hardoon et al., 2004] David R Hardoon, Sandor Szedmak,
and John Shawe-Taylor. Canonical correlation analysis:
An overview with application to learning methods. Neuralcomputation, 16(12):2639–2664, 2004.
[He and Lawrence, 2011] Jingrui He and Rick Lawrence. A
graph-based framework for multi-task multi-view learning. Proceedings of the 28th International Conference on
Machine Learning, ICML 2011, pages 25–32, 2011.
[Hoerl and Kennard, 1970] Arthur E Hoerl and Robert W
Kennard.
Ridge regression: Biased estimation for
nonorthogonal problems. Technometrics, 12(1):55–67,
1970.
[Jalali et al., 2010] Ali Jalali, Sujay Sanghavi, Chao Ruan,
and Pradeep K Ravikumar. A dirty model for multi-task
learning. In Advances in neural information processing
systems, pages 964–972, 2010.
[Jin et al., 2013] Xin Jin, Fuzhen Zhuang, Shuhui Wang,
Qing He, and Zhongzhi Shi. Shared Structure Learning forMultiple Tasks with Multiple Views. In Machine Learning
and Knowledge Discovery in Databases, pages 353–368,
Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
[Kan et al., 2016] Meina Kan, Shiguang Shan, Haihong
Zhang, Shihong Lao, and Xilin Chen. Multi-view discriminant analysis. IEEE transactions on pattern analysis andmachine intelligence, 38(1):188–194, 2016.
[Li and Huan, 2018] Xiaoli Li and Jun Huan. Interactions
modeling in multi-task multi-view learning with consistent task diversity. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM ’18, pages 853–861. ACM, 2018.
[Liu et al., 2009a] Han Liu, Mark Palatucci, and Jian Zhang.
Blockwise coordinate descent procedures for the multi-

task lasso, with applications to neural semantic basis discovery. In Proceedings of the 26th Annual International
Conference on Machine Learning, pages 649–656. ACM,
2009.
[Liu et al., 2009b] Jun Liu, Shuiwang Ji, and Jieping Ye.
Multi-task feature learning via efficient l2, 1-norm minimization. In Proceedings of the Twenty-Fifth Conference
on Uncertainty in Artificial Intelligence, UAI ’09, pages
339–348. AUAI Press, 2009.
[Lozano and Swirszcz, 2012] Aurelie C Lozano and Grzegorz Swirszcz. Multi-level lasso for sparse multi-task regression. In Proceedings of the 29th International Coference on International Conference on Machine Learning,
pages 595–602. Omnipress, 2012.
[Lu et al., 2017] Chun-Ta Lu, Lifang He, Weixiang Shao,
Bokai Cao, and Philip S. Yu. Multilinear Factorization
Machines for Multi-Task Multi-View Learning. In Proceedings of the Tenth ACM International Conference on
Web Search and Data Mining - WSDM ’17, pages 701–
709, 2017.
[Muslea et al., 2006] Ion Muslea, Steven Minton, andCraig A Knoblock. Active learning with multiple views.
Journal of Artificial Intelligence Research, 27:203–233,
2006.
[Nesterov, 2013] Yurii Nesterov. Introductory lectures on
convex optimization: A basic course, volume 87. Springer
Science & Business Media, 2013.
[Sindhwani et al., 2005] Vikas Sindhwani, Partha Niyogi,
and Mikhail Belkin. A co-regularization approach to semisupervised learning with multiple views. In Proceedings of
ICML workshop on learning with multiple views, volume
2005, pages 74–79. Citeseer, 2005.
[Sun and Jin, 2011] Shiliang Sun and Feng Jin. Robust cotraining. International Journal of Pattern Recognition and
Artificial Intelligence, 25(07):1113–1126, 2011.
[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage
and selection via the lasso. Journal of the Royal StatisticalSociety. Series B (Methodological), pages 267–288, 1996.
[Wang et al., 2016] Xin Wang, Jinbo Bi, Shipeng Yu, Jiangwen Sun, and Minghu Song. Multiplicative multitask feature learning. Journal of Machine Learning Research,
17(80):1–33, 2016.
[Xie and Sun, 2015] Xijiong Xie and Shiliang Sun. Multiview twin support vector machines. Intelligent Data Analysis, 19(4):701–712, 2015.
[Zhang and Huan, 2012] Jintao Zhang and Jun Huan. Inductive multi-task learning with multiple view data. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’12,
page 543, 2012.
[Zhou et al., 2018] D. Zhou, J. Wang, B. Jiang, H. Guo, and
Y. Li. Multi-task multi-view learning based on cooperative multi-objective optimization. IEEE Access, 6:19465–
19477, 2018.