Association Graph Learning for Multi-Task Classification with Category Shifts
Authors: Jiayi Shen
Zehao Xiao
Xiantong Zhen
Cees G. M. Snoek
Marcel Worring
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-task learning
category shifts
association graph

Abstract: In this paper, we focus on multi-task classification, where related classification tasks share the same label space and are learned simultaneously. In particular, we tackle a new setting, which is more realistic than currently addressed in the literature, where categories shift from training to test data. Hence, individual tasks do not contain complete training data for the categories in the test set. To generalize to such test data, it is crucial for individual tasks to leverage knowledge from related tasks. To this end, we propose learning an association graph to transfer knowledge among tasks for missing classes. We construct the association graph with nodes representing tasks, classes and instances, and encode the relationships among the nodes in the edges to guide their mutual knowledge transfer. By message passing on the association graph, our model enhances the categorical information of each instance, making it more discriminative. To avoid spurious correlations between task and class nodes in the graph, we introduce an assignment entropy maximization that encourages each class node to balance its edge weights. This enables all tasks to fully utilize the categorical information from related tasks. An extensive evaluation on three general benchmarks and a medical dataset for skin lesion classification reveals that our method consistently performs better than representative baselines.

Association Graph Learning for Multi-Task Classification with Category Shifts
Jiayi Shen', Zehao Xiao', Xiantong Zhen Cees G. M. Snoek', Marcel Worring'
AIM Lab, University of Amsterdam, Netherlands, shen, 2. z.xia,, C.G.M. C.G.M.Snoer Snoek, m. worringg morring/@uva.ni @uva.nl. 2 Inception Institute of Artificial Intelligence, Abu Dhabi, UAE, zhenxt@gmail com
Abstract
In this paper, we focus on multi-task classification, where related classification tasks share the same label space and are learned simultancously. In particular, we tackle a new setting, which is more realistic than currently addressed the literature, where categories shift from training test data. Hence, individual tasks do not contain complete training data for the categories in the test set. To generalize such test data, it is crucial for individual tasks leverage knowledge from related tasks. To this end, we propose learning an association graph to transfer knowledge among tasks for missing classes. We construct the association graph with nodes representing tasks, classes and instances, and encode the relationships among the nodes in the edges guide their mutual knowledge transfer. By message passing on the association graph, our model enhances the categorical information each instance, making it more discriminative. To avoid spurious correlations between task and class nodes in the graph, we introduce an assignment entropy maximization that encourages each class node to balance its edge weights. This enables all tasks to fully utilize the categorical information from related tasks. An extensive evaluation on three general benchmarks and a medical dataset for skin lesion classification reveals that our method consistently performs better than representative baselines.'
Introduction
Multi-task learning aims to simultaneously solve several related tasks [9, 69] by sharing information and has attracted much attention in recent years. In this paper, we focus on multi-task classification under the multi-input multi-output setting [70. 37, 46, 69]. In this setting, the label space is shared but each task operates on different type of visual modality or the same modality collected from different environments or equipment. As a consequence, each task follows different data distributions for the shared labels. The intuition behind multi-task classification is that tasks having the same label space provide partial knowledge of the distributions that can be shared among all tasks to reach a better view of the full distribution, which in turn benefits the individual tasks.
real- world challenge for multi-task classification is category shift, where the categories phase are shared but during training not all classes are present for each individual task. This challenge common in various realistic scenarios, such as skin lesion classification [31, 64), fault diagnosis [56, 14], remote sensing scene classification [38]. For example, in skin lesion classification, data provided by different hospitals or healthcare facilities should lead to the same set of diagnoses [31, 64]. Unfortunately, due to patient populations proprietary use regulations, these tasks do not share
testing
"Currently with United Imaging Healthcare, Co., Ltd., China. CCod: https: thub. com/ autumn9999//TC-- wit-witthh with- Category- Shiffs..gftt....
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Conventional Multi-Tsk Tasti-TaskCaa Classification
Mult-Task Task Clsssffiattion with Category Shifts
yulr
yule
X
yolr
yyle
Artistic
SAA
RAS.
Clipart
SAX
2.% 4G5
1.A%%
Product
SAX
4.%
Real_world
Figure I: Comparison between multi-task classification without (left) and with category shifts, x denotes the training set. Each row and column of the training data corresponds to one task and one category. Each visual modality (e.g., Artistic) corresponds to one task. y and yite denote the label spaces in training and test phases, respectively. We address the category shifts in multi-task classification, where instances from several categories are unavailable during individual task training.
the same diagnosis categories at training time. For example, some institutions miss instances from melanoma and basal cell carcinoma while others lack dermatofibroma and benign keratosis [64]. In this case, it is beneficial to expand the diagnostic scope of different hospitals or healthcare facilities by simultaneously learning their training data and improving the overall prediction accuracy. Motivated by these realistic scenarios, we propose a new multi-task setting, namely multi-task classification with category shifts. Figure 1 shows comparisons between multi-task mallti-tsk classification with and without category shifts. The goal of the proposed setting is to explore rask-relaesdness with an incomplete training label space improve the generalization ability of the categorical information at test time.
To deal with the category shifts, we propose to learn an association graph to transfer knowledge among tasks. The association graph is constructed over three different types of nodes: task, class and instance nodes. To model the complex relationships among these heterogeneous nodes, we apply different learnable metric functions to construct the edge weights among different types of nodes. To propagate knowledge between the nodes, we apply message passing to update each node in the association graph according to their relationships. Essentially, the association graph stores the task and class-specific knowledge during training time and enhances each instance feature by associating it with other task and class nodes during inference. In the constructed association graph, the relationships between class and task nodes tend to be biased due to the category shifts. This hinders tasks in utilizing the categorical information for missing classes. To avoid these spurious correlations between task and class nodes, we introduce assignment entropy maximization. This regularization of the association graph encourages each class node to balance its edge weights with all tasks, enabling them to fully utilize the categorical information.
We evaluate our model on three multi-task classification benchmarks and a medical dataset for skin lesion analysis to demonstrate that the proposed model performs better under category shifts. We also provide detailed analyses to show how the proposed association graph enhances the categorical information of each instance with transferred knowledge.
2 Problem Statement
We first formally introduce the new problem setting of multi-task classification with category shifts. We consider I' related classification tasks (D.) Each task contains a training set D and a test set Di= We define D = . ). where X," denotes the set of training data from the t-th task and y is the corresponding label space in the training phase and f E {1, 2, ..., T'.. Likewise, we define define the entire label space of all tasks as y The conventional
D " In addition, multi-task classification setting, e.g., [37, 46], is a specific instantiation of our setting where all tasks share the entire label space at both training and test time D =//
Definition 1 (Category Shifts in Multi-Task Classification). For each task, the training label space is a proper subset of the test label space Di C Yif, where i (1,2,.,TT) The union of the training label spaces of all tasks y= >=U?_>"'' yt determines the label space for the test phase.
We provide visual illustration of the difference between multi-task classification without and with category shifts in Figure 1. The goal of the proposed setting is to explore task-relatedness in the presence of missing classes to improve the generalization ability of categorical information. To
2

---

study the impact of different degrees of category shifts, we introduce the missing rate that formally measures the degree of category shifts, with higher missing rates yielding more severe category shifts, and therefore more challenging conditions for each task.
Definition 2 (Missing Rate). Given T related classification lasks with entire label space y, the missing rate Y is the average rate of the number of missing classes with respect to the size of the
entire label space, 7= - 1- [ D4
Having defined the problem setting, we are now ready to present the first multi-task classification method tailored to handle category shifts.
3 Methodology
3.1 Learning the association graph for knowledge transfer
To deal with the category shifts, we propose to learn an association graph to transfer knowledge among tasks for each class. The necessity of designing the graph is due to the category shifts requiring knowledge transfer among tasks and classes, which varies across different classes. We construct an undirected graph over three types of nodes: task, class and instance nodes. The association graph stores the task and class-speciff specific knowledge from the training data in the task and class nodes. The edges encode the relationships between the nodes, enabling the relevant knowledge to be transferred to each instance node. Since the nodes are heterogeneous, we apply different learnable metric functions to compute edge weights for different types of nodes. To better explain the construction, we provide an illustration of the association graph in Figure 2.
The rationale behind our model that relevant knowledge is transferred among nodes in the asso- ciation graph to enhance the categorical information of each instance, making the instance more discriminative. During training. the model learns the ability to update all nodes in the graph by trans- ferring the relevant knowledge. At inference time, the ability is generalized to each test instance of either observed or missing categories. Thus, each test instance is refined with the relevant knowledge stored in the association graph, which reduces the category shifts from the training to the test set. For clarity, we introduce the main components of the association graph: task and class graphs.
Task graph. Given I' related tasks, we introduce the task graph model the relationships between tasks. The features of the nodes in the task graph are task-specift representations, each of which aggregates all features from the corresponding task. We define the node of the t-th task follows:
N
V2 = Ntr Σ E(xi),
(1)
where x; is training instance belonging the f-th task and N the number of training instances from the corresponding task. È is feature extractor shared by all tasks, which embeds each instance into a d-dimensional feature space. E is feature extractor shared by all tasks, which embeds each instance into a d-dimensional feature space.
With the task nodes, we further define the edges between the task nodes v; and V as Ar(V;, Vy)- The weight of the edge is determined by the learnable similarity between the task nodes, which is formulated follows:
Av v2) = a (W vil/a-vyl//)))//// + by),
where WT and by are the learnable parameters for the task graph. ar is a scalar and a is the sigmoid function used to normalize the edge weight between 0 and 1. The edge weight indicates the proximity between the i-th and j-th tasks. We denote the task graph as GT= AT). In the task graph, Vv=(v E [1,T]} E Rxxd is the of all task nodes and A7=(A7 v )ii [1,2]] E RTx, is the corresponding adjacency matrix, which characterizes task relationships.
Class graph. Likewise, define the class graph Ge=(Ve,Ae)) where each node represents the corresponding categorical information. Here = e [1, CJ] Bxdd where C' denotes
3

---

Association graph
Adjacency matrix
V1
AT
AC-T
K1
V2
Va
Ax-T
Ac
Task graph
Ax-c
kg
AC
A
Class graph
Instance node
Ax-T
Ax-c
The illustrative association graph (left) and the adjacency matrix (right). The association graph contains the task graph, the class graph and instance node. Nodes in the task and class graphs store the task and class-specific knowledge during training. Edges in the association graph encode the various relationships among nodes, facilitating knowledge transfer between them. Single and double lines denote the edges among the homogeneous nodes and the MULTIPLE edges among heterogeneous nodes, respectively.
Figure 2:
size entire label space of tasks. We define the features of the node of the e-th class as:
N
1
kg
E(xm),
Ntr
where Xm training instance from e-th class and Norr the number of training instances of the corresponding class. The edge weight between class nodes (k,,,,, k,) is formulated as:
(4) where We and be are the learnable parameters for the class graph. ac is fixed scalar. The motivation for designing the metric function with learnable parameters is to explore the task-specific class-specifie semantic information in the corresponding graph. The adjacency matrix of the class graph is (A=(Ac j E [1) E RCC
Ac,k, kj) o oWe(k,; kyl/ac + bc)),
Association graph. Having defined the task and class graphs, we build the association graph by connecting both graphs with each instance node, as shown in Figure 2. We use association graph to enhance instance feature learning by leveraging task representations (nodes in the task graph) and categorical information (nodes in the class graph). To do so, we query each instance in the task and class graphs. Formally, we define instance node as V (x=(E(x)) E Bg
We build up the connection of the instance node to each node of the task graph and the class graph, which gives the instance node access to the knowledge stored in the task and class graph. The edge between the instance node {xx) and task node V, is denoted by Ax~7(x, vı). The weight of the edge is obtained by the normalized similarity between the instance and task node,
Ax-7=softmax
[(x) Vv).. The corresponding adjacency matrix Ax- vz)l
[1, RIxT Likewise, we define the adjacency matrix between instance nodes and the class graph as Ax-c=(Axxc kx) [1, C}} R lxc
In the association graph, further build the edges cross the task and class graphs to enable knowledge transfer between them. Formally, we define the edge connecting class node and a task node as:
exp( - llkk - v 12/22
Ac (ke, ((lkevv)) Vo) Er=1 exp(-Il(t - a////p/l//
where op is a fixed scaling factor. This metric function without learnable parameters aims to alleviate the overfitting of the learned connections between the task and its observed classes. The adjacency matrix between the task and class graphs is denoted by Ac-T=(AcwT vr)le E [1, C] e [1,2"] CCT, Thus, the whole association graph G=(VAA) with three types of nodes can be formulated as:
AT
AT
A
CNT
XNT
== V
AC-T
Ac
A
(6)
Ax~T
Ax~c

---

Knowledge transfer by message passing. With the constructed association graph, we perform message passing the association graph via multi-layer Graph Neural Network (GNN). general. our model can work with various GNN architectures. In this work, we apply GraphSAGE [23]. To simplify, we use h, to represent one node in the association graph g, which could be task, class or instance node. The 1-th layer of GNNs can be written as:
U'Concat (Mean (ReTUU (. h, E Maahh))) nn nn-"))
(7)
where h, denotes the node embedding by the I-th GNN layer and U² and W are learnable weight matrices of the 1-th GNN layer. 1 E {1,2,.... ..., L with L denoting the number of GNN layers. h!") is initialized the nodes defined above. NV& (h,) denotes the top-k neighbors of the node h, according to the adjacency matrix between all nodes A. By message passing, each instance is refined with the categorical information stored in the association graph. As a result, the instance will gain more discriminative and informative representations. The enhanced instance feature is formulated as fq). where
(x)) We compute the prediction for the enhanced feature as f, denotes the corresponding task-specific classifier.
3.2 Assignment entropy maximization
Category shifts in multi-task classification yield spurious correlations between tasks and their corre- sponding observed classes. This means the edges between each task and its observed classes have considerably higher weights than other edges. The knowledge transfer between tasks and classes will be dominated by these spurious correlations, hindering their missing classes from exploiting the categorical information.
To tackle this problem, we propose assignment entropy maximization to encourage each class node to balance the weights of its edges with all tasks. Formally, the assignment entropy for the c-th class formulated as:
H(ke) Σ ACTT (ke, Vz) log (kwr(ke, Ve).
(8)
By maximizing the assignment entropy, weights are balanced for each class, which enables all tasks to fully utilize the categorical information for their missing classes. Intuitively, each class node is task-agnostic when the assignment entropy reaches its maximum values. In this ideal case, the model eliminates the spurious correlations between the class and its corresponding tasks in the graph.
By combining the assignment entropy maximization and cross-entropy minimization of the classifiers, we have the final objective as follows:
1 C (D.) + 8 CCAE(k
==
E (D, T 1=1
1
c=1
where log and LAR(K,)=- H(k)). is introduced balance the importance of the cross-entropy and assignment entropy losses. We provide the training and inference algorithms in the supplemental materials.
Related Works
Multi-task learning. Multi-task learning [9] aims learn several related tasks simultaneously and improve their overall performance. The task relatedness is learned by many different aspects of the model, e.g., the loss functions [34, 29], gradient space [45, 65], parameter space [37, 4], or representation space [2, 40, 22]. The basic idea of sharing information from multiple tasks has been successfully applied under different settings, including the single-input multi-output setting [45, 65, 48], the multi-input multi- setting [2, 37, 46), and using meta- [16, 54, 1]. In the single-input multi-output setting [24, 15], tasks are defined by different supervision information included in the same input. In meta-learning, tasks are sampled from one task distribution and different tasks have different category spaces [54, 72]. In multi-input multi-output, tasks follow different data distributions since they are collected from different visual modalities or equipment [37, 46, 69, 70]. remains unexplored to investigate category shifts in multi-task classification, despite being common in various realistic scenarios.
5

---

Category shifts. Category shifts denotes that training data collected from different domains may not completely share their categories, which is first proposed by [60] for domain adaptation. [36] and [42] challenge domain generalization with category shifts, where the change of domains is always followed by the change of categories. Category shift is a common scenario in real-world applications since it relaxes the requirement on the shared category set among source domains [60, 10]. As result, category shift has drawn increasing attention and has been applied to a wide range of learning tasks, including fault diagnosis [14], skin lesion classification [31] and remote sensing image classification [38]. In this paper, we develop a new multi-task learning scenario in which individual tasks do not contain complete training data for the categories in the test set. Unlike domain adaptation and generalization, which only focus on the unidirectional knowledge transfer from source domains to target domain, our multi-task classification encourages simultaneous bidirectional knowledge transfer between any paired domain to enable efficient predictions for both tasks. To the best of our knowledge, we are the first to address category shifts in multi-task classification.
Exploring graph structure. Several recent works prove that graphs are effective in modeling label correlation [11, 32, 30, 33]. task relationships [63, 8, 25], meta-paths [27] and representation learning [61]. For multi-label classification, [32] formulates the multi-label predictions as a conditional graphical lasso inference problem, while [11] utilizes a graph convolutional network to propagate information between multiple labels and consequently learn inter-dependent classifiers for each of the image labels. For multi-task learning, some works learn the relationship between multiple tasks by message passing over a graph neural network [35, 21]. [41, 6] explore the graph structure to produce higher quality node embeddings on the graph-structure and [63] design the hand-crafted and automatically constructed knowlenn graph to provide meta knowledge for each task. Different from these methods, we address the new challenge, category shifts in multi-task classification, which yields novel graph construction. Particularly, our work shares the high-level goal with [33] in terms of joint inference over multiple heterogeneous graphs with within- -ighin relations and pross-graph interactions. With respect to the form of information exchange, the main differences are: [33] uses the spectral graph product and label propagation operators exchange information between heterogeneous nodes, while our work performs message passing (e.g., GNNs). Moreover, [33] infers the unobserved multi-relations with observed multi- relations across the graphs. In contrast, our work refines each instance node with the categorical information from heterogeneous nodes. [27, 61] are related to our work since both of them construct new graph structures enable knowledge transfer across graphs. [27] multiplies two adjacency matrices (edge types) of the heterogeneous graph to automatically learn new meta-paths. In contrast, our method constructs the connections between heterogeneous nodes by directly computing the similarities between any pair-wise nodes with different metric functions. [61] fixes the edges between heterogeneous nodes to one or zero. By contrast, our method constructs the connections between each heterogeneous node through the similarity between these nodes.
data. For few-shot learning, [62]
Heterogeneous GNNs and relational graph models. Heterogeneous GNNs models [67, 57, 27, 17, 66, 39] have similar spirit to our work in dealing with heterogeneous graphs, however the problem settings and technical implementations are fundamentally different. HetGNN-based methods focus on graph data (c.g., academic graph and review graph data) and need pre-processing modules for each node type to encode heterogeneous contents as a fixed-size embedding. Most HetGNN-basce methods [67, 57, 27, 17] depend on a hierarchical architecture to aggregate content embeddings of heterogeneous neighbors for each node, which contains node-level (intra-metapath) and semantic- aggregations. Our work constructs an association graph from the non-gram non-graph data and enables pair-wise nodes to transfer knowledge whether they are from the same type or not. Thus, our method is suitable for solving the complex knowledge transfer in multi-task classification with category shifts. Moreover, relational graph models [44, 51, 71, 7] are also related to our work since both aim fully utilize the structure information in the graph. However, the main difference between them is that most relational graph models [44, 51, 71, 7] are designed for homogeneous graphs, while the proposed graph model handles heterogeneous nodes.
level
generalization. These methods are relevant in the sense that both deal with several domains that share the same label space. However, out-of-distribution generalization meth- ods 158, 47, 73, 59, 55, 68, 74, 3, 12, 13) focus on the single-directionll knowledge transfer from the source domain(s) to the target dsmain(s), while our setting aims to learn a bi-directional knowledge transfer between wisr domains (tasks). Moreover, due to the various missing classes per task,

---

the bi-directional knowledge transfer among domains (tasks) varies across different pairs of classes. Thus, the knowledge transfer in our setting more complex than --d-trr-iittttttonn generalization.
Experiments and Results
Datasets. We conduct experiments on three common multi-task classification benchmarks and skin lesion classification dataset to evaluate the effectiveness of our proposed method.
Office- Home [53] contains images from four domains/tasks: Artistic. Clipart, Product and Real- world. Each task contains images from 65 object categories collected under office and home settings. There about 15, 500 images in total.
Office-Cal [18] contains the ten categories shared between Office-31 [43] and Caltech-256 [19]. One task uses data from Caltech- and the other three tasks use data from Office-31, whose images were collected from three distinct domains/tasks, namely Amazon, Webeam and DSLR. There are 8 ~ 151 samples per category per task, and 2, 533 images in total.
ImageCLEF [37]. the benchmark for the ImageCLEF domain adaptation challenge, contains 12 common categories shared by four public datasets/tasks: Caltech-256, ImageNet ILSVRC 2012, Pascal VOC 2012, and Bing. There are 2, 400 images in total.
Skin-Lesion contains three skin lesion classification tasks: HAM10000 [49]. Dermofit [5] and Derm7pt [28]. Tasks are collected from different hospitals healthcare facilities. In this dataset, each task contains a subset of the following classes: melanocytic nevus, melanoma, basal cell carcinoma, dermatofibroma, benign keratosis and vascular lesion.
Experimental setup. We explore the effect of varying degrees of category shifts in these datasets by different missing rates in the training label spaces. For the three common multi-classfficttion benchmarks, we set the missing rates for the three benchmarks as 75%, 50%, 25% 0%, denoting that each task cannot access the training data of 75% (or 50%, 25%, 0%) of the categories. For simplicity, we use the same missing rate for all tasks in each setting, which is achieved by assigning the same number of missing classes for each task. Since the union of the training label space of all tasks equals the test label space in the proposed setting, 75% is the largest missing rate for the common multi-task classification datasets that have data from four tasks. With the 75% missing rate, data from each category is only accessible in one task. By contrast, when the missing rate is set to 0%, the problem degrades to the conventional multi-task classification without category shifts, i.c., each task has complete training data from all classes. Since Skin-Lesion has three tasks, we set the missing rates as 67%, 33%, 0%. For a fair comparison, the assignment of missing classes for different missing rates and datasets is shared for all methods. We use ResNet-18 18 [26] the backbone for all experiments and deploy the graph in the output space the backbone. We provide the code and the missing class assignment of each dataset in the supplemental materials.
Metrics. The average multi-task classification accuracy (%, top-1) along with 95% confidence intervals from five runs are reported across all tasks. In order to evaluate the model's general- ization power from observed classes to missing classes of each task, test instances come from observed classes and missing classes. We re- port the average accuracy of missing classes and observed classes of all tasks as Am and Ag, re-
60
Harmonic Mean (%)
58
56
54-
spectively. Moreover, we apply the harmonic mean to show the overall performance on both missing and observed classes, which is denoted 2xAmXA. by == Am A.
52
Atlention graph
Assoc ation graph
Layers
Figure 3: Comparisons between the attention graph and proposed association graph. Our association graph consistently performs better than the atten- tion graph with different message passing layers.
Benefit of the association graph. To show the
benefit of the proposed association graph, we conduct experiments with increasing numbers of the message passing layers, where L=0 denotes
7

---

w/o Graph
Rasing cines
w/ Graph
Artistic
Clipart
Product
Real_wordd world
Figure 4: Benefit of the association graph in feature learning. Visualization of features without (upper row) and with the graph (lower row) are shown, where each column corresponds to a task from Office-Home Different shapes denote different classes, while observed and missing classes are in blue and red, respectively. The proposed graph distinguishes each missing class from the observed classes, which tend to collapse together due to the category shifts.
Table 1:
Benefit of the assignment entropy maximization our model on Office -ome under the setting of missing 75% classes for each task. The assignment entropy maximization improves the overall performance on both missing and observed classes.
Method
Am
A.
Average Assignment Entropy
w/o CAB
44.65
0.2
87.59
0.3
58.26 = 0,31
0.13 0.05
w/ LAF
47,51
= 0.39
87.16
0.34
60.59 =: 0.24
0.84 0.07
the model without the graph and thus without knowledge transfer among graph nodes, shown in Figure 3. As the number L increases, our model performs increasingly better with the peak L-4, surpassing the model without graphs by large margin. By stacking multiple layers of GNNs, each node can eventually incorporate the knowledge from the large-hop neighbors across the entire graph to reduce the generalization gap between the training and test sets. The results demonstrate that the task and class-specific knowledge stored in the graph is important enhance the categorical information in the features, which enables them to be more discriminative. In the following experiments, we set L=4 for our model.
To better understand the benefit of the association graph in feature learning, we visualize the distribu- tions of samples from observed and missing classes of each task Office-Home by t-SNE [50]. Figure shows that the association graph reduces the overlap of the distributions of missing classes (shapes in red) and updates the features of each class be more clustered. We therefore conclude that the association graph enhances the categorical information of the instance features from both observed classes and missing classes, making them more distinguishable.
Association graph vs. attention graph. We compare the association graph with the attention graph [52], which incorporates the same self-attention strategy for all nodes. Different from the attention graph, the proposed association graph adopts different learnable metric functions for different types of nodes. In Figure 3, the association graph performs consistently better than the attention graph with different numbers of message passing layers. The association graph with different learnable metric functions suitable for modeling complex relationships among different types of nodes, leading to improvements over the attention graph.
Benefit assignment entropy maximization. We investigate the benefit of the proposed assign- ment entropy maximization in our model. As shown in Table 1, the assignment entropy maximization regularization considerably improves the overall performance. This reasonable since the assignment entropy maximization balances the edge weights between each class and all task nodes. Thus, the spurious correlation between task and class nodes reduced, which enables each task to fully utilize the categorical information provided by other tasks.
8

---

Importance of knowledge transfer by message passing. To investigate the importance of knowl- edge transfer, we conduct experiments on Office-Hoo with different neighbor sizes for each node during message passing, which reflects the amount of the transferred knowledge.
With the size of 0, each node does not utilize the transferred knowledge. With the maximum size (which is 70 this dataset), each node aggregates the knowledge from all other nodes on the graph. As shown in Table 2, we find that the best performance happens with the largest neighbor size, which indicates the importance of passing messages throughout the whole graph for knowledge transfer.
Table 2: Performance with different neighbor sizes on Office -ome. The largest size performs best.
We
Am
Ag
H
0
36,84
0.11
83.09
0.93
49.82
0.76
8
45.48
6.21
84.82
1 0.67
58.45
0.34
16
46.14
0.34
84.07
= 0.55
58.83
:: 0.28
32
46.10
0.5
84.89
58.80
64
47.51
85,95
60.20
70 (max.)
47.51
0.39
87.16
0.34
60.59
Table 3: Comparative results with different missing rates Office-Home, Office-Caltech and ImageCLEF. Our method a consistent top performer on missing and observed classes.
Missing Rate (1)
Office Office-Home
Office-Caltens Cach
ImageCLEF A,
Method
A.
Ao
II
A.
A,
II
Am
II
STL
0.00
88.25
0.00
0.00
98.53
0.00
95.00
0.00
ERM [20]
36.45
83.53
49.32
47.43
97.28
62.98
71.94
80.00
75.55
PCGrud [65]
75%
36.99
83.30
49.56
49.84
96.43
64.93
71.94
83.33
76.92
WeighLosses [29]
37.39
82,92
50.26
49.39
96.43
64.46
72.22
80.83
76.08
Ours
47.51
87.16
60.59
55.47
98.12
70.55
75.28
85.00
79.45
STL
0.00
84.37
0.00
0.00
98.61
0.00
0.00
88.33
0.00
ERM [20]
50,96
81.89
62.14
77.33
97.43
85.09
76.67
84.58
80.36
PCGrad [65]
50%
50.95
82.52
62.39
80.29
97.43
87.28
74.58
82.92
78.46
WeighLosses [29]
51.65
82.38
62.84
76.54
97.27
84.60
75.42
85.42
79.94
Ours
54.65
83.57
65.54
88.65
98.15
92.83
78.33
87.08
82,20
STL
0.00
82.06
0.00
0.00
98.07
0.00
0.00
83.06
0.00
ERM [20]
54.09
81.34
64.51
94.27
97.42
95.76
74.17
85.00
78.90
PCGrad [65]
25%
52.43
80.81
63.18
92.96
97.92
95.20
76.67
82.22
79.12
WeighLosses [29]
53.60
81.38
64.03
93.84
97.81
95.69
76.67
83.89
79.88
Ours
56.74
82.94
67.12
97.35
98.51
97.92
80.00
85.28
82.48
STL
79.29
98.13
81.67
ERM [20]
80.99
98.22
84.79
PCGrad [65]
81.41
98.02
82.71
WeighLosses [29)
81.78
98.24
82.75
Ours
82.01
98.26
86.04
Effect of different degrees of category shifts. We evaluate the proposed method on the three benchmarks with different missing rates in Table 3. ERM [20] is simple baseline that mixes all training data together and trains the shared model. Our method achieves the best overall performance on all three benchmarks under each missing rate in terms of the harmonic mean of the accuracy of observed and missing classes. We note that 75% the most severe category shifts in multi-task classification, which demonstrates there are overlap categories between tasks. On Office- Home with the 75% missing rate, our model surpasses the second best method, i.e., WeighLosses [29], by large margin of 10.33%, in terms of the harmonic mean. The consistent improvements on all benchmarks with different missing rates demonstrate that the association graph effective in addressing the category shifts for multi-task classification. More detailed results with 95% confidence intervals are provided in the supplemental materials. Moreover, we also provide the results on the realistic medical dataset, i.e., Skin-Lesis on with missing rates of 67%, 33% and 0% in Table 4. Our model achieves the best overall performance, which again confirms the effectiveness of our method.
It is worth mentioning that with 7=0% the setting reduces to traditional multi-task learning without category shifts, where classes are observed by each task during training. this case, the results of missing classes A,, and the harmonic mean H are not available. Our method still outperforms other baselines in all datasets. We conclude that our association graph better utilizes the shared knowledge to improve overall performance for all tasks, which also holds for settings without category shifts.

---

Table 4: Comparative results with different missing rates the medical dataset Skin-Lesion Our method achieves the best overall performance on both missing and observed classes.
Method
67%
33%
7
0%
An
A.
H
Am
A.
H
Am
A.
H
STL
0.00
97.99
0.00
0.00
87.32
0.00
84.33
ERM [20]
8.74
93.95
15.16
15.52
84.24
25.96
83.48
PCGrad [65]
8.04
91.62
14.51
14.28
82.77
23.53
84.11
Lossel
Josses [29]
7.73
89.68
13.07
14.25
85.56
24.35
84.20
Ours
10.82
90.29
18.17
16.58
86.62
27.21
85.98
Conclusion
We address category shifts in multi-task classification, which is challenging yet realistic since individual tasks often do not contain complete training data for the categories in the test set. To tackle this, we propose to learn an association graph transfer knowledge among tasks for missing classes, which enhances the categorical information of each instance, making them more discriminative. To avoid the spurious correlations between task and class nodes, we introduce assignment entropy maximization, enabling all tasks to fully utilize the categorical information for missing classes. To the best of knowledge, are the first to address the challenge multi-task classification. We conduct ablation studies to demonstrate the effectiveness of the proposed association graph and assignment entropy maximization in our model. The superior performance on three multi-task classification benchmarks and the medical dataset for skin lesion classification further substantiates the benefits our proposal.
Acknowledgment
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
References
[1] Ahdollahzadeh, T. Malekzadeh, and N.-M. M. Cheung. Revisit multimodal meta-lcarning through lens of multi-task learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[2] A. Argyriou, Evgeniou, and M. Pontil. Multi-task feature learning. In Advances in Neural Information Processing Systems, volume 19, 2006.
[3] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.0289, 2019.
[4] B. Bakker and T. Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4:83 99, 2003.
[5] L. Ballerini, R. B. Fisher, B. Aldridge, and J. Rees. color and texture based hierarchical k-nn approach to the classification of non-melanoma skin lesions. Color medical image analysis, pages 63-86. Springer, 2013.
[6] D. Buffelli and F. Vandin. Graph representation learning for multi-task settings: a meta-leaming approach. arXiv preprint arXiv:2201.03322 2022.
[7] D. Busbridge, D. Sherburn, Cavallo, and N. Hammerla. Relational graph attention networks. arXiv preprint arxiv: ArXiv:1904 1eee::9444001111 2019.
[8] K. Cao, J. You, and J. Leskavee. Relational multi-task learning: Modeling relations between data and tasks. In International Conference Learning Representations, 2021.
[9] R. Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.
[10] Z. Chen, P. Wei, J. Zhuang, G. Li. and L. Lin. Deep cocktail networks. International Journal of Computer Vision, 129(8)22328 222222222223211112 2021.
10

---

[11] Z.-M. Chen, X.-S. Wei, P. Wang. and Y. Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5177-5186, 2019.
[12] Y. Du, J. Xu, H. Xiong, Q. Qiu, X. Zhen, C. G. M. Snoek, and L.. Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pages 200-216. Springer. 2020.
[13] Y. Du, X. Zhen, L. Shao, and C. G. M. Snock. Hierarchical variational memory for few-shot learning across domains. arXiv preprint 2rXiv:2/12.08/1 2021.
[14] Y. Feng, J. Chen, S. He. Pan, and Z. Zhou. Globally localized multisource domain adaptation for cross domain fault diagnosis with category shift. IEEE Transactions on Neural Newworks and Learning Systems, 2021.
[15] C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[16] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[17] X. Fu, J. Zhang, Z. Meng, and King. Magnn: Metapath aggregated graph neural network for heteroge- ncous graph embedding. In Proceedings of The Web Conference 2020, pages 233 2331-2341, 2020.
[18] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kereel for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2066-2073, 2012.
[19] G. Griffin, A. Holub, and P. Perona. Caltech- object category dataset. 2007.
[20] Guirajani and D. Lopez-Pazz Paz. search of lost domain generalization. arXiv preprint arXiv:2007001334, 2020.
[21] P. Guo, C. Deng. L. Xu, X. Huang, and Y. Zhang. Deep multi-task augmented feature learning via hierarchical graph neural network. In Joint European Conference Machine Learning and Knowledge Discovery Databases, pages 538-553. Springer, 2021.
[22] P. Guo, C.-Y. Lee, and D. Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pages 3854-3884 3863. PMLR, 2020.
[23] W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30, 2017.
[24] H. Hazimeh, 7. 7hao, A. Chowdhery, M. Sathiamoorthy, Y. Chen, R. Mazumder, I.. Hong, and E. Chi. Dsclect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
[25] J. He and R. Lawrence. A graphbased framework for multi-task multi-view learning. In International Conference on Machine Learning, 2011.
[26] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[27] Z. Hu, Y. Dong, K. Wang, and Y. Sun. Heterogeneous graph transformer. In Proceedings of The Web Conference, pages 2704-2710, 2020.
[28] Kawahara, Daneshvar, G. Argenziano, and G. Hamarnh. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538-546, 2018
[29] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7492-7491, 2018.
[30] C. W. Lee, W. Fang, C.-K. Yeh, and Y.-C F. Wang. Multi-label zero-shot learning with structured knowl- edge graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1576 1585, 2018

---

[31] H. Li, Y. Wang, R. Wan, S. Wang. T.-Q. Li, and A. classification with linear-dependency regularization. In Advances in Neural Information Processing Systems, volume 33, pages 31-8-1129, 2020.
Kot. Domain generalization for medical imaging
[32] Q. Li, M. Qiao, W. Bian, D. Tao. Conditional graphical lasso for multi-label image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2977-2986, 2016.
[33] H. Liu and Y. Yang. Cross-graph learning of multi-relational associations. In International Conference on Machine Learning, pages 2235-2243, PMLR, 2016.
[34] L. Liu, Y. Li, Z. Kuang, J. Xue. Y. Chen, W. Yang. Q. Liao, and W. Zhang. Towards impartial multi-task learning. In International Conference Learning Representations, 2020.
[35] P. Liu, J. Fu, Y. Dong, X. Qiu, and J. C. K. Cheung. Learning multi-task communication with message passing for sequence learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2019.
[36] Y. Liu, 7. Xiong, Li, Y. Lu, X. Tian, and 7..J. 7ha. Category-stitch learning for union domain generalization. ACM Transactions on Multimedia Computing, Communications, and Applications, 2022.
[37] M. Long, Z. Cao, J. Wang, and P. S. Yu. Learning multiple tasks with multilinear relationship networks. In Advances in Neural Information Processing Systems, volume 30, 2017.
[38] X. Lu. T. Gong, and X. Zheng. Multisource compensation network for remote sensing cross-domaiee scene classification, IEEE Transactions on Geoscience and Remote Sensing, 58(4):2504 2515, 2019.
[39] Q. Lv, M. Ding, Q Liu, Y. Chen, W. Feng, S. He, C. Zhou, J. Jiang, Y. Dong. and J. Tang. Are really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1150-1 160, 2021.
[40] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3994-4003, 2016.
[41] R. Nassif, S. Vlaski, C. Richard, J. Chen, and A. H. Sayed. Multitask learning over graphs: An approach for distributed, streaming machine learning. IEFE Signal Processing Magazine, 37:1--4-25, 2020.
[42] M. M. Rahman, C. Fookes, and S. Sridharan. Deep domain generalization with feature-norm network. arXiv preprint arXiv:2104.15:: 135881 2021.
[43] Saenko, Kulis, Fritz, and T. Darrell. Adapting visual category models new domains. In European conference on computer vision, pages 213-226. Springer. 2010.
[44] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. Berg, I. Titov, and M. Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.
[45] O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, volume 31, 2018
[46] J. Shen, X. Zhen, M. Worring, and L. Shao. Variational multi-task learning with gumbel-softune priors. In Advances in Neural Information Processing Systems, volume 34, 2021.
[47] Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui. Towards out-of-distribution generalization: survey. arXiv preprint arXiv:210s 132.2108.......... 2021.
[48] G. Strezoski, N. Noord, and M. Worring, Many task learning with task routing. Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 1375-1384, 2019.
[49] P. Tschandl, C. Rosendahl, and H. Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):199, 2018.
[50] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008
[51] S. Vashishth, S. Sanyal, Nitin, and P. Talukdar. Composition based multi-relational graph convolutional networks. arXiv preprint 191.009111. 2019.
12

---

[52] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks arXiv preprint arXiv:1710.10900 2017.
[53] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[54] H. Wang. H. Zhao, and B. Li. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. International Conference on Machine Learning, pages (1991-11002. PMLR, 2021.
[55] Wang, C. Lan. C. Liu, Y. Ouyang. T. Qin, W. Lu, Y. Chen, W. Zeng. and P. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
[56] Q. Wang. G. Michau, and O. Fink. Missing-clase class-robust domain adaptation by unilateral alignment. JEEE Transactions on Industrial Electronics, 68(1):663 671, 2020.
[57] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu. Heterogeneous graph attention network. In The world wide web conference, pages 2022-2032 2019.
[58] Z. Xiao, J. Shen, X. Zhen, L. Shao, and C. G. M. Snoek. A bit more bayesian: Nomain-invarianl learning with uncertainty. In International Conference on Machine Learning, pages 11351-11361. PMLR, 2021.
[59] Z. Xiao, X. Zhen, L. Shao, and C. G. M. Snoek. Learning to generalize across domains on single test 2022.
samples. arXiv preprint 2202:
[60] R. Xu, Z. Chen, W. Zuo, J. Yan, and L. Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3964-3973, 2018.
[61] L. Yang and S. Hong. Omni-gramular ego-semantic propagation for self-supervised graph representation learning. arXiv preprint 220.::2205.17746 2022.
[62] H. Yao, Y. Wei, J. Huang, and Z. Li. Hierarchically structured meta-learning. In International Conference Machine Learning, pages 7045-7054, PMLR, 2019.
[63] H. Yao, X. Wu, Z. Tao, Y. Li, B. Ding, R. Li, and Z. Li. Automated relational meta-learning. arXiv preprint arXiv:2001.000 00744,, 2020.
[64] C. Yoon, G. Hamarneh, and R. Garbi. Generalizahle feature learning in the presence of data bias and domain class imbalance with application skin lesion classification. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 365-373. Springer, 2019.
[65] T. Yu, Kumar, A. Gupta, S. Levine, Hausman, and C. Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824-5836, 2020.
[66] S. Yun, M. Jeong, R. Kim, Kang, and H. J. Kim. Graph transformer networks. In Advances Neural Information Processing Systems, volume 32, 2019.
[67] C. Zhang, D. Song. C. Huang, A. Swami, and V. Chawla. Heterogeneous graph neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 793-803, 2019.
[68] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17 09410, 2017.
[69] Y. Zhang and Q. Yang. Learning sparse task relations in multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[70] Y. Zhang, Y. Zhang, and W. Wang. Deep multi-task learning via generalized tensor trace norm. arXiv preprint 2XX::2002.04799 2020.
[71] Z. Zhang, F. Zhuang, H. Zhu, Z. Shi. H. Xiong, and Q. He. Relational graph neural network with hicrarchical attention for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[72] X. Zhen, Y. Du, H. Xiong, Q Qiu, C. G. M. Snoek, and L. Shao. Learning to learn variational semantic memory. In Advances in Neural Information Processing Systems, volume 33, pages 9122-9134, 2020.
13

---

[73] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey, arXiv preprint urXiv:2133.02503 2021.
[74] K. Zhou, Y. Yang. Y. Qiao, and T. Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.080 02008,008,00.0002000 2021.
14


Reviews:
[{"review_id": "cqb-zJpQOd1", "reviewer": "Reviewer_QjDy", "summary": "This paper aims to tackle the problem about the multi-task classification with category shifts. It proposes learning an association graph to transfer knowledge among tasks for missing classes, and constructs the association graph with nodes representing tasks, classes and instances. The proposed method encodes the relationships among the nodes in the edges to guide the knowledge transfer between them.  Moreover, to avoid spurious correlations between task and class nodes in the graph, the authors introduce an assignment entropy maximization that encourages each class node to balance its edge weights, which enables all tasks to fully utilize the categorical information from related tasks. An extensive evaluation on three general benchmarks and a medical dataset for skin lesion classification reveals that the proposed method consistently performs better than representative baselines.", "questions": "* The author should clarify the relationships between the proposed graph method and the heterogeneous GNNs.\n* The author should clarify the relationships between the proposed setting and out-of-distribution (OOD) generalization problem, and add more OOD methods as baselines for fair comparisons. \n* The author should add more related works involving above two areas.", "limitations": "The computational cost of the proposed method seems a lot, the authors do not provide necessary analysis and comparisons about it. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* This paper develops a new multi-task learning scenario, in which individual tasks do not contain complete training data for the categories in the test set.\n* This paper utilize an association graph to propagate  and transfer the knowledge between the  constructed nodes, which is reasonable.\n\n**Weaknesses**\n* The construction and processing of the proposed association graph are similar to another research area-heterogeneous GNNs, and the author does not mention much about them.\n* The proposed setting is a bit like a combination of multi-task learning and out-of-distribution generalization, which is of limited novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V9R_vAqP0d4", "reviewer": "Reviewer_XBgC", "summary": "The paper addresses the problem of multi-task classification with category shifts where training data for different tasks would not have same categories. To mitigate category shift problem, the authors construct association graph where different tasks, classes and instances are nodes and use message passing to transfer knowledge between them. Assignment entropy maximization is also designed to balance weights of edges. The experiments conducted on three general benchmarks and a realistic medical dataset for skin lesion classification demonstrate the effectiveness of the method.", "questions": "1. How does the graph structure help to transfer knowledge? I wonder what would happen if you directly use the class knowledge as extra information to help with prediction. \n2. Considering results from Table 2, the best performance is achieved when the node can aggregate knowledge from all other nodes. Does that mean that the knowledge of each node can directly be incorporated into the input instance and local structure information of association graph is not important?", "limitations": "More experiments and explanations need to be included to show how the association graph structure information can help with the category drifts problem.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Very well written paper, addressing category shifts problem in multi-task classification, which is realistic and interesting. The proposed method is also a reasonable way to mitigate this problem by constructing association graph and transferring knowledge, because the missing class for one task may appear in other tasks.\n2. The paper is well developed and easy to follow. The proposed method is clearly presented.\n3. Extensive experiments are conducted on both three general benchmarks and real-world datasets. Results show that the proposed method outperforms other baselines. Ablation tests are also conducted to demonstrate the effectiveness of association graph and assignment entropy maximization modules in the method.\n\nWeaknesses:\n1. More explanations need to be included to show the necessity of using graph to transfer knowledge. They can just combine all class knowledge directly with input instance instead of constructing a complex network. The benefit of association graph structure needs to be stated.\n2. More experiments can also be added to deal with the problem stated above and show the benefit of association graph structure. For example, what happens if the knowledge of class is directly combined with input instance or the association graph is complete.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2rmzvbT-7To", "reviewer": "Reviewer_n1qJ", "summary": "This article discusses the problems of category shifts in multi-task learning. To solve the problem, the author prposes to build a heterogeneous graph called association graph, where nodes represent tasks and classes and edges indicate their relationships. There are three kinds of edges: task-task, class-class, and task-class. The authors use the average of the corresponding sample representations to denote the tasks and classes. In addition, they propose three learnable metric functions to calculate the weights of the three kinds of relationships mentioned above. Intuitively, the edge weight of the relationship between the task and the class that does not appear in its training set should be low. Because the association graph is used for message passing, the authors hopes that the task can benefit from the information of unseen class nodes. To tackle the category shifts problems, they propose an entropy maximization mechanism to ensure that the relationships from a class to each task tend to be equal.  As a part of the loss functions, entropy maximization trains the model together with the classification loss of each task. In the testing phase, the instance node is inserted into the association graph by calculating the relationship with each task and each class. Afterwards, the instance representation is updated through message passing and then input to the classifier of the corresponding task to obtain the predicted label. To evaluate the proposed model, ablation experiments and visualization are conducted to show the competitive performance of the proposed model.", "questions": "1)\tWhat is the specialty  of the designed metric function? Why the task graph and class graph need to be learnable to build edges, while the association graph does not use learnable to build edges between task and class? The author should discuss the motication and the difference behind these metric functions, rather than just show the formula.\n2)\tThe associated graph is a heterogeneous graph. Have you considered or tried to use a heterogeneous GNN model. If so, how about the effect?\n", "limitations": "The authors described the limitations of their work and did not discuss any potential negative societal impacts of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1)\tThis article explicitly defines the problem called category shifts in the multi-task classification.\n2)\tIt proposed to address the category shifts problem with the association graph, which is a heterogeneous graph to describe the relationship between tasks and classes and enables tasks to benefit from the information propagated by unseen classes.\n3)\tThe experiment part is relatively complete, where they have tested different parameters, and conducted ablation experiments. At the same time, they also provide the source code to ensure the reproducibility of the model.\n\nWeaknesses\n1)\tUsing assignment entropy as the part of loss function is far-fetched. It assumes that the weight of the edge from each task to a class tends to be the same in the association graph, that is, a class passes message to different tasks with equally importance. Obviously, it is not reasonable in practical application. As mentioned in the paper, due to patient populations or proprietary use of regulations, different tasks do not have the same diagnosis categories distribution, that also means, a class has different probabilities of appearance in different tasks. Therefore, the importance of tasks that the class appeared in training set should be higher than that of tasks that the class have not appeared in training set. However, the association graph trained by assignment entropy maximization assumes that each task should have the same importance to a class, which is against reality.\n2)\tThe scenarios of category shifts problems are limited in the practical application. It requires the assumption that related tasks share the same label space in the testing space. In the evaluation part, the training data is fabricated by sample masking to simulate this scenario. Can the author give some examples in real life applications, such as giving some practical data and calculating their missing rate (definition 2)?\n3)\tThe association graph is constructed as a small graph node with only about 70 nodes (Office-home dataset), but it requires 4 or even 6 layers of GNN for message passing? The association graph should be visualized to reveal the relationship between tasks and classes after training.\n4)\tAssignment entropy maximization is regarded as the objective to optimize the matching the association graph and multi-task, but the investigation is rather heuristic. In the ablation experiment (section of benefit of assignment entropy maximization), the advantages of using assignment entropy maximization should be evaluated. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0cujrSQCPww", "reviewer": "Reviewer_dn35", "summary": "The paper proposes a new problem setup where category shift exists in multi-task classification. The authors try to solve it by building graphs that represent tasks, instances and classes as nodes and assigning learnable weights among them. They also tried assignment entropy maximization to overcome the spurious features. The results show that using Graphsage on the proposed graph improve the performance significantly on several benchmarks under different category missing rate. \n", "questions": "-- How would a simple baseline of mixing everything together and training the model, work?\n\n-- Have you tried using relational graph models?\n", "limitations": "Yes, the paper discusses potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ The paper is well written and addresses an important problem.\n\n++ The definition and representation of the graph are clear and easy to understand.\n\n++ The idea for solving the spurious features (3.2) (to boose the connection of tasks with their unseen categories) is novel. The idea can be tried on other domain generalization tasks. \n\n++ The performance is also consistent when a shift does not exist, which may imply a potential theoretical proof behind the method.\n\n--The graph model may need some ablation study on the effectiveness of each proposed architecture.\n\n-- The main limitation I am worried about is the extensibility of the model. It might be a little challenging for a graph model to expand especially since the out-of-domain classes can be theoretically infinite. In this case, the missing rate also does not make sense.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept