Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the problem of multi-task classification with category shifts where training data for different tasks would not have same categories. To mitigate category shift problem, the authors construct association graph where different tasks, classes and instances are nodes and use message passing to transfer knowledge between them. Assignment entropy maximization is also designed to balance weights of edges. The experiments conducted on three general benchmarks and a realistic medical dataset for skin lesion classification demonstrate the effectiveness of the method.
strengths_and_weaknesses: Strengths:
1. Very well written paper, addressing category shifts problem in multi-task classification, which is realistic and interesting. The proposed method is also a reasonable way to mitigate this problem by constructing association graph and transferring knowledge, because the missing class for one task may appear in other tasks.
2. The paper is well developed and easy to follow. The proposed method is clearly presented.
3. Extensive experiments are conducted on both three general benchmarks and real-world datasets. Results show that the proposed method outperforms other baselines. Ablation tests are also conducted to demonstrate the effectiveness of association graph and assignment entropy maximization modules in the method.

Weaknesses:
1. More explanations need to be included to show the necessity of using graph to transfer knowledge. They can just combine all class knowledge directly with input instance instead of constructing a complex network. The benefit of association graph structure needs to be stated.
2. More experiments can also be added to deal with the problem stated above and show the benefit of association graph structure. For example, what happens if the knowledge of class is directly combined with input instance or the association graph is complete.

questions: 1. How does the graph structure help to transfer knowledge? I wonder what would happen if you directly use the class knowledge as extra information to help with prediction. 
2. Considering results from Table 2, the best performance is achieved when the node can aggregate knowledge from all other nodes. Does that mean that the knowledge of each node can directly be incorporated into the input instance and local structure information of association graph is not important?
limitations: More experiments and explanations need to be included to show how the association graph structure information can help with the category drifts problem.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to tackle the problem about the multi-task classification with category shifts. It proposes learning an association graph to transfer knowledge among tasks for missing classes, and constructs the association graph with nodes representing tasks, classes and instances. The proposed method encodes the relationships among the nodes in the edges to guide the knowledge transfer between them.  Moreover, to avoid spurious correlations between task and class nodes in the graph, the authors introduce an assignment entropy maximization that encourages each class node to balance its edge weights, which enables all tasks to fully utilize the categorical information from related tasks. An extensive evaluation on three general benchmarks and a medical dataset for skin lesion classification reveals that the proposed method consistently performs better than representative baselines.
strengths_and_weaknesses: **Strengths**
* This paper develops a new multi-task learning scenario, in which individual tasks do not contain complete training data for the categories in the test set.
* This paper utilize an association graph to propagate  and transfer the knowledge between the  constructed nodes, which is reasonable.

**Weaknesses**
* The construction and processing of the proposed association graph are similar to another research area-heterogeneous GNNs, and the author does not mention much about them.
* The proposed setting is a bit like a combination of multi-task learning and out-of-distribution generalization, which is of limited novelty.
questions: * The author should clarify the relationships between the proposed graph method and the heterogeneous GNNs.
* The author should clarify the relationships between the proposed setting and out-of-distribution (OOD) generalization problem, and add more OOD methods as baselines for fair comparisons. 
* The author should add more related works involving above two areas.
limitations: The computational cost of the proposed method seems a lot, the authors do not provide necessary analysis and comparisons about it.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new problem setup where category shift exists in multi-task classification. The authors try to solve it by building graphs that represent tasks, instances and classes as nodes and assigning learnable weights among them. They also tried assignment entropy maximization to overcome the spurious features. The results show that using Graphsage on the proposed graph improve the performance significantly on several benchmarks under different category missing rate. 

strengths_and_weaknesses: ++ The paper is well written and addresses an important problem.

++ The definition and representation of the graph are clear and easy to understand.

++ The idea for solving the spurious features (3.2) (to boose the connection of tasks with their unseen categories) is novel. The idea can be tried on other domain generalization tasks. 

++ The performance is also consistent when a shift does not exist, which may imply a potential theoretical proof behind the method.

--The graph model may need some ablation study on the effectiveness of each proposed architecture.

-- The main limitation I am worried about is the extensibility of the model. It might be a little challenging for a graph model to expand especially since the out-of-domain classes can be theoretically infinite. In this case, the missing rate also does not make sense.


questions: -- How would a simple baseline of mixing everything together and training the model, work?

-- Have you tried using relational graph models?

limitations: Yes, the paper discusses potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This article discusses the problems of category shifts in multi-task learning. To solve the problem, the author prposes to build a heterogeneous graph called association graph, where nodes represent tasks and classes and edges indicate their relationships. There are three kinds of edges: task-task, class-class, and task-class. The authors use the average of the corresponding sample representations to denote the tasks and classes. In addition, they propose three learnable metric functions to calculate the weights of the three kinds of relationships mentioned above. Intuitively, the edge weight of the relationship between the task and the class that does not appear in its training set should be low. Because the association graph is used for message passing, the authors hopes that the task can benefit from the information of unseen class nodes. To tackle the category shifts problems, they propose an entropy maximization mechanism to ensure that the relationships from a class to each task tend to be equal.  As a part of the loss functions, entropy maximization trains the model together with the classification loss of each task. In the testing phase, the instance node is inserted into the association graph by calculating the relationship with each task and each class. Afterwards, the instance representation is updated through message passing and then input to the classifier of the corresponding task to obtain the predicted label. To evaluate the proposed model, ablation experiments and visualization are conducted to show the competitive performance of the proposed model.
strengths_and_weaknesses: Strengths
1)	This article explicitly defines the problem called category shifts in the multi-task classification.
2)	It proposed to address the category shifts problem with the association graph, which is a heterogeneous graph to describe the relationship between tasks and classes and enables tasks to benefit from the information propagated by unseen classes.
3)	The experiment part is relatively complete, where they have tested different parameters, and conducted ablation experiments. At the same time, they also provide the source code to ensure the reproducibility of the model.

Weaknesses
1)	Using assignment entropy as the part of loss function is far-fetched. It assumes that the weight of the edge from each task to a class tends to be the same in the association graph, that is, a class passes message to different tasks with equally importance. Obviously, it is not reasonable in practical application. As mentioned in the paper, due to patient populations or proprietary use of regulations, different tasks do not have the same diagnosis categories distribution, that also means, a class has different probabilities of appearance in different tasks. Therefore, the importance of tasks that the class appeared in training set should be higher than that of tasks that the class have not appeared in training set. However, the association graph trained by assignment entropy maximization assumes that each task should have the same importance to a class, which is against reality.
2)	The scenarios of category shifts problems are limited in the practical application. It requires the assumption that related tasks share the same label space in the testing space. In the evaluation part, the training data is fabricated by sample masking to simulate this scenario. Can the author give some examples in real life applications, such as giving some practical data and calculating their missing rate (definition 2)?
3)	The association graph is constructed as a small graph node with only about 70 nodes (Office-home dataset), but it requires 4 or even 6 layers of GNN for message passing? The association graph should be visualized to reveal the relationship between tasks and classes after training.
4)	Assignment entropy maximization is regarded as the objective to optimize the matching the association graph and multi-task, but the investigation is rather heuristic. In the ablation experiment (section of benefit of assignment entropy maximization), the advantages of using assignment entropy maximization should be evaluated. 

questions: 1)	What is the specialty  of the designed metric function? Why the task graph and class graph need to be learnable to build edges, while the association graph does not use learnable to build edges between task and class? The author should discuss the motication and the difference behind these metric functions, rather than just show the formula.
2)	The associated graph is a heterogeneous graph. Have you considered or tried to use a heterogeneous GNN model. If so, how about the effect?

limitations: The authors described the limitations of their work and did not discuss any potential negative societal impacts of the work.
review_title: null

Meta-review: metareview: This paper focuses on a variant of multi-task classification that it argues is new and of practical interest. In this setting, multiple tasks share a label space, such as in the case of a domain shift setting in which classes exist in multiple domains. The distinguishing feature of this setting is that not all classes have observed instances in all tasks. The paper proposes a graph-based approach, in which nodes representing tasks, classes, and instances are connected in an "association graph." A graph neural network is then learned over the association graph. Learnable metric functions weight the edges. Then representations of instances are classified using task-specific classifiers. Experiments on three benchmark tasks and a skin lesion task show that the proposed approach can consistently outperform both multi-task methods and sensible baselines like empirical risk minimization on all data.

The reviewers generally found the paper well-written, the new setting interesting and important to study, and the experimental evaluation thorough. During the review process, the authors added several other experiments and discussion justifying the use of a graph structure to effectively share knowledge across tasks.
confidence: Less certain
Decision: Accept