Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a federated learning (FL) framework for COVID-19 xray image analysis, which leverages the natural scalibility of visual transformer to establish a multi-task FL process. The proposed method jointly learns from different clients of various x-ray-based tasks including classification, detection and segmentation in a split learning manner, in which each client preserves their only transformer head and tail while jointly learning a shared transformer body maintained in the global server. Experimental results show that the established multi-task federated learning framework with transformer improves individual performances for each COVID-19 diagnosis task and are comparable with centralized training model.
main_review: 1. Concern on technical novelty. This paper provides good insight of utilizing the natural configuration of transformer to promote multi-task federated learning by sharing the transformer body across tasks. However, the technical novelty of this paper are limited. Both the federated learning and split learning techniques in this paper are proposed previously, and the capacity of transformer for multi-task learning has also been well explored in many vision-language tasks. Though integrating transformer into federated scenario has some merits, it is insufficient to serve as the major contributions considering technical aspect.

2. Concern on privacy issue. The split learning procedure in this framework requires to send the data features of local clients into global server. Given that the networks parameters of local transformer head and tail are also attainable in global server for aggregation purpose, the sharing of samples features could lead to privacy leakage to some extent.

3. Transformer shows advantage on its flexibility which is the key enabling the establishment of multi-task federated learning, but does transformer also advantage on the task performance than the expert network on each task? I understand that authors utilize transformer as it enables to learn from different tasks under federated setting, but the architecture of transformer may not fit each task and could sacrifice the performance compared with federated training task-specific expert network at clients of the same task.

4. What is the criteria to select the global model for validation. Since the transformer body are shared among all clients, how do the authors determine the best model for testing? Will the performances at all three tasks be taken into consideration? 

5. There are some mistakes in the experimental results, e.g., in Table 2, authors mark the proposed method with “0.916+-0.038” as bold in “Others”, but the baseline federated learning methods performs better “0.926+-0.018”. 

6. It is unclear what the proposed method “W/o transformer body” mean. Without the server-side transformer body, does the remaining architecture only contain the transformer head and tail?
limitations_and_societal_impact: Authors have discussed the limitation and societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a federated split task-agnostic (FESTA) framework that can take an effective use of the vision transformer's (ViT) to perform several CXR tasks simultaneously. The authors discover that the design of ViT is ideal for split learning, in which a network is designed into a client-server mechanism. They integrate ViT into their framework by combining the benefits of federated and split learning through its decomposable design. According to the authors, a model trained with the FESTA framework can leverage robust representations from multiple related tasks to improve individual task performance.
main_review: The work is somewhat a combination of a few existing techniques. The authors, however, make good use of previous studies to create a usable framework. They illustrate the usefulness of the framework by running comprehensive experiments on a variety of datasets and tasks.

The authors demonstrate that a model trained with the FESTA framework can leverage robust representations from multiple related tasks to improve individual task performance. The problem, however, the test set is quite a small one containing only 6 COVID images in the external/test set. Regarding this issue, I would like to see the result of the COVID classification task by considering ‘Hospital 3’ as an external/test set and ‘Hospital 1’ as part of the training/validation set.

The submission is clearly written and well-organized. However, it would strengthen the study and clarify the performance more precisely if the authors could compare their results with the winning solutions (e.g., segmentation and detection tasks) from the Kaggle platform. The readers could get a good understanding and performance of the proposed model compared to those of the Kaggle platform.

The findings are important, and I think the research community is likely to use or expand on them. Certainly, the submission does a better job of addressing a challenging task than previous work.

Overall, the study is an interesting one and the authors make an effort to support their claims with adequate experiments and promising results. However, the experimental setting for COVID classification, in my opinion, is not up to mark. According to the paper, the classification is the main task to achieve through FESTA, yet the test set comprises of only six COVID images. As a result, I am a bit skeptical about the robustness of the model with respect to the results. Nonetheless, considering the idea and the merit of the study, experiments conducted by the authors and performance shown in the manuscript, I believe this work is marginally above the threshold to accept. Hence, I vote for accepting the paper.
limitations_and_societal_impact: The authors acknowledge that this is more of a proof-of-concept study and that privacy concerns may arise. Furthermore, they have not done the necessary tests to evaluate the robustness of their approach against the problems of significant bottlenecks in communication, stragglers, and fault tolerance. However, because these issues are not the central theme of the work, they may be overlooked for the time being.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper combines: split learning, federated learning, multi-task learning, vision transformers, and COVID-19 chest x-ray (CXR) classification. Specifically, the authors propose the Federated Split Task-Agnostic (FESTA) framework, motivated by privacy concerns of training models with patient level healthcare data (CXRs). This framework is for training large deep learning models based on private datasets distributed over clients (i.e. hospitals). Here, each client has a local copy of a "head" and "tail" of a vision transformer model, while the server has a shared body of a vision transformer. The training process is as follows:
1.) clients perform forward passes using their local data through their head networks and send the embedded representations to the server
2.) the server performs a forward pass with the embedded representations and sends the resulting representations back to their respective clients
3.) the clients finish the forward pass through their tail networks, compute the loss, and send gradients back through the server
4.) server continues backprop, client finishes backprop, everyone updates their weights (the server's network weights will be updated considering data from _all_ clients)
5.) occasionally, all the weights from the clients head and tail networks will be averaged together (FedAvg)

Additionally, the authors consider the case where some clients are training on different tasks (e.g. image classification, segmentation, detection) using the same modality (CXRs). Here, step 5 is applied over clients with the same task.
main_review: The main claims of this paper are:
1.) That the split-learning setup *and* federated-learing training setup result in models that are as good as training with a centralized dataset.
2.) That ViT models are "ideally suited" for the split learing/federated learning (FESTA) training setup.

Claim #1 is supported to some extent by the results shown in Table 2. In average performance the Data-centralized approach, FESTA, and federated learning perform within a stdev of each other, while Split learing performs worse. 

Claim #2 is not supported. In Table 5 the authors show results with and without the "Transformer body" and both are within a standard deviation of each other. Further, the "without Transformer body" approach is not described. Does the server use a different model when a transformer body is not used? Would a CNN perform just as well?

The classification task setup uses 3 public datasets and 4 private datasets collected from hospitals / labeled by radiologists. One of the 4 private datasets is held out for evaluation. Data leakage in COVID-19 classification tasks has been a huge problem, with many preprints claiming wonderful model performance. The authors should consider holding out CXRs from all 4 hospitals as an experiment (especially if the same radiologists labeled all 4) to test the generalization capabilities of the models. This is a major limitation of the application of this work.

Finally, as the authors point out, split learning is not sufficient to guarantee data privacy, while the premise of the paper is a system that can be used in a federated manner among clients with private non-IID datasets. This is another major limitation of the application of this work.

Overall, this work presents an interesting application of a combination of different methodologies. Such systems will likely be necessary to train large medical image models that are able to generalize to new imagery. It needs to be improved with experiments demonstrating the value of transformers specifically, or rewritten (with experiments) to consider other architectures.   Because the paper is motivated by a privacy sensitive application, more discussion with potential solutions to the privacy problems in SL need to be added.

## Minor details

- The entire FESTA row is bolded in Table 2 even though the mean performance in the Others column is less than other values. Please specify what is bolded in Table captions.
- Be careful of copying text from [1] (cited as [33] in the text). I noticed that the data descriptions from these papers were very similar, e.g. sharing, "With a total of 224,316 CXR images from 65,240 subjects, the 32,387 lateral view images were excluded, leaving 29,420 PA and 161,427 AP view".

[1] Park, Sangjoon, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, and Jong Chul Ye. "Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification." arXiv preprint arXiv:2104.07235 (2021).
limitations_and_societal_impact: The authors discuss this in Section 6 and suggest, "This work is more like a proof-of-the-concept study, rather than a ready-to-use solution for industry", while strongly motivating this work from an application standpoint (the COVID pandemic). This is flawed reasoning as "industry" will look to venues like NeurIPS for ideas, and the flaws from research published at NeurIPS (especially application focused papers) can be directly translated to society.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a multi-task learning framework with a shared feature encoder (body) and separate input embedding (head) and application-wise decoder (tail) components for each task in a distributed/federated learning setting. A number of multi-head self-attention modules from the transformer model are adopted here as the body part. The Federated Split Task-Agnostic Learning (FESTA) paradigm is proposed to compute/update the model parameters of head/tail parts on the client-side and the body parts on the server-side. A list of chest x-ray datasets is employed for the experiments, including datasets for COVID-19 classification (normal, pneumonia, and COVID-19), pneumothorax segmentation, and pneumonia detection. Superior results of the proposed method are reported in comparison to other learning paradigms. The manuscript is overall well-prepared, while several things could be improved and addressed(listed below). 
main_review: - One of the limits of the proposed method is that a large computation resource is required on the server-side. 
- Additionally, the communication cost between client and server is more than doubled compared to the FedAvg algorithm, which also increases the risk of data privacy breaches.  The output of the Embedding model (the head, with both activations and gradients) could be used to reconstruct the inputs (patient images) easier than using only gradients in FedAvg.
- The implementation detail about the head, body, and tail parts for each task could be enhanced, e.g., what's the architecture of the tail part in the classification model, a single FC layer?
- Are the results show in Table 2 with a single task? Add multi-task ones as well?
- I am seriously concerned that the ablation studies are conducted using FedAvg, not the proposed method?
- On lines 171-174, I am not sure why the CheXpert dataset is introduced with much more details than other datasets while it is not even used in the experiments. And its label is not generated by experts but by an NLP-based labeler. On the other side, the authors should reveal more details about the data from 4 hospitals, e.g., geographic info.
- I will suggest the authors tune down the claim for the Non-iid settings since the datasets do not have the "extreme" non-iid setting as stated on line 246, and I did not see any particular design for the non-iid setting in this work. 
- H, T for "det" and "seg" are mismatched with the input images in Fig. 1(lower left)
- Bolded results better suggest the best in the column. 
- It will be helpful to discuss the relationship between the proposed method and the one listed below (first arxiv version on Dec. 2020). Both share a lot in the model architecture.
Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu Z, Ma S, Xu C, Xu C, Gao W. Pre-trained image processing transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 (pp. 12299-12310).

I will be happy to raise my ratings if the author could discuss/address some of the concerns.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The main concerns raised by the reviewers were: 1) comparison to multi-task, single-task, task-specific experts and CNN-based multi-task models. 2) more detailed discussions on the privacy issues 3) a discussion/comparison of communication costs. Overall, the reviewers found that the authors' response did a good job in addressing all these concerns. Indeed, the new results/discussion significantly strengthens the paper and should be included in a revised version.
Decision: Accept (Poster)