Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose to solve the gradient conflict in training and dense activation in inference in multi-task learning using a co-designed mixture-of-expert model. In algorithm level, the authors propose to replace ViT layer with mixture of expert layer and sparsely activate them during training to alleviate the gradient conflict issue, while during inference, the author propose a novel computation reordering scheme to better support the sparse activation for different tasks. Experiments on GPU and FPGA show the proposed co-design achieves better performance than existing methods. 
strengths_and_weaknesses: Strengths:
- The co-design for Mixture-of-Expert to enable multi-task learning is a timely attempt. 
- The achieved results is promising in both accuracy and efficiency. 

Weaknesses:
- Using mixture-of-expert to deal with multi-task learning is not the novel idea, but the co-design attempt is interesting to me. 
- Is the mentioned double buffered computation strategy similar to Ping-pong buffer? If so it would be better to add the reference. 
questions: - How are different experts shared between different tasks? Will the scaling in task number requires more experts to maintain a satisfied performance? 
- What is the training cost of the proposed model compared with baseline methods? Does it need extra training to learn the expert-task mapping and fully train the large number of experts? 
- What is the performance improvement breakdown for the proposed hardware design? 
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper presents a multi-task MoE and model-accelerator co-design. At the algorithm level, the method adopts a task-level mixture-of-expert approach that can effectively reduce training and inference cost. At the model-accelerator level, the paper proposes a computation reordering scheme that is tailored for memory-constrained MTL and achieves little switching overheads. The paper observes significant inference FLOPs reduction and memory requirement reduction, compared to a FPGA baseline.
strengths_and_weaknesses: Strengths:
- The paper has a very thorough and complete related work and reference section. 
- Multi-task MoE is an important field that generates numerous important works. This paper seems to be a right direction addressing important problems. Co-designing a software/hardware systems can be the right problem to solve. 
- The proposed reordering reduces memory footprint at small overheads. 

Weaknesses:
- The paper is rephrasing many contributions of conditional computation/mixture-of-experts and multi-task MoE. Those contributions are not unique contributions from this paper. For example, token-MoE has been prevalent for a long time and more recently, task-MoE has been demonstrated useful in many tasks including machine translations (e.g. Machine translation from Google). [1] 

- The proposed reordering scheme is a system-level contribution which might not best suited for NeurIPS. Even though, there is a co-design claim, the reviewer feels that task-level MoE and and hardware execution reordering should be decoupled. The same reordering scheme can be applied to other types of networks equally. In order to prove co-design is effective, some co-optimization should be applied which can require some joint search space on the MoE model and reordering scheme. The paper should show that co-designing these two parts can be useful and is better than separately optimizing each of them.

- The paper listed challenges round "handle many different tasks" and "switch between tasks" and how different tasks can interfere each other when training on a shared model backbone. An example given in the paper is the autonomous driving setting where potentially hundreds of tasks are running on the shared model backbone. However, the paper does not prove it's own point by evaluating only on a small set of tasks. 

[1] "Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference", https://aclanthology.org/2021.findings-emnlp.304.pdf
questions: 1. In table 1, did you implement all the related work similarly on FPGAs? How did you generate the energy numbers? If all the baselines are also implemented in FPGA, then the quality of models are largely determined by implementation not necessarily by the method itself. 
2. Two out of six tasks are worse than cross-stitch, any reasons?
3. Why make the total FLOPs only half of the baseline models (Table 1)? A fair comparison would be with similar FLOPs and compare the accuracy.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies a novel setting of MTL: training with multiple tasks hoping them to boost each other, and inference with a single task each time. The setting is practically convincing as most MTL systems switch between tasks instead of executing all tasks at the same time. The authors presented a model-accelerator co-design framework, involving both algorithm and hardware innovations.
strengths_and_weaknesses: Strong points:
1. The overall problem and idea are novel and interesting. Most MTL papers assume all tasks to be trained and tested in the same bundle. This paper allows for more flexibility. 
2. The introduction MoE is well motivated from two aspects: efficiency for single-task inference (sparse activation); and avoiding MTL training conflicts (grouping similar modules). Using MoE is hence natural in this new setting, and to my best knowledge, is novel for the MTL field. The authors also designed a task-dependent gating network for this new purpose.
3. The authors meanwhile present interesting hardware innovations. To my best knowledge, not too many hardware works were done in the MTL domain. This work proposed a novel computation reordering mechanism tailored for memory-constrained MTL and MoE, which allows scaling up to any number of experts and also achieves zero-overhead switching between tasks. It sounds very promising and might be of independent research interest. 
4. Experiments are solid, involving customized FPGA design on real hardware, and evaluating two major MTL datasets (NYUD-v2, PASCAL-Context). The proposed co-design can save memory and energy costs by up to one order of magnitude, with no worse accuracies than baselines. Besides, more tasks seem to save more (PASCAL versus NYUD), which shows great scalability to larger numbers of tasks.
5. The abstract and intro were written elegantly to give a proper big picture. The authors also did a good job in making figures. As another plus, codes are provided in the supplementary: I did not have time to carefully check but from quick reading, the code quality looks fine.


Weak points:
1. My main concern is that the MTL baselines used are all somehow weak. Even though the problem setting is new, and I know not many peer algorithms did exactly the same job, there are still some. For example, the authors should check “Task Adaptive Parameter Sharing for Multi-Task Learning” (CVPR 2022). That paper has important similarity to this submission, in learning modularized models for MTL while allowing for single-task inference. Although it does not use dynamic MoE and has no hardware co-design, I believe some level of comparison and discussion is necessary. There might be more in literature. 
2. One other baseline-related issue is that the authors only compare with encoder-focused MTL models, and claim that is because decoder-focused ones are in general heavier (but also, more performant). I think the authors should at least compare with or discuss some decoder-focused methods and demonstrate the trade-offs here, rather than completely ignoring. For example, is it because decoder-focused MTL models cannot fit into FPGA, really? 
3. Why jumping to a ViT backbone? Looking at Table 1, the ViT baseline is already stronger than most prior works built on ResNet 18 (as expected). Can you compare with other MTL works with the same ViT backbone, or can you replace ViT with Res18 in your co-design framework? That would help disentangle your own contributions.
4. If some MTL applications do require to activate all or most tasks all the time, will your framework still be advantageous? For example, what if always executing 2, 3, or 5 tasks simultaneously on PASCAL? I understand this will change your problem target, but I suggest the authors to include some discussions and clarity for future readers. 
questions: Please see the weakness part.
limitations: No, the current version claimed it has no negative social impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper describes an efficient multi-task neural network for vision applications that can be deployed on resource-constrained devices. The model consists of a mixture of vision transformer experts that are sparsely activated by a GELU activation function. The main innovation is that mixture of expert layers are computed expert-by-expert instead of token-by-token, which reduces latencies of loading expert weights.
strengths_and_weaknesses: ## Strengths
* Computing mixture-of-expert layers seems like a simple yet effective why to increase the efficiency of a vision transform mixture of expert architecture
* The paper is well well
* Experiments are solid

## Weaknesses
* Only two datasets (NYUD, PASCAL) are considered in experiments--no real world datasets
questions: * How does the model accuracy and efficiency (Latency, Energy, Memory) depend on the total number of experts N and top experts K?
* Is the softmax in equation 3 needed, given that you are only interested in the top-k activations?
* How are the top-k experts computed at training and inference time? What is the runtime?
limitations: Yes
review_title: null

Meta-review: metareview: This paper presents a model-accelerator co-design framework to enable on-device Multi-task Learning (MTL). At the model level, customized mixture-of-expert (MOE) layers are introduced for MTL, which alleviate gradient conflict at training time and improve the efficiency at inference time via sparse activation. At the accelerator level, the paper proposes computation reordering which allows zero-overhead switching between tasks. The algorithm is verified the on popular multi-task datasets, and the accelerator is implemented on commercial FPGAs, demonstrating improved efficiency.

The paper is very well written, the details on the algorithm and hardware implementation are clearly explained. The author chose a particular setting of MTL, then design the model and tailor the parameters to enable efficient on-device MTL. The work is complete, covering from algorithm design to hardware implementation with sufficient innovations. 

Reviewers have raised concerns such as
1). Evaluation on small datasets. During rebuttal period, the authors provide more experimental results from the large-scale Taskonomy dataset.
2). Overclaiming. For example, double buffering is a well-known technique for dataflow optimization. The technique itself is by no means novel. However, I think using it to solve a practical problem still has value.

Overall, it is a solid paper and is recommended for acceptance.

confidence: Certain
Decision: Accept