M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Authors: hanxue liang
Zhiwen Fan
Rishov Sarkar
Ziyu Jiang
Tianlong Chen
Kai Zou
Yu Cheng
Cong Hao
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-task learning
mixture of experts
vision transformer
hardware co-design

Abstract: Multi-task learning (MTL) encapsulates multiple learned tasks in a single model and often lets those tasks learn better jointly. Multi-tasking models have become successful and often essential for many sophisticated systems such as autonomous driving and indoor robots. However, when deploying MTL onto those real-world systems that are often resource-constrained or latency-sensitive, two prominent challenges arise: (i) during training, simultaneously optimizing all tasks is often difficult due to gradient conflicts across tasks, and the challenge is amplified when a growing number of tasks have to be squeezed into one compact model; (ii) at inference, current MTL regimes have to activate nearly the entire model even to just execute a single task. Yet most real systems demand only one or two tasks at each moment, while flexibly switching between tasks per need: therefore such “all tasks activated” inference is also highly inefficient and non-scalable in practice. 
In this paper, we present a model-accelerator co-design framework to enable efficient on-device MTL, that tackles both training and inference bottlenecks. Our framework, dubbed M³ViT, customizes mixture-of-experts (MoE) layers into a vision transformer (ViT) backbone for MTL, and sparsely activates task-specific experts during training, which effectively disentangles the parameter spaces to avoid different tasks’ training conflicts. Then at inference with any task of interest, the same design allows for activating only the task-corresponding sparse “expert” pathway, instead of the full model. Our new model design is further enhanced by hardware-level innovations, in particular, a novel computation reordering scheme tailored for memory-constrained MTL that achieves zero-overhead switching between tasks and can scale to any number of experts. Extensive experiments on PASCAL-Context and NYUD-v2 datasets at both software and hardware levels are conducted to demonstrate the effectiveness of the proposed design. When executing the practical scenario of single-task inference, M³ViT achieves higher accuracies than encoder-focused MTL methods, while significantly reducing 88% inference FLOPs. When implemented on a hardware platform of one Xilinx ZCU104 FPGA, our co-design framework reduces the memory requirement by 2.40×, while achieving energy efficiency (as the product of latency and power) up to 9.23× times higher than a comparable FPGA baseline.

ViV Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
Hanxue Liang"; Zhiwen Fan**, Rishov Sarkar?, Ziyu Jiang', Tianlong Chen', Kai Zou', Yu Chenge, Cong Hao², Zhangyang Wang' University of Texas Austin, GGooggaa Institute of Technology Texas A&M University, Protagolabs Inc, 'Microsoft Research zhiwenfawaa an, tianlong. chen, atlaswang) edu
frishov. sarkar, callie. haotegtt edu, jiangziyudtamu edu kz@protagolabes com, yu. chengtmicrosoof..
Abstract
Multi-task learning (MTL) encapsulates multiple learned tasks in single model and often lets those tasks learn better jointly. However, when deploying MTL onto those real-world systems that are often resource-constrained or seeesiy-sensitive, two prominent challenges arise: (i) during training, simultaneously optimizing all tasks often difficult due to gradient conflicts across tasks, and the challenge amplified when growing number of tasks have to be squeezed into one compact model; (ii) at inference, current MTL regimes have to activate nearly model even just execute single task. Yet most real systems demand only one two tasks at each moment, and switch between tasks as needed: therefore such "all tasks activated" inference is also highly inefficient and non-scalable.
entire
In this paper, we present model-accelerator co-design framework enable ef- ficient on-device MTL, that tackles both training and inference bottlenecks. Our framework, dubbed M customizes mixture-offexperts (MoE) layers into a vision transformer (VIT) backbone for MTL, and sparsely activates task-specific experts during training, which effectively disentangles the parameter spaces avoid different tasks' training conflicts. Then at inference with any task of interest, the same design allows for activating only the cor-ccorrspnniing sparse "expert" pathway, instead of the full model. Our new model design is further enhanced by hardware-leved innovations, in particular, novel computation reordering scheme tailored for memory constraiind MTL that achieves over-eee switching be- tween tasks and can scale to any number of experts. Extensive experiments on PASCAL-Context [ and NYUD-v2 (2) datasets at both software and hardware levels are conducted demonstrate the effectiveness the proposed design. When executing single-task inference. ViT achieves higher accuracies than encoder- focused MTL methods, while significantly reducing 88 inference FLOPs. When implemented on a hardware platform of one Xilinx ZCU104 FPGA, co-design framework reduces the memory requirement by 2.40 while achieving energy efficiency up 9.23x higher than comparable FPGA baseline. Code is available at: https:://git //tph/bgtttubbbbb com CttpVVVTTA-rroo//XV///
Introduction
Vision Transformers (ViTs) 6 as the latest performant deep models, have achieved impressive performance on various computer vision tasks 7 8 [9. These models are specially trained or tested
"quual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

for only one or few tasks; however, many real-worid applications require one compact system that can handle many different tasks efficiently, and often need to swiftly switch berween tasks per demand. For example, an autonomous driving system [00] needs to perform and switch between many tasks such as drivable area estimation, lane detection, pedestrian detection, and scene classification: apparently both single task inference and rross-task switching need to happen at ultra-low latency. As another example, smart-home indoor robots [I]] are expected to address semantic segmentation, navigation, tracking, or other tasks in varying contexts, with very limited on-board resources. Multi- task learning (MTL) [2] solves multiple tasks simultaneously within a single model and learns improved feature representations [15] shared by related tasks [16 17). Therefore, accomplishing realistic efficient MTL is becoming a key knob for building real-time sophisticated AI systems.
Despite the promise, challenges persist to build an efficient MTL model suitable for real-world applications: O during training, prior works [8 20 indicate the competition of different tasks in training may degrade MTI,, since the same weights might receive and be confused by conflicting update directions. Specifically, [19 reveals that negative cosine similarities between different tasks' gradients are detrimental. 22 confirm that conflicting gradients not only slow down convergence but also bias the learned representations against some tasks. That is only getting worse on compact models owing to their limited modeling capacity. To tackle the cross-task conflicts, solutions have been proposed by varying learning rate speeds of different tasks [20), using "cross "cross-stitch" sharing [23], or re- ee-balancing task gradients [19 24] 20 25). However, they either require task sask-speiffic design or significantly increase the model complexity which contradicts our efficiency goal. @ at inference, existing MTL regimes typically activate the entire backbone model unconditionally. However, many real systems only need to call upon one or a few tasks at each moment, hence the "all activated"'
inference is heavily inefficient and non-scalable. For example, current regimes [14 23 26 have
to activate the whole gigantic ResNet (28) encoder even just to execute a single monocular depth estimation task or so. If the number of tasks scale up [2] and the backbone keeps growing bigger, the "per task" inference efficiency of the resultant MTL model could become catastrophically poor.
To tackle these bottlenecks, we propose a model-accelerator co-design framework that enables efficient on-device MTL. Specifically, in the software level, we propose to adapt mixture of experts (MoE) layers (30 31] into the MTL backbone, as MoE can adaptively divide-and-congee
the entire model capacity into smaller sub-models [30,32]. Here, we replace the dense feed-forward network in the VIT with sparsely activated MoE experts (MLPs). A task-dependent gating network will be trained to select the subset of experts for each input token, conditioning on tasks. During training, this task-dependent routing principle effectively disentangles the parameter spaces, balancing feature reuse and automatically avoiding different tasks' training conflicts. Meanwhile, at the inference stage with any task of interest, this design naturally allows for sparse activation of only the experts corresponding to the task instead of the full model, thus achieving highly sparse and efficient inference for the specific task. In the hardware level, we propose a novel computation reordering mechanism MTL and MoE, which allows scaling up to any number of experts
tailored for and also achieves eerr-hver switching between tasks. Specifically, based on ViT, we push tokens to per-expert queues to enable espert-by-exper computation rather than loken-by-tokee implement a double- buffered computation strategy that hides the memory access latency required to load each expert's 's weights from off-chip memory, regardless of task-specific expert selection. This design naturally incurs no overhead for switching between frames or tasks in FPGA.
We then
To validate the effectiveness, we evaluate our performance gain using the ViT-small backbone on the NYUD-v2 and PASCAL-Context datasets. On the NYUD-v2 dataset with two tasks, our model achieves comparable results with encoder-focused MTL methods while reducing 71 % FLOPs for single-task execution. When we evaluate on the PASCAL-Context dataset with more tasks, our model achieves even better performance (2.71 vs. 0.60) and reduces 88% inference FLOPs. We found the MTL performance gain brought by MoE layers consistently increases as the task count grows. When implemented on a hardware platform of one Xilinx ZCU104 FPGA, our co-design framework reduces the memory requirement by 2.40x x while achieving energy efficiency (as the product of latency and power) up to 9.23x higher than comparable FPGA baselines and up to 10.79x higher than the GPU implementation. Our contributions are outlined below:
We target the problem of efficient MTL, and adopt the more realistic inference setting (activating one task at time, while switching between tasks). We introduce MoF! as the unified tool to attain two goals: both resolving cross-task training conflicts (better MTL. performance), and sparsely activating paths for single-task inference (better efficiency).
2

---

Specifically for MTL, the MoE layer is accompanied with task-dependent gating network to make expert selections conditioning on the current task.
We implement the proposed MTL MoE ViT framework on hardware platform of one Xilinx ZCU104 FPGA, which enables us to exploit a memory- -eatore-ffficentt computation reordering scheme that consolidates per-expert only one expert's 8 weights are needed on-chip at time. Our design is scalable to any number of experts while requiring no frame-switching or task-switching overhead.
Settiply-an--CCumulll
(MAC) operations such that
We conduct extensive experiments to justify its inference effectiveness in both accuracy and on-edge efficiency metrics. Our framework, dubbed ViT, achieves higher accuracies than encoder-focused MTL methods, while significantly reducing 88% inference FLOPs; on hardware, it reduces the memory requirement by x and costs up to 9.23x and x less energy compared to the FPGA and GPU baselines, respectively.
2 Related Works
Multi-task Learning
The generic multi-task learning problem has been studied for long history.
Some non-deep learning-base based methods propose to use distance metric [33 34 355, probabilistic
prior 36 37
39 40 to model the common information among tasks. With the emergence of the
deep learning technique, MTL [14 23 42
43 44 is performed to learn shared representation
among tasks. The emergence of VIT further makes it possible to extend the task range from only vision tasks to other modalities tasks (e.g.. text, audio) 45 46
48 49] Current MTL models can be roughly categorized into two types based on where the task interactions take place in the network. The encoder-focused architectures [23 41 26 27] only share information in the encoder, before decoding each task with an independent task ssecificcific head. Cross-stitch networks (23) introduce linear combination in each layer. NDDR-CK CNN [26] improves it by dimensional reduction. MTAN [27] leverages an attention mechanism to learn between tasks. TAPS [50] adapts a base model to a new task by modifying small task-specific subset of layers. The second type, decoder-focused models [43 44 (52), make initial task predictions in decoder and then leverage features from these initial predictions to further improve output. Although they report higher performance, their models consume a large number of FLOPs, according to [14]. This makes it difficult to deploy them onto those real-world systems that are often resstrce-onstrrained
47
latency-sensitive. And they need to execute all the tasks for initial prediction, which is heavily inefficient in the common scenario when only one or few tasks are needed. Hence, we focus on encoder-focusee architecture this work. are also proposed to handle the MTL training conflicts problem.
Many methods (25 20 53 27
Mixture of Experts (MoE) MoE contains series of sub-models (i.e., experts) and performs conditional computation in an input-dependent fashion [54 55
58] based on learned or deterministic routing policies (59 58). The traditional dense MoEs suffer from intensive compu-
56
57
tational costs since they select all experts [60]. Recent studies [30 61 62] in natural language
processing (NLP) propose sparse MoE that sparsely activates a few experts during both training and inference, thus substantially reducing the cost and allowing gigantic language models even with trillions of parameters [62]. Unfortunately, such a sparse-gate -aate manner still has limitations of unstable training and imbalanced selections among experts. Various solutions are invented from regu- larization 633 61 62] and optimization (4 65] perspectives. Moreover, MoE has drawn increasing
70 71,, where it mainly focuses on considerably smaller network backbones compared the ones in NLP. For instance, (68) and 69 formulate the channel and kernel of convolutional layers as experts and establish the MoE framework. Several pioneer investigations also explore MoE for multi-task learning, which are related to this work. Particularly, [17 72 73 introduce task-specific gating networks to choose different parts of models for processing information from each task. They present certain possibilities of using MoE solve MTL problems in some cases like classification for medical signals (72), digital number images (MNIST) 73, , and recommendation systems [7.. We make a further attempt to adapt MoE into a compact model for dense prediction multi-task learning, along with
popularity in computer vision (60] 66,
67
68
69
co-design.
Vision Transformer There are growing interests in exploring the use of transformers 774 for computer vision tasks since its success in the natural language processing 75) 76), including image generation 777 78), generative adversarial networks [79] 80), image classification [77
8
82 83
84 82 35 semantic segmentation [8] 86), object detection 66 [8]]. 3D data processing
88
novel view synthesis [9] 92), and many others [93
94
95
96
3

---

Hardware FPGA acceleration of Transformer- models has attracted increasing attention. 99 [00] note that transformers are computation- and memory-intenste and are too large to fit on the FPGA. Therefore, various model compression methods have been proposed, such as activation quantization, token pruning, block-circulant matrices (BCM) for weights, block weight pruning. Such compression methods are lossy and require compression-aves -ware training to regain accuracy. To our best knowledge, there is no existing FPGA accelerator for MoE in a Transformer-basee model. The MoE mechanism exposes great challenges to FPGA since requires swift expert switching between tokens and frames, which may introduce significant overhead of memory and parameter loading. In this work, however, we propose novel expert-by-exppe by-expert prppuuttioo-reordrriiii approach that can reduce the overhead to negligible despite the number of experts, and does not require model compression re- tr-trinning.
Pioneering works [97 98
block-balanced weight pruning, and column-balane
Method
Overview We first describe the standard Vision Transformer and MoEs, and then show the proposed MoE ViT design for MTI. To enable dynamically adapting between different tasks with minimum overhead FPGA, we detail the hardware implementation. Figure shows the whole framework.
(a) MoE ViI Design
(b) Hardware Design
Task
Task B
Both
NotActivated
Decoder A
Decoder B
Intermidiate impat Unitial imput
Self-Attentin
Gating Function
Experts
Load
Parameters
taken
Compute Expert
Intermellate
Time
output
Linear Projection
Final outpot
Figure 1: The overall structure of the proposed ViT pipeline. The input image is split into fixed-size patches, embedded, and combined with position embeddings. In training, the MTL MoE ViT adaptively activates the model by sparsely selecting relevant experts using its task-dependent routers. During inference, only one task will be performed at time. The hardware collects all patches allocated for each expert and processes them expert-by-expert with the "load parameters" and "compute expert" modules.
Router
Task
Expert,
Task-conditionel embedding
Router
Expert,
Layer Norm
A leetion
Layer Norm
Expert,
L Layer
Attention
Layer Norm
Expert
Expert
Norm
Expertya
Router
Task B
Expert,
Expert
(a) Multi-gate MoE layer design.
(b) Task-conditioned MoE layer design.
Figure 2: The proposed two variants of MTL MoE layers. In the left figure, each task selects its experts using own router. In the right one, all tasks share one router, while task-specific embedding is concatenated with the token embedding to formulate the input of the shared router.
3.1 Task-dependent MoE VIT Design
Vision Transformer The representative Vision Transformer architecture first splits the input image into non-overlapped patches and projects the patches to higher hidden dimension using one

---

convolutional layer. The projected patches (a.k.a. tokens) are then passed through several consecutive transformer layers. Each layer contains a self-attention module and a forward network (MLPs). The self-attention is computed using the scaled-dot product:
QQ" VC
Attention (Q, K,K, = softmaa
E RN- are the query, key and value matrices computed from input tokens; V and C' indicate the token number and the hidden dimension. In our experiments, we adopt the DeiT 4 as the backbone encoder, which is data-efficient ViT variant that distills tokens to ensure the student learns from the teacher through attention.
where K, K,
Mixture of Experts Layer A Mixture of Experts (MoE) layer typically consists group of N experts S1. f2,-.... JN along with router R. (or gating network) to select the corresponding experts. The experts network stands for multi-layer perceptrons [62] [OT in ViTs. The router R plays key role within our MoE ViT design as it determines task routings via only sparsely activating relevant experts. We adopt representative router called top-K gating (30) based on VIT. With input æ, the resultant output of MoE layers can be formulated as the summation of the selected top K experts from N expert candidates using a router:
y Σ R(2)x fx(x),,
R(x)) TopK (s) K));
TopK K) = { e if o is in the top K elements
(4)
0 otherwise
where G represents the learnable network within the router, for which we employ a single-layer MLP in practice. The softmaxx-) together with TopK(:, sets all elements of the vector zero except the elements with the largest values. In practice, we choose K = out of = 16 expert candidates. Each expert is computed with get (Wiz), where gelu is the GELU activation [102]. Wi and W2 are two learnable weight matrices. Note that we scale down the expert size by four times compared to that in standard VIT MIP layers make the computation FL.OPs equivalent. We also employ the
load and important balancing loss with the weight of 0.01 following (30) to avoid always picking the same experts while ignoring others. This loss term is also employed for the two task-dependent MTL MoE designs that we introduce next.
Multi-gate MoE VIT for MTL MoE brings training dynamics to balance between large capacity and efficiency, by selecting only subset of experts using the router. To adapt vanilla MoE into our dense prediction MTL framework, we first propose to assign each dense prediction task router R; to specify its own experts, denoted as multi-gate MTL MoE VIT:
V/s Σ R²(x) fa(x:)
(5)
where i denotes task index. Expert candidates 9h are shared across tasks. The flow chart of the multi-gate variant is shown in Figure aaa): task-dependent routers take as input the shared token embedding and do their expert selections.
fask-conditioned MoE VIT for MTL Conditional encoding has been widely applied to multi- modal [103] and multi-task [104] models. To achieve task-dependent routing with one gating network, we propose the task-conditioned MTL MoE VIT shown in Figure 2b)) Specifically, suppose we have n tasks training. We manually define a n-ddimensional one-hot task-conditioned vector: The vector is fed into two-layer MLP to extract a di-einnnsl task embedding, which is then concatenated with token embeddings to form the task- dependent input for the router in the MoE layer:
K
yi Σ
R(x, 14) Su((r),
(6)
1=1
t, ReLU((((zee)
where T indicates the two-layer MLPs to extract task conditioned embeddings, e, E {0, 17", and Σラ=1 Gj 1. We denote this conditional design as task-conditioned MTL MoE VIT, in which backbone model parameters do not proportionally increase if we include more tasks in training.
5

---

Layer Norm
Vaya
Load Weights
Self-Attention
Compute Conne-Coneetted Layer
Head
Compute QKV
scht
Mon
Linear Projection
Mol: Layer
Gating Function
Load Expert
Combine
Layer 8ppo
Head 12
Compute QKV
Experts
Mon
Compute Expert
Per Token
Figure 3: Hardware implementation a ViT block of VIT. The hardware implementation
consists of a layer norm unit, self-attention unit containing 12 independent heads followed by linear projection, a unit to compute the fully-connected layers standard ViT layer, and unit to select and compute the experts in MoE layer. Numerical indicators within the figure indicate the path through which data flows during the computation of a single layer of MiT,, either a VIT layer or MoE layer. This hardware is shared across all blocks.
3.2 Circuit-level Implementation
We co-design the hardware support MTL MoE ViT. We design a layer-wise implementation of ViT on Xilinx ZCU104 FPGA, diagram of which is shown in Figure 3 The design computes layers sequentially but parallelizes computation steps within each layer. By proposing novel computation reordering scheme, our hardware design features memory-efficient expert computation that also achieves zero-overhead task switching and frame switching.
Challenges of Naive Method A straightforward (but naive) implementation would compute the output for each token in the order it appears in the input sequence: all tokens choose any K experts ostensibly the only way to avoid data loading overhead would be to keep
out of the N candidates, weights of all N experts on-chip at all times. However, this requires extreme on-chip memory usage, scaling with O(N) and typically exceeding FPGA on-chip memory capacity unless N is very small.
Challenges of Cache- based Method We can adopt cache to store several experts on-chip at any given time. However, this on-demand approach incurs long delays from off-chip DRAM accesses whenever the cache needs to be repopulated with an expert's weights. Further, we experimentally found that all experts are likely to be activated at least once across all tokens, exhibiting a cache- unfriendly access pattern. Therefore, although cache-based design alleviates memory inefficiency. incurs severe delays by frequently loading the weights of experts.
Proposed Solution: Memory-efficient Computation Reordering The crux of the problem lies in the unpredictability the set of experts that will be needed by tokens at any given time. We address this problem at its root by designing novel computation reordering scheme that flips the compute pattern on its head: rather than computing the MoE layer token-by-token, we instead compute expert-by-expert. The overall flow chart of the reordering scheme shown in Figure 4
Expert selection
Computation reordering
Double- buffeinn and pre-fetching
Token
Experts
Rearder
Expert
Tokens
([
1)
1. Compute partial results for all tokens that use expert
3. Swap between iterations
2. Meanwhile, load parameters the nexd expert
Prefetch Buffer]
Figure 4: The computation reordering flow used by M for hardware memory efficiency. The MoE gating function selects K experts for each token, which are used to route tokens to per-expert queues. This is followed by double-bufferede computation flow that computes expert's results its entire token queue while loading another expert's 's parameters, swapping buffers between iterations.
Specifically, we propose to add each token to a queue for its selected top- K experts, instead of computing the token output immediately. Our hardware then makes use of the per-expert queues via

---

Table 1: Comparisons with encoder-focused MTL architectures on the PASCAL-Context dataset.
Seg.
Norm. H. Parts
Sal.
Edge
Am
FLOPS Energy
Model
Backbone
(mloUt) (mErr)) (mloU)) (mloUy) (odsF) f (%)
(G) 4
(W-s)
STL-B
ResNet-18
66.2
13.9
59.9
66.3
68.8
0.00
167
1.029
MTL-B
ResNet-18
63.8
14,9
58.6
65.1
69.2
-286
167
1.029
Uncertainty [3] (MTL-B)
ResNet-18
65.4
16.5
59.2
65.6
68.6
-4.60
167
1.029
DWA 6
(MTTL-B)
ResNet-18
63.4
14.9
58.9
65.1
69.1
-2.94
167
1.029
GradNorm
(MTL-B)
ResNet-18
64.7
15.4
64.5
67.0
-3.97
167
1.029
MGDA
27
(MTL-B)
ResNet-18
15.6
57.9
62.5
61.4
-6.81
167
1.029
MTAN
27
ResNet-18
63.7
14.8
58.9
65.4
69.6
-2.39
212
5.306
Cross-Stitch 23
ResNet-18
66.1
13.9
60.6
66.8
69.9
+0.60
647
6.001
NDDR-CNN
26
ResNet-18
65.4
13.9
60.5
66.8
69.8
+0.39
747
5.034
M-VIT (MTL-B)
ViT-small
70.7
15.5
58.7
64.9
68.8
-1.77
83
3.062
M (+MoE)
MoE ViT-small
72.8
14.5
62.1
66.3
71.7
+2.71
84
7.446
M (+MoE+Codesign) MoE ViT-small
72.8
14.5
62.1
66.3
71.7
+2.71
0.690
Houble-buffered computation flow, also known as ping-pong buffering [105 with an expert's s weights from off-chip memory accesses, while another already-loaded buffer is used to compute another expert's s results for its entire token queue. After both operations finish, the buffers are swapped, and the process repeats.
one buffer is filled
Scalability and Efficiency Our approach hides nearly all latency from off-chip memory accesses to load expert weights, and it uses O(1) on-chip memory with respect to K and N, making it scalable to any number of experts. Additionally, our method's efficiency does not rely on any specific usage given task,
pattern of experts for a given frame between frames and between tasks. Task switches and frame switches our hardware design do not change computation flow at all, and there is no specific step taken to execute the switch.
naturally achieve zero-overhead switches
Experiments
4.1 Experiment Setup
To evaluate the propose method, we conduct experiments on two popular dense labeling MTL benchmarks, i.e. NYUD-v2 and PASCAL-Context [ Both datasets are described below.
[ contains a total of 10,103 images, for the five tasks of edge detection (Edge), semantic segmentation (Seg.), human parts segmentation (H.Parts), surface normals (Norm.), and saliency detection (Sal.). The NYUD-v2 dataset [2] an indoor dataset which consists of RGB-D images of 464 indoor scenes. There are 795 images for training and 654 images for testing, both with annotation for semantic segmentation (Seg.) and monocular depth estimation (Depth).
Datasets The
For software level evaluation, we adopt the standard evaluation metrics following 14 106 [5]. Particularly, we use mean intersection over union (mloU) for semantic segmentation, human parts segmentation, and saliency; mean error (mErr) for surface normals estimation, root mean square error (rmse) for depth estimation; and optimal dataset F-measure (odsF) [107] for edge detection. Following [14], we use Am to evaluate a MTL model m as the average per task drop with respect to the STL model b over all tasks: Am 7 Σ (-1) - Mr.i)/Ma.i. where Mm,i and Mb,; are the metrics of task for the model m and respectively, and 4 if lower value means better performance.
Evaluation Metrics
To evaluate mooel--cceelraate design. consider latency, energy usage (as the product of latency and power), and on-chip memory usage for single-task inference using a batch size of 1.
Network Configuration and Implementation Details We evaluate our model based on several versions of ViT backbone 4 including ViT-tiny, ViT-small, and ViT-base. Our FPGA designs target the Xilinx ZCU104 FPGA at 300 MHz clock frequency, consuming 10 of power. The GPU baselines are measured the NVIDIA Quadro RTX 8000. The results reported below based on ViT-small. Please refer to the supplementary materials for training setup, more details on network configuration, results on ViT-tiny and ViT-base, and details of our target hardware platforms.
7

---

MTL-
AV Til
M-VIT
M MAXT
ATAANi PM
MTL-8
M-VI
M VIT
Figure 5: Qualitative result on PASCAL-Context. We compare between vanilla MTL-B, M-VIT and ViT models, and our model outperforms baseline on edge detection, semantic segmentation and human parts segmentation and saliency detection.
Table 2: Comparisons with encoder-focused MTL architectures the NYUD-v2 dataset.
Seg. (mloU)+
Depth
Am
FLOPS
Energy
Model
Backbone
(rmse)
(%) t
(G) 4
(W-s) +
STL-B
ResNet-50
43.9
0.585
0.00
192
2.145
MTL-B
ResNet-50
44.4
0.587
+0.41
192
2.145
Uncertainty
(MTT.-B)
ResNet-50
44.0
0.590
-0.23
2.145
DWA 53
(MTL--)
ResNet-50
44.1
0.591
-0.28
192
2.145
Grad
20 (MTL-B)
ResNet-50
44.2
0.581
+1.45
192
2.145
MGDA
2 (MTL-B)
ResNet-50
43.2
0.576
+0.02
192
2.145
MTAN
2F
ResNet-50
45.0
0.584
+1.32
320
5.036
Cross-Stitch
23
ResNet-50
44.2
0.570
+1.61
310
4,221
NDDR-CNI
26
ResNet-50
44.2
0.573
+1.38
340
4.244
M-VIT (MTL-B)
ViT-small
40.9
0.631
-6.27
2.097
M (+MoE)
MoE ViT-small
45.6
0,589
+1.59
100
8.189
V (+Moo+Coo-desinn)
MoE ViT-small
45.6
0.589
+1.59
100
0.845
4.2
Comparison with State-of-the-art Dense Prediction MTL
As target an efficient MTL system under the single-task inference setting, we conduct experiments on encoder-focused architectures (more details in Section 2). MTL-B (14) is vanilla multi-task learning baseline model which is composed of shared backbone in combination with task-specific heads. Several sate--of-theart (SoTA) encoder- focused MTL models, including MTAN [27]. Cross- Stitch (23] and NDDR-CV -NN (26), improve MTL-B by proposing feature sharing methods in the encoder. Our methods are all conducted on vanilla MTL-B, namely, applying MTL on VIT (M-ViT), adding task-dependent MoE design (MTViT), and adding hardware co-design on FPGA (M*) We also compare with previous works that handle the multi-task training conflicts problem, including uncertainty weighting (25), GradNorm (0), DWA (3), and MGDA 27), and they are evaluated on MTL-B. Single task learning baseline (STL-B) is used for MTL performance evaluation Am.

---

Table 3: Effect of task-dependent MoE design. MiViT-Single, M'ViT-Multi, and M cond. refer to the MTL MoE model with single router, multi routers, and task-conditioned router, respectively.
Seg. (mloUt)
Norm. (mErr)
H. Parts (mloU)f
Sal. (mloUyt
Edge
A
FLOPS
PASCAL-Context
(odsF)
(%)
(G) L
STL-B
66.2
13.9
59.9
66.3
68.8
0.00
167
M-ViT (MTL-B)
70.7
15.5
58.7
64.9
68.8
-1.76
83
Ving-li
71.5
14.8
61.2
65.9
71.5
+1.40
84
MOViT-Multi.
72.8
14.5
62.1
66.3
71.7
+2.71
84
VVViT--ask-cond.
72.0
14.4
61.3
65.8
71.8
+2.22
85
NYUD-v2
Seg. (mloUyt
Depth
FLOPS
(rmse)]
(%)
(G) L
STL-B
43.9
0.585
0.00
192
M-VIT (MTL-B)
40.9
0,631
-6.27
100
M ViT-Sinee
45.3
0,600
+0.31
100
ViT-MMuult.
45.6
0.589
+1.59
100
M WiT-TTask-cood
45.3
0.595
+0.74
101
As multi- gatlti-gateeoee MoE shows better performance than tondicond MoE, the reported M2 ViT and V results are based on the multi-gan galtti-gae design.
Results on PASCAL-Context Dataset As shown in Table even using vanilla MTL-B frame- work, introducing MoE (M² can achieve the highest performance over all previous encoder- focused works (+2.71 MTL performance); meanwhile, significantly reduces their single task inference FLOPs (particularly, reducing Cross-Stitch by 88%). Comparing against Uncertainty [25], DWA 533 GradNorm [20]. and MGDA (27). the superior performance of V demonstrates its strong capacity in handling training conflict. Moreover, leveraging the Model-Accelerator co-design helps to consume less than one-tenth the energy cost when deploying our model on FPGA. Some qualitative results are shown in Figure
Results on NYUD-v2 Dataset On this dataset, M can reduce previous SoTA's inference FLOPs by 68% while achieving comparable MTL performance. Introducing MoE to M-ViT helps to enlarge the model capacity without increasing inference FLOPs, which results in MTL performance boost from -6.27% +1.59% With Model-Accelerane co-design, also a nearly tenfold increase in energy efficiency. Results are shown in Table
4.3 Effect of Task-dependent MoE Design
To evaluate the effectiveness of our task-dependent MoE design, we compare between several models in Table including STL-B, MTL ViT (M-ViT), M with one gating function for all the tasks, V with multi gates, and ViT with task-conditioned token input. Results on both PASCAL- Context and NYUD-v2 datasets show that adding MoB layers into ViT with only one gating function for all tasks can already improve the M- ViT model. Making MoE selection task-dependent can further improve the performance, where multi-gating performs better than task-conditioned gating design. Particularly, comparing our model performance with STL-B as well as previous SoTAs in Table I and 2, we find that our MoE model better demonstrates its effectiveness when more tasks need to be encapsulated in the system. More results about our model's performance on different numbers of tasks can be found in the supplement.
4.4 Hardware Performance Results
Results comparing hardware performance metrics on FPGA and GPU are shown in Table We first discuss our memory efficiency. The naive approach described in Section 3.2 would require 11.610 MiB of on-chip memory (too much for our FPGA platform). but our compute-reordering design achieves the same result using only 4. 4.840 MiB, demonstrating a 2.40x reduction. Moreover, when comparing against memory-constrainee MoE ViT on FPGA without our compute-reordering method (using the cache-base based method described in Section 3.2h, we see that our method takes 9.23 less latency and energy the PASCAL-Contes dataset and 8.88 less latency and energy NYUD-v2.

---

Table Quantitative comparisons of hardware metries between FPGA and GPU implementations.
CR indicates usage of our memory-efficient computation reordering method (Section 3.2)
Memory (MiB)
PASCAL-Context
NYUD-v2
Platform
Backbone
CR
Latency (ms) Energy (W-s) Latency (ms) Energy (W-s)
GPU
ResNet- 18
21.336
3,489
1.029
GPU
ResNet-50
44.939
7.270
2.145
GPU
ViT-small
42.058
10.381
3.062
7.110
2.097
GPU
MoF ViT-small
82.747
25.239
7.446
27.760
8.189
FPGA
ViT-small
4.828
68.931
0.689
84.418
0.844
FPGA
MoE ViT-small
4.840
637.478
6.375
750.557
7.506
FPGA
MoE ViT-small
4.840
69.033
0.690
84.538
0.845
Our compute-reoree reordering ViT on FPGA also beats all GPU baselines in energy efficiency; e.g., beats MoE VIT on GPU by 10.79x on PASCAL-Context and by 9.69 on NYUD-v2. For discussion on the latency breakdown of ViT, please refer the supplement.
5 Conclusion, Discussion of Limitation and Broader Impact
In this paper, we propose MTL mixture-ou centure-offexte experts layers into a VIT backbone, we sparsely activate task-specific experts in training mitigate MTL gradient conflicts. For inference, we can activate only the sparse "expert" pathway relevant to the task of interest for efficiency, and can further achieve zero-overhead switching between tasks with our hardware-level co-design. Extensive experiments that show MT surpasses the pop-performing mncoer--ocuusee MTT. methods, reduces 88% FLOPs, and saves more than 8x energy over our baseline. The limitation of our work is that V so far mainly evaluated on academic datasets; will try real applications like autonomous driving in the future. For broader impact, our work can reduce the resource and energy consumption needed for MTL regimes, while still maintaining SOTA performance, which can effectively serve the goal of Green AI.
accelleace co-design for efficient on-device MTL. By customizing
Acknowledgment
Zhiwen Fan, Rishov Sarkar, Cong Hao and Zhangyang Wang are in part supported by the DARPA In-Pixel Intelligent Processing (IP2) program.
References
[1] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891-898, 2014.
[2] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pages 746-760. Springer, 2012.
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11923 2020.
[4] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficien image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10012-10022, 2021.
10

---

[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European conference on computer vision, pages 213-229. Springer, 2020.
[7] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense predic- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22179-12188 2021.
[8] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
[9] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[10] Dong-Gyu Lee. Fast drivable areas estimation with multi-task learning for real-time au- tonomous driving assistant. Applied Sciences, 11(22):10713, 2021.
[11] Wonsuk Kim and Junhee Scok. Indoor semantic segmentation for robot navigating on mobile. In 2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN), pages 22-25. IEEE, 2018.
[12] Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many- task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv: 1.015777 2016.
[13] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[14] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021.
[15] Kevin Swersky, Jasper Snock, and Ryan P Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 26, 2013.
[16] Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 3712-3722, 2018.
[17] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1930- 1939, 2018
[18] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv: 2015.
[19] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 1155824--836, 2020.
[20] Zhao Chen, Vijay Badrinarayanan, Chen-Yr Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803. PMLR, 2018
[21] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039 13:2039-2050, 2020.
[22] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.0587,, 2020.
11

---

[23] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994 4003, 2016.
[24] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270-287, 2018.
[25] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[26] Yuan Gao, Jiayi Ma. Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9205-3214, 2019.
[27] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 1871-1888 2019.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[29] Introducing Pathways: A next-generation Al architecture.
Sechmologyy/aiinttroee httprroddcciinnpphhhaaysseee pathways-next- next- neet-geeeeation ai wrw.itecthttrre////c
[30] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 776538638 2017.
[31] Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojeiech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895-9907, 2021.
[32] Yoshua Bengio. Deep learning of representations: Looking forward. In International confer- ence on statistical language and speech processing, pages 1-37. Springer, 2013.
[33] Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Learning Research, 8(1), 2007.
[34] Laurent Jacob, Jean-philippe Vert, and Francis Bach. Clustered multi-task learning: A convex formulation. Advances neural information processing systems, 21, 2008.
[35] Jiayu Zhou, Jianhui Chen, and Jieping Ye. Clustered multi-task learning via alternating structure optimization. Advances in neural information processing systems, 24, 2011.
[36] BJ Bakker and TM Heskes. Task clustering and gating for bayesian multitask learning. 2003.
[37] Kai Yu, Volker Tresp, and Anton Schwaighofer. Learning gaussian processes from multiple tasks. In Proceedings of the 22nd international conference on Machine learning, pages 1012-1019, 2005.
[38] Su-In Lee, Vassil Chatalbashev. David Vickrey, and Daphne Koller. Learning a meta-level prior for feature relevance from multiple related tasks. In Proceedings of the 24th international conference on Machine learning, pages 489-496, 2007.
[39] Hal Daumé III. Bayesian multitask learning with latent hierarchies. arXiv:0907.078. 2009.
arXiv preprint
[40] Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi-task learning. arXiv preprint arXiv:1206.6417, 2012.
12

---

[41] Sebastian Ruder. Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi- task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligncee volume 33, pages
2019.
[42] Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, and John Paisley. Joint es-mri reconstruc- tion and segmentation with unified deep network. In International conference on information processing in medical imaging, pages 492-504. Springer, 2019.
[43] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided recdiction-addddisilllnn network for simultaneous depth estimation and scene parsing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 675-684, 2018
[44] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattem-affinitive propagation across depth, surface normal and semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4106 4106-4115, 2019.
[45] Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv: 2017.
[46] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.115500 2019.
[47] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[48] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439-1449, 2021.
[49] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint urXiv:2111.111 14322, 2021.
[50] Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto. Task adaptive parameter sharing for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7561-7570, 2022.
[51] Zhenyu Zhang. Zhen Cui, Chunyan Xu. Zequn Jie, Xiang Li, and Jian Yang. Joint task- recursive learning for semantic segmentation and depth estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 235-251, 2018.
[52] Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. In European Conference on Computer Vision, pages 527-543. Springer, 2020.
[53] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
[54] Robert A Jacobs, Michael I
Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[55] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[56] Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural networks, 12(9):122--125) 1999.
[57] Seniha Esen Yuksel, Joseph N. Wilson, and Paul Gader. Twenty years of mixture of experts.
IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177-1199 2012.
13

---

[58] Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E Weston. Hash layers for large sparse models. In A. Beygelzimer, Y. Dauphin, Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[59] Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan. Tricks for training sparse translation models. arXiv preprint 10.00200 2021.
[60] David Eigen, Marc' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv: 111.44431, 2013.
[61] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[62] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03900 2021.
[63] Jakob Vogdrup Hansen. Combining predictors: comparison of five meta machine learning methods. Information Sciences, 19(1-2):91-105, 1999.
[64] Mike Lewis. Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pages 1265-6274, PMLR, 2021.
[65] Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXivv2202.01116, 2022.
[66] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large- scale image categorization. In European Conference on Computer Vision, pages 516-532. Springer, 2016.
[67] Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6865-6877 2017.
[68] Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in artificial intelligence, pages 552-562. PMLR, 2020.
[69] Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019.
[70] Alhabib Abbas and Yiannis Andreopoulos. Biased mixtures of experts: Enabling computer vision inference under data transfer limitations. IEEE Transactions on Image Processing, 29:7656-7667, 2020.
[71] Svetlana Pavlitskaya, Christian Hubschneider, Michael Weber, Ruby Moritz, Fabian Huger, Peter Schlicht, and Marius Zollner. Using mixture of expert models to gain insights into semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 342-343, 2020.
[72] Raquel Aoki, Frederick Tung, and Gabriel L Oliveira. Heterogeneous multi-task learning with expert diversity. arXiv preprint arXiv:2 106. 10595, 2021.
[73] Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong. and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34, 2021.
[74] Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages $124-2133
2016.
14

---

[75] Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint urXiv:1606.01933 2016.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 0008-600880000 2017.
[77] Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nla Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[78] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[79] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
[80] Kwonjoon Lee, Huiwen Chang. Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04589 2021.
[81] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[82] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988, 2021.
[83] Xiangxiang Chu, Bo Zhang, Zhi Tian, Xiaolin Wei, and Huaxia Xia. Do we really need explicit position encodings for vision transformers? arXiv e-prints, pages arXiv-2102, 2021.
[84] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint 1xx:x:2033.00 12, 2021.
[85] Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In International Conference on Learning Representattonss 2022.
[86] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequee
perspective with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 688 1-890, 2021.
[87] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-based object detection. arXiv preprint arXiv:2012.09955 2020.
[88] Kevin Lin, Lijuan Wang. and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1954-1963, 2021.
[89] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16259- 16268, 2021.
[90] Meng- Mao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7(2):187-199 2021.
[91] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ra- mamoorthi. Vision transformer for nerf-based view synthesis from single input image. arXiv preprint 2200:2207.007366 2022.
[92] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all nerf needs? arXiv e-prints, pages arXiv-2207, 2022.
15

---

[93] Fuzhi Yang. Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 5791-5800, 2020.
[94] Wenqing Zheng, Qiangqiang Guo, Hao Yang, Peihao Wang, and Zhangyang Wang. Delayed propagation transformer. universal computation engine towards practical control in cyber- physical systems. Advances in Neural Information Processing Systems, 34, 2021.
[95] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint
2019.
[96] Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Laurent Itti, and Vibhav Vineet. Dall-e for detection: Language-driven context image synthesis for object detection. arXiv preprint arXiv:2206.09592, 2022.
[97] Mengshu Sun, Haoyu Ma, Guoliang Kang, Yifan Jiang, Tianlong Chen, Xiaolong Ma, Zhangyang Wang. and Yanzhi Wang. Vaqf: Fully automatic boftware-hardware co-design framework for low-bit vision transformer. arXiv preprint arXiv:2201.00 2201006618, 2022.
[98] Bingbing Li, Santosh Pandey, Haowen Fang, Yanjun Lyv, Ji Li, Jieyang Chen, Mimi Xie, Lipeng Wan, Hang Liu, and Caiwen Ding. Ftrans: energy-efficient acceleration of transform- ers using fpga. In Proceedings of the ACM/EEE International Symposium on Low Power Electronics and Design, pages 175-180, 2020.
[99] Panjie Qi, Yuhong Song, Hongwu Peng, Shaoyi Huang, Qingfeng Zhuge, and Edwin Hsing- Mean Sha. Accommodating transformer onto fpga: Coupling the balanced model compression and ingaeemmplemnnnaiioon optimization. In Proceedings of the 2021 on Great Lakes Symposium on VLSI, pages 163-168, 2021.
[100] Hongwu Peng, Shaoyi Huang, Tong Geng, Ang Li, Weiwen Jiang, Hang Liu, Shusen Wang, and Caiwen Ding. Accelerating transformer-based deep learning models on fpgas using column balanced block pruning. In 202/ 22nd International Symposium on Quality Electronic Design (ISQED), pages 142-148. IEEE, 2021.
[101] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34, 2021.
[102] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1xXiv:1606.08411 2016.
[103] Chao Xiong, Xiaowei Zhao, Danhang Tang, Karlekar Jayashree, Shuicheng Yan, and Tae- Kyun Kim. Conditional convolutional neural network for modality- face recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 3667-3675, 2015.
[104] Jonathan Pilault, Amine Elhattami, and Christopher Pal. Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters & less data. arXiv preprint 0tte:20090091333 2020.
[105] Xilinx. Specifying arrays as ping-pong buffers or FIFOs. In Vitis High-Level Synthesis User Guide, pages 326-327. 2021.1 edition, August 2021.
[106] Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Luc Van Gool. Task switching network for multi-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8291-8300 2021.
[107) David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE transactions on pattern analysis and machine intelligence, 26(5):530-549, 2004.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] please see section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] please see section [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our work does not contain theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] Our work does not contain theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] We used publicly available data in all our experiments. Meanwhile we either provide the detailed implementations or cite the papers of them following the authors instructions (please refer to our implementation details and appendix). All of our codes will be released.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We provides all the training details in appendix.
(c) Did you report error bars (c.g.. with respect the random seed after running experi- ments multiple times)? [Yes] We test our model on PASCAL-Context and NYUD-v2 for three times and report their mean results.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We describe the details of computation resources the appendix.
4. If you using existing assets (e.g., code, data, models) curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used publicly available data, i.e., PASCAL-Context, in our experiments. We cited the corresponding papers published by the creators.
(b) Did you mention the license of the assets? [N/A] The license of ImageNet, PASCAL- Context, NYUD-v2 are included the paper that we have cited.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The ImageNet we used are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you're data.
using/curating? [N/A] We did not collect/curate
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] ImageNet and the used MTL datasets are already publicly available and broadly adopted. We do not think there are any issues of personally identifiable information or offensive content
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "r9cUm6ih1ZV", "reviewer": "Reviewer_V1Gi", "summary": "The paper presents a multi-task MoE and model-accelerator co-design. At the algorithm level, the method adopts a task-level mixture-of-expert approach that can effectively reduce training and inference cost. At the model-accelerator level, the paper proposes a computation reordering scheme that is tailored for memory-constrained MTL and achieves little switching overheads. The paper observes significant inference FLOPs reduction and memory requirement reduction, compared to a FPGA baseline.", "questions": "1. In table 1, did you implement all the related work similarly on FPGAs? How did you generate the energy numbers? If all the baselines are also implemented in FPGA, then the quality of models are largely determined by implementation not necessarily by the method itself. \n2. Two out of six tasks are worse than cross-stitch, any reasons?\n3. Why make the total FLOPs only half of the baseline models (Table 1)? A fair comparison would be with similar FLOPs and compare the accuracy.", "limitations": "Yes. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper has a very thorough and complete related work and reference section. \n- Multi-task MoE is an important field that generates numerous important works. This paper seems to be a right direction addressing important problems. Co-designing a software/hardware systems can be the right problem to solve. \n- The proposed reordering reduces memory footprint at small overheads. \n\nWeaknesses:\n- The paper is rephrasing many contributions of conditional computation/mixture-of-experts and multi-task MoE. Those contributions are not unique contributions from this paper. For example, token-MoE has been prevalent for a long time and more recently, task-MoE has been demonstrated useful in many tasks including machine translations (e.g. Machine translation from Google). [1] \n\n- The proposed reordering scheme is a system-level contribution which might not best suited for NeurIPS. Even though, there is a co-design claim, the reviewer feels that task-level MoE and and hardware execution reordering should be decoupled. The same reordering scheme can be applied to other types of networks equally. In order to prove co-design is effective, some co-optimization should be applied which can require some joint search space on the MoE model and reordering scheme. The paper should show that co-designing these two parts can be useful and is better than separately optimizing each of them.\n\n- The paper listed challenges round \"handle many different tasks\" and \"switch between tasks\" and how different tasks can interfere each other when training on a shared model backbone. An example given in the paper is the autonomous driving setting where potentially hundreds of tasks are running on the shared model backbone. However, the paper does not prove it's own point by evaluating only on a small set of tasks. \n\n[1] \"Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference\", https://aclanthology.org/2021.findings-emnlp.304.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XgORurZT_Ud", "reviewer": "Reviewer_YbW8", "summary": "In this paper, the authors propose to solve the gradient conflict in training and dense activation in inference in multi-task learning using a co-designed mixture-of-expert model. In algorithm level, the authors propose to replace ViT layer with mixture of expert layer and sparsely activate them during training to alleviate the gradient conflict issue, while during inference, the author propose a novel computation reordering scheme to better support the sparse activation for different tasks. Experiments on GPU and FPGA show the proposed co-design achieves better performance than existing methods. ", "questions": "- How are different experts shared between different tasks? Will the scaling in task number requires more experts to maintain a satisfied performance? \n- What is the training cost of the proposed model compared with baseline methods? Does it need extra training to learn the expert-task mapping and fully train the large number of experts? \n- What is the performance improvement breakdown for the proposed hardware design? ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The co-design for Mixture-of-Expert to enable multi-task learning is a timely attempt. \n- The achieved results is promising in both accuracy and efficiency. \n\nWeaknesses:\n- Using mixture-of-expert to deal with multi-task learning is not the novel idea, but the co-design attempt is interesting to me. \n- Is the mentioned double buffered computation strategy similar to Ping-pong buffer? If so it would be better to add the reference. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L8Al9lhUbm5", "reviewer": "Reviewer_SvCo", "summary": "The paper describes an efficient multi-task neural network for vision applications that can be deployed on resource-constrained devices. The model consists of a mixture of vision transformer experts that are sparsely activated by a GELU activation function. The main innovation is that mixture of expert layers are computed expert-by-expert instead of token-by-token, which reduces latencies of loading expert weights.", "questions": "* How does the model accuracy and efficiency (Latency, Energy, Memory) depend on the total number of experts N and top experts K?\n* Is the softmax in equation 3 needed, given that you are only interested in the top-k activations?\n* How are the top-k experts computed at training and inference time? What is the runtime?", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* Computing mixture-of-expert layers seems like a simple yet effective why to increase the efficiency of a vision transform mixture of expert architecture\n* The paper is well well\n* Experiments are solid\n\n## Weaknesses\n* Only two datasets (NYUD, PASCAL) are considered in experiments--no real world datasets", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G_NAXruH3I2", "reviewer": "Reviewer_DMLe", "summary": "This paper studies a novel setting of MTL: training with multiple tasks hoping them to boost each other, and inference with a single task each time. The setting is practically convincing as most MTL systems switch between tasks instead of executing all tasks at the same time. The authors presented a model-accelerator co-design framework, involving both algorithm and hardware innovations.", "questions": "Please see the weakness part.", "limitations": "No, the current version claimed it has no negative social impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points:\n1. The overall problem and idea are novel and interesting. Most MTL papers assume all tasks to be trained and tested in the same bundle. This paper allows for more flexibility. \n2. The introduction MoE is well motivated from two aspects: efficiency for single-task inference (sparse activation); and avoiding MTL training conflicts (grouping similar modules). Using MoE is hence natural in this new setting, and to my best knowledge, is novel for the MTL field. The authors also designed a task-dependent gating network for this new purpose.\n3. The authors meanwhile present interesting hardware innovations. To my best knowledge, not too many hardware works were done in the MTL domain. This work proposed a novel computation reordering mechanism tailored for memory-constrained MTL and MoE, which allows scaling up to any number of experts and also achieves zero-overhead switching between tasks. It sounds very promising and might be of independent research interest. \n4. Experiments are solid, involving customized FPGA design on real hardware, and evaluating two major MTL datasets (NYUD-v2, PASCAL-Context). The proposed co-design can save memory and energy costs by up to one order of magnitude, with no worse accuracies than baselines. Besides, more tasks seem to save more (PASCAL versus NYUD), which shows great scalability to larger numbers of tasks.\n5. The abstract and intro were written elegantly to give a proper big picture. The authors also did a good job in making figures. As another plus, codes are provided in the supplementary: I did not have time to carefully check but from quick reading, the code quality looks fine.\n\n\nWeak points:\n1. My main concern is that the MTL baselines used are all somehow weak. Even though the problem setting is new, and I know not many peer algorithms did exactly the same job, there are still some. For example, the authors should check “Task Adaptive Parameter Sharing for Multi-Task Learning” (CVPR 2022). That paper has important similarity to this submission, in learning modularized models for MTL while allowing for single-task inference. Although it does not use dynamic MoE and has no hardware co-design, I believe some level of comparison and discussion is necessary. There might be more in literature. \n2. One other baseline-related issue is that the authors only compare with encoder-focused MTL models, and claim that is because decoder-focused ones are in general heavier (but also, more performant). I think the authors should at least compare with or discuss some decoder-focused methods and demonstrate the trade-offs here, rather than completely ignoring. For example, is it because decoder-focused MTL models cannot fit into FPGA, really? \n3. Why jumping to a ViT backbone? Looking at Table 1, the ViT baseline is already stronger than most prior works built on ResNet 18 (as expected). Can you compare with other MTL works with the same ViT backbone, or can you replace ViT with Res18 in your co-design framework? That would help disentangle your own contributions.\n4. If some MTL applications do require to activate all or most tasks all the time, will your framework still be advantageous? For example, what if always executing 2, 3, or 5 tasks simultaneously on PASCAL? I understand this will change your problem target, but I suggest the authors to include some discussions and clarity for future readers. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept