SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then we proceed to bound  $S_{1,T}, S_{3,T}, S_{4,T}$ . Under either Assumptions 1, 3, or Assumptions 1, 2 with  $\ell_f$ ,  $\ell_F$  defined in Lemma 1, we have that for all  $z \in S$  and  $\lambda \in \Delta^M$ ,  $\| \nabla F_z(x_t)\lambda \| \leq \ell_f$ , and  $\| \nabla F_z(x_t) \| \leq \ell_F$ . Then  $S_{1,T}, S_{3,T}, S_{4,T}$  can be bounded below
$$
S _ {1, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2} \leq M \ell_ {f} ^ {4} \tag {C.42a}
$$
$$
S _ {3, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t}) \lambda_ {t} \| \| \nabla F _ {S} (x _ {t}) ^ {\top} \nabla F _ {S} (x _ {t}) \lambda_ {1} \| \leq \ell_ {F} ^ {2} \ell_ {f} ^ {2} = M \ell_ {f} ^ {4} \tag {C.42b}
$$
$$
S _ {4, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {z t, 3} (x _ {t}) \lambda_ {t + 1} \| ^ {2} \leq \ell_ {f} ^ {2} \tag {C.42c}
$$
which proves that
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} ^ {*} (x _ {t}) \| ^ {2} \leq \frac {1}{\alpha T} c _ {F} + \frac {3}{2} \gamma M \ell_ {f} ^ {4} + \frac {1}{2} \alpha \ell_ {f, 1} \ell_ {f} ^ {2}. \tag {C.43}
$$
We arrive at the results by  $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_A\|\nabla F_S(x_t)\lambda_t^*(x_t)\| \leq \left(\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_A\|\nabla F_S(x_t)\lambda_t^*(x_t)\|^2\right)^{\frac{1}{2}}$  from the Jensen's inequality and the convexity of the square function, as well as the subadditivity of square root function.
(a) MGDA
(b) Static
(c) MoDo
Figure 5: Convergence of MGDA, static weighting and MoDo to the empirical (gray, upper) and population (green, lower) Pareto fronts. Horizontal and vertical axes in figures in the first / second row are the values of the two empirical / population objectives. Three colormaps are used for the trajectories from three initializations, respectively, where the same colormaps represent the trajectories of the same initializations, and darker colors in one colormap indicate earlier iterations and lighter colors indicate later iterations.
D Additional experiments and implementation details
Compute. Experiments are done on a machine with GPU NVIDIA RTX A5000. We use MATLAB R2021a for the synthetic experiments in strongly convex case, and Python 3.8, CUDA 11.7, Pytorch 1.8.0 for other experiments. Unless otherwise stated, all experiments are repeated with 5 random seeds. And their average performance and standard deviations are reported.
D.1 Synthetic experiments
D.1.1 Experiments on strongly convex objectives
Implementation details. Below we provide the details of experiments that generate Figure 3. We use the following synthetic example for the experiments in the strongly convex case. The  $m$ -th objective function with stochastic data sample  $z$  is specified as
$$
f _ {z, m} (x) = \frac {1}{2} b _ {1, m} x ^ {\top} A x - b _ {2, m} z ^ {\top} x \tag {D.1}
$$
where  $b_{1,m} > 0$  for all  $m \in [M]$ , and  $b_{2,m}$  is another scalar. We set  $M = 3$ ,  $b_1 = [b_{1,1}; b_{1,2}; b_{1,3}] = [1;2;1]$ , and  $b_2 = [b_{2,1}; b_{2,2}; b_{2,3}] = [1;3;2]$ . The default parameters are  $T = 100$ ,  $\alpha = 0.01$ ,  $\gamma = 0.001$ . In other words, in Figure 3a, we fix  $\alpha = 0.01$ ,  $\gamma = 0.001$ , and vary  $T$ ; in Figure 3b, we fix  $T = 100$ ,  $\gamma = 0.001$ , and vary  $\alpha$ ; and in Figure 3c, we fix  $T = 100$ ,  $\alpha = 0.01$ , and vary  $\gamma$ .
D.1.2 Experiments on nonconvex objectives
Implementation details. The toy example is modified from [29] to consider stochastic data. Denote the model parameter as  $x = [x_1, x_2]^\top \in \mathbb{R}^2$ , stochastic data as  $z = [z_1, z_2]^\top \in \mathbb{R}^2$  sampled from the standard multi-variate Gaussian distribution. And the individual empirical objectives are defined as:
$$
f _ {z, 1} (x) = c _ {1} (x) h _ {1} (x) + c _ {2} (x) g _ {z, 1} (x) \text {a n d} f _ {z, 2} (x) = c _ {1} (x) h _ {2} (x) + c _ {2} (x) g _ {z, 2} (x), \text {w h e r e}
$$
$$
h _ {1} (x) = \log (\max (| 0. 5 (- x _ {1} - 7) - \tanh (- x _ {2}) |, 0. 0 0 0 0 0 5)) + 6,
$$
$$
h _ {2} (x) = \log (\max  \left(\left| 0. 5 \left(- x _ {1} + 3\right) - \tanh  \left(- x _ {2}\right) + 2 \right|, 0. 0 0 0 0 0 5\right)) + 6,
$$
(a)  $\gamma = 0$
(b)  $\gamma = 10^{-12}$
(c)  $\gamma = 10^{-4}$
Figure 6: Trajectories of MoDo under different  $\gamma$  on the contour of the average of objectives. The black  $\clubsuit$  marks initializations of the trajectories, colored from red (start) to yellow (end). The background solid/dotted contours display the landscape of the average empirical/population objectives. The gray/green bar marks empirical/population Pareto front, and the black  $\star$ /green  $\star$  marks solution to the average objectives.
(d)  $\gamma = 10^{-2}$
$$
\begin{array}{l} g _ {z, 1} (x) = \left(\left(- x _ {1} + 3. 5\right) ^ {2} + 0. 1 * \left(- x _ {2} - 1\right) ^ {2}\right) / 1 0 - 2 0 - 2 * z _ {1} x _ {1} - 5. 5 * z _ {2} x _ {2}, \\ g _ {z, 2} (x) = \left((- x _ {1} - 3. 5) ^ {2} + 0. 1 * (- x _ {2} - 1) ^ {2}\right) / 1 0 - 2 0 + 2 * z _ {1} x _ {1} - 5. 5 * z _ {2} x _ {2}, \\ c _ {1} (x) = \max  (\tanh  (0. 5 * x _ {2}), 0) \text {a n d} c _ {2} (x) = \max  (\tanh  (- 0. 5 * x _ {2}), 0). \tag {D.2} \\ \end{array}
$$
Since  $z$  is zero-mean, the individual population objectives are correspondingly:
$$
f _ {1} (x) = c _ {1} (x) h _ {1} (x) + c _ {2} (x) g _ {1} (x) \text {a n d} f _ {2} (x) = c _ {1} (x) h _ {2} (x) + c _ {2} (x) g _ {2} (x), \text {w h e r e}
$$
$$
g _ {1} (x) = \left(\left(- x _ {1} + 3. 5\right) ^ {2} + 0. 1 * \left(- x _ {2} - 1\right) ^ {2}\right) / 1 0 - 2 0,
$$
$$
g _ {2} (x) = \left(\left(- x _ {1} - 3. 5\right) ^ {2} + 0. 1 * \left(- x _ {2} - 1\right) ^ {2}\right) / 1 0 - 2 0. \tag {D.3}
$$
The training dataset size is  $n = |S| = 20$ . For all methods, i.e., MGDA, static weighting, and MoDo, the number of iterations is  $T = 50000$ . The initialization of  $\lambda$  is  $\lambda_0 = [0.5, 0.5]^\top$ . The hyperparameters for this experiment are summarized in Table 5.
Figure 5, in addition to Figure 1, shows the trajectories of different methods from different initializations to the empirical and population Pareto fronts (PF). With the visualized empirical and population PFs, it is clear in Figure 5a, the first row, that the three trajectories of MGDA all converge to the empirical PF, but, it stops updating the model parameter as soon as it reaches the empirical PF. However, due to the difference between the empirical and population PFs caused by finite stochastic training data, as shown in Figure 5a, the second row, not all three solutions from MGDA has small population risk, implied from the distance of the solution (yellow point) to the population PF colored in green. For static weighting method with uniform weights in Figure 5b, one trajectory is able to converge to the center of the empirical PF, which is the optimal solution of the uniform average of the two objectives. However, the other two get stuck and oscillate around suboptimal parameters for a long time, corresponding to the clusters of scattered points in the figure. Nevertheless, in the second row of Figure 5b, one empirically suboptimal solution (on the trajectory with red to yellow colormap) is able to achieve small population risk. This example demonstrates that even though static weighting method does not have small distance to CA direction, it might still be able to achieve small testing error. Finally, for MoDo in Figure 5c, the first row shows that MoDo is slower than MGDA in convergence to the empirical PF, since it only approximately solves the CA direction using
Table 5: Summary of hyper-parameter choices for nonconvex synthetic data.
Static	MGDA	MoDo
optimizer of xt	Adam	Adam	Adam
xt step size (αt)	5 × 10-3	5 × 10-3	5 × 10-3
λt step size (γt)	-	-	10-4
batch size	16	full	16
(c)  $\gamma = 10^{-4}$
Figure 7: Convergence of MoDo to the empirical (gray, upper) and population (green, lower) Pareto fronts under different  $\gamma$ . Horizontal and vertical axes in figures in the first / second row are the values of the two empirical / population objectives. Three colormaps are used for the trajectories from three initializations, respectively, where the same colormaps represent the trajectories of the same initializations, and darker colors in one colormap indicate earlier iterations and lighter colors indicate later iterations.
stochastic gradient. The second row shows that all three solutions of MoDo can achieve relatively small population risk, demonstrating a good generalization ability.
To demonstrate how the choice of  $\gamma$  has an impact on the performance of MoDo, we further conduct experiments with different  $\gamma$ , and show that MoDo can recover static weighting ( $\gamma = 0$ ) and approximate MGDA ( $\gamma = 10^{-2}$ ). Results are visualized in Figure 6, which plots the trajectories over the contours of the average objectives, and Figure 7, which plots the empirical and population PFs. Figure 6a and Figure 7a together verify that MoDo with  $\gamma = 0$  behaves the same as static weighting, cf. Figure 1d and Figure 5b. Like static weighting, for two of the initializations, they either cannot go through the valley or stuck in the valley for a long time. And if the trajectory converges, it will converge to the optimal solution of the average empirical objectives. In addition, Figure 6d and Figure 7d together indicate that MoDo with  $\gamma = 10^{-2}$  behaves similarly as MGDA, cf. Figure 1c and Figure 5a. Like MGDA, it can go through the valley without getting stuck in it for a long time, and then converge to the empirical PF for all three initializations. But also like MGDA, it will stop updating or oscillating around the parameters as soon as it reaches the empirical PF, resulting in a worse population risk for one of the solutions with a trajectory colored from purple to pink to yellow. We also conduct experiments on  $0 < \gamma < 10^{-2}$ . With  $\gamma = 10^{-12}$ , as shown in Figures 6b, and 7b, MoDo is able to go through the valley and converge to the empirical PF for all initializations, but very slowly. And all initializations converge to the optimal solution of the average objectives. While with  $\gamma = 10^{-4}$ , as shown in Figures 6c, and 7c, MoDo is able to go through the valley and converge to the empirical PF for all initializations quickly. And it stops updating the parameters as soon as it reaches the empirical PF. The solutions of the three initializations are different from the optimal solution of the average objectives. But compared to the case when  $\gamma = 10^{-2}$ , the three solutions are closer to the optimal solution of the average objectives. This is because the weighting parameter  $\lambda$  does not change too much during the optimization procedure with a small  $\gamma$ .
D.2 Multi-task supervised learning experiments
In this section we present experiment details and additional results for comparison of static weighting, MGDA, and MoDo algorithms, under synthetic and real world multi-task supervised learning problems. We use MNIST, Office-31, Office-home, and NYU-v2 datasets.
D.2.1 MNIST dataset experiments
Implementation details. Below we provide the details and additional experimental results on MNIST image classification. We simulate a synthetic multi-objective optimization problem using
different loss functions applied for training an image classifier for MNIST handwritten digit dataset. We consider three loss functions: cross entropy, mean squared error (MSE), and Huber loss. The model architecture is a two-layer multi-layer perceptron (MLP). Each hidden layer has 512 neurons, and no hidden layer activation. The input size is 784, and the output size is 10, the number of digit classes. The training, validation, and testing data sizes are  $50\mathrm{k}$ ,  $10\mathrm{k}$ , and  $10\mathrm{k}$ , respectively. Hyperparameters such as step sizes are chosen based on each algorithm's validation accuracy performance, as given in Table 6.
We then discuss the results of the experiments in Table 7, which shows the performance of the last iterate for each method. Experiments are repeated 10 times with average performance and standard deviations reported. Observed from Table 7, for cross-entropy loss, MGDA performs the worst while static weighting performs the best. On the other hand, for Huber loss, MGDA performs the best while static weighting performs the worst. This is not surprising as cross-entropy loss has the largest scale and Huber loss has the smallest scale among the three losses. Since equal weights are assigned for all three objectives in the static weighting method, it tends to focus more on optimizing the loss with the largest scale. While for MGDA, it is the other way around. Compared to MGDA, MoDo performs much better on cross-entropy loss, and in the meantime, it achieves comparable performance on Huber loss. Comparing their PS population risks  $R_{\mathrm{pop}}$  and the decomposed PS optimization errors  $R_{\mathrm{opt}}$  and PS generalization errors  $R_{\mathrm{gen}}$ , MGDA has the smallest PS population risk and PS optimization error. One potential reason is that MGDA performs best on Huber loss, with smaller gradients. Another reason is that the generalization errors for all the algorithms are similar and not dominating compared to optimization errors in this setting, making the PS population risk close to the PS optimization error. Overall, MoDo demonstrates a balance among objectives with different scales and performs reasonably well on all three objectives since it combines the properties of static weighting and MGDA.
Table 6: Summary of hyper-parameter choices for MNIST image classification
Static	MGDA	MoDo
optimizer of xt	SGD	SGD	SGD
xt step size (αt)	0.1	5.0	1.0
λt step size (γt)	-	-	1.0
batch size	64	64	64
Table 7: MNIST classification with cross-entropy, MSE, and Huber loss as objectives.
Method	Cross-entropy	MSE loss	Huber loss	Rpop(10-3)↓	Ropt(10-3)↓	|Rgen|(10-3)↓
Loss (10-3)↓	Loss (10-3)↓	Loss (10-3)↓	Loss (10-3)↓
Static	306.9±3.9	13.2±0.14	2.2±0.03	2.1±0.56	1.9±0.5	0.2±0.19
MGDA	363.6±4.1	13.5±0.13	1.9±0.01	1.3±0.24	1.1±0.2	0.2±0.13
MoDo	317.9±3.4	13.1±0.13	2.1±0.05	2.1±0.38	1.9±0.4	0.1±0.09
D.2.2 Office-31 and Office-home dataset experiments
Implementation details. We give the details for the experiments conducted using Office-31 and Office-home datasets, which consist of multi-domain image classification tasks. Both of these are multi-input single-task learning problems. Office-31 and Office-home consist of 31 and 65 image classes, respectively. The image domains for Office-31 are; "Amazon", which includes object images from Amazon, "DSLR", which includes high-resolution images of objects, and "Webcam", which includes low-resolution images from objects. The image domains for Office-31 are "Art", "Clipart", "Product", and "Real-world" which include images of objects taken from the respective image domains. We use the MTL benchmark framework LibMTL [26] to run experiments on both of the aforementioned datasets. For both datasets, we tune the step size of  $x$  and weight decay parameters for Static and MGDA algorithms and tune the step sizes of  $x$  and  $\lambda$  for the MoDo algorithm. We use batch size 64 to update static weighting and MGDA, and use 2 independent samples of batch size 32 to update MoDo, for both Office-31 and Office-home. A summary of hyper-parameters used for
Table 8: Summary of hyper-parameter choices for Office-31 task
Static	MGDA	MoDo
optimizer of xt	Adam	Adam	Adam
xt step size (αt)	10-4	10-4	10-4
λt step size (γt)	-	-	10-3
weight decay	10-3	10-7	10-5
batch size	64	64	64
Office-31 and Office-home for each algorithm are given in Table 8 and Table 9, respectively. All other experiment setup is shared for all algorithms, and the default LibMTL configuration is used.
The results of Office-31 and Office-home experiments are given in Tables 3 and 10, respectively (average over 5 seeds, the error indicates standard deviation). Here, we use the average per-task performance drop of metrics  $S_{A,m}$  for method  $\mathcal{A}$  with respect to corresponding baseline measures  $S_{B,m}$  as a measure of the overall performance of a given method. Specifically, this measure is
$$
\Delta \mathcal {A} \% = \frac {1}{M} \sum_ {m = 1} ^ {M} (- 1) ^ {\ell_ {m}} \left(S _ {\mathcal {A}, m} - S _ {\mathcal {B}, m}\right) / S _ {\mathcal {B}, m} \times 100, \tag{D.4}
$$
where  $M$  is the number of tasks. Here,  $\ell_{m} = 1$  if higher values for  $S_{A,m}$  are better and 0 otherwise. We use the best results for each task obtained by dedicated independent task learners of each task as  $S_{B,m}$ . The independent task learners are tuned for the learning rate and weight decay parameter. For Office-31,  $S_{B,m}$  values are  $87.50\%$  for "Amazon",  $98.88\%$  for "DSLR", and  $97.32\%$  for "Webcam". For Office-home,  $S_{B,m}$  values are  $66.98\%$  for "Art",  $82.02\%$  for "Clipart",  $91.53\%$  for "Product", and  $80.97\%$  for "Real-world". It can be seen from Tables 3 and 10 that static weighting outperforms MGDA method in some tasks and also in terms of  $\Delta A\%$ . However, by proper choices of hyperparameters, MoDo performs on par or better compared to both static weighting and MGDA, and hence achieves the best overall performance in terms of  $\Delta A\%$ .
Table 9: Summary of hyper-parameter choices for Office-home task
Static	MGDA	MoDo
optimizer of xt	Adam	Adam	Adam
xt step size (αt)	10-4	10-4	10-4
λt step size (γt)	-	-	10-3
weight decay	10-3	10-6	10-5
batch size	64	64	64
Table 10: Classification results on Office-home dataset.
Method	Art	Clipart	Product	Real-world	ΔA% ↓
Test Acc ↑	Test Acc ↑	Test Acc ↑	Test Acc ↑	Test Acc ↑	Test Acc ↑
Static	64.14 ± 1.40	79.57 ± 1.09	90.00 ± 0.50	78.94 ± 0.87	2.85 ± 1.08
MGDA	61.71 ± 1.33	73.95 ± 0.43	90.17 ± 0.27	79.35 ± 1.15	5.29 ± 0.47
MoDo	65.50 ± 0.55	79.44 ± 0.29	89.72 ± 0.94	79.65 ± 0.67	2.24 ± 0.48
D.2.3 NYU-v2 dataset experiments
Implementation details. We give the details for the experiments conducted using NYU-v2 dataset, which consists of image segmentation, depth estimation, and surface normal estimation tasks. Unlike Office-31 and Office-home datasets, this is a single-input multi-task learning problem. The
dataset consists of images from indoor video sequences. We use the MTL benchmark framework LibMTL [26] to run experiments with this dataset. We tune step size of  $x$  and weight decay parameters for static weighting and MGDA algorithms, and tune step sizes of  $x$  and  $\lambda$  for MoDo algorithm. We use batch size 4 to update static weighting and MGDA, and use 2 independent batches of size 2 to update MoDo. Experiments were run for 50 epochs for all methods. Since there was no validation set for NYU-v2, we averaged and reported the test performance of the last 10 epochs. A summary of hyper-parameters used for each algorithm is given in Table 11. All other experiment setup is shared for all algorithms, and the default LibMTL configuration is used.
Table 11: Summary of hyper-parameter choices for NYU-v2 task
Static	MGDA	MoDo
optimizer of xt	Adam	Adam	Adam
xt step size (αt)	10-4	10-4	10-4
λt step size (γt)	-	-	10-3
weight decay	10-4	10-6	10-5
batch size	4	4	4
Table 12: Segmentation, depth, and surface normal estimation results on NYU-v2 dataset.
Method	Segmetation	Depth	Surface Normal	ΔA% ↓
(Higher Better)	(Lower Better)	Angle Distance (Lower Better)	Within t° (Higher better)
mIoU	Pix Acc	Abs Err	Rel Err	Mean	Median	11.25	22.5	30
Static	52.02 ± 0.69	74.21 ± 0.57	0.3984 ± 0.0032	0.1645 ± 0.0010	23.79 ± 0.10	17.44 ± 0.15	34.07 ± 0.17	60.17 ± 0.31	71.48 ± 0.29	3.98 ± 0.70
MGDA	46.39 ± 0.17	70.27 ± 0.24	0.4269 ± 0.0024	0.1737 ± 0.0009	22.34 ± 0.03	15.70 ± 0.08	37.71 ± 0.21	63.96 ± 0.11	74.50 ± 0.06	6.25 ± 0.38
MoDo	52.64 ± 0.19	74.67 ± 0.08	0.3984 ± 0.0020	0.1649 ± 0.0018	23.45 ± 0.06	17.09 ± 0.05	34.79 ± 0.11	60.90 ± 0.13	72.12 ± 0.11	3.21 ± 0.34
The results of NYU-v2 experiments are given in Tables 12 (average over 3 seeds, the error indicates standard deviation). Again, we use the average per-task performance drop of metrics  $S_{\mathcal{A},m}$  for method  $\mathcal{A}$  with respect to corresponding baseline measures  $S_{\mathcal{B},m}$  as a measure of the overall performance of a given method. We use the best results for each task obtained by dedicated independent task learners of each task as  $S_{\mathcal{B},m}$ . The independent task learners are tuned for the learning rate and weight decay parameter. For segmentation task,  $S_{\mathcal{B},m}$  values are 53.94% for "mIoU", and 75.67% for "Pix Acc". For depth estimation task,  $S_{\mathcal{B},m}$  values are 0.3949 for "Abs Err", and 0.1634 for "Rel Err". For surface normal estimation task,  $S_{\mathcal{B},m}$  values are 22.12 for "Angle Distance - Mean", 15.49 for "Angle Distance - Median", 38.35% for "Within 11.25°", 64.30% for "Within 22.5°", and 74.70% for "Within 30°". It can be seen from Table 12 that MoDo outperforms both MGDA and static weighting in some tasks and also in terms of  $\Delta A\%$ . The worse performance of MGDA in terms of  $\Delta A\%$  can be due to the large bias towards the Surface normal estimation task, which seems to affect other tasks adversely.
D.2.4 Additional experiments for comparison with other MOL baselines
In this section, we provide a comparison between MoDo and other popular MOL baselines. For this purpose, we use the same benchmark datasets as the previous section and use the experiment setup provided in [25] to run experiments with MoDo. Hence, we use experiment results provided by [25] for other baselines for comparison. Additionally, we implement MoCo [10], which is not included in [25]. For results in this section we report two holistic measures:  $\Delta A_{\mathrm{st}}$ , which measures performance degradation w.r.t. static scalarization (similar to [25], but lower the better), and  $\Delta A_{\mathrm{id}}$  which measures performance degradation w.r.t. independent task learners (as defined in our paper)
The results on Office-31, Office-home, and NYU-v2 are given in Tables 13, 14, and 15, respectively, where MoDo outperforms all the baselines for most tasks, and has a better overall performance in  $\Delta A_{\mathrm{st}}\%$  and  $\Delta A_{\mathrm{id}}\%$ . The hyper-parameters of MoDo for the above experiments are in Table 16.
Table 13: Comparison with other methods on Office-31 dataset.
Method	Amazon	DSLR	Webcam	ΔAst% ↓	ΔAid% ↓
Static (EW)	81.02	96.72	96.11	0.00	2.96
MGDA-UB [39]	81.02	95.90	97.77	0.40	3.32
GradNorm [6]	83.93	97.54	94.44	-0.19	2.80
PCGrad [48]	82.22	96.72	95.55	0.40	3.35
CAGrad [29]	82.22	96.72	96.67	0.01	2.96
RGW [25]	84.27	96.72	96.67	-0.81	2.18
MoCo [10]	85.30	97.54	97.22	-1.70	1.32
MoDo (ours)	85.47	98.36	96.67	-1.86	1.17
Table 14: Comparison with other methods on Office-home dataset.
Method	Art	Clipart	Product	Real-world	ΔAt% ↓	ΔId% ↓
Static (EW)	62.99	76.48	88.45	77.72	0.00	5.02
MGDA-UB [25]	64.32	75.29	89.72	79.35	-1.02	4.04
GradNorm [6]	65.46	75.29	88.66	78.91	-1.03	4.04
PCGrad [48]	63.94	76.05	88.87	78.27	-0.53	4.51
CAGrad [29]	63.75	75.94	89.08	78.27	-0.48	4.56
RGW [25]	65.08	78.65	88.66	79.89	-2.30	2.85
MoCo [10]	64.14	79.85	89.62	79.57	-2.48	2.68
MoDo (ours)	66.22	78.22	89.83	80.32	-3.08	2.11