SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In short, though it is possible to derive fast approximation algorithms to approximate the gradient update on  $w$  for some of the existing gradient manipulation methods, it often involves much more complicated computation compared to that of FAMO.
C FAMO Pseudocode in PyTorch
We provide the pseudocode for FAMO in Algorithm 2. To use FAMO, one just first compute the task losses, call get_weighted_loss to get the weighted loss, and do the normal backpropagation through the weighted loss. After that, one call update to update the task weighting.
D Toy Example
We provide the task objectives for the toy example in the following. The model parameter  $\theta = (\theta_{1},\theta_{2})\in \mathbb{R}^{2}$  and the task objectives are  $L^1$  and  $L^2$ :
$$
L ^ {1} (\theta) = 0. 1 \cdot \left(c _ {1} (\theta) f _ {1} (\theta) + c _ {2} (\theta) g _ {1} (\theta)\right) \text {a n d} L ^ {2} (\theta) = c _ {1} (\theta) f _ {2} (\theta) + c _ {2} (\theta) g _ {2} (\theta), \text {w h e r e}
$$
$$
f _ {1} (\theta) = \log \left(\max  \left(| 0. 5 (- \theta_ {1} - 7) - \tanh  (- \theta_ {2}) |, 0. 0 0 0 0 0 5\right)\right) + 6,
$$
$$
f _ {2} (\theta) = \log \left(\max  \left(| 0. 5 (- \theta_ {1} + 3) - \tanh  (- \theta_ {2}) + 2 |, 0. 0 0 0 0 0 5\right)\right) + 6,
$$
$$
g _ {1} (\theta) = \left(\left(- \theta_ {1} + 7\right) ^ {2} + 0. 1 * \left(- \theta_ {2} - 8\right) ^ {2}\right) / 1 0 - 2 0,
$$
$$
g _ {2} (\theta) = \left(\left(- \theta_ {1} - 7\right) ^ {2} + 0. 1 * \left(- \theta_ {2} - 8\right) ^ {2}\right) / 1 0 - 2 0,
$$
$$
c _ {1} (\theta) = \max  \left(\tanh  \left(0. 5 * \theta_ {2}\right), 0\right) \text {a n d} c _ {2} (\theta) = \max  \left(\tanh  \left(- 0. 5 * \theta_ {2}\right), 0\right).
$$
Algorithm 2 Implementation of FAMO in PyTorch-like Pseudocode
class FAM0:   
def __init__(self, num_tasks, min.losses,  $\alpha = 0.025$ $\gamma = 0.001$  #min_LOsses (num_tasks,) the loss lower bound for each task. self.min_LOsses  $=$  min_LOsses self.xi  $=$  torch.tensor([0.0] \* num_tasks, requires_grad=True) self.xi_opt  $=$  torch.optim.Adam([self.xi], lr=α, weight Decay=γ)   
def get_weighted_loss(self, losses): #losses (num_tasks,) z  $=$  F softmax(self.xi,-1) D  $=$  losses - self.min_LOsses  $+1\mathrm{e} - 8$  c  $= 1 / (\mathbf{z} / \mathbf{D}).$  sum().detach() loss  $=$  (c  $\ast$  D.log(  $\ast$  z).sum() return loss   
def update(self, prev.losses, curr.losses): #prev_LOsses (num_tasks,) #curr_LOsses (num_tasks,) delta  $=$  (prev.losses - self.min_LOsses + 1e-8).log( ) - (curr_LOsses - self.min_LOsses + 1e-8).log( with torch_enable_grad(): d  $=$  torch.autograd.grad(F softmax(self.xi,-1), self.xi, grad_outputs  $\equiv$  delta-detach())[0] self.xi_opt.zero_grad() self.xi_grad  $=$  d self.xi_opt.step
We followed the exact experimental setup from NASHMTL [32]. Therefore, the numbers for baseline methods are taken from their original paper. In the following, we provide FAMO's result with error bars.
Method	Segmentation	Depth	Surface Normal	Δm% ↓
mIoU ↑	Pix Acc ↑	Abs Err ↓	Rel Err ↓	Angle Dist ↓	Within t° ↑
Mean	Median	11.25	22.5	30
FAMO (mean)	38.88	64.90	0.5474	0.2194	25.06	19.57	29.21	56.61	68.98	-4.10
FAMO (stderr)	±0.54	±0.21	±0.0016	±0.0026	±0.06	±0.09	±0.17	±0.19	±0.14	±0.39
Table 5: Results on NYU-v2 dataset (3 tasks). Each experiment is repeated over 3 random seeds and the mean is reported. The best average result is marked in bold. MR and  $\Delta m\%$  are the main metrics for MTL performance.
Method	μ	α	εHOMO	εLUMO	〈R2〉	ZPVE	U0	U	H	G	cv	Δm%↓
MAE ↓
FAMO (mean)	0.15	0.30	94.0	95.2	1.63	4.95	70.82	71.2	71.2	70.3	0.10	58.5
FAMO (stderr)	±0.0046	±0.0070	±3.074	±2.413	±0.0211	±0.0871	±2.17	±2.19	±2.19	±2.21	±0.0026	±3.26
Table 6: Results on QM-9 dataset (11 tasks). Each experiment is repeated over 3 random seeds and the mean is reported. The best average result is marked in bold. MR and  $\Delta m\%$  are the main metrics for MTL performance.
Method	CityScapes	CelebA
Segmentation	Depth	Δm%↓	Δm%↓
mIoU ↑	Pix Acc ↑	Abs Err ↓	Rel Err ↓
FAMO (mean)	74.54	93.29	0.0145	32.59	8.13	1.21
FAMO (stderr)	±0.11	±0.04	±0.0009	±1.06	±1.98	±0.24
Table 7: Results on CityScapes (2 tasks) and CelebA (40 tasks) datasets. Each experiment is repeated over 3 random seeds and the mean is reported. The best average result is marked in bold. MR and  $\Delta m\%$  are the main metrics for MTL performance.