Dropped Scheduled Task: Mitigating Negative Transfer in Multi-task Learning using Dynamic Task Dropping
Authors: Aakarsh Malhotra
Mayank Vatsa
Richa Singh
Venue: Accepted by TMLR ()

Abstract: In Multi-Task Learning (MTL), K distinct tasks are jointly optimized. With the varying nature and complexities of tasks, few tasks might dominate learning. For other tasks, their respective performances may get compromised due to a negative transfer from dominant tasks. We propose a Dropped-Scheduled Task (DST) algorithm, which probabilistically “drops” specific tasks during joint optimization while scheduling others to reduce negative transfer. For each task, a scheduling probability is decided based on four different metrics: (i) task depth, (ii) number of ground-truth samples per task, (iii) amount of training completed, and (iv) task stagnancy. Based on the scheduling probability, specific tasks get joint computation cycles while others are “dropped”. To demonstrate the effectiveness of the proposed DST algorithm, we perform multi-task learning on three applications and two architectures. Across unilateral (single input) and bilateral (multiple input) multi-task net- works, the chosen applications are (a) face (AFLW), (b) fingerprint (IIITD MOLF, MUST, and NIST SD27), and (c) character recognition (Omniglot) applications. Experimental results show that the proposed DST algorithm has the minimum negative transfer and overall least errors across different state-of-the-art algorithms and tasks.

Published in Transactions on Machine Learning Research (12/2022)
Dropped Scheduled Task: Mitigating Negative Transfer in Multi-task Learning using Dynamic Task Dropping
Aakarsh Malhotra
aakarshm@tittd.ac.ik
Deportment of Computer Science LeT-Delhi, New Delhi, India 110020
Mayank Vatsa
meatsa@hitj.commmm ac.
IIT Jodhpur, Rajasthan, India 342037
Richa Singh
richa@ittj.or oc. oc.in
IIT Jodhpur, Rajasthan, India 342037
Reviewed OpenReview: https: oppeeeve/pennrevieuu net/ forum? id= AA
Abstract
In Multi-Task Learning (MTL), K distinct tasks are jointly optimized. With the vary- ing nature and complexities of tasks, few tasks might dominate learning, For other tasks, their respective performances may get compromised due to a negative transfer from domi- nant tasks. We propose a Dropped-Schedduled Task (DST) algorithm, which probabilistically "drops" specific tasks during joint optimization while scheduling others to reduce negative transfer. For cach task, scheduling probability decided basod on four different met- rics: (i) task depth, (ii) number of ground-truth samples per task, (iii) amount of training completed, and (iv) task stagnancy. Based the scheduling probability, specific tasks get joint computation cycles while others are "dropped". To demonstrate the effectiveness of the proposed DST algorithm, we perform multi-task learning on three applications and two architectures. Across unilateral (single input) and bilateral (multiple input) multi-task net- works, the chosen applications are (a) face (AFLW), (b) fingerprint (IITTD MOLF, MUST, and NIST SD27), and (c) character recognition (Omniglot) applications. Experimental re- sults show that the proposed DST algorithm has the minimum negative transfer and overall least errors across different state-of-the-art algorithms and tasks.
1 Introduction
Machine learning and deep learning aim to optimize objective to achieve an end goal. Depending the use-case scenario, the goal may vary from regression task to classification task. However, in the real world, there are scenarios where many objectives need to be solved together. For example, a self-driving car needs to decipher a road sign while tracking the car in front of it while simultaneonsly avoiding an incoming pedestrian on the road (Yang al. 2018 Chowdhuri et. al.. 2019 Chang et. al. (021). Similarly, an explainable deep model can predict an organ's X-ray/Cr MRI is infected while segmenting the discase to assist the doctor (Amyar et al., 2020 Zhang al. 2020 Malhotra al. 2022a).. In such scenarios, algorithms need to process input data, like road view from lidar sensor organ X-ray/CT/ scan, to perform multiple tasks.
In the scenarios defined above, Multi-Task Learning (MTL) Caruana 1997 Zhang et al. 2021a Vanden- hende et al., 2021) is an ideal choice. MTL is a joint optimization of distinct tasks. The aim to share information across related tasks by utilizing shared representations. In a deep convolutional neural network (CNN), tasks in MTL often share network parameters or layers design scalable and robust solutions. In MTL, these K tasks be placed for training at different stages (depth) of the network. However,

---

Published in Transactions on Machine Learning Research (12/2022)
MTL
Shared Layers
DST
7L
-155
VL
0.75
VL. 0.75
VL
0.75
VL. 0.75
Task Specific
Layers
Negative
Positive Transfer
NO Transfer
Positive Transfer
Transfer
Task Training
ON
ON
ON
OFF
ON
ON
Figure 1: Some tasks can dominate learning MTL, resulting in negative transfer to other tasks. The proposed Dropped Scheduled Task (DST) algorithm reduces negative transfer by dynamically selecting dominating tasks and "dropping" them. This way, other lasks get more compute cycles and learns without negative transfer.
level, different tasks share parameters and use the shared representations to perform the respective K end task. The shared representations enable the generalization of the solution (Caruana, 1997 Ndirango & Lee 2019) while saving time and cost to deploy each of the K task models separately. MTL offers fiexibility with lasks getting trained logether at different stages (depth) and even different. loss functions such MSE cross-entropy. With different depths, tasks requiring coarse features may be trained with smaller network depth than more intricate tasks that require considerable network depth and more epochs. However, these variabilities in joint optimizations may lead to negative transfer (Wang et al. 2019 Zhang et al. 2021b).
1.1 Background and Problem Formulation
In this subsection, we provide the notations, background, and problem definition for multi-task learning and negative transfer. In MTL, the aim is to optimize for a set of tasks simultaneously on an ith input X₁. X2 is an instance in z-dimensional imput dataset E NNx with a task-specific labels Y, N Note that 1 S |Yi] < N translates to the fact that each task may have different number of labeled data samples. However, cach sample has at least one task with labeled ground truth. The MTL network is denoted by s(X; 0*)), where 0 are the nelw parameters al tth epoch, consisting of shared and task-specific parameters, given as 8%, and of respectively. Here, t refers to individual tasks, where 1 S K.
MTL aims to learn the network parameters by simultancously optimizing all K task's losses. Henco, the net optimization for the 7th instance becomes L; = Σ fi,, where L1; is the individual loss term for the th task. In more general sense, the net optimization can be written as linear combination of individual losses C. EE WyL,,- The weighting term W, be used for loss scaling or task prioritization {Gong et al, 2019 While jointly optimizing with individual task's objectives, source and target domains are learned together. During this joint learning, tasks can assist each other for improved learning and generalization However, due to imbalanced learning between tasks, overall MTL performance or performance for a subset of task(s) can occasionally be sub-par compared to its single task learning (STL) counterpart due to a negative transfer Rudder 2017
In & general sense, a negative transfer can be defined as scenario of transfer learning where the target task demonstrates lower performance due to knowledge transfer from source task compared to the case where the target task is trained individually. In the context of MTL, source and target domains are trained together and interchangeably knowledge is updated. With multiple tasks optimized together, a subset of lask(s) may dominate training. Here, the performance for these dominant tasks improves while the tasks outside the dominant group observe lower performance (Liu et al. 2019a),, referred to as negative transfer in MTL. We describe negative transfer in MTL as a scenario where a subset of target task(s) reports sub-optimal performance compared their respective performance when trained in the absence of source fask(s). More specifically, the absence of a source task be defined scenario where the target task
2

---

Published in Transactions on Machine Learning Research (12/2022)
Table Correlation of task-specific gradients at the shared layer on the five tasks the AFLW database Koestinger
et al. 2011). The five tasks are: Tı: gender prediction, T2: wearing spectacles not prediction, Ts: landmark localization, TĄ: pose estimation, and T5: denoising.
72
T3
T4
71
1.0
0.012
0.001
0.005
0.006
T2
0.012
1.0
0.001
0.006
-0.007
33
0.001
0.001
1.0
0.117
0.044
74
0.005
0.006
0.117
1.0
0.004
13
0.006
-0.007
0.044
0.004
1.0
is trained individually (single task learning: STL), the larget task is trained in an MTL setup without the source task in consideration. This can be seen in Figure under the MTL setup. For instance, outliers in an alrcady learned, well-performing, casier task can negatively affect another challenging task that is still underperforning. Alternatively, incorrect predictions from complex lasks may negatively alfect correctly predicting easier tasks (Lee et al. 2018). Furthermore, some unrelated tasks might get left out during the training.
To formally persuade the problem of negative transfer, let us consider task interference on the shared param- eters 8% During parameter updates for each sample i (or minibatch), the gradients Va flow backwards from task-specific 0% to the shared parameters 0% The parameters 0% are updated by a linear combination of individual task's gradients, as given by: vith I. = Σ Vol4,- However, at these shared layers, the gradients of different tasks may interfere with opposite update directions. The disagreement in the gradient directions between the tasks may millify the overall gradient, limiting performance for subset of task(s). For instance, seen for MTL in Figure I gradients from TA mullify gradients for TB and Tc at the shared layer.
To motivate the task interference with respect to counter directions of task gradients, we showcase vanilla MTL of five lasks of the AFIW database (noestinger et al., 2011) using the architecture shown in Figure 2 While more details follow in Section 2.2 the five tasks are: Ti: gender prediction, T2: wearing spectacles or not prediction, T3: landmark localization, Ta: pose estimation, and Ts: denoising. We obtain gradients with respect to each task for each minibatch throughout training for a randomly selected parameter in the shared encoder layer. This yields a vector of gradient values for each task. Using these vectors, quantify the correlation of gradients across each pair of tasks (Dwivedi & Roig 2019 As seen in Table D there is limited consensus in task gradients, which supports the impact of negative transfer caused by interfering gradients.
Recent studies aim to achieve global optima and address negative transfer by manipulating the training process. For limiting negative transfer and controlling information sharing while training an MTL network, the existing approaches can be categorized under three categories: (i) task grouping, (ii) task prioritization, and (iii) curriculum learning. The upcoming subsection elaborates on each of these methods.
1.2 Literature Review
The commonly used approaches to optimize training process includes: (i) task grouping (Kumar & Daume III 2012), (ii) task prioritization (Gong et al., 2019 and (iii) curriculum learning (Bengio et al. 2009). Each method aims to control the priority of the tasks to improve the overall performance. For thess existing approaches, we describe cach of these three hoods is below.
Task Grouping: In this approach, algorithms aim to group tasks by identifying their relatedness. Post grouping, the optimization is done by selecting group of tasks sub-network for better multi-task training, Rescarch studies compute grouping by mcasuring affinitics (Zhang & Yeung 2014 Sttndley et al.] 2019 cross-Lask consistencies (Zamir et al.) 2020), or probability of concurrently simple difficult tasks {Lu el. al. 2017). Post grouping, the algorithm may give different sub-networks (Lu et al.) 2017 Standley et al.) 2019) or orchestrate MTL (Zhang & Young 2014 Zamir et al.) (2020). Instead of a single level, a few studies (Han) & Zhang 2015 Chen et al. [2018a] Zamir et al. (2018) further segregated task grouping into different levels to model complex relationships amongst. tasks. Though not exactly on the lines of grouping, Maninis et al.
3

---

Published in Transactions on Machine Learning Research (12/2022)
2019 proposed attentive single-tasking while training a multi-task network. The idea was to select single task to execute using residual adapter so that the MTL network could learn focussed features relevant to the task and disregard irrelevant features. In 2020, Chen et al. (2020) suggested dropping gradients of a group of tasks based on a positive sign purity metric. The idea was to drop cither negative or positive gradients from different lasks so that they do not neutralize each other during training. Similarly, Navon et al. [202)] formulated MTL as a Nash-MTLL algorithm. Nash-MTL performed learning as a bargaining game, where tasks negotiate to obtain consensus for the gradient direction. Recently, Liu et al. (2022) built
dynamic task relationship based on meta-loss during training. The association of tasks is dynamically decided based on the validation loss values. The result is a dynamic task-specific weighting parameter À against each task's loss.
Weight based Task Prioritization in MTL: In previous subsection, we defined MTL as C, = - Ct., To prioritize tasks, research studies weigh each task individually and optimize as: E. - = WeCe, The term Wa, is altered based on wncompleteness/harte haaddness of each lask. Assuming Lask-specific loss is informative enough, studies use loss value to dynamically balance training. GradNorm (Chen et al. 20188) altered the gradients of the network to prioritize tasks. Thoy calculated the ratio of current loss and the starting loss for all the tasks. Each task's gradients are altered based on the calculated lask-wise ratio against the expected ratio value across all tasks. LBTW (Liu et al.) 2019a) uses the ratio of current loss and the starting loss to determine completeness. The term (ratio)" acts as weight for each task. Other weighting strategies include studies by Senerr & Koltun 2018) that finds weights leading to Pareto-optimal solution, and Kendall et al. 2018) that uses homoscedastic uncertainty for task weighing. Similarly, Liu et al. (2019b) introduced Dynamic Weight Average (DWA) to calculate relative descending rate. DWA is then used to weigh individual (2018) added task-wise weights inlo learnable parameters. They constrained these weights with a regularization term to enforce non- non-trivial solutions. Other than loss values, studies (Guo ct al. 2018 Jean et al., 2019) also utilize train or validation set performance to determine completeness or hardness of the task. In some scenarios, weights are dynamically modified to achieve performance closer to pre-computed Single Task Learning (STL).
tasks. Liebel & Körner
Curriculum Learning: In 1993, Elman 1993 discussed the concept of starting small to improve the training process for multiple lask subsets. However, the term "curriculum learning" was coined by [Bengio et al. 2009). The idea was to learn easier sramples/sample followed by introducing more challenging cases. Results showed a speed up in the training process by finding good local minima for non-convex learning, [Lee] & Grauman (2011) use the idea to sequentially discover difficult categories using the carlier discovered casier ones without supervision. [Pentina et al. (2015) performed MTL by sequentially learning individual tasks. They regularized parameters of each introduced task to be similar to just previously learned tasks. The idea was that consecutive related tasks in the order of difficulty should have similar parameter representation. Recently, Graves et al. (2017) utilized two signals as a reward to tune curriculum learning. These signals were the amount of increase in accuracy and increase in model complexity.
Despite advancements in task grouping, task prioritization, and curriculum learning, negative transfer re- mains a predominant issue in MTL. The limitations of existing approaches are: (i) they seldom consider network depth while training that can cause negative transfer. Further exacerbated negative transfer may be caused by task-wise varying count of ground-truth annotated training samples (Ge et al.) 2014 Wu et al. 2020), (ii) the weighted approaches may leave some tasks in a stagnant condition, and lastly, (iii) negligible weight to early completed tasks can result in their catastrophic forgetting. Hence, this research aims to discover potential issues that can cause a nogative transfer and address them using the proposod optimal training approach.
Research Contributions: In this research, we propose Dropped-SSheduued Task (DST) to address negative transfer by dynamically selecting dominating tasks and "dropping" them. Our approach selects four factors that cause negative transfer and resolves them using the corresponding four metrics. These include (i) network depth, (ii) ground-truth availability count, and current loss values to determine (ini) task-wise learning completeness, and lastly, (iv) task-wise stagnation. Using these parameters, the DST occasionally "drops" a subset of tasks by a task-specific activation probability. The term "dropping" refers to not considering a subset of tasks during joint optimization, i.c., preventing error propagation through the dropped task(s). As shown in Figure] [ holding/ /dropiigg/ quick learners gives a fair chance to complex tasks, and all
4

---

Published in Transactions on Machine Learning Research (12/2022)
the tasks are learned without negative transfer. Dropping can also prevent dominance//overfiting (Wan et al, 2013 Srivastava et al. 2014 Ghiasi et al. 2018) of the completed tasks. Along these lines, recent work by Sun et al. (2021) suggested performing one task at a time during MTL optimization. However, different combinations of tasks learning together can act as a regularization method and result in the generalization of Ndirango & Lee 2019). The related tasks may further assist other data scarce tasks and assist in their learning as well (Kapidis et al. 2021 p. The contributions of the DST algorithm are
the solution (Caruana, 1997
as follows:
The DST algorithm considers negative transfer caused due to task-wise varying network depths and limited samples for some tasks. Due to network depth as a parameter, DST is shown to work in unilateral and bilateral models.
The DST algorithm computes completeness and stagnation of the task, allowing only subset of tasks to remain active. Further, DST allows prolonged stagnated earlier finished tasks to be occasionally active in later training stages, reducing catastrophic forgetting.
Due to task dropping, 2 combinations from K tasks are possible. Different combinations during training act as implicit regularization move tasks out of local minima.
For experimental results, three applications are chosen related to faces, (latent) fingerprints, and character recognition. The applications are chosen considering: (i) their diverse nature and com- plexities (classification, image-to-image translation, regression, contrastive loss, sogmentation), (ii) different output depths (encoder Maaderr/C//aa /CC/ /deeddrr)r and architecture (unilateral/billatrra) (iii) varying difficulty of similar tasks (segmentation for fingerprints easier than latent fingerprints). and (iv)
large number of tasks (Omniglot).
2 Dropped-Scheduleed Task (DST)
Negative transfer in MTL can be due to variety of factors. The proposed DST algorithm is based four important factors that can negatively affect learning in MTL. Based on the four factors, the proposed DST algorithm devises four metrics, based on which a task-specific activation probability is computed. Consequently, during training, the DST algorithm occasionally "drops" subset of tasks based on the computed task-specific activation probability.
The four factors are network depth, ground-truth sample count, task incompleteness, and lask stagnation. For each tth task in MTL, these four factors are quantified by a respective metric to quantify the drop rate. For 9th task, P(a..) quantifics drop rate based on task depth and Pie.s) quantifics drop rate based on training sample count. Additionally,
wtMh epoch, P(u,k.t) and P(r.A.) quantifies drop rate based on task incompleteness and stagnancy, respectively. These individual metrics tell us how a certain task is overpowering or underpowered. These metrics are combined to define task-wise activation probability P(k,)); where P(R,t) ranges in [0, 1]. P(k,t) tells what are the chances of 1th task to remain active on A epoch. The following section explains the proposed DST algorithm and the four metrics. It is followed by details of the chosen tasks and respective architectures.
2.1 Dropping Mechanism to Hold Overpowering Tasks
In the proposed DST algorithm, a task can remain active at 1tth epoch by a probability P(x.o)- Otherwise, the tasks are dropped by a probability (1 - P(..)). P(k,,) is a weighted combination of five different metrics (one being a regularizer). The following subsections explain how the factors like network depth, ground- truth sample count, task incompleteness, and task stagnation affect MTL. Further, we explain how these adverse factors can be translated into individual metries, which eventually are combined to get task activation probability P(..))
The source code is available at: http://gethhbbccmm//aaaaaaaaaaaa
5

---

Published in Transactions on Machine Learning Research (12/2022)
2.1.1 Metric based Network Depth
He et al. (2016) highlighted that inereasing the layers in a deep network may alleviate the problem of vanishing gradients. Vanishing gradients refer to scenario during backpropagation where the gradient diminishes as the network depth increases. Few studies utilizing multi-task networks have addressed the concerns using& residual connection (Subramanian et al., 2018: Ding et al. 2019 Li et al.) 2020) or some additional loss terms QQiu et al., 2022). These approaches work well when all tasks are placed at the same depth. In an MTL sctup with tasks of different depths, the effect of gradients can differ on the initial layers. For a shallower task, the gradient would dominate the initial layers. On the other hand, gradients would relatively vanish for deeper tasks.
To give an equal chance, one way is to increase the gradient. GradNorm (Chen et al. 2018b) showed a mechanism to alter gradients for faster training. Considering task-wise network depths, our study gives more computation cycles (keeping lasks active) for deeper tasks. It reduces the dominance of shallower tasks. Based network depth d for task t, the metric P(d,)) is:
d
P(d.t)
(1)
max (d)) 1SISK
A higher value of P(a,,) represents that the task considered is deeper and increases the chances of computation cycles.
2.1.2 Metric based on Training Sample Count
While training a deep network, the model may overfit if the training samples are low and trained for long duration (higher number of epochs). On the contrary, the network may underfit if it is trained for lesser epochs with larger number of training samples. For MTT, not all tasks may have an equal number of ground-trut -tutth labels. The task with fewer annotated samples might benefit (positive transfer) from the large corpus of samples provided by another lask with more labeled samples. However, a larget task with more labeled samples might undergo negative transfer due to a source task with fewer instances (Wu all 2020). This can be seen from Lee et al. (2016) and our experiments, where highly confident task faces negative transfer due to a low-confident task trained with fewer training samples. Hence, each task requires computing cycles proportional to the number of labeled instances.
We propose to consider task wise ground-truth count to decide computation cycles for each task. cnsures a reduction in computation cycles for tasks with fewer annotated training samples by dropping them more often. On the other hand, increases computation cycles for the task with more annotated training samples. The metric P(e,t) is given as:
Ple.)) max (ct)
(2)
IStSK
Here, P(e.2) the ratio of the ground-truth count C of the considered 8th task divided by the maximum ground-truth availability count of all tasks. A higher value of Pie..) represents that the lask has more labeled instances, resulting into task having more computation cycles.
2.1.3 Metric based on Task Incompleteness
To dynamically perform lask scheduling, the proposed algorithm assumes that task-specific losses are de- scriptive for task balancing. A ratio of each task's current loss to its initial loss shows how much the task has learned. The expected value of this ratio across tasks denotes the average completion rate. Using the for each task.
expected ratio, we can establish relative mupplteeees/incoopleeennee
Let the value of loss for task at the htth epoch be V(R.))- The initial loss value after the 1st epoch is represented as: V(I.)). Hence, the amount of "incompleteness" during training for task at the 4 epoch can be defined as:

---

Published in Transactions on Machine Learning Research (12/2022)
V(R..)
(1.0))
(3)
V(1.))
For a task whose learning is incomplete with negligible decrease in loss, the value for ((k,)) would be close to I. On the contrary, a task that has completely learned would have a small value towards O. Further, it could be case where task has digressed and has its V(R.I) 2 V(1,)- This would result in ((x.t) 2 1. Using (( the relative incompleteness of each task t can be found as:
P(u,k.)) min (1, ((K.t) ((T)))))
(4)
Here, (()()) is the expected value of ((k.e) across all tasks tth epoch. The variable u in P(u,k,) is an alias for incompleteness measure, defined by ((.,))- ((u,k.)) penalizes faster learning lasks but does not reward slow task as the value is upper bound to 1. It makes all tasks that are "uncompleted" and slower than estimated ((T to obtain the highest activation chance while roducing the compute cycles of faster tasks.
2.1.4 Metric based on Task being Stagnant
While training, some lasks may become stagnant in an MTL setting. Stagnancy is defined as a phase where the rate of fall of training loss is negligible. An easy, quick learning task may become stagnant due to training completion. Such quick learning tasks should be refrained from further training to avoid overfitting, On the other hand, a few tasks may get left out due to not getting priority in MTL. Such tasks require more explicit emphasis with more compute cycles. Hence, the metric presented in this subsection aims empower incomplete stagnant tasks.
We now devise metric to quantify incompleteness and stagnance together. For each task, stagnancy be computed by monitoring the decline in loss values over a few epochs. For the same, we calculate the local rate of change of losses using the loss value in the current and previous epoch. The local rate of change of loss R(,,)) for th task at 1tth epoch can be defined as:
V(N..) - V(K-1.0) Vk 2.
R(kk)) P(u,k.))
(5)
V(k-1.0)
Here, V(K..) represents the loss value for task at the 4th epoch. The term P(u,,,)) is taken from Eq. to encode task incompleteness. The multiplication by Plo... ensures that for two equally stagnant tasks, priority given to the relatively incomplete one. On the other hand, the term V----VieLL is responsible to encode stagnancy across two simultaneous epochs.
Va-10
Next, we need to compute an incomplete task is in a stagnant phase rather than being stagnant in just two simultaneous epochs. Using the local rate of change R(K,))s we compute the global exponential moving average as:
R(Rks) (R (1 3) if 3.
if k 2;
R(,,0)
(6)
Here, B E (0, 1] is the discount factor. Larger values of prioritizes more recent rate of change. H( is assigned value of R1)-110) if RiR.) 0. Here, RR,,) (0, xx), where a value close to 0 indicates an incomplete and stagnant task. The metric based on loss stagnancy is defined as:
if l:
P(r.k.))
E(R(N)) Rip.
(7)
min
if k: 2.

---

Published in Transactions on Machine Learning Research (12/2022)
Here, (x) denotes the expected value of Rl(x) across all tasks at the kth epoch. The variable r in ((r,k)) is an alias for stagnancy measure, as defined by R((..)- Stagnant and incomplete tasks obtain a value of P(r,k.)) closer to
2.1.5 Regularization
The above four metrics control chances task activation based on network depth, ground-truth sample count, task incompleteness, and task stagnation. prevent a task to completely remain OFF, resulting in catastrophic forgetting, each task needs get some chance be active. To do so, regularization metric is given as ((t.))
2.1.6 Final Metric for Dropping and Scheduling
Using the five metrics defined above, the final probability to schedule a particular lask l at h epoch is:
P(r. = (d.)).) + XcP(an) + (aP (r, + A
(8)
Here, are non-negative weights given individual metrics such that Σ 1. These A can be altered to address specific variations of data, network, and learning in MTL setup. For our experiments, details and ablation around X; are shown in Section 3.2 and 4.4 respectively. Using P(r,) defined above, G(k,)) is sampled as:
G(v.)) ~ Bernoulli(P
(9)
G(ka)) a 1xK vector of independent Bernoulli random variables for tasks. For a task such that < K, the 8th value in the vector denotes an ON/OFF bit for the 8th task at hthh epoch (during training).
( G(R.)) @ C(R,))-
(10)
Here, denotes element-wise multiplication. The vector G(k.)) is sampled and multiplied element-wise with individual task losses. The multiplication accounts for sampling and scheduling tasks from all sets of tasks at each epoch.
The proposed DST algorithm schedules and gives compute eycles to one or more tasks. The advantages of the proposed DST algorithm as follows:
Dynamic scheduling according to task-wise loss values.
With K tasks being ON or OFF, each combination from 2 combinations is possible. This acts as an implicit regularizer that may take a particular task out of its local minima due to varying combinations of active tasks.
For two stagnant tasks, DST provides more compute cycles to the relatively incomplete task.
The DST method considers network depth and ground-truth training sample count for each task.
The occasional OFF may lake easier tasks backward and again take them to their minima when they are ON. This prevents their overfitting. In later stages of training, the learning allows the network to gradually switch ON the completed tasks at the very end (since they have been stagnant over time) avoiding their catastrophic forgetting.
Next, we discuss the applications considered in this study and their corresponding architectures. We show MTL on faces, (latent) fingerprints, and character recognition. The face and character recognition use the hard parameter sharing MTL approach. On the other hand, (latent) fingerprint rocognition uses a soft parameter sharing MTL approach. We refer to the face application architecture unilateral MTL. In unilateral MTL, an image input is fed, and the tasks segregate out after few shared layers. The same idea
8

---

Published in Transactions on Machine Learning Research (12/2022)
is followed in the application of character recognition. (Latent) fingerprints use bilateral MTL for various tasks. In bilateral MTL, two images are taken input and processed with soft. parameter sharing. We elaborate on the details of these architectures in the following two subsections.
Encoder
Decoder
Noisy Face Input Image
T5: Denoising
T1: Gender
T4: Pose
prediction
estimation
12: Spectacles
T3: Landmark estimation
prediction
Conv. +Batch ReLu Max Pool
Global Average Poaling (GAP) FC+SoftMax
Max UnPool
FC+ReLu
Sigmoid
Figure 2: The -ecoder MTL architecture for face applications. The five tasks (i) gender prediction, (ii) spectacles prediction, (iii) 21 point (x.y) landmark prediction, (iv) pose estimation, and (v) image denoising.
2.2 Unilateral MTL: Face Attribute Analysis
For application of attribute analysis in faces, deep MTL has been explored (Han et al. 2017 Ranjan et al. 2017). Unilateral MTL takes a face image as input and performs MTL. For MTL using face images, the selected tasks are (i) gender, (ii) wearing spectacles not, (ii)) landmarks, (iv) pose estimation, and (v) image denoising. Let the 7th input to the network shown in Figure 2 be noisy face image X Let fi represent the sub-network that predicts the gender probability P(ga (X the 7th noisy input X Then, the loss function for the first task TiF, of gender prediction is:
Tif, lg.loo
(1 g.) [())))())))))))).
(11)
where, Ii E (0, 1). Similarly, the loss for the second task of predicting if the face is "wearing spectacles/or not" can be given
T2r; = - -(1
(1- - (()))))
(12)
where, 8, 10, The above two tasks Tir, and T2r; binary classification tasks. The other three tasks are regression tasks.
The task T3f, predicts 21 distinct facial landmarks, each represented as a {z, y} tuple. Hence, for each input X 42 values are predicted using a sub-network fs as:
Y fa(''))
Vj 1 15 42.
(13)
These values are optimized using Mean Squared Error (MSE) for the third task Tar, given as:
42 Σ (, - 11,))
TEF. = Σp, 1=1
(14)
Here, Yi, are the ground truth landmarks and Yi, are the predicted landmarks. Due to facial pose partial faco, some landmarks may be missing. To ensure such cases keep the loss unaffected, the error is multiplied with Py (1 if the 5th landmark is present, else 0). The error normalized by the total number of landmarks present (Er)).
The fourth task T4F; predicts pose angular values. The head pose defined as three rotation angles: yaw, pitch, and roll. Unlike landmarks, each of the three angles present. Hence, the MSE for the fourth task Tar;, given

---

Published in Transactions on Machine Learning Research (12/2022)
Segmentation
Probe Latent Fingerprints
Matching'Companison
Gallery Fingerprints
Classification into orientation types
Figure 3: Different task associated with latent fingerprint analysis.
1
"0 (Ya, Ya,)². TAF. 3
(15)
Here, Ya, is the ground-truth while Va, the predicted angle. The error normalized by total pose types, i.., 3.
Lastly, T&A is a regression task to denoise the noisy input x Denoising is achieved using the sub-network fs- For each pixel j, Tsr, is given as:
TEF. (Xe, - fo(x,,
(16)
3-1
where is the total pixel count and Nr, is the clean face image.
Overall Loss Function: For images with unavailable ground truth labels for task, respective sub- networks may not be active during training. Further, the proposed DST algorithm may "switch OFF" (or drop) task. Hence, the final loss C, is computed as:
L. Σ Gewieaii
(17)
Here, Zr are the switches to denote the presence of the ground-truth label for ith image of the 1th task. Further, G2 is gate to control training for Uth task by the DST algorithm. The values of these switches (gates) are cither 0 or 1. wt is a static weight normalization for cach task by scaling the loss values to the same expected starting loss value. The term we optional, and the proposed DST algorithm also works for varied starting loss values across tasks.
2.3 Bilateral MTL: (Latent) Fingerprint Analysis
Bilateral MTL inputs a pair of images to simultaneously process them for performing joint MTL. We chose latent fingerprint analysis as requires important tasks (Malhotra al.) 2018) of recognition, orientation classification, and segmentation. These tasks explained in Figure 3 The selected tasks are (i) fingerprint orientation estimation, (ii) fingerprint segmentation, (iii) latent fingerprint orientation estimation, (iv) latent fingerprint segmentation, and (v) pairwise matching. The orientation of the (latent) fingerprint is pattern in which the ridges flow. They are classified into six categories, as shown in Figure 10 in appendix. As shown in Figure let the 7th input to the Siamese net work be a pair of images, fingerprint Xp, and latent fingerprint Xt,-
Let f be the sub-network that predicts the fingerprint orientation probability P(o, in one of the pre-defined six orientations. Then, the first task can be defined as:
10

---

Published in Transactions on Machine Learning Research (12/2022)
Encoder
Decoder
T2. Fingerprint Segmentatiion
Inpul
Fingerprint
T1: Fingerprint Crientation estimation
Contrastive
T3: Latent Fingerprint Orientation estimation
Loss
Input
Fingerprint
T4. Latent Segmentation
Encoder
Decoder
Conv. Batch Norm+ReLu
Global Average Pooling (GAP)
Max UnPool
Max Pool
FC+ReLu
SofSttMMax
Sigmold
Figure 4: The MTL architecture for fingerprint applications.
The tasks are (i) fingerprint orientation estimation,
(ii) Fingerprint segmentation, (iii) latent fingerprint orientation estimation, (iv) Latent fingerprint segmentation, and matching.
TaF, = - Σ oglog(((o{XP)))
(18)
Similarly, using P(o:)XL,). loss function for latent fingerprint orientation prediction (Tsi,) can also be defined.
The second and fourth tasks are semantic segmentation of fingerprint Xp, and latent fingerprint Xz, respec- tively. For fingerprint segmentation, let ground truth segmentation mask be Sp, while the predicted mask be Sp:- Then, the second task T2P; defined as:
T2p: = -
Σ Sp (z, y) (og (z, (1 Sp,) log(1 1 Sp.
(19)
Similarly, Tax, can be defined for latent fingerprint segmentation. Fach of the two segmentation tasks, Ter; and TaL, operate pixel level (I, The classification for each pixel is a 2-class problem, where the masks predicted using sub-networks f' and Sí (with identical weights). Lastly, Ts, calculates pairwise contrastive loss using Euclidean distance D; between encoder representations of Xp, and Xt,
Ts. = (Y.), (1,) + +(1 - (1-Y)), max(0, m D.)²
(20)
Here, m denotes the margin parameter. Y; = 1 for genuine pair, while Y; = 0 for an imposter pair.
Overall Loss Function: Taking into account the missing ground truth labels (Z), proposed task gating (G), and loss value normalization (w), the net loss L', is computed as:
C G ZTTTP, GrweZaT2p, + GzwaZyTaa + Z. GswzZo,Tb.
(21)
3 Experimental Details
3.1 Databases and Protocols
In this study, two kinds of experiments are performed. For the first experiment with unilateral MTL, we have taken two case studies with two different datasets. The first case study is on faces, while the second is
11

---

Published in Transactions on Machine Learning Research (12/2022)
Table 2: Train-test test split for the AFLW database, with unequal ground-truth label count selected for each task.
Task
Task Depth
Training
Testing (Count.)
Task
No.
Count
(%)
1
Gender
17
8434
50
2
Spectacles
17
12731
75
3
Landmark
17
12727
75
7384
4
Pose angle
17
6692
40
5
Denoising
26
17000
100
Table 3: Consolidated fingerprint database used for experiments. The labels counts are as per the source availability.
Task
Task Depth
Training
Testing (Count)
Task
No.
Count
(%)
FP Orientation
17
33340
79.12
FP Segmentation
26
33340
79.12
LFP Orientation
17
32308
76,76
13577
LFP Segmentation
26
9756
23.15
Matching
13
42140
100
character recognition, with experiments performed on the AFLW and OmniGlot datasets, respectively. The second experiment of Bilateral MTL is performed on latent fingerprints. A summary of the dalasets is given below.
1. Unilateral MTL: For faces, we use the AFLW datasct (Kocstinger al.) 2011) having 24,384 usable faces. We use 70% (17,000) for training and 30% (7,384) for testing. The cropped face images are deteriorated by adding speckle noise to 15% of pixels. Table 2 shows that varying proportion of training labels are selected for each task. For the testing set, no tags are discarded.
Omniglot database (et al., 2015) consists 50 handwritten alphabets, each of which has its respective character recognition tasks. Omniglot a relevant database for the application of MTL since a large number of classes can illustrate the performance of the proposed method as a function of the number of classes. Furthermore, the tasks are related knowledge of the alphabets can help classes learn from each other. The dataset consists of grayscale images of resolution 105x105, with 20 instances for each character. We follow the standard MTL protocol suggested in the literature (Liang et al., 2018 Mcyerson & Mükkuulainen 2018: Prellberg & Kramer, 2020). A fixed random training and testing split is created by selecting split of 20% as the test set from each alphabet, with the remaining 80% considered training, The architecture and experimental details are highlighted in Section 4.3
2. Bilateral MTL: We use a consolidated set from three fingerprint datasets: NIST SD 27 (NIST- SD-27 2000), IITTD MOLF (Sankaran et al., 2015), and MUST (Malhotra et al. 2022b), with numbers shown in the Table 3 The pairs for the fifth task are considered in subject-disjoint split. Details for each of the database is listed below:
NIST SD 27 (NIST-SD-27 2000) has 258 latent fingerprints with respective inked fingerprint, resulting in 258 genuine pairs. An equal number of imposter pairs are added, making a total of 516 pairs. In NIST SD 27, there are segmentation masks for latent impressions. Further, each fingerprint and latent fingerprint are manually annotated for orientation. All of the 516 pairs are included in training set, with labels available for only four tasks (TI, T3, T4, and T5).
HITD MOLF (Sankaran et al., 2015) has 4,400 latent fingerprints (DB4). A genuine and an imposter pair with a fingerprint is formed for each latent fingerprint, totalling 8,800 pairs. Each pair has only the label for the fifth task (match/ non-match) and all 8,800 pairs are included in training set.
HITD MUST Latent Fingerprint database (Malhotra et al.) 2022b): As per the defined protocol for the MUST database, there is subject disjoint train-test split. Using the train identities, 32,824 train pairs are formed (with an equal number of genuine and imposter pairs). The knowledge of genuine and imposter pairs affirms the match/ monchhhhn-mtch information for the fifth task. Finally, the MUST database's
12

---

Published in Transactions on Machine Learning Research (12/2022)
test set is used for the testing procedure for the multi-task network. The test set has 13,577 query latent fingerprints, which are matched against all the test gallery fingerprints. The ground-truth labels for the testing set have information for all pairs for orientation (TI, T3) and matching (T5). However, for a few pairs, the segmentation masks are missing.
3.2 Implementation Details
The pseudo-code for lask sampling of activation bits using the DST algorithm is in the appendix. The models are trained on machine with Intel Core i7 with 128 GB RAM and NVIDIA GeForce RTX 2080Ti GPU in PyTorch implementation. Learning rale is set at 10-5 with Adam optimizer. The encoder in MTL is initialized with VGG16 (Simonyan & Zisserman 2014] pre-trained weights of the ImageNet dataset. During training, error backpropagation and weight update are performed from the second epoch to obtain initial loss estimates. 3 in Eq. 0 is set as 0.1 to ensure slower change for the 'stagnant' parameter. Further, all X in Eq. are kept as 0.2 to provide equal weight, to each metric. For face MTL, the input is a cropped face image of size 224x224x3 The model trained for 100 epochs with a batch size of 16. For fingerprint MTL, the input is a fingerprint-latent fingerprint image pair at resolution x400280x3. The model is trained for 75 epochs with a batch size of 4.
4 Results and Analysis
This section shows the efficacy of the proposed DST algorithm. The chosen applications are: (a) multi- task face (AFLW (Koestinger et al., 2011),, (b) multi-task fingerprint (WITD MOLF (Sankaran et al, 2015), MUST (Malhotra et al. 2022b), and NIST SD27 (NIST-SD-27 2000)), and (c) multi-task character recognition (Omniglot fet al. 2015)). Subsequently, in Section 4.4 we present an ablation study to study the effects of each selected metric, X; Wt, network depths and initialization, and varying sample counts.
As seen in Table 4 to, Table 5] and Table 7 the proposod DST algorithm reports minimum average errors across all the three applications (face: 10.75%, fingerprint: 18.87%, character: 10.41%). We compare the results against. STT., MTL and random task drop. Furthermore, the results are also compared against other task prioritization or scheduling algorithms (Lee et al. 2018 Chen et al. 2018b Guo et al. 2018 Jean et al. 2019 Liu et al. 2019a Maninis et al. 2019: Chen et al. 2020 Liu et al. 2022) with varying necessary parameters. Furthermore, the comparative results show that a = 0.5 are the desired parameters for GradNorm (Chen et al. DST, we also observe that LBTW (Liu et al. 2019a), Gradient Drop (Chen et al.] 2020), and Auto-A {Liul let al., (022) result in competitive performance across the three applications. We derive information from the train set for all algorithms that require task-wise performance/los values weigh/schedul tasks.
2018b) and LBTW (Liu et al. 2019a). After the best performanc by the proposed
4.1 Unilateral MTL: Faces
When MTL is performed for different tasks with input as face images, the results are shown in Table Visually, prediction outputs for denoising and landmark prediction are shown in Figure 8 and 9 of the appendix, and briefly in Figure 7 on this main paper.
We observe that the regression tasks (T3, T4, and T5) undergo negative transfer compared to STL. On the contrary, gender classification improves by 1.15% over corresponding STL. The gender prediction task has 40% labeled samples. Hence, learning from other tasks helps in a performance gain for gender task. By allowing a negligible drop in gender classification performance, DST improves performance for all the other four tasks and reports the least average error. For landmark localization, the DST algorithm provides 1.3% lower error than the STL. This can be seen from loss and task activation plots in Figure 0 The relatively more computation cycles for landmark localization with the occasional absence of others help task 3 move towards its minima. Concurrently, the occasional ON/OFF of tasks 1, 2, and limits trained tasks from overfitting, Once trained towards the end, the DST algorithm also reduces the activation probability of T3.
Further, to intuitively illustrate the effect of DST, we study two tasks using their T-Sne plot in Figure When tasks of gender classification (T1) and "wearing spectacles" (T2) are trained as vanilla MTT. we get
13

---

Published in Transactions on Machine Learning Research (12/2022)
T1: Gender Classification
T2: Wearing Spectacles
800
Parelodalt AoTL Pomele.OT
ISNE Lower Dimension N2
60
000
Weattng Spinne
Dimers
Wissting g Spancos DET
40
20
20
LowE
200
20
80
200
200
40
-50
-60
-10
40
ISNE Lower Dimension #1
tSNE Lower Dimension #1
Female- MTL
Female DST
No MTL
No DST
Male MTL
Male DST
Yes MTL
Yes DST
Figure 5: L-SNE plot for gender classification (TT) and "wearing spectacles" (T2) in vanilla MTL & and with the proposed DST algorithm. The axis represent a 2D projection of high dimensional features (encoder representation of size 1x512), obtained by the t-SNE: algorithm. Notice the two green clusters in the right t-SNE plot. As the gender task dominates, we see that it overpowers the T2 of "wearing spectacles", resulting in three clusters for T2: (i) not wearing spectacles (red), (ii) females with spectacles (left green), and (ii) males with spectacles (right green). Without gender classification, we would have seen just the two groups of clusters. Using the proposed DST algorithm, we observe the lower blue-yellow representation points. While gender classes are separated similarly (left. t-SNE plot), the wearing vs. non-wearing spectacles is now just two clusters (right t-SNE plot).
the upper kidney-like representation cloud (green-red points). As gender task dominates, it overpowers the "wearing spectacles" task. In the T2 plot the right, green points illustrate wearing spectacles using the vanilla MTL algorithm, while red is not wearing spectacles. Hence, when the person is "wearing spectacles", two clusters are formed (one for men with spectacles and the other for women with spectacles). The consequence is that for spectacles, there are three clusters of data, (i) not wearing spectacles, (ii) females with spectacles, and (iii) males with spectacles. Without gender classification, we would have seen just the two groups of clusters. Using the proposed DST algorithm, Pie.e) metric reduces the dominance of T1 and T2 gets more chance. T2 then makes relevant weight updates with limited influence of T1, allowing T2 to separate the "wearing spectacles" class as one cluster. Hence, we now observe in Figure 5 the lower blue-yellow representation points. While gender classes are scparated similarly, the wearing vs. non-wearing spectacles is now just two clusters.
Compared to other task prioritization algorithms, the proposed DST algorithm has the best overall perfor- mance and the least negative transfer. However, for a 2 1, GradNorm and LBTW drift away from the optimal solution and fail to work on the easier lasks (Gender and Spectacles). DTP (Guo el al., 2018
operates on key performance indicators. On the other hand, Deep Asymmetric MTL by Lee et al. (2018) aims to focus more reliable predictors from casy tasks. Hence, the results for Deep Asymmetric MTL is much improved upon easier tasks of gender and spectacles. However, overall, the average error deteriorates to 11.52, mainly due to lower performance on the difficult task of landmark localization. Even though task may have saturated due to an upper bound on performance, DTP still prioritizes tasks with lesser accuracy (as it has lower performance than others). While DTP has overall higher errors, we observe that it provides the near-to-best accuracy of 79.87% for the lower-performing task of gender recognition. While the overall performance of Adaptive Schodule (Jean al.) 2019) competitive, the performance platcaus out when a task is getting closer to its STL performance. This limits the performance gain, even if there could be a scope for improvement (e.g., landmark localization).
Compared with other scheduling or dropping-basee studies, Maninis et al. 2019) suggested attentive single- task scheduling while Chen et al. (2020) suggested Gradient drop. As seen by the performance of Allentive Tasking (Maninis et al.) 2019), scheduling single tasks results in close to optimal outcomes for easier tasks (as they are fast learners). However, a difficult task such as T3 of landmark localization remains incomplete. Finally, a dominant task causing negative transfer can have high gradient, increasing the chances of its gradient propagation in Gradient Drop (Chen et al. 2020
To counter the effect of the dominant. task,
14

---

Published in Transactions on Machine Learning Research (12/2022)
DST identifies such tasks by their properties (depth, data count, incompleteness, and stagnation). Then, DST reduces its dominance by dropping them stochastically. Furthermore, occasional switching "ON" of task with opposite gradient can also assist in regularization, which DST achieves by its stochasticity. Experimentally, Gradient Drop (Chen et al. 2020 has a net error of 11.07% while DST outperforms with a 10. 75% error on the AFLW database.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Table Classification accuracy (Gender and Spectacles) and normalized error (Coordinates, Posc, and Denoising) on the AFLW test set. (L) signifies that lower value is desired, while (1) signifies that higher value is desired.
Average Error
10.75 1 0.08
10.87 0.03
1 11.29 0.12
11.52 0.31
11:30 0.56
12:24 1 0.11
15.08 + 0.11
11.01 + 0.02
12:34 0.06
11.01 0.27
10.98 J 0.00
11.38 E 0.23
11.07 + 0.16
10 82 0.11
1 0.34
11.21 1 0.03
Normallzer Error (% (L)
T5: Denolsing
20.83 = 0.01
21.09 + 0.01
21.06 + 0.14
21.17 1 0.05
21.28 = 0.13
21.05 $ 0.01
21.20 + 0.01
21.10 I 0.01
21.10 E 0.02
21.08 + 0.02
21.06 I 0.00
21.23 # 0.13
21.09 1 0.00
21.05 + 0.01
21.17 =
21.23 1
T4: Pose
0.18 0.01
0.27 0.04
0.37 + 0.06
0.62 1 0.65
1.87 T 0.21
13.73 = 0.04
0.30 0.00
2.23 1. 0606
0.25 + 0.02
0.24 I 0.02
0.28 + 0.03
0.26 + 0.03
0.26 0.02
0.43 + 0.13
0.25 1 0.01
0.24 + 0.02
13: Landmark
8.21 0.01
10.31 + 0.29
10.93 = 0.07
10.73 + 1.49
13.74 1 0.03
3588 = 0.05
8.53 + 0.17
13. 1: 0.29
8.94 + 1-39
8.37 + 0.04
10.22 = 0.71
9.35 + 0.17
7.13 4 0.41
9.87 + 1.07
8.59 1 0.12
6.01 + 0.32
Classifccatoon lon Accuracy (% (1)
T2: Spectac les
96,82 = 0.01
96. .0.00 + 0.22
95,95 = 0.14
95.99 + 26
95.42 1. 0.2 23
94.05 + 0.30
95.94 = 0.44
95, 1: 0 04
95.48 + 0.25
96.02 I 0.11
96.17 + 0.20 29
96. 05 + 0.21
96.21 1. 0.17
95.93 + 0.23
96.14 1 0.32
96. +: 0.05
T1: Gender
78.05 0.13
79 19.20 01.3
0.46
80.16 1 0.63
79.41 = 0.45
78.87 0.51
09 0.04
79.87 + 0.24
78.77 + 0.53
78.51 + 0.62
79.27 0.44
78. 16 0.27
78.06 0.63
77.48 1 0.59
78 78,26 0.13
78 78.90
79 79.48
2018b
20 8b
20180
2010
(Jan el al. 2019
Lee a.. 2018
Liu ct aL 2019a
GradNorm (o = 0.5 Chen et al.
Gram aa = 1.0 Chen et a..
GradNorm (a = 1.5 (Chn S
Maninss et a..
Gradient Drop (Chen al. 2020
Indiviuualy Trained (STL)
2018
Auto-A (Liu el. al. 2022
Random (P())) 0.50)
Random (P(a.) 0.75)
Liu
LETW (a D.5)
1.0)
DTP Guo et al
Adaptive Scherble
Attentive Thskingg
Experiment
Asymmetric
Name
LETW
MTL
DST
Training
Method
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
Jointly
16

---

Published in Transactions on Machine Learning Research (12/2022)
Loss
0.45
0.4
Task:
Task:
0.3
0.25
0.2
0.1
0.05
90
Epoch
(a)
Task activation probability
0.9
0.8
Man Laman
Task:
Task:
0.6
0.5
10
30
60
80
100
Epoch
(b)
Task 1 Activation Signal
30
40
80
Task Ac Activation Signal
20
30
40
70
80
90
100
Task 3 Activation Signal
20
30
40
50
70
80
90
100
Task Activation Signal
Task Activation Signal
0
10
20
30
40
50
60
70
80
90
100
E poch
(c)
Figure 6: For AFLW database, (a) the training loss, (b) the task activation probability, and (c) the task-wise activation using DST algorithm.
17

---

Published in Transactions on Machine Learning Research (12/2022)
Table Classification accuracy (Orientation). ToU (Segmentation), and EER (Pairwise matching) on the IITD MUST test: set. (1) signifies that a lower value desired, while (1) signifies that higher value desired.
23 76 1 0.65
20 66 + 0.88
1 1.03
19.32 0.21
26.2 + 1.07
37.91 # 3.30
20:47 1 0,20
25.02 1 0.10
20.98 F 0.20
20.63 + 1.10
19.30 1 0.38
20.02 1 0.92
19.47 + 0.12
20.68 # 0.62
19.74 + 0.45
18.87 1 0.42
Average (%)
21 11.72
EER (%) (I)
T5: Siamese Verlficatio
15. 5583 0.24
22 26 + 0.33
30.47 1 0.93
35.43 + 0.45
33.98 I 1.04
21.65 1 1,55
37.68 1 0.30
0.10
23.30 + 1.50
21.69 1 0.82
21.25 1 1.22
20.72 + 0.13
23.37 + 0.14
22.22 1 0.39
20.28 1 0.53
21.24
22.36
T4: Latent Segmentation
79.43 + 1.17
83 83.06 = 0.01
83 06 + 0.77
82.94 + 0.02
80.97 = 0.35
83.12 + 0.49
80.67 + 3.02
$1.07 + 0.06
82 + 0.57
82.48 + 051
83.12 1 0.48
81.69 + 0.64
79.80 = 1.17
82.74 + 1.05
82.69 1 1.18
83 $3.35 1
YoU (x100)
T2: Fingerprint Segmentation
90 .6 + 1.02
94.37 + 1.
94.72 + 0. 0.23
94.71 + 0.32
93.14 = 0-19
91.35 + 0.28
94.39 E 0.41
93.61 0.16
94.41 0.15
94.37 1 0.01
94.22 29 1 0.23
94.01 + 0.08
93. 53.50 = 0.48
94.39 + 0.05
94.45 1 0.32
62
Classifccatoo Accuracy (%) (1)
T3: Orlentaation
44 24 + 0.43
55.77 + 1.24
57 81 +. 1.42
1. 0.90
50.28 +. 1.10
21.45 + 2.76
.2.92 + 0.00
51.95 1. 0.47
57.67 E 0.64
56. + 1.14
30 40 +: 1.19
57.18 1: 0.81
59.31 + 0.94
23 = 0.32
50 +. 0.02
59.66 1 0.75
T1: Orientation Fingerprint
82 40 0.58
85.78 + 2.91
1.17
0.33
5.62
4.90
85,90 +: 0.66
81.05 1: 2.93
85.12 + 0.34
86. 02 2.15
88.03 +: 0.78
86.57 1: 3.07
88.35 + 0.63
88. 43 +: 4.27
86.86 0.76
89.14 1 1.29
86. 30
66
76.47
15.88
Chen cl. a.. 200 8D
20 Sb
Attentive Tasking Manims et al 2019)
201
Asymmetric MTL Lee al. 2018
Chen et a..
2019h LBTW (a 0.5) Lin m.
2019a
Jean et al 2019
Gradient Drop Chen et al. 2020
Individually Trained (STL)
(Chen
LBTW (a = 10 Liu
2018
Auto-A Liu al. 22122
0.50)
Random (P = 0.75)
GrudNorm (a 0.37
GradNorn (a 1.00
GradNorm (a 1.3)
Adaptive Schedmle
Random (P..)))
DST (Proposed]
DTP GoD et
Experimens Name
MTL
Training Method
Jointly
Jointly
Jointly
Joinily
Jointly
Jointly
Joinnly
Joimtly
Johnnly
Jointly
Jointly
Joinnly
Jointly
Jointly
Joinlyy
18

---

Published in Transactions on Machine Learning Research (12/2022)
Table Segmentation performance (loU) for DST compared against different encoder-decoder deep architectures.
NegNet
UNet
MTL.
DST
Badrinarayanan al. 2017
Ronneberger et al.
2015)
Fingerprint Latent
0.91-10.01 0.79:+0.01
0.9210.01 0.80+:0.22
0.9410.00
0.94
0.83
0.83+0.011
4.2 Bilateral MTL: Fingerprints
When MTL is performed for different tasks with input as pair of fingerprint images, the results are shown in Table 5 followed by segmentation task results in Table G Visually, prediction outputs for latent and fingerprint segmentation is shown Figure 12 and 13 of the appendix and briefly Figure 7 Figure 14 in appendix shows the ROC curve for the fifth task of latent fingerprint comparison.
Under bilateral MTL with application fingerprints, the fifth task of pairwise matching has the largest number of annotated ground-truth samples. Training other tasks in presence of pairwise matching lask improves their performances. However, the fifth task (pairwise matching) undergoes a negative transfer duc to the segmentation tasks, which have far lower samples. The DST algorithm ensures performance gain orientation and verification tasks with a negligible drop in the performance of latent fingerprint segmentation task (T4). Overall, with improvement four tasks, the proposed DST algorithm reports the least overall error across algorithms. Visually, as seen in Figure 7 DST gives good segmentation results for fingerprint segmentation (T2) and sufficient latent fingerprint segmentation results (T4). We compare results of the DST algorithm with MTT. and other encoder-decoder algorithms for segmentation, as shown in Table D We observe that the proposed DST algorithm reports the best loU for fingerprint segmentation. Further, similar to MTL, the DST algorithm also overpowers the slandard SegNel and U-Net for latent fingerprint segmentation. Hence, with a minimal reduction in the performance of latent segmentation, the DST algorithm limits negative transfer. The training graphs of DST for fingerprints are shown in Figure in the appendix.
The proposed DST algorithm has the overall best performance of 18.87% compared to other task prioriti- zation algorithms for fingerprint applications. Similar to faces, GradNorm and LBTW provide competitive performances when = 0.5. However, they report higher errors for 2 1, where passibly the solution drifted away from the optima. Asymmetric MTL., while considering easier tasks as reliable, does give one of the best results for segmentation tasks, However, has the highest overall error due to sub-par performance on the difficult task of verification (T5). DTP aims to prioritize lower pwwerpperforiig tasks. Thus, DTP pro- vides a near-to-best, performance of 57.67% for latent. fingerprint orientation classification (T3) as it is the worst-performing task across five tasks of fingerprints (when trained in MTL setting). However, due to T3, well-performing tasks T1, T2, and T4 observe higher drop in performance. Hence, DTP reports an overall error of 20.98% compared to 18.87% by DST. Attentive single-tasking and Auto-A provide one of the best performances after the proposed DST algorithm. This can be attributed to better disentangling and reduced interference, as unlike faces and character recognition, the tasks are more diverse in fingerprints. Gradient Drop provides an overall error of 20.02%. While T2, T4, and T5 are comparable, DST outperforms due to higher accuracy in T1 and T3. This possibly results from more "ON" time for T3 (latent orientation) in DST, which is similar to T1 (fingerprint orientation) and improves its performance as well(refer to Figure 11(c) of the appendix).
4.3 Unilateral MTL: Character Recognition
We illustrate the results of multi-task character recognition using the Omniglot datasct, which is commonly used to highlight performance on a high number of tasks in MTL. In this study, since face and fingerprint had architecture similar to VGG16, we also utilized VGG (encoder only) based architecture for this application. Given all the tasks are classification tasks for Omniglot character recognition, all tasks are placod at the same depth. A VGG encoder is used as the shared backbone, followed by a branch for each task after the last encoder layer to classify character in the MTL setup. The branch consists of three dense layers, with the
19

---

Published in Transactions on Machine Learning Research (12/2022)
Noisy Dencising Landmarks
Segmentation
Segmentation
Input
Asymmetric
GradNorm
LBTW
DTP
Adaptive Schedule
Attentive
Tasking
Gradient
Drop
Auto-A
DST (Proposed)
Figure 7: Sample outpuls from the proposed DST and other comparative algorithums for lace denoising, landmark localization, and segmentation tasks. Additional instances shown in the appendix.
20

---

Published in Transactions on Machine Learning Research (12/2022)
Table 7: Average error (%) on the Omniglot dataset.
Algorithm
Avg. Error (%)
MTL
11.26
Asymmetric MTL [Lee et al. 2018)
10.81
GradNorm (a 0.5] Chen ct
2018b
10.84
LBTW (a = 0.5) Liu al. 2019a
10.59
DTP (Gno et al. 2018)
10.77
Adaptive Schedule (Jean et al. 2019
10.74
Attentive Tasking (Manimis et al. 2019
10.67
Gradient Drop (Chen al. 2020
10.61
Auto-A (Liu al. 2022
10.52
DST (Proposed)
10.41
Table Ablation results on AFLW dataset. TI: gender, T2: spectacles, T3: landmark, T4: pose, & T5: denoising,
Accuracy (%)(t)
Normalized Error
(%)(L
Details
T1
T2
T3
T4
T5
Avg.
Proposed
78.26
96 96.18
6.91
0.24
21.05
10.75
Small MTT.
70.31
96.11
5.69
0.29
25.56
13.02
Small DST
70.51
96.47
5.28
0.25
24.61
12.63
Early stop
77.97
91.71
8.99
0.26
21.05
12.12
Just P(u.k.c)
79.43
96.23
8.97
0.47
21.02
10.96
P (c.4)
77.89
95.67
8.47
0.29
21.09
11.26
dense output layer having units equal to the number of classes. As mentioned earlier, the shared encoder in MTL is initialized with VGG16 (Simonyan & Zisserman 2014] pre-trained weights of the ImageNet dataset.
Using this architecture, MTL yields an error of 11.26%. Table also shows the performance of other task prioritization algorithms. Compared to just 10.41% error by the proposed DST algorithm, other task prioritization algorithms have relatively higher crror in the range of 10.52% to 10.84%. For reference, with fixed ResNet18 archilecture, Prellberg & Kramer 2020 illustrated the average baseline error on Omniglot
as ~~ 10.70%
4.4 Ablation study
The proposed DST algorithm involves multiple variability points that need to be tested to validate the effectiveness of DST. Table B and Table g highlight ablation results under different settings. With the ablation study, we check the impact of weight initialization, network depths, dala count, w. in Eq- [7 varying (from Eq. 8) and, consequently, the effect of each of the four metrics. We also check a closely possible extension of DST, where the tasks get dropped with fixed static probability. Each of these ablations is presented in this subsection. These ablation experiments are conducted on the AFLW dataset.
4.4.1 Random Drop
A static random drop is closer to the dynamic task dropping of DST for comparison. To perform ablation and check a random drop can achieve a good performance, we drop lasks with a probability of 0.25 and 0.50. This implies that with a throughout fixed activation probability P(a.)). the tasks are stochastically dropped with a constant probability 1 P(L.)- Static dropping gives slightly better performance than MTL, but not that of DST, as highlighted in the bottom three rows of Table 4 and 5] Random drop reports the sccond-best results after the DST algorithm for few lasks. For instance, T2 of AFLW (wearing spectacles?) has just 0.04% lower performance for random drop than DST. Likewise, T4 of latent fingerprint segmentation also has a difference of 0.05% in loU. One reason could be the fact that for these tasks, the DST algorithm has ((P(R.)) closer to the static random drop probability. However, of overall performance, DST outperforms static random drop.
21

---

Published in Transactions on Machine Learning Research (12/2022)
Table 9: Ablation study on the AFLW Koestinger et al. 2011) test: set by leaving out: or keeping certain metrics. Classification accuracy (Gender and Spectacles) and normalized error (Coordinates, Pose, and Denoising) reported different probability weights. A (L) signifies that lower value is desired, while (f) signifies that higher value is desired. Only values reported here in interest of space.
Average Error
11.20
11.15
11.04
11.25
11.03
11.13
11.16
11.33
10.96 06
10,96
11.07
10.08
11.52
11.03
11,22
10.75
Normalized Error (%) (4)
Denoising
TG:
21.00
21.10
2 21.07
21 11.07
21.05
21.09
21.10
21.02
21.00
21.09
21.05
21.07
21.06
21 10.07
21.05
T4:
Pose
0:27
0.26
0.29
0.27
0.29
0.34
0.32
0.47
0.27
0.30
0.27
0.34
0.31
0.28
0.24
Coordinates
T3:
10.31
9.11
8.86
0.50
9.80
9.05
9.60
8,07
8,75
8.41
8.68
10. 10.00
8,30
9.82
6.91
8
Spectacles
Accuracy (%) (1)
T2::
96.37 .7
96.18 18
38
96. 22
96.27
96. 96.30
95. 95.93 .3
96. 14
96. 25
966. 11
96.18 18
Classification
96
96
966
Gender
T1:
79.20
78.37 .7
78.86
78
70.14
79.10
79.06
77.90 .99
79.43
79.04
78.
79.42
78 78.56
78.30
78.96
78.20 26
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.10
0.10
0.20 20
Individual Probability
A
0.25
0.25
9.25
0.25
0.00
P.00
0.00
1.00
D-30
0.40
0.10
0.30
0.
0. 2.20
Weights
An
0.25
0.25
0.00
0.25
0.25
0,00
0.00
1.00
0.00
0.30
0.40
0.10
0.30
0.15
20
A
0.25
0,00
0.225
0.25
0.00
1.00
0.00
0.00
0.20
0.10
0.40
0.15
0.30
0.20
Aa
Traditional MTL
0.00
0.25
(.25
0.25
0.25
1.00
0.00
0.00
0.00
0,20
0.10
0.40
0.15
0.30
0.20
Regularization
V
with larizaiion
Varying weights w/o regularization
Varying weights
Only dynamic metrics
Details
Leave-one metric out
metrics
Proposed
22

---

Published in Transactions on Machine Learning Research (12/2022)
4.4.2 Network Initialization
We used encoder with VGG16 ImageNet pro-trained weights for stable initialization (to mitigate high initial loss (V(1.)))- Alternatively, we also estimate initial loss with N 10 different initializations and observe that. it slightly reduces the effect of randomness. yields similar average error of 10.70%.
4.4.3 Varying Network Depth
Negative transfer is more likely when a task creates dominance due to being shallower. To show stability for different network structure, T1 FC branch is preponed to 7th cony instead of original 13th. Position of other tasks remain the same. With MTL (Row-2, Table S) we observe an adverse effect on the deepest task T5. Tl gets deteriorated due to information loss (larger GAP resolution). T3 gol improved with less interference in learning at encoder. With proposed DST (Row-3, Table ()), the dominance of shallower task T1 gets reduced on the deepest task T5. Overall, with other network architectures, observe improved performance with DST.
4.4.4 Overfitting
A cause for negative transfer could be overfitting. To counter overfitting, we: (i) early stop, or (ii) prob- abilistically drop early completed tasks (only Plw,k.t) as metric). For first case, completed tasks are early stopped when its MTL loss value reaches its optimal STL loss value (Row 4, Table 8 Here, classification tasks suffer other regression tasks (T3, T5) stay longer to make undesirable weight updates for T1 and T2. Secondly, with only P(u,k.)), observe overall sub-par performance with T3 being the most affected (Row 5, Table 8). T3 got limited ON time despite its stagnation (due to absence of P(r,k,)).
4.4.5 Inverse case of data count
We also perform ablation for the inverse case where task with lower ground-truth labeled samples is preferred over a task with more labeled samples (against the original assumption of Eq. p where task with more labeled samples is preferred). If the performance turns out to be better than the proposed DST, this could mean that task with lower ground-truth labeled samples leads to positive transfer. The selection criterion is updated as follows:
min (c) 1stsk
P' (e.))
(22)
C
The results from the above formula are reported in row 6 of Table g We observed higher average error of 11.26% than 10.75% from the proposed DST. Hence, our original assumption holds that tasks with fewer samples can hamper generalizability while higher sample tasks improve generalization (also shown by (Wul
et al. 2020
4.4.6 We Eq.
scales losses to standard initial value such that task-wise losses are in the same range for Figure 6a Removing it has a minimal effect on the DST algorithm as both task incompleteness (Section 2.1.3 and stagnation (Section 2.1.4p operate on loss ratios. With we the error MTL reduces by 0.54% while without We, the reduction is by 0.40% only.
4.4.7 Varying À and Metrics
A detailed ablation varying À values in Eq. g is shown in Table It also includes keeping certain X = 0 (not considering a metric) and keeping specific A = (considering only one metric). As we see in the rows titled OOnly dynamic metricsqq, the greatest contribution towards performance is due to dynamic parameters (P(r.k,)) and P(r..,e)). In these cases, one of the lowest errors 10.96% observed. While other static metrics encode intrinsic propertics of model and amount of training labels, P(u.k.t) and Pirna, dynamically favors tasks during training that require more computational cycles. We observe the same behavior when Ao and X are relatively given higher weights, and
lower error of 10.93% is seen.
23

---

Published in Transactions on Machine Learning Research (12/2022)
5 Conclusion
Due to joint optimization in Multitask Learning, some task(s) may face negative transfer. We propose Dropped Scheduled Task (DST) algorithm to reduce the impact of negative transfer. Based on the scheduling probability, the DST algorithm gives computation cycles to specific tasks while the others are "dropped", For each task, the scheduling probability is decided based on four different metrics. These metrics rely on: (i) lask depth, (ii) the number of ground-truth samples per Lask, (iii) task training completion, and (iv) task stagnancy. For experimental results, three MTL applications are chosen related to faces, (latent) fingerprints, and character recognition. Due to the occasional task dropping of learned dominating tasks, more computation cycles are given to stagnant /slower hroowerwwaaerrr /waker tasks, helping them to move towards their minima. In the process, the dropped task(s) refrain from overfitting, resulting in generalizable solution. Overall, the results show minimum negative transfer and overall least error across different algorithms and tasks.
This research focuses on the metrics defined in the DST algorithm for vanilla MTL. As for future impli- cations of this study, these metrics can also have broader implications and use cases. For instance, task incompleteness and lask stagnation metrics can be extended for the case of single-task classification models, where the aim is to increase the performance for a weaker class. Similarly, the metric task depth and training sample count can become vital in MTL with auxiliary tasks. In such scenario, these metries can be used as it is or altered to prioritize the main task more compared to other auxiliary tasks.
Acknowledgments
A. Malhotra acknowledges IIT Jodhpur for hosting him during this research. A. Malhotra is partially supported through Visvesvaraya Ph.D. Scheme by MEITY, Govt. of India. M. Vatsa is supported through the Swarnajayanti Fellowship by the Government of India.
References
Amine Amyar, Romain Modzelewski, Hua Li, and Su Ruan. Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation. Computers in Biology and Medicine, 126:104037, 2020.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A Deep Convolutional Secoder--ecoder Architecture for Image Segmentation. IEEE Trunsactions on Pallern Analysis and Machine Intelligence, 29(12):2481225)) 2495, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum Learning. In Interna- tional Conference on Machine Learning, pp. 41 48, 2009.
Rich Caruana. Multitask learning. Machine learning, 28(1):41 75, 1997.
Xuepeng Chang, Huihui Pan, Weichao Sun, and Hujjun Gao. Yoltrack: Multitask learning based real-time multiobject tracking and segmentation for autonomous vehicles. TEFE Transactions on Neural Networks and Learning Systems, 32(12):5332 5333, 2021.
Junkun Chen, Kaiyu Chen, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Exploring Shared Structures and Hicrarchies for Multiple NLP Tasks. arXiv preprint arXi::1808.0790 07658, 2018a.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient Normaliza- tion for Adaptive Loss Balancing in Deep Multitask Networks. In International Conference on Machine Learning, pp. 794-803, 2018b.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Xiv preprint wrXiv:2010.0680 2020.
24

---

Published in Transactions on Machine Learning Research (12/2022)
Sauhaarda Chowdhuri, Tushar Pankaj, and Karl Zipser. Multinet: Multi-modal multi-task learning for autonomous driving. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1496 1504. IEEE, 2019.
Xinghao Ding, Yunshu Chen, Zhen Tang, and Yue Huang. Camera identification based on domain knowledge- driven deep multi-task learning. IEEE Access, 1125878-25890 2019.
Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12387-12396, 2019.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Elsevier Cognition, 48(1):71199) 19779,99991911 1993.
Lake et al. Human-level concept learning through probabilistic program induction. Science, 350(6266): 2015.
Liang Ge, Jing Gao, Hung Ngo, Kang Li, and Aidong Zhang. On Handling Negative Transfer and Imbalanced Distributions in Multiple Source Transfer Learning. Statistical Analysis and Data Mining: The ASA Data Science Journal, 7(4):254-27) 2014.
Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock: A Regularization Method for Convolutional Networks. In Advances in Neural Information Processing Systems, pp. 10727 10737, 2018
Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndirango, Gokee Keskin, and Oguz H Elibol. A comparison of loss weighting strategies for multi task learning in deep neural networks. TEEE Access, 7:141627-141632, 2019.
Alex Graves, Mare G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukeuoglu. Automated Cur- riculum Learning for Neural Networks. In International Conference on Machine Learning, pp. 1311-1320, 2017.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic Task Prioritization for Multitask Learning. In Europeon Conference on Computer Vision, pp. 270-287, 2018.
Hu Han, Anil K Jain, Fang Wang, Shiguang Shan, and Xilin Chen. Heterogencous face attributc estimation: A deep multi-task learning approach. JEEE transactions on pullern analysis and machine intelligence, (11):2597 22099, 2017.
Lei Han and Yu Zhang, Learning Multi-Level Task Groups in Multi-Taan Learning. In AAAI Conference on Artificial Intelligence, pp. 2638-26444 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sébastien Jean, Orhan Firat, and Melvin Johnson. Adaptive Schoduling for Multi-Task Learning. arXiv preprint arXiv:1909.06.1 06434, 2019.
Georgios Kapidis, Ronald Poppe, and Remco Veltkamp. Multi-datasot, multitask learning of egocentric vision tasks. IEEE Transactions on Pottern Analysis Machine Intelligence, (01):1-1, 2021.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Martin Koestinger, Paul Wohlhart, Peter M Roth, and Horst Bischof. Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization. In IEEE International Conference on Computer Vision Workshops, pp. 2144-2151, 2011.
Abhishek Kumar and Hal Daume III. Learning Task Grouping and Overlap in Multi-Task Learning. arXiv preprint ar orx::11206 6417, 2012.
25

---

Published in Transactions on Machine Learning Research (12/2022)
Giwoong Lee, Eunho Yang, and Sung Hwang. Asymmetric Multi-task Learning Based on Task Relatedness and Loss. In International Conference on Machine Learning, pp- 230-238, 2016.
Hae Beom Lee, Funho Yang, and Sung Ju Hwang. Deep asymmetric multi-task feature learning. In Inter- national Conference on Machine Learning, pp. 2956 2964. PMLR, 2018.
Yong Jae Lee and Kristen Grauman. Learning the Easy Things First: Self-paced Visual Category Discovery. In IEEE Conference Computer Vision and Pattern Recognition, pp. 1721-1728, 2011.
Yinshuo Li, Jianyong Song, Wenkai Lu, Patrice Monkam, and Yile Ao. Multitask learning for super-resolution of seismic velocity model. TEEE Transactions on Geoscience and Remote Sensing, 59(9):8022-80)) 8033, 2020.
Jason Liang, Elliot Meyerson, and Risto Miikkulainen. Evolutionary architecture search for deep multitask networks. In Genetic and Evolutionary Computation Conference, pp. 466 473, 2018.
Lukas Liebel and Marco Körner. Auxiliary Tasks in Multi-task Learning. arv preprint MXXi:1885.06333 2018.
Shengehao Liu, Yingyu Liang, and Anthony Gitter. Boss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning. In AAAT Conference on Artificial Intelligence, volume 33, pp. 9977- 9978, 2019a.
Shikum Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference Compuler Vision and Pollern Recognition, pp. 1871 1880, 2019b.
Shikun Liu, Stephen James, Andrew Davison, and Edward Johns. Auto-X: Disentangling dynamic task
relationships. Transactions on Machine Learning Research, 2022.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive Feature Sharing in Multi-task Networks with Applications in Person Attribute Classification. In IEEE Conference Compuler Vision and Pallern Recognation, pp. 5334 5343, 2017.
Aakarsh Malhotra, Anush Sankaran, Mayank Valsa, and Richa Singh. Learning Representations for Uncon- strained Fingerprint Recognition. Deep Learning in Biometrics, pp. 197-221, 2018
Aakarsh Malhotra, Surbhi Mittal, Puspita Majundar, Saheb Chhabra, Kartik Thakral, Mayank Vatsa, Richa Singh, Santanu Chaudhury, Ashwin Pudrod, and Anjali Agrawal. Multi-task driven explainable diagnosis of covid-19 using chest x-ray images. Paltern Recognition, 122:108243, 2022a.
Aakarsh Malhotra, Mayank Vatsa, Richa Singh, Keith B. Morris, and Afzel Noore. Multi-Surface Multi- Technique (MUST) Latent Fingerprint Database.
www..bbbbbbr..gg/ollloo.o ptrssources/mussee 2022b.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive sugle-tasking of multiple tasks. In /CVF Conference on Computer Vision and Pattern Recognàtion, pp. 1851-18600, 2019.
Meyerson and Miikkulainen. Beyond shared hicrarchies: Deep multitask learning through soft layer ordering. In International Conference on Learning Representations, 2018.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as bargaining game. International Conference on Machine Learning, 2022.
Anthony Ndirango and Tyler Lee. Generalization in multitask deep neural classifiers: statistical physics approach. Advances in Neural Information Processing Systems, 32, 2019.
NIST-SD-27. Fingerprint minutiae from latent and matching temprint images. In National Institute of Standards and Technology. https://wwwr https://www.nistt mist. hist.gov/srd gow/srd/nistsd27, /nisssss7, 2000.
Anastasia Pentina, Viktoriia Sharmanska, and Christoph Н Lampert. Curriculum Learning of Multiple
Tasks. Tn IEFE Conference on Computer Vision and Pattern Recognition, pp. 5492 5492-5500, 5500, 2015.
26

---

Published in Transactions on Machine Learning Research (12/2022)
Prellberg and Kramer. Learned weight sharing for deep multi-task learning by natural evolution strategy and stochastic gradient descent. In International Joint. Conference on Newral Networks, 2020.
Yuanhang Qiu, Ruili Wang, Feng Hou, Satwinder Singh, Zhizhong Ma, and Xiaoyun Jia. Adversarial multi- task learning with inverse mapping for speech enhancement. Applied Soft Compuling, 120:108568, 2022.
Rajeev Ranjan, Vishal M Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE transactions pottern analysis and machine intelligence, 41 (11):1211333, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical
Image Segmentation. In International Conference Medical Image Computing and ComputeerAssisttee Intervention, pp. 234-241. Springer, 2015.
Schastian Ruder. An overview of multi-task learning in doep neural networks. arXiv preprint arXiv:1706.0500 05698998. 2017.
Anush Sankaran, Mayank Vatsa, and Richa Singh. Multisensor Optical and Latent Fingerprint Database. IEEE Access, 3:653 (633-66,2220 2015.
Ozan Sener and Vladlen Koltun. Multi-Task Learning as Multi-Objective Optimization. In Advances in Newral Information Processing Systems, pp. 527-538, 2018.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-scale Image Recog- nition. arXiv preprinl 14X.i::40091555 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhovsky, Πla Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way Prevent Neural Networks from Overfitting. The Journal of Machine Learning Research, 15(1):1999-1958, 2014.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which Tasks Should Be Learned Together in Multi-task Learning? arXiv preprint NNN5:1905.0755 2019.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations, 2018.
Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Lue Van Gool. Task switching network for multi-task learning. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 8291-8300, 2021.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2021.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of Neural Networks using DropConneet. In International Conference on Machine Learning, pp. 1058-1066, 2013.
Zirui Wang, Zihang Dai, Barnabás Póezos, and Jaime Carbonell. Characterizing and Avoiding Negative Transfer. In IEEE Conference on Computer Vision and Pattern Recognation, pp. 11293- 2019.
Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and Improving Information Transfer Multi-Task Learning. In International Conference on Learning Representations, 2020.
Zhengyuan Yang, Yixuan Zhang, Jerry Yu, Junjie Cai, and Jiebo Luo. End-to-end multi-modal multi-task vchicle control for self-driving cars with visual perceptions. In 2018 24th International Conference on Pallern Recognition (ICPR), pp. 2289 2294. IEEE, 2018.
Amir R Zamir, Alexander Sax, William Shen, Lconidas J Guibas, Jitendra Malik, and Silvio Savaresc. Taskonomy: Disentangling Task Transfer Learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
27

---

Published in Transactions on Machine Learning Research (12/2022)
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjic Cao, Jitendra Malik, and Leonidas J Guibas. Robust Learning Through Cross- Task Consistency. In JEEE Conference Computer Vision and Pattern Recognition, pp. 11197 11206, 2020.
Ranran Zhang, Xiaoyan Xiao, Zhi Liu, Yujun Li, and Shuo Li. Mrin: Multi-task relational learning network for mri vertebral localization, identification, and segmentation. IEEE journal of biomedical and health informatics, 24(10) 2002-9922291 2020.
Rui Zhang, Hongyuan Zhang, and Xuelong Li. Robust multi-task learning with flexible manifold constraint. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2150 22570 2021a. doi: 10.1109/ TPAMM.2020.300770 2020.3007637.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. orXiv preprint arXiv:2009.0m 00909, 2021b.
Yu Zhang and Dit- Yan Yeung. Regularization Approach to Learning Task Relationships in Multitask
Learning. ACM Transactions on Knowledge Discovery from Data, 8(3):1 31, 2014.
28

---

Published in Transactions on Machine Learning Research (12/2022)
A
Appendix
Algorithm 1: Sampling Task-wise activation bits (ON/OFF) with Dropped Scheduled Task (DST)
Algorithm
Result:
G(R,): 1xK vector of independent Bernoulli random variables for aetivating/dropplle
individual tasks
Given K tasks, wise network depth de, and task wise annotated sample count c₁
Initialize network weights W
Initialize a value for B E [0, 1]
Initialize Ad, Ae; Aus Ar, A [0, 1] such that Xa d Au + A, Ao for each task t do Set the probability based on network depth P(a.))
di
max
(di)
1St5K
Set the probability based on training sample count P(o,t)
Ct
max (ci)
15 K
Set regularization probability P(b,t) t)
end
for each epoch & do
for each task: do if k -==11 then
Calculate task-wisc loss value without weight update and store as V(1.t)
Set. P(a,k.)) =1 Set P(r,k.t) =
else
ViE.n Vis.
1(x,t)
Calculate lask incompletness probability P(a,k.0) = min(1, Ja.
Bda))
Vik 1)
Calculate local rate of change R(Kk,)) P(nnk.))
V(--19))
if k then Set global rate of change R(,,)) R((,o) else
Update global rate of change R( BR(K,)) + (1 B)
end
Calculate task stagnancy probability P(r.h.)) min(1, B(H((:)) R/
end
Calculate the final probability for task scheduling P(N.2) = >aPia.e) + XeP(e.)) + X₂ P(..k + X, A(r,.,) Ao P(..))
Sample task ÓN/OFF bit G(R..) Bernoull((((x.))
end
end
29

---

Published in Transactions on Machine Learning Research (12/2022)
Ground-iruth
Prodicted
Ground-truth
Predicted
Figure 8: Landmark localization results from the DST algorithm for the AFTW database.
Noley
Cleen ground-ruth
Predicted
Figure 9: Image denoising results from the DST algorithm for the AFLW database.
30

---

Published in Transactions on Machine Learning Research (12/2022)
Figure 11: For the consolidated latent fingerprint database, the task wise (a) training loss, (b) activation probability; and (c) activation using DST algorithm.
70
70
Figure 10: The six types of fingerprint orientation patterns (also known as level-1 features) used for six class classification tasks.
60
60
69
60
(1) Twin Loop
Task Activation Signal
50
Task Activation Signal
50
Task Activation Signal
Tack Activatioon Signal
50
Activation Signal
[
M00 U,
40
40
Epoch
(c)
30
30
20
20
(e) Right Loop
13
10
70
(d) Left Loop
60
Task activation probabality
50
+0
(b)
(c) Tented Arch
30
20
10
0.8
0.2
0.5
0.4
Arch
Tack:
Task
Task
70
60
50
Loss
40
(a) Whorl
30
10
1.8
1.6
1.2
0.0
0.2
31

---

Published in Transactions on Machine Learning Research (12/2022)
(a) Fingerprint
(b) Ground-truth mask
(c) Predicted mask
Figure 12: Few sample oulputs for fingerprinl segmentation (lask 2), compared against the ground-truth masks.
(a) Latent Fingerprint
RT
(b) Ground-trut -ruth mask
(c) Predicted mask
Figure 13: masks.
Few sample outputs for latent fingerprint segmentation (task 4), compared against the ground-truth
32

---

Published in Transactions on Machine Learning Research (12/2022)
0.9
0.8
True Accept Rate (TAR)
0.7
0.6
STL
MTL
0.5
GradNorm ==0.5
GradNorm ==1.0
0.4
GradNorm a=1.5
a=0.5
LBTW a=1.0
DTP
0.2
Adaptive Schedule
Random P=0.50
0.1
Random P=0.75
DST (Proposed)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
False Accept Rate (FAR)
Figure 14: ROC curve for latent fingerprint matching (task 5).
33


Reviews:
[{"review_id": "Uy3hibO2xJ", "reviewer": "Reviewer_psS9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The auhtors propose a different way for multi-task training. The novel point is from the task \"dropping\", which is different from previous works. In this drop way, the tasks can be dynamically decided to be training or not. \n2. The authors define different metrics for deciding the probability of the task drop, and these metrics are reasonably defined by considering the different aspects of the task and the training status.\n3. The experiments show effectiveness in some applications. \n\nWeaknesses:\n1. Though the four metrics are reasonably defined by considering the different views. One feeling is that these metrics are somehow heuristic and straightforward. It's hard to see the technique's novelty in terms of these metrics.\n2. One big concern of the paper is the definition of \"negative transfer\". While the authors claim the negative transfer needs to be avoided and this is the main motivation for this method. It is important to mathematically define what \"negative transfer\" is. The best way is with some statistical measurement. \n3. In terms of writing, there are some typos that need to be revised. For example, \"Furthermore, The xxx\". Besides, $k$ and $K$ represents differently. It is highly suggested to use unified notation. For example, $k$ and $K$ for layer, and $t$ and $T$ for tasks. \n4. It is also noted in Table 3, the results of multi-task training are not good as individual training. Therefore, this is not to show the effectiveness of multi-task training. Besides, compared to some previous works, the improvements are not consistent. This is a serious problem, but more modifications of the method are encouraged to achieve better performance. Figure 5 is not so clear, for example, the legend in the figure, and the comparison of the figure is not so obvious. It can be removed then. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a multi-task learning framework, which differs from previous work in that the authors define several metrics for \"dropping\" the task in the multi-task training framework so that the tasks are dynamically dropped. The authors claim that this dropped scheduled task can help remove the negative transfer from dominant tasks. The four metrics are (1) task depth, (2) number of ground-truth samples per task, (3) amount of training completed, (4) task stagnancy. Based on these metrics, a scheduling probability is decided then the tasks can be dropped during training. Then the authors conduct experiments on three types of applications with multiple tasks. The results show that the method is effective compared to previous works. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See above. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b3uyWyHlYt", "reviewer": "Reviewer_doDg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The main strength is the comprehensive investigation of the proposed DST algorithm. A series of experiments and analyses are conducted to investigate the DST algorithm. \n- The proposed DST algorithm is verified to be effective compared with several reported MTL algorithms. The overall average performance exceeds the baselines. \n- Ablation and visualization give some insights into the proposed five metrics and their effect. \n\n## Weaknesses\n- From the algorithm and method perspective, the proposed DST is simple and ad-hoc, involving the manually designed metrics at training time or based on training statistics. I am concerned about whether these metrics can be generalized when the test statistics are different from the training. \n- Although DST outperforms several baselines, except for Gradient Drop, all others are before 2020. The authors mentioned several other related methods of mitigating negative transfer, e.g., Liu et al. (2022), Sun et al. (2021), and Lee et al. (2018) (earlier one, but also focus on negative transfer). Is it possible to compare any of these new/related methods in the experiment? \n- There is a lacking of Omniglot settings and experiment details either in the method, experiment, or appendix section. \n- Formulation Equation issues:   \n(1) Incorrect formulation, Eq. (11), (12), and (18) are incorrect. For example, Eq. (11) should be $T_{1F_i} = -g_i \\log (P(g_i|X_{F_i}^{'}))  -(1-g_i) \\log (1-P(g_i|X_{F_i}^{'}))$, or alternatively define $g_i \\in \\mathbb{R}^{2}$ as a 2-dim formation.    \n(2) No definition for symbols, e.g., before sec 2.1,  $P_{(d,t)}, P_{(c,t)}, P_{(u,k,t)}$, and $P_{(r,k,t)}$ used before defined. Also, it is unclear what are $u$ and $r$ mean in the metrics.   \n(3) There is no punctuation after each equation.   \n(4) In Eq. (20), how is $D_i$ implemented? \n- Typos:   \nFor example, \"easier tasks(Lee et al., 2018)\", and \" explored(Han et al., 2017; Ranjan et al., 2017).\" miss a space before the left parentheses. \n- Formation issues:   \n(1) Figure 2 is not in the proper place;   \n(2) Unexpected space areas on Pages 11, 13, and 16;  \n(3) The rotated results table is OK but seems a little uncomfortable for the audience.\n\nI acknowledge the efforts regarding experiments the authors made in this paper, but I think the proposed method is kind of simple and ad-hoc. Also, there remain several issues that show the paper is not ready. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with the negative transfer problem in multi-task learning. The authors tackle the problem from a training tasks schedule perspective and propose the Dropped Scheduled Task (DST) algorithm. \n\nSpecifically, for each training task, the drop probability is determined by four metrics (two statics and two dynamics): network depth, ground-truth samples, training incompleteness, and training stagnancy.   \nThe former two are static ones according to the model and dataset statistics, and the latter two are dynamic ones either based on the per-epoch loss values or on the first-order difference between consecutive epoch loss values.   \nIn addition, a constant regularization metric is given.   \nThe final dropping probability is calculated from the weighted sum of all five metrics. \n\nThree sets of experiments are performed to verify the proposed algorithm: unilateral multi-task face applications, multilateral fingerprint applications, and multi-task Omniglot character recognition. Comparison with several baseline methods shows the effectiveness of the proposed DST algorithm. Ablation studies give intuitions about each designed metric in mitigating negative transfer. \n\nOverall, the contribution lies in two aspects: (1) the designed metrics to determine the dropping probability in the multi-task setting, (2) experiments on three multi-task settings show the effectiveness. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Recent and most related comparison methods\n- Correct the formulations\n- Correct other formations and typos\n\nOthers, please refer to the above weaknesses. \n", "broader_impact_concerns": "N/A ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KkslymurWq", "reviewer": "Reviewer_P6J7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe method is simple to implement and yields good results on considered tasks. \n\nWeakness:\nThe approach targets only limitations of MTL that are due to the training procedure, which could be overcome by scheduling the tasks. More precisely, it does not reason about potential incompatibilities between tasks which would render joint training a lost cause. In other words, should we expect there to exist _any_ algorithm without negative transfer? This limitation is reflected in the four metrics used for scheduling, none of which are directly concerned with the compatibility of learning targets. Naturally, such compatibility is hard to measure a priori, but in its absence it is difficult to assess _when_ we should expect the proposed method to work, and why. \n\nThe justification for the proposed approach is very informal, for example, by referring broadly to concepts like vanishing gradients due to network depth. Similarly, \"task incompleteness\" is defined as the ratio of the loss in the first epoch and the loss in the kth epoch, normalized by on the \"expectation\" of this ratio over tasks. It is is not clear to me that the different losses in 2.2 have comparable scales/units which would make such a normalization justified.\n\n(+ a few clarity issues in the next section)\n\nQuestions:\nRelated to the previous point, the ablation study on AFLW indicates that using only p(u,k,t) (task incompleteness) for scheduling achieves performance close to the full model. A potential take-away is the most important factor is to make sure that no task is left behind. Does this hold also for other datasets/MTL tasks? Do other MTL algorithms target this idea? In some sense, it appears opposite to the curriculum learning idea of focusing on easier tasks in the beginning of training. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a number of heuristic rules for dynamically removing the influence of specific tasks during multi-task learning (MTL). This is claimed to limit the effect of \"negative transfer\"---situations where the learning of one task contributes adversely to the learning of another. The heuristics are used to define a probability that the influence of a task is paused during one epoch of training. In a series of MTL experiments, the authors find that the proposed method compares favourably to baselines from the literature. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The concept of \"negative transfer\" is fundamental to the paper yet never formally defined. The authors cite Wang et al (2019) who attempt to define negative transfer, but, as a reader less familiar with the details of the MTL literature, I am not sure whether this definition is agreed upon in the community. Nevertheless, I think it would be appropriate to state _a_ definition in the paper.\n\nIn Figure 1, negative transfer is illustrated in terms of transfer between pairs of tasks A and B in the context of another task C, where A can have a negative effect on B and B has a positive effect on C. This type of transfer is never defined in the context of multiple (>2) tasks, and never estimated from what I can tell. A clarification would be useful. \n\nIn Figure 7, examples of the output from the DST algorithm are given, but no examples from other algorithms. This makes comparison difficult. Perhaps a few examples from other algorithms could be added?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-task learning framework, in which the authors define several metrics for "dropping" the task in the multi-task training framework so that the tasks are dynamically dropped. While the specific technical design is a little bit heuristic (multiple reviewers raised their concerns that the technical novelty and depth of the paper is limited, and its implication to more general tasks and its value to general audience are questionable), this is not the main evaluation criteria of TMLR. In terms of the two key criteria (claims and evidence, audience), the authors have done a reasonably good job. Therefore, I believe this paper is acceptable, provided that the authors can make necessary changes according to the detailed review comments.
Decision: Accept (poster)