## 3 Methodology
### Sequential and Intensive Weighted Language Modeling (SIWLM)  
We denote the task that we target for improved performance as a central task and the remainder as auxiliary tasks. Furthermore, a task focused on in each iteration is termed a focusing task. SIWLM is learning scheme that adjusts the weights to be multiplied by the loss functions. In this scheme, all tasks have fixed initial weights, and each task is weight-adjusted from the initial weight during training. SIWLM consists of two stages: Sequential Weighted Learning (SWL) and Intensive Weighted Learning (IWL). In the SWL stage, all tasks used for the MTL have opportunities to be dominantly trained. In the IWL stage, only the central task that aims to obtain higher performance through MTL is intensively learned. Algorithm 1 describes the process of SIWLM in more detail. Each mini-batch consists of one type of dataset, and the model is aware of the type of each batch during training. Model training has two stages: SWL for odd epochs and IWL for even epochs. ```
0: Dataset \(D_{n}\), task type \(T_{n}\), initial weight \(iw_{n}\) when \(n\) is a dataset id (1\(\leq n\leq N\), \(N\) is the number of datasets)
1: Initialize shared layers as BERT pretrained model
2:for\(T\) in \(T_{1}\), \(T_{2}\),..., \(T_{N}\)do
3: Append a specific layer for \(T\)
4:endfor
5: Merge \(D_{1}\), \(D_{2}\),..., \(D_{N}\) into \(D\)
6: Divide \(D\) into mini-batches \(B\)={\(b_{1}\), \(b_{2}\),..., \(b_{k}\)} (\(k\) is the number of mini-batches and each mini-batch consists of only one dataset)
7:for\(epoch\) in 1, 2,..., \(N\)do
8:if\(epoch\) is odd then\(\triangleright\) SWL
9:\(i\gets 0\)
10: Set the central task to a focusing task
11:for\(b\) in \(B\)do
12:\(n\leftarrow\) dataset id of \(b\)
13:if\(i\) is 20 then
14: Change the focusing task
15:\(i\gets 0\)
16:endif
17:\(aw_{n}\gets 1\) (\(aw_{n}\) denotes an adjusted weight)
18:if\(b\) is not the focusing task then
19:\(aw_{n}\gets iw_{n}\) * 0.1
20:endif
21: Compute \(loss\)
22:\(loss\gets loss*aw_{n}\)
23: Update model
24:\(i\gets i+1\)
25:endfor
26:else\(\triangleright\) IWL
27:for\(b\) in \(B\)do
28:\(n\leftarrow\) dataset id of \(b\)
29:\(aw_{n}\gets 1\)
30:if\(b\) is not the central task then
31:\(aw_{n}\gets iw_{n}\) * 0.1
32:endif
33: Compute \(loss\)
34:\(loss\gets loss*aw_{n}\)
35: Update model
36:endfor
37:endif
38:endfor
```  
**Algorithm 1** Sequential and intensive weighted learning. In the SWL stage, one task to be trained with high weight is selected as a focusing task every 20 iterations. Because all tasks are sequentially selected as the focusing task, all tasks are assigned equal opportunities to be learned in-depth. We concentrate on the focusing task by computing adjusted weights, multiplying 0.1 to initial task weights except for the focusing task. Moreover, to update the model to reflect the weights of tasks, the computed loss values are multiplied by the adjusted weights. This approach is iterated over all iterations, enabling the model to learn all tasks. After the SWL stage, we use IWL to concentrate on the central task. In the IWL stage, the weight of the auxiliary tasks is multiplied by 0.1, which is the purpose of performance improvement in the central task. In contrast to in the SWL stage, the adjusted task weights are maintained throughout all iterations without any change. The task weight used in IWL is also multiplied by the loss to adjust the impact of each task on learning representation. In this stage, because the central task has a high weight during all iterations, the central task can be learned intensively. Figure 2 illustrates an example of applying SIWLM to eight tasks in GLUE. This example demonstrates how SIWLM is applied when RTE is a central task and the initial weights for each task are set to [6:1:1:1:1:1:1:1:1]. In the SWL stage, all learning tasks used for learning are sequentially selected as focusing tasks every 20 iterations. During the first 20 iterations, RTE is selected as the focusing task, and the weight of each task except RTE is multiplied by 0.1.