SlowGuess/ACL_2020_OCR huggingface dataset:
Yuren Mao†, Shuang Yun‡, Weiwei Liu‡*, Bo Du‡
$\ddagger$ School of Computer Science, Wuhan University
†School of Computer Science and Engineering, University of New South Wales
yuren.mao@unsw.edu.au, liuweiwei863@gmail.com,
{yunshuang2014, dubo}@whu.edu.cn
Multi-task Learning methods have achieved significant progress in text classification. However, existing methods assume that multi-task text classification problems are convex multi-objective optimization problems, which is unrealistic in real-world applications. To address this issue, this paper presents a novel Tchebycheff procedure to optimize the multitask classification problems without any convex assumption. The extensive experiments back up our theoretical analysis and validate the superiority of our proposals.
Multi-task Learning (MTL) aims to learn multiple related tasks simultaneously, and obtain better performance than learning each task independently by setting inductive bias across tasks. (Caruana, 1993; Bakker and Heskes, 2003; Ben-David and Schuller, 2003; Ando and Zhang, 2005). It has achieved great success in various applications ranging from computer vision (Kendall et al., 2018) to text classification (Liu et al., 2016, 2017; Xiao et al., 2018).
Existing MTL methods for text classification, usually set up the inductive bias across tasks by designing a parameterized hypothesis class that shares some parameters across tasks (e.g. shares some hidden layers in a Neural Network), and cast the multi-task text classification problem as a multi-objective optimization problem.  $L_{1}$ -metric method is one of the most popular strategies for solving the multi-objective optimization problem. Specifically, it learns the parameters by minimizing a weighted linear combination of per-task losses. And this method is able to find an arbitrary Pareto optimal solution in the Pareto set if the problem is convex. Unfortunately, for a non-convex problem, this
(a) Convex case.
Figure 1: Graphical interpretation of Pareto optimization for weighted linear combination based MTL. The points of tangency between the line of linear combination and Pareto front are Pareto optimal points. (a). in the convex case, all the Pareto optimal points are achievable; (b). in the non-convex case, the Pareto optimal points located at the concave part of the Pareto front are unachievable.
(b) Non-convex case.
method excludes many Pareto optimal solutions from its search scope. To illustrate the issue, it is instructive to consider a 2-tasks learning case shown as Figure 1. From Figure 1, we can see that for a non-convex problem, the Pareto points located at the concave part of the Pareto front are unachievable. According to the uniform convergence properties of MTL (Baxter, 2000), the exclusion of Pareto optimal solutions may degenerate the generalization performance of multi-task text classification.
To address the non-convexity problems, this paper proposes a novel Tchebycheff procedure to improve the performance of multi-task text classification. To validate the superiority of the proposed method, we conduct the experiments on two classical text classification problems: sentiment analysis on reviews (Blitzer et al., 2007) and topic classification on news (Lang, 1995). The results show that our proposed method can converge and outperform several state-of-the-art multi-task text classification methods.
4217
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4217-4226 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
The family of Pareto optimality methods, including  $L_{1}$ -metric methods (weighted sum methods) (Maurer et al., 2016; Chen et al., 2018; Kendall et al., 2018) and multiple-gradient descent algorithm (MGDA) (Sener and Koltun, 2018), have become one of the most prevalent Multi-task Learning (MTL) strategies. In multi-task text classification,  $L_{1}$ -metric methods are widely used (Liu et al., 2016, 2017; Xiao et al., 2018; Yadav et al., 2018). However, for non-convex problems, the  $L_{1}$ -metric methods are likely to exclude the optimal hypothesis from the hypothesis class.
To handle the non-convex case, MGDA leverages the Karush-Kuhn-Tucker conditions and provides Pareto stationary points as solutions. However, the solutions are not sufficient to be Pareto optimal. A novel MTL method, which can achieve Pareto optimal without any convex assumption, is necessary to compensate for disadvantages in the  $L_{1}$ -metric and MGDA. In this paper, a novel Tchebycheff procedure is proposed to achieve Pareto optimal without any convex assumption.
3 MTL as Multi-objective Optimization
Consider a multi-task learning problem with  $T$  tasks over an input space  $\mathcal{X}$  and a collection of task spaces  $\{\mathcal{Y}^t\}_{t=1}^T$ . There is also a parametric hypothesis  $h = \{f^t\}_{t=1}^T \circ g = \{f^t(g(x, \theta^{sh}), \theta^t)\}_{t=1}^T: \mathcal{X} \to \{\mathcal{Y}^t\}_{t=1}^T$  for each task, where  $\theta^{sh}$  represents the parameters shared between tasks,  $\theta^t$  represents the task-specific parameters,  $g(\cdot, \theta^{sh}): \mathcal{X} \to \mathbb{R}^K$  is the feature map used across different tasks.  $K$  is the dimension of the representation space. The functions  $g(\cdot, \theta^{sh}): \mathcal{X} \to \mathbb{R}^K$  and  $f^t(\cdot, \theta^t): \mathcal{X} \to \mathcal{Y}^t$  are chosen from respective hypothesis classes  $\mathcal{G}$  and  $\mathcal{F}$ .  $h$  is in hypothesis classes  $\mathcal{H}$ . The choice of representation and specialized predictors is based on the data observed for all the tasks. The data takes the form of a multisample  $\overline{D} = \{D_t\}_{t=1}^T$ , with  $D_t = (\overline{X}_t, \overline{Y}_t)$  and  $(\overline{X}_t, \overline{Y}_t) = \{x_i^t, y_i^t\}_{i=1}^{n_t} \sim \mathcal{P}_t^{n_t}$ .
The task-specific training loss is denoted by  $\mathcal{L}^t (f^t (g(\overline{X}_t,\theta^{sh}),\theta^t),\overline{Y}_t)$  :  $\mathcal{V}^t\times \mathcal{V}^t\to \mathbb{R}^+$  . Correspondingly, the empirical loss of the task  $t$  is defined as  $\hat{\mathcal{L}}^t (\theta^{sh},\theta^t) = \frac{1}{n_t}\sum_{i = 1}^{n_t}\mathcal{L}^t (f^t (g(x_i^t,\theta^{sh}),\theta^t),y_i^t)$  We also denote the transpose of the vector/matrix by superscript , the logarithms to base 2 by log.
Figure 2: Comparison between  $L_{1}$  and  $L_{\infty}$  metric.  $L_{1}$  metric cannot achieve Pareto optimal points lying on the concave part while  $L_{\infty}$  metric can.  $L_{\infty}$  metric finds the set of weak Pareto optimal points, which includes the set of Pareto optimal points.
3.1 Multi-objective Optimization
MTL can be formulated as a multi-objective optimization problem that optimizes a collection of possibly conflicting objectives (Sener and Koltun, 2018). We formulate the optimization objective of MTL as a vector-valued loss  $\mathbf{L}$ :
$$
\min  _ {\theta^ {s h}; \theta^ {1}, \dots , \theta^ {t}} \mathbf {L} \left(\theta^ {s h}; \theta^ {1}, \dots , \theta^ {T}\right), \tag {1}
$$
where  $\mathbf{L}(\theta^{sh};\theta^1,\dots,\theta^T) = (\hat{\mathcal{L}}^1 (\theta^{sh},\theta^1),\dots,\hat{\mathcal{L}}^T (\theta^{sh},\theta^T))'$ . The goal of multi-objective optimization is to achieve the (weak) Pareto optimality.
Definition 1 (Pareto optimality for MTL). The Pareto optimality for MTL is defined as:
(i) A solution  $\theta$  dominates a solution  $\overline{\theta}$  if  $\hat{\mathcal{L}}^t (\theta^{sh},\theta^t)\leq \hat{\mathcal{L}}^t (\overline{\theta}^{sh},\overline{\theta}^t)$  for all tasks  $t$  and  $\pmb {L}(\theta^{sh};\theta^1,\dots,\theta^t)\neq \pmb {L}(\overline{\theta}^{sh};\overline{\theta}^1,\dots,\overline{\theta^t})$
(ii) A solution  $\theta^{*}$  is called Pareto optimal if there exists no solution  $\theta$  that dominates  $\theta^{*}$ .
Definition 2 (Weak Pareto optimality for MTL). A solution  $\theta$  is weakly Pareto optimal if there does not exist another solution  $\overline{\theta}$  such that  $\hat{\mathcal{L}}^t (\overline{\theta}^{sh},\overline{\theta}^t)< \hat{\mathcal{L}}^t (\theta^{sh},\theta^t)$  for all tasks  $t$ .
The set of (weak) Pareto optimal solutions are different trade-offs between tasks. The Pareto optimal set is a subset of the weakly Pareto optimal set.
3.2 Method of the Global Criterion
Global criterion is a standard technique for finding (weak) Pareto optimality, which optimizes all tasks
Figure 3: An original hard parameter sharing network model.
together by minimizing a weighted  $L_{p}$ -objective shown as (2).
$$
\min  _ {\theta^ {s h}; \theta^ {1}, \dots , \theta^ {t}} | | \left(w _ {1} \bar {\mathcal {L}} ^ {1}, \dots , w _ {T} \bar {\mathcal {L}} ^ {T}\right) | | _ {p}, \tag {2}
$$
where  $1 \leq p \leq \infty$ ,  $\bar{\mathcal{L}}^t = |\hat{\mathcal{L}}^t(\theta^{sh}, \theta^t) - l_t^*|$ ,  $w_t \geq 0$  and  $\sum_{t=1}^{T} w_t = 1$ .  $l_t^*$  is the ideal empirical loss of training task  $t$ .  $p = 1, 2$  or  $\infty$  are widely used choices. The  $L_\infty$  is a Tchebycheff metric. The state-of-the-art multi-task text classification methods use the  $L_1$  metric.
3.3  $L_{1}$ -metric versus  $L_{\infty}$ -metric
Non-convex Multi-objective Optimization:  $L_{\infty}$  metric can find every Pareto optimal solution without convex assumption. By contrast, the  $L_{1}$  metric excludes some Pareto optimal solutions when the problem is non-convex (Miettinen, 1998). It can be interpreted geometrically in a two-dimensional case shown as Figure 2. From Figure 2, we can see that a Pareto optimality is achieved at the point of tangency between the Pareto front and the surface formulated by  $L_{p}$  metric.  $L_{1}$  metric cannot be tangency to the Pareto optimal points located at the concave part of the Pareto front.
In practice, most of the multi-task text classification problems are non-convex multi-objective problems, especially when the Deep Neural Network involved. According to the uniform convergence properties of MTL (Baxter, 2000), the exclusion of Pareto optimal solutions will lead to the degenerated performance. Therefore, we use the  $L_{\infty}$  metric to boost the performance.
Weak Pareto optimality: The solution of a  $L_{\infty}$ -metric objective is weakly Pareto optimal. Figure 2 provides geometrical interpretation. Empirical risk combinations formulate the upper bound of the generalization error of MTL (Baxter, 2000). Weakly Pareto optimal set, which contains more candidate
Figure 4: An adversarial hard parameter sharing network model.
empirical risk combinations than the Pareto optimal set, can achieve a lower generalization error than Pareto optimal set.
Therefore, this paper presents to use  $L_{\infty}$ -metric to improve the performance of multi-task text classification.
4 Tchebycheff Procedure for Multi-task Text Classification
Many multi-task neural network models can be used in multi-task text classification, such as hard parameter sharing networks (Caruana, 1997) and soft parameter sharing networks (Liu et al., 2017; Xiao et al., 2018). This paper adopts a hard parameter sharing network model, because it has the lowest computational cost among the models.
4.1 Hard Parameter Sharing Network
Original hard parameter sharing network: A hard parameter sharing network learns multiple related tasks simultaneously by sharing the hidden layers across all tasks, while keeping task-specific output layers for each task shown as Figure 3.
The shared layers can be formulated by any feature extractor (e.g. long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997), TextCNN (Kim, 2014)), while the task-specific output layers are task dependent. In multi-task classification, the task-specific layers are usually formulated by fully connected layers ending with a softmax function.
Adversarial hard parameter sharing network: Cutting edge work (Liu et al., 2017) shows that adding an adversarial module to a MTL model can improve the performance. We extend the original hard parameter sharing network with an adversarial module shown as Figure 4. The adversarial module is essentially a task discriminator in the representation space, which discriminates which
Algorithm 1: Tchebycheff Procedure
Input: data  $D_{t} = (\overline{X}_{t},\overline{Y}_{t})$  , the number of training epochs  $N_{e}$
Initialization: Train each task  $t$  independently, get  $\overline{l^t}$  (the loss corresponding to the highest verification accuracy) and initialize  $\theta_0^{sh}$  with the hidden layers of task 1.
for  $i = 1$  to  $N_{e}$  do
$$
\hat {t} = \arg \max _ {t} \{\min _ {\theta^ {1}} w _ {1} \hat {\mathcal {L}} ^ {1} (\theta_ {i - 1} ^ {s h}, \theta^ {1}), \dots \}
$$
$$
\theta_ {i} ^ {s h}, \theta_ {i} ^ {t} = \arg \min _ {\theta^ {s h}, \theta^ {\hat {t}}} \hat {\mathcal {L}} ^ {\hat {t}} (\theta^ {s h}, \theta^ {\hat {t}})
$$
$$
\theta_ {N _ {e}} ^ {t} = \arg \min _ {\theta^ {t}} \hat {\mathcal {L}} ^ {t} (\theta_ {N _ {e}} ^ {s h}, \theta^ {t})
$$
$$
\text {r e t u r n} \theta_ {N _ {e}} ^ {s h}, \theta_ {N _ {e}} ^ {1}, \dots , \theta_ {N _ {e}} ^ {T}
$$
task a sample  $x$  belongs to and can be formulated as (3).
$$
D (x; W, b) = \operatorname {s o f t m a x} \left(W ^ {\prime} g \left(x, \theta^ {s h}\right) + b\right), \tag {3}
$$
where  $W\in \mathbb{R}^{K\times K}$  and  $b\in \mathbb{R}^K$
4.2 Tchebycheff Loss
To boost the performance in non-convex problems, we use the Tchebycheff  $(L_{\infty})$  metric to formulate the optimization objective.
The scales of empirical risks for different tasks can vary significantly. To normalize the scales, we divide each empirical risk in the MTL model with the empirical risk of learning the corresponding task independently, which typically have similar scale. That is, we define the weight  $w_{t}$  in (2) as (4).
$$
w _ {t} = \frac {1}{\bar {l} ^ {\bar {t}} \sum_ {i = 1} ^ {T} \frac {1}{\bar {l} ^ {i}}}, \tag {4}
$$
where  $\overline{l^t}$  is the empirical risk of learning task  $t$  independently. In practice, we set  $\overline{l^t}$  to be the training loss of training task  $t$  independently and achieving the highest accuracy in verification.
In the ERM (Empirical Risk Minimization) paradigm, it is reasonable to assume that the minimum empirical loss of each task equals 0. That is,  $l_{t}^{*} = 0$  in (2). Further more, the empirical losses are non-negative. This paper presents the Tchebycheff Loss for multi-task text classification as (5).
$$
\hat {\mathcal {L}} _ {c h e b} = \max  _ {t} \left\{w _ {1} \hat {\mathcal {L}} ^ {1} \left(\theta^ {s h}, \theta^ {1}\right), \dots , w _ {T} \hat {\mathcal {L}} ^ {T} \left(\theta^ {s h}, \theta^ {T}\right) \right\}, \tag {5}
$$
where  $w_{t}$  is defined in (4).
Algorithm 2: Adv Tchebycheff Procedure
Input: data  $D_{t} = (\overline{X}_{t},\overline{Y}_{t})$  , the number of training epochs  $N_{e}$ $\alpha$
Train the discriminator with  $\theta_{i - 1}^{sh}$  and get  $\hat{\mathcal{L}}_D^i$
$$
\mathbf {i f} \hat {\mathcal {L}} _ {D} ^ {i} \leq \alpha \text {t h e n}
$$
$$
\hat {t} = \arg \max  _ {t} \left\{\min  _ {\theta^ {1}} w _ {1} \hat {\mathcal {L}} ^ {1} \left(\theta_ {i - 1} ^ {s h}, \theta^ {1}\right), \dots \right\}
$$
$$
\theta_ {i} ^ {s h} = \arg \min  _ {\theta^ {s h}} \hat {\mathcal {L}} _ {D}
$$
$$
\theta_ {N _ {e}} ^ {t} = \arg \min _ {\theta^ {t}} \hat {\mathcal {L}} ^ {t} \left(\theta_ {N _ {e}} ^ {s h}, \theta^ {t}\right)
$$
return  $\theta_{N_e}^{sh},\theta_{N_e}^1,\dots,\theta_{N_e}^T$
4.3 Tchebycheff Loss for Adversarial MTL
The empirical loss of the discriminator can be formulated as (6).
$$
\hat {\mathcal {L}} _ {D} = \max  _ {W, b} \sum_ {t = 1} ^ {T} \frac {1}{n _ {t}} \sum_ {i = 1} ^ {n _ {t}} \mathbb {1} _ {y _ {i} = t} \log D \left(x _ {i} ^ {t}; W, b\right), \tag {6}
$$
where  $\mathbb{1}_{y_i = t}$  is the indicator function which equals to 1 when  $y_{i} = t$  otherwise 0.
In the adversarial MTL setting, we add the loss of the discriminator into the Tchebycheff loss. In the Tchebycheff procedure, we optimize  $\theta^{sh}$  with the discriminator when  $\hat{\mathcal{L}}_D > \alpha$ , where  $\alpha$  is a hyper parameter. (7) is the Tchebycheff loss for Adversarial MTL.
$$
\hat {\mathcal {L}} _ {c h e b A d v} = \max  \left\{\mathbb {1} _ {\hat {\mathcal {L}} _ {D} \leq \alpha} \hat {\mathcal {L}} _ {c h e b}, \mathbb {1} _ {\hat {\mathcal {L}} _ {D} > \alpha} \hat {\mathcal {L}} _ {D} \right\}, \tag {7}
$$
4.4 Tchebycheff Procedure
By minimizing the Tchebycheff loss (5) or (7), we can learn a (adversarial) hard parameter sharing network model. The training process of the model is defined as an (adversarial) Tchebycheff procedure, which is formulated as Algorithm 1 (Algorithm 2 for the adversarial model).
The networks are trained with backpropagation. In the adversarial Tchebycheff procedure, the dis
criminator is trained by using a gradient reversal layer (Ganin and Lempitsky, 2015).
The computational cost of training a hard parameter sharing network model with Tchebycheff procedure is higher than training it with a  $L_{1}$  metric. The extra cost comes from the process of selecting the task with maximum loss. However, it can be easily reduced by parallelly computing loss of each task.
In this section, firstly, we conduct a synthetic experiment to validate our theory analysis. Then, we perform experimental studies on two real-world applications: sentiment analysis and topic classification. The implementation is based on PyTorch (Paszke et al., 2019). The code can be found in the supplementary materials.
5.1 Synthetic Experiment
In this section, two 2-objective optimization problems, problem 1 and 2, are introduced to evaluate the performance of the  $L_{1}$  metric method and the  $L_{\infty}$  metric method. Problem 1 is a convex 2-objective optimization problem, while problem 2 is a non-convex 2-objective optimization problem.
$$
\begin{array}{l} \min  _ {x _ {1}, x _ {2}} \quad (x _ {1}, x _ {2}) ^ {\prime} \\ s. t. \quad x _ {2} \geq 1 / x _ {1} \\ x _ {1} \geq 0, x _ {2} \geq 0. \\ \end{array}
$$
$$
\min  _ {x _ {1}, x _ {2}} \quad (x _ {1}, x _ {2}) ^ {\prime}
$$
$$
s. t. x _ {2} \geq 1 / x _ {1} + 5 / (e ^ {(x _ {1} - 1) ^ {2}} + 1)
$$
$$
x _ {1} \geq 0, x _ {2} \geq 0.
$$
Let  $w_{1} \in \{0.01, 0.02, 0.03, \dots, 0.99, 1\}$  and  $w_{2} = 1 - w_{1}$ . We solve problem 1 by using the  $L_{1}$  metric method (minimizing  $w_{1}x_{1} + w_{2}x_{2}$ ) and  $L_{\infty}$  metric method (minimizing  $\max(w_{1}x_{1}, w_{2}x_{2})$ ) respectively. The results are shown in Figure 5. Then, we compare the  $L_{1}$  metric method with the  $L_{\infty}$  metric method in solving the non-convex problem 2. Figure 6 shows the results. Experimental results verify the superiority of the  $L_{\infty}$  metric method at handling non-convex case.
5.2 Real World Applications
5.2.1 Datasets
Sentiment Analysis<sup>1</sup>. We evaluate our algorithm on product reviews from Amazon. The dataset (Blitzer et al., 2007) contains product reviews from 14 domains: apparel, baby, books, camera photo, DVDs, electronics, health personal care, kitchen appliances, magazines, music, software, sports outdoors, toys, games and video. We consider each domain as a binary classification task. Reviews with rating  $>3$  are labeled positive, those with rating  $<3$  are labeled negative. Reviews with rating  $=3$  are discarded as the sentiments are ambiguous and hard to predict. The training/testing/validation partition is randomly split into  $70\%$  training,  $10\%$  testing, and  $20\%$  validation.
Topic Classification  ${}^{2}$  . We select 16 newsgroups from the 20 Newsgroup dataset, which is a collection of approximately 20,000 newsgroup documents. We formulate the 16 newsgroups into four 4-class classification tasks (shown as Table 1). The training/testing/validation partition is randomly split into  ${60}\%$  training,  ${20}\%$  testing,and  ${20}\%$  validation.
Table 1: Data Allocation for Topic Classification Tasks.
TASKS	NEWSGROUPS
COMP	OS.MS-WINDOWS.MISC, GRAPHICS, SYS.MAC.HARDWARE, WINDOWS.X .
REC	SPORT.BASEBALL, SPORT.HOCKEY AUTOS, MOTORCYCLES .
SCI	CRYPT, ELECTRONICS, MED, SPACE .
TALK	POLITICS.MIDEAST, RELIGION.MISC, POLITICS.MISC, POLITICS.GUNS.
5.2.2 Network Model
We implement our (adversarial) Tchebycheff Procedure via a deep MTL network with hard parameter sharing strategy (Caruana, 1997). As shown in Figures 3 and 4, all tasks have task-specific output layers and share the feature map layers. In the adversarial Tchebycheff Procedure, an extra adversarial module is added in the deep MTL network.
In our experiments, TextCNN (Kim, 2014) is used to build feature extraction module. The TextCNN is structured with 3 parallel convolutional layers with kernels size of 3, 5, 7, respectively.
$^{1}$ https://www.cs.jhu.edu/~mdredze/datasets/sentiment/
<sup>2</sup>http://qwone.com/~jason/20Newsgroups/
(a).  $x_{2} = 1 / x_{1}$
(b).  $L_{1}$  metric
(c).  $L_{\infty}$  metric
Figure 5: Convex case. (a) shows the Pareto front of problem 1. (b) shows the Pareto optimal points that the  $L_{1}$  metric method achieves with different  $w_{1}$  and  $w_{2}$ . (c) shows the Pareto optimal points that the  $L_{\infty}$  metric method achieves with different  $w_{1}$  and  $w_{2}$ . Both the  $L_{1}$  and  $L_{\infty}$  metric method can find all Pareto optimal points.
(a).  $x_{2} = 1 / x_{1} + 5 / (exp((x - 1)^{2}) + 1)$
(b).  $L_{1}$  metric
Figure 6: Non-convex case.(a) shows the Pareto front of problem 2. (b) shows the Pareto optimal points that the  $L_{1}$  metric method achieves with different  $w_{1}$  and  $w_{2}$ . (c) shows the Pareto optimal points that the  $L_{\infty}$  metric method achieves with different  $w_{1}$  and  $w_{2}$ . The  $L_{1}$  metric method excludes the Pareto optimal points located at the concave part, while the  $L_{\infty}$  metric method can find all Pareto optimal points.
The extracted feature representations are then concatenated and classified by the task-specific output module, which has one fully-connected layer.
The adversarial module is built with one fully connected layer whose output size equals to the number of the tasks. It is noteworthy that the adversarial module connects to the shared layers via a gradient reversal layer (Ganin and Lempitsky, 2015). The gradient reversal layer multiplies the gradient by  $-1$  during the backpropagation, which optimizes the adversarial loss function (6).
5.2.3 Training Parameters
We train the deep MTL network model according to Algorithms 1 and 2 respectively. We set  $\alpha$  be 2.5 and 1 for sentiment analysis and topic classification respectively. The learning rates are  $1e - 4$  and  $3e - 4$  for sentiment analysis and topic classification respectively. We use Adam optimizer (Kingma and Ba, 2015) and train 3000 epochs for both sentiment analysis and topic classification. The batch size is 256. We use dropout with a probability of 0.5
for both adversarial modules and all task-specific output modules.
5.2.4 Results and Analysis
Classification Accuracy
We compare our proposed methods with baselines and some state-of-the-art methods: (i) Single Task: solving tasks independently, (ii) Uniform Scaling: minimizing a uniformly weighted sum of loss functions, (iii) MGDA: using the MGDA-UB method proposed by (Sener and Koltun, 2018). (iv) Adversarial MTRL: using the adversarial MTL framework proposed by (Liu et al., 2017).
We report results over 10 runs by plotting classification accuracy of each classification task for sentiment analysis and topic classification in Figures 7 and 8 respectively. Figures 7 and 8 visually compare the classification accuracy performances between all the methods. The numerical results validate that the proposed (adversarial) Tchebycheff procedure outperforms the state-of-the-art methods.