SlowGuess/ACL_2020_OCR huggingface dataset:
Figure 7: Classification accuracy of Single Task Learning (single), Uniform Scaling (uniform), MGDA (mgda), Adversarial MTRL (mtrl_adv), Tchebycheff procedure (tchebycheff) and adversarial Tchebycheff procedure (tchebycheff_adv) on sentiment analysis dataset. Each colored cluster shows the classification accuracy performance of a method over 10 runs. Adversarial Tchebycheff procedure has a better average performance than Tchebycheff procedure. Our proposed methods outperform Single Task Learning in all tasks and outperform Uniform Scaling, MGDA, Adversarial MTRL in most tasks. (Adversarial) Tchebycheff procedure's average performance dominates the state-of-the-art methods.
Figure 8: Classification accuracy of Single Task Learning (single), Uniform Scaling (uniform), MGDA (mgda), Adversarial MTRL (mtrl_adv), Tchebycheff procedure (tchebycheff) and adversarial Tchebycheff procedure (tchebycheff_adv) on topic classification dataset. Each colored cluster shows the classification accuracy performance of a method over 10 runs. Adversarial Tchebycheff procedure has a better average performance than Tchebycheff procedure. Our proposed methods outperform Single Task Learning in comp, sci, talk and outperform Uniform Scaling, MGDA, Adversarial MTRL in sci, talk. (Adversarial) Tchebycheff procedure's average performance dominates the state-of-the-art methods.
To verify the convergence of the proposed (adversarial) Tchebycheff procedure, we plot curves of training loss for each task and discriminator in
Figure 9 for topic classification. The (adversarial) Tchebycheff procedure obtains similar convergence curves in sentiment analysis. The results verify that our method converges rapidly. From
Figure 9: Convergence curve of task specific loss achieved by adversarial Tchebycheff procedure for topic classification. The empirical loss decreases rapidly in the first 500 epochs and then tend towards convergence. The absolute value of loss of the discriminator is higher than  $\alpha$  after 500 epochs
Figure 10: Color map for the Tchebycheff procedure in the training process for sentiment analysis. In the first 500 epoch, all tasks appear evenly. Then, the frequency of occurrence of each task is various.
Figure 11: Color map for adversarial Tchebycheff procedure in the training process for topic analysis. The adversarial module only appears in the first 500 epoch.
Figure 9, we can see that the adversarial module only works in the first 500 epochs.
Tchebycheff Procedure Visualization
We visualize the Tchebycheff procedure and adversarial Tchebycheff procedure with color maps
as shown in Figures 10 and 11. In the color maps, each task has a specific color and each epoch is colored by the task with the maximum loss. Here, we display the color maps for sentiment analysis.
Figures 10 and 11 show that the (adversarial) Tchebycheff procedure is a dynamic procedure,
Table 2: Average training time (second/epoch) comparison between Uniform Scaling method (uniform), MGDA (Sener and Koltun, 2018), Adversarial MTRL (adv MTRL) (Liu et al., 2017), Tchebycheff procedure (TP), adversarial Tchebycheff procedure (adv TP), Multi-processing Tchebycheff procedure (MTP-TP) and adversarial Multi-processing Tchebycheff procedure (adv MTP-TP).
LEARNING TASK	UNIFORM	MGDA	ADV MTRL	TP	MTP-TP	ADV TP	ADV MTP-TP
SENTIMENT ANALYSIS	1.5	3.3	2.8	3.1	2.3	3.1	2.3
TOPIC CLASSIFICATION	0.7	1.4	1.3	1.5	1.3	1.5	1.3
which changes optimization objective according to its strategy  $(L_{\infty}$  metric) in each epoch and finally achieves better performance. The procedure is totally different from existing methods, which optimize all tasks together.
We run the code on a server with a 2.2GHz Intel CPU and a single NVIDIA GeForce RTX 2080Ti GPU. The results of the average training time for each epoch in (adversarial) Tchebycheff procedure (TP) are shown in Table 2. From Table 2, we can see that the (Adversarial) Tchebycheff procedure is slower than the Uniform Scaling method and Adversarial MTRL (Liu et al., 2017).
In an adversarial Tchebycheff procedure, optimizing the adversarial task (4.5s per epoch for sentiment analysis and 2.1s per epoch for topic classification) is more time-consuming than optimizing a single task (3.5s per epoch for sentiment analysis and 1.5s per epoch for topic classification). However, optimizing the adversarial module appears less than 100 epochs. The extra computational cost resulted from the adversarial training can be ignored.
We are able to accelerate the (adversarial) Tchebycheff procedure with Multi-processing. In Multi-processing (adversarial) Tchebycheff procedure, we accelerate the procedure of selecting the task by computing the loss of each task in different processes. We implement the code by using the multiprocessing package in PyTorch. From Table 2, we can see that Multi-processing (adversarial) Tchebycheff procedure outperforms MGDA and Adversarial MTRL.
Most of multi-task text classification problems are non-convex multi-objective optimization problems. However, existing methods ignore the nonconvexity and solve the problems using convex optimization methods. To address this issue, this
paper presents an (adversarial) Tchebycheff procedure for multi-task text classification without any convex assumption. Numerical experiments show that our proposed methods can converge and outperform state-of-the-art methods.
In the Tchebycheff Procedure, we choose the weight for each task according to the empirical risk of learning the corresponding task independently. Obtaining the empirical risk is a little laborious. In the future, it would be fruitful to develop a novel weighting strategy for the Tchebycheff Procedure.
This work is supported by the National Natural Science Foundation of China under Grants 61976161 and 61822113, and the Fundamental Research Funds for the Central Universities under Grant 41300082.
Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853.
Bart Bakker and Tom Heskes. 2003. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 4:83-99.
Jonathan Baxter. 2000. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198.
Shai Ben-David and Reba Schuller. 2003. Exploiting task relatedness for multiple task learning. Springer.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL.
Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In ICML, pages 41-48.
Rich Caruana. 1997. Multitask learning. Machine Learning, 28(1):41-75.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pages 793-802.
Yaroslav Ganin and Victor S. Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In ICML, pages 1180-1189.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pages 7482-7491.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In ICML, pages 331-339.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In *IJCAI*, pages 2873-2879.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In ACL, pages 1-10.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. 2016. The benefit of multitask representation learning. Journal of Machine Learning Research, 17:81:1-81:32.
Kaisa Miettinen. 1998. Nonlinear multiobjective optimization. Kluwer.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS.
Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. In NeurIPS, pages 525-536.
Liqiang Xiao, Honglun Zhang, and Wenqing Chen. 2018. Gated multi-task network for text classification. In NAACL, pages 726-731.
Shweta Yadav, Asif Ekbal, Sriparna Saha, Pushpak Bhattacharyya, and Amit P. Sheth. 2018. Multi-task learning framework for mining crowd intelligence towards clinical treatment. In *NAACL*.