SlowGuess/NAACL_2021_OCR huggingface dataset:
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Roser Saurí and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. *Language Resources and Evaluation*, 43:227-268.
Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1537-1547, Denver, Colorado. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jy Wu, Jason Chuang, Chris Manning, Andrew Ng, and Chris Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, USA.
Anders Søgaard and Yoav Goldberg. 2016. Deep multitask learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 231-235, Berlin, Germany.
Veronika Vincze, György Szarvas, Richard Farkas, György Móra, and János Csrík. 2008. The Bioscope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC bioinformatics, Suppl 11.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Bo Wang, Maria Liakata, Arkaitz Zubiaga, and Rob Procter. 2017. TDParse: Multi-target-specific sentiment recognition on twitter. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 483-493, Valencia, Spain. Association for Computational Linguistics.
B. L. Welch. 1947. The generalization of 'student's' problem when several different population variances are involved. Biometrika, 34(1/2):28-35.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. *Language Resources and Evaluation*, 39(2-3):165-210.
Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83.
Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2514-2523, Melbourne, Australia. Association for Computational Linguistics.
Meishan Zhang, Yue Zhang, and Duy Tin Vo. 2015a. Neural networks for open domain targeted sentiment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), pages 612-621.
Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015b. Neural networks for open domain targeted sentiment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 612-621, Lisbon, Portugal. Association for Computational Linguistics.
A Class Distribution of the Sentiment Datasets
Train	Dev	Test
pos	neu	neg	both	pos	neu	neg	both	pos	neu	neg	both
Laptop	19.9	43.2	36.9	-	18.0	40.6	41.4	-	26.0	53.5	20.5	-
\( Laptop_{Neg} \)	-	-	-	-	17.1	35.4	47.5	-	26.7	23.3	50.0	-
\( Laptop_{Spec} \)	-	-	-	-	50.7	16.2	33.1	-	38.2	20.5	41.4	-
Restaurant	15.8	60.0	24.2	-	12.3	65.2	22.5	-	11.5	66.6	21.9	-
\( Restaurant_{Neg} \)	-	-	-	-	16.4	32.5	51.1	-	15.0	32.2	52.8	-
\( Restaurant_{Spec} \)	-	-	-	-	30.0	29.0	41.0	-	16.9	39.7	43.5	-
MAMS	45.1	30.2	24.7	-	45.5	30.3	24.3	-	45.5	29.9	24.6	-
MPQA	13.3	43.9	39.1	3.7	17.0	42.5	37.0	3.5	19.2	33.2	41.4	6.3
Table 7: Sentiment class distribution statistics as a percentage of the number of targets (samples), for the sentiment datasets used in the experiments. pos, neu, neg, and both represent the sentiment classes positive, neutral, negative, and both respectively.
B Examples of Auxiliary Tasks
you	might	not	like	the	service
NEGCD	Bscope	Iscope	Bcue	Bscope	Iscope	Iscope
NEG5FU	Bscope	Iscope	Bcue	Bscope	Iscope	Iscope
SPEC	Bscope	Bcue	Bscope	Iscope	Iscope	Iscope
UPOS	PRON	AUX	PART	VERB	DET	NOUN
DR	nsubj	aux	advmod	root	det	obj
LEX	OPRON	O AUX	OADV	BV-vemotion	ODET	BN-n.ACT
Table 8: A toy example sentence with the labels from each auxiliary task
C Additional Main Result Tables
Laptop	acc-s	GloVe	NEGCD	DR	LEX	NEG\(S_{FU}\)	SPEC	UPOS	STL
71.90(1.32)	70.66(1.55)	70.36(2.24)	70.30(1.86)	68.11*(2.19)	69.60*(1.98)	70.80(2.02)
TL	75.30(0.54)	74.75(1.14)	74.56(1.49)	74.36(1.74)	74.47*(0.82)	74.70*(1.02)	76.85(1.96)
F1-s	GloVe	65.00(1.36)	63.19(2.32)	63.07(3.51)	62.60*(2.74)	59.83*(2.46)	61.51(2.66)	61.90(3.32)
TL	66.92(2.41)	67.76(1.75)	67.26(2.27)	66.00(3.03)	66.92*(1.21)	66.63*(1.61)	69.91(2.72)
extraction	GloVe	76.00*(0.99)	75.98*(1.17)	77.99(1.14)	76.43(1.57)	75.81(1.57)	77.81(1.10)	76.76(1.69)
TL	83.51(1.09)	83.32(0.94)	83.88(0.88)	84.21(1.81)	83.29(1.15)	83.48(1.30)	82.90(0.72)
targeted	GloVe	54.65(1.37)	53.67(0.94)	54.85(0.99)	53.73(1.93)	51.65*(2.32)	54.17(2.26)	54.37(2.56)
TL	62.89(1.18)	62.29(1.32)	62.55(1.66)	62.61(1.79)	62.03(1.14)	62.35(0.77)	63.70(1.14)
MPQA	acc-s	GloVe	78.18(2.72)	74.37(3.47)	75.94(3.02)	77.38(4.91)	72.82*(3.88)	73.83*(2.30)	73.01*(3.41)
TL	71.84(3.46)	72.01(3.01)	73.08(4.01)	70.96(2.03)	72.79(3.13)	72.61(3.84)	70.47(1.51)
F1-s	GloVe	42.03(1.50)	39.96(1.66)	40.58(1.28)	41.16(2.32)	39.19(1.94)	39.90*(1.06)	39.00(1.86)
TL	39.92(1.15)	40.27(0.80)	41.13(3.05)	39.17(0.79)	39.84(1.56)	39.90(1.66)	39.25(0.68)
extraction	GloVe	24.17(1.44)	22.93*(1.57)	24.58(1.36)	22.84*(1.58)	22.90*(2.49)	25.34(0.46)	24.77(3.55)
TL	30.98*(2.40)	30.71*(1.10)	31.19*(2.69)	31.41*(1.16)	31.41*(0.51)	31.88*(2.62)	34.99(1.10)
targeted	GloVe	18.88(1.17)	17.03*(1.12)	18.66(1.22)	17.60*(0.57)	16.70*(2.26)	18.70(0.25)	18.11(2.83)
TL	22.25*(2.00)	22.09*(0.70)	22.74(1.68)	22.30*(1.19)	22.86*(0.98)	23.05*(0.88)	24.66(1.07)
Table 9: acc-s,  $F_{1}$ -s, extraction ( $F_{1}$ -a) and full targeted ( $F_{1}$ -i) results for Laptop and MPQA test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a 95% confidence level.
MAMS	acc-s	GloVe	NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
81.72(1.12)	81.42(0.32)	81.24(0.44)	81.10(0.46)	80.90(0.70)	81.58(0.96)	81.70(0.79)
TL	84.59(0.50)	84.81(0.78)	83.99*(0.84)	83.73(0.68)	84.28(0.39)	83.90*(0.51)	84.67(0.56)
F1-s	GloVe	81.05(1.15)	80.71(0.43)	80.53(0.60)	80.39(0.69)	80.22(0.80)	80.81(1.01)	80.94(0.88)
TL	84.24(0.51)	84.39(0.80)	83.59*(0.87)	83.30(0.67)	83.89(0.32)	83.44*(0.60)	84.24(0.58)
extraction	GloVe	76.49(0.99)	76.21*(0.39)	76.49(1.06)	76.87(0.64)	76.83(0.48)	76.97(0.54)	77.36(0.19)
TL	77.04(0.35)	76.76*(0.13)	76.98(0.84)	77.64(0.36)	76.54*(0.79)	77.33(0.61)	77.59(0.35)
targeted	GloVe	62.50(0.42)	62.05*(0.32)	62.14*(0.83)	62.34*(0.54)	62.16(0.71)	62.79(0.37)	63.20(0.65)
TL	65.17(0.35)	65.10(0.63)	64.65*(0.88)	65.00*(0.48)	64.50*(0.79)	64.88(0.46)	65.70(0.55)
Restaurant	acc-s	GloVe	83.02(1.82)	83.23(1.69)	83.26(0.89)	83.80(0.78)	83.01(1.16)	83.36(1.09)	83.65(0.48)
TL	87.40(0.67)	87.63(0.76)	87.37(0.90)	87.26(0.96)	87.36(0.48)	87.00*(0.56)	87.32(0.66)
F1-s	GloVe	66.75(3.75)	67.79(3.00)	67.59(1.39)	67.75(1.92)	67.35(3.02)	67.13(2.31)	68.00(1.61)
TL	72.27(1.14)	72.96(1.79)	73.73(2.60)	72.12(2.30)	73.90(2.82)	71.61(1.13)	73.47(1.10)
extraction	GloVe	78.33(1.55)	79.34(1.60)	79.13(0.93)	78.53(0.96)	78.48(0.81)	78.84(0.78)	78.42(0.85)
TL	81.27(0.90)	81.53(1.01)	82.13(1.35)	82.08(1.09)	81.85(1.22)	80.89*(1.37)	80.94(1.18)
targeted	GloVe	65.06(2.66)	66.06(2.63)	65.89(1.32)	65.82(1.31)	65.16(1.50)	65.73(1.46)	65.60(1.06)
TL	71.04(1.13)	71.45(1.47)	71.77(1.88)	71.63(1.64)	71.51(1.16)	70.38(1.63)	70.68(1.53)
Table 10: acc-s,  $F_{1}$ -s, extraction ( $F_{1}$ -a) and full targeted ( $F_{1}$ -i) results for MAMS and Restaurant test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a 95% confidence level.
Laptop	acc-s	GloVe	NEGCD	DR	LEX	NEG\(S_{FU}\)	SPEC	UPOS	STL
77.47(1.43)	77.53(0.69)	76.52(1.38)	78.34(2.23)	77.32(1.06)	76.67(1.77)	77.22(1.05)
TL	81.22(1.42)	79.63*(1.56)	80.54(2.27)	80.05*(1.28)	80.87(1.64)	79.31*(1.12)	81.89(1.07)
F1-s	GloVe	71.55(0.81)	70.75(2.75)	70.61(2.46)	72.04(3.85)	70.70(1.83)	67.82(1.20)	67.27(2.80)
TL	74.53(3.09)	74.84(1.87)	74.89(3.32)	73.70(1.98)	75.22(2.03)	71.90*(1.77)	75.33(1.61)
extraction	GloVe	74.87(1.15)	73.55(2.09)	75.70(0.96)	74.81(1.18)	74.75(0.57)	74.15*(1.84)	75.06(1.06)
TL	80.82(1.35)	80.89(0.83)	80.39(1.02)	81.86(1.24)	80.05*(0.59)	81.37(0.99)	81.23(0.82)
targeted	GloVe	57.99(0.69)	57.02(1.53)	57.92(1.08)	58.62(2.19)	57.80(1.10)	56.82(0.71)	57.97(1.24)
TL	65.62*(0.76)	64.42(1.64)	64.73(1.30)	65.52*(0.52)	64.72*(0.97)	64.55*(1.53)	66.51(0.43)
MPQA	acc-s	GloVe	87.75(3.15)	88.65(4.20)	87.64(3.71)	89.11(3.29)	86.85(1.46)	88.16(3.20)	85.29*(0.89)
TL	88.63(1.83)	90.08(2.94)	87.23(1.80)	85.62(4.14)	88.71(2.89)	88.75(1.56)	88.01(1.36)
F1-s	GloVe	54.18(8.15)	59.87(14.46)	51.51(8.83)	56.39(10.28)	48.09(4.83)	52.32(9.96)	45.92(3.06)
TL	52.83*(3.40)	55.80(5.28)	59.03(5.99)	54.48(9.33)	56.55(3.28)	53.82(7.01)	55.74(6.52)
extraction	GloVe	20.68(0.65)	21.00(1.73)	20.78(1.52)	20.48*(0.85)	19.54*(2.18)	21.73(0.74)	20.11(2.38)
TL	32.33(3.71)	30.39*(1.09)	31.75(1.78)	32.18(1.29)	30.65*(1.52)	31.00*(1.92)	33.38(0.67)
targeted	GloVe	18.14(0.58)	18.57(1.13)	18.18(1.10)	18.23(0.55)	16.94*(1.66)	19.16(0.89)	17.16(2.07)
TL	28.60(2.82)	27.35*(0.72)	27.67*(1.29)	27.53*(1.30)	27.15*(0.92)	27.49*(1.38)	29.39(0.95)
Table 11: acc-s,  $F_{1}$ -s, extraction  $(F_{1}-a)$  and full targeted  $(F_{1}-i)$  results for Laptop and MPQA development split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and TL at a  $95\%$  confidence level.
MAMS	acc-s	GloVe	NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
GloVe	80.73*	80.98	80.89*	80.68*	80.87*	81.36	82.14
TL	(0.22)	(1.17)	(0.45)	(0.48)	(0.53)	(0.82)	(0.77)
F1-s	GloVe	84.37	83.99*	84.73	84.18	84.17	83.79*	84.43
TL	(0.22)	(0.25)	(0.34)	(0.49)	(0.53)	(0.40)	(0.41)
extraction	GloVe	80.20*	80.48*	80.38*	80.14*	80.38*	80.76	81.67
TL	(0.24)	(1.15)	(0.50)	(0.57)	(0.56)	(0.95)	(0.77)
GloVe	84.14*	83.72*	84.46	83.96*	83.93	83.54*	84.12
TL	(0.24)	(0.21)	(0.35)	(0.47)	(0.47)	(0.42)	(0.45)
targeted	GloVe	78.93	79.11	79.24	79.00	78.86	78.68	79.15
TL	(0.66)	(0.52)	(0.62)	(0.47)	(0.67)	(0.35)	(0.39)
GloVe	77.81	77.86	77.62*	78.32	77.59*	78.54	78.35
TL	(0.48)	(0.59)	(0.34)	(0.31)	(0.21)	(0.38)	(0.40)
GloVe	63.72*	64.06*	64.10*	63.74*	63.76*	64.01*	65.01
TL	(0.63)	(0.66)	(0.48)	(0.28)	(0.21)	(0.62)	(0.44)
GloVe	65.65	65.39*	65.77*	65.93	65.31*	65.81	66.15
TL	(0.39)	(0.47)	(0.44)	(0.20)	(0.56)	(0.41)	(0.54)
Restaurant	acc-s	GloVe	78.42*	78.78	79.58	78.75	78.31*	79.14	78.76
TL	(0.78)	(0.67)	(0.89)	(0.52)	(0.78)	(0.41)	(0.37)
GloVe	81.90	81.90	81.53	81.89	81.02	80.47*	81.77
TL	(0.69)	(0.69)	(0.86)	(0.84)	(0.85)	(1.10)	(0.46)
F1-s	GloVe	62.89	64.01	65.37	62.49*	62.54*	63.00*	63.15*
TL	(2.84)	(2.56)	(1.47)	(0.86)	(2.28)	(1.64)	(1.94)
GloVe	67.98	69.26	69.09	68.18	67.54	67.14*	69.37
TL	(3.46)	(1.07)	(1.72)	(1.90)	(3.88)	(1.05)	(0.97)
extraction	GloVe	78.22*	79.20	78.85	78.38*	78.21	79.62	79.18
TL	(0.78)	(1.07)	(1.08)	(0.73)	(1.37)	(0.48)	(0.76)
GloVe	81.69	81.84	82.56	82.25	82.07	82.48	82.33
TL	(0.71)	(0.88)	(0.79)	(0.22)	(0.68)	(0.61)	(0.52)
targeted	GloVe	61.34*	62.39	62.75	61.73*	61.25*	63.02	62.36
TL	(0.73)	(0.95)	(1.26)	(0.77)	(1.26)	(0.42)	(0.60)
GloVe	66.90	67.03	67.31	67.36	66.49	66.38*	67.32
TL	(0.40)	(1.06)	(0.32)	(0.66)	(0.52)	(1.31)	(0.55)
Table 12: acc-s,  $F_{1}$ -s, extraction  $(F_{1}-a)$  and full targeted  $(F_{1}-i)$  results for MAMS and Restaurant development split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a  $95\%$  confidence level.
D Additional Negation and Speculation Result Tables
NEGCD	DR	LEX	NEGSEFU	SPEC	UPOS	STL
LaptopNeg
F1-s	GloVe	40.88(2.17)	38.11*(1.71)	38.89(3.32)	42.33(1.19)	39.46*(2.98)	38.11*(2.02)	37.13*(2.78)
TL	44.81(2.40)	45.05(3.47)	44.58(2.29)	43.53(2.74)	43.83(1.90)	44.77(1.54)	45.08(2.68)
RestaurantNeg
F1-s	GloVe	46.58(3.24)	44.16*(2.18)	42.74*(1.09)	47.65(1.35)	44.00(3.99)	44.78(1.85)	44.14*(1.89)
TL	52.85*(1.69)	54.41(1.51)	54.08(3.87)	52.54*(1.99)	55.63(1.65)	52.16*(2.01)	53.59*(1.95)
Table 13:  $F_{1}$ -s results for the negation test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.
NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
F1-s	GloVe	32.74(2.35)	33.14(0.98)	35.24(2.16)	33.62(2.59)	33.83(1.62)	33.21(1.00)	31.99*(1.92)
TL	33.02(4.07)	33.33(1.56)	33.14(2.10)	31.72*(0.88)	33.25(2.14)	32.71(1.38)	34.08(1.40)
F1-s	GloVe	55.27(3.82)	57.77(2.91)	56.27*(2.36)	55.59*(1.09)	57.35(3.75)	56.55(3.14)	57.32(2.30)
TL	58.84*(1.58)	60.95(0.98)	62.36(2.86)	58.44*(2.24)	60.52(3.25)	59.23(1.81)	60.74(2.32)
Table 14:  $F_{1}$ -s results for the speculation test split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.
LaptopNeg	acc-s	GloVe	NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
37.07(3.78)	32.99*(0.96)	33.98(2.60)	36.64(3.17)	34.83(1.09)	32.09*(3.41)	29.45*(1.46)
TL	41.84*(0.54)	39.38*(2.69)	40.85*(2.50)	45.40(1.70)	43.00*(1.57)	38.77*(2.64)	41.15*(3.28)
F1-s	GloVe	34.02(2.05)	31.48*(2.66)	33.06(2.21)	34.61(3.34)	33.66(1.13)	28.92*(4.11)	24.92*(1.33)
TL	38.26*(1.67)	38.05(3.06)	39.42(4.02)	42.32(3.45)	41.39(1.97)	36.73(3.67)	38.92(3.45)
extraction	GloVe	72.49(1.45)	74.40(1.64)	73.91(0.99)	71.25*(1.20)	73.83(0.93)	73.33(1.90)	74.60(1.51)
TL	80.94(1.64)	81.31(1.29)	81.21(1.36)	81.94(1.46)	79.00*(0.87)	81.92(1.36)	82.75(1.80)
targeted	GloVe	26.87(2.75)	24.56(1.21)	25.12(1.99)	26.14(2.55)	25.71(0.68)	23.54(2.66)	21.98*(1.26)
TL	33.86*(0.71)	32.04*(2.54)	33.14*(1.56)	37.20(1.59)	33.98*(1.47)	31.78*(2.46)	34.01*(2.29)
RestaurantNeg	acc-s	GloVe	46.02(4.88)	43.13*(3.24)	41.06*(3.36)	49.02(1.31)	41.65*(4.06)	44.02*(3.09)	42.69*(2.01)
TL	53.79(2.56)	54.40(3.63)	52.25(3.63)	54.42(3.18)	53.16(1.92)	54.31(2.49)	52.25(2.51)
F1-s	GloVe	40.03(5.30)	38.56*(3.10)	37.54(3.69)	41.05(2.45)	37.01(4.16)	38.03(3.40)	38.45(2.28)
TL	48.03(3.48)	49.13(2.62)	48.31(3.71)	49.18(3.58)	48.80(2.03)	49.42(2.76)	48.06(2.12)
extraction	GloVe	81.74(0.77)	82.37(0.64)	82.36(0.80)	80.61*(0.72)	81.34(1.48)	82.38(1.00)	81.32(0.37)
TL	84.08(0.72)	82.87*(0.66)	84.32(0.68)	83.71(0.55)	83.45(1.09)	84.02(1.12)	84.84(0.99)
targeted	GloVe	37.61(3.86)	35.54*(2.81)	33.82*(2.78)	39.51(1.03)	33.88*(3.38)	36.27*(2.64)	34.71*(1.58)
TL	45.23(2.19)	45.07(2.90)	44.04(2.77)	45.57(2.91)	44.38(2.05)	45.62(1.88)	44.31(1.74)
Table 15: acc-s,  $F_{1}$ -s, extraction ( $F_{1}$ -a) and full targeted ( $F_{1}$ -i) results for the negation development split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The  $\star$  represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a  $95\%$  confidence level.
LaptopSpec	acc-s	GloVe	NEGCD	DR	LEX	NEGSFU	SPEC	UPOS	STL
33.56*	35.00	31.70*	34.67	37.24	31.59*	32.57*
(2.39)	(2.94)	(1.64)	(2.21)	(2.48)	(1.57)	(1.94)
TL	34.73	35.02	35.16	32.79	34.24	34.27	34.28
(1.94)	(3.50)	(2.48)	(1.43)	(1.95)	(1.56)	(1.19)
F1-s	GloVe	32.66	33.99	30.72*	33.20	34.99	29.88*	30.75
(2.33)	(3.14)	(1.33)	(2.36)	(2.91)	(1.50)	(2.02)
TL	33.60	34.14	33.83	30.92	33.36	32.45	32.47
(2.30)	(3.47)	(2.83)	(1.48)	(1.98)	(1.32)	(1.67)
extraction	GloVe	71.10	69.42	72.61	70.46	69.33*	70.58*	73.04
(1.97)	(2.25)	(0.95)	(1.41)	(1.58)	(2.46)	(2.48)
TL	80.50	80.25	79.91*	80.83	79.35*	80.95	82.00
(0.95)	(0.90)	(0.61)	(0.61)	(1.34)	(0.43)	(1.32)
targeted	GloVe	23.88	24.26	23.02*	24.44*	25.82	22.32*	23.80
(2.05)	(1.81)	(1.35)	(1.81)	(1.82)	(1.73)	(1.77)
TL	27.96	28.11	28.08	26.51	27.16	27.74	28.12
(1.60)	(2.92)	(1.81)	(1.34)	(1.39)	(1.31)	(1.32)
RestaurantSpec	acc-s	GloVe	35.54*	40.09	37.98	37.18	37.97	38.32	37.23
(0.90)	(2.95)	(2.38)	(2.14)	(2.28)	(1.31)	(1.65)
TL	38.80	38.50	40.72	40.84	39.69	39.49	40.55
(2.17)	(1.19)	(1.18)	(2.30)	(1.69)	(0.89)	(1.16)
F1-s	GloVe	31.46*	35.99	34.37	32.53	34.02	32.92	33.18
(1.47)	(3.98)	(2.89)	(1.82)	(2.03)	(1.74)	(1.15)
TL	33.00*	33.28	35.92	35.26	34.47	35.08	35.40
(3.34)	(1.65)	(1.84)	(3.37)	(3.78)	(1.78)	(1.59)
extraction	GloVe	78.26*	80.38	80.33	79.75*	79.99*	81.98	80.59
(0.92)	(1.86)	(1.13)	(1.05)	(1.20)	(1.04)	(1.29)
TL	83.91	83.93	84.60	84.77	83.40	85.03	84.85
(0.67)	(1.03)	(0.63)	(0.29)	(1.58)	(1.47)	(0.83)
targeted	GloVe	27.82*	32.25	30.51*	29.64*	30.37	31.43	30.02
(0.79)	(2.79)	(2.06)	(1.60)	(1.91)	(1.42)	(1.74)
TL	32.56	32.31*	34.45	34.62	33.08	33.59	34.41
(1.83)	(1.13)	(1.00)	(1.96)	(1.01)	(1.20)	(1.23)
Table 16: acc-s,  $F_{1}$ -s, extraction ( $F_{1}$ -a) and full targeted ( $F_{1}$ -i) results for the speculation development split, where the values represent the mean (standard deviation) of five runs with a different random seed. The bold values represent the best model, while highlighted models are those that perform better than the single task baseline. The * represent the models that are statistically significantly worse than the best performing model on the respective dataset, metric and embedding at a 95% confidence level.
E Hyperparameter Search Space
GPU Infrastructure	1 GeForce GTX 1060 6GB GPU
CPU Infrastructure	AMD Ryzen 5 1600 CPU
Number of search trials	30
Search strategy	uniform sampling
Best validation span F1/F1-i	0.6156
Training duration	14232 sec
Model implementation	https://bit.ly/3lAz6yf
Hyperparameter	Search space	Best assignment
embedding	GloVe 300D	GloVe 300D
embedding trainable	False	False
number of epochs	150	150
patience	10	10
metric early stopping monitored	Span F1/F1-i	Span F1/F1-i
batch size	32	32
dropout	uniform(float[0, 0.5])	0.5
1stlayer LSTM hidden dimension	uniform-integer[30, 110]	60
main task LSTM hidden dimension	50	50
skip connection between embedding and main task layer	True	True
learning rate optimiser	Adam	Adam
learning rate	loguniform(float[1e-4, 1e-2])	1.5e-3
gradient norm	5.0	5.0
regularisation type	L2	L2
regularisation value	1e-4	1e-4
Table 17: STL search space and best assignment using the Laptop dataset.
GPU Infrastructure	1 GeForce GTX 1060 6GB GPU
CPU Infrastructure	AMD Ryzen 5 1600 CPU
Number of search trials	30
Search strategy	uniform sampling
Best validation span F1/F1-i	0.6017
Training duration	18473 sec
Model implementation	https://bit.ly/3lAz6yf
Hyperparameter	Search space	Best assignment
embedding	GloVe 300D	GloVe 300D
embedding trainable	False	False
number of epochs	150	150
patience	10	10
metric early stopping
monitored	Span F1/F1-i	Span F1/F1-i
batch size	32	32
dropout	uniform-float[0, 0.5]	0.27
Shared/1stlayer LSTM hidden
dimension	uniform-integer[30, 110]	65
main task LSTM hidden
dimension	50	50
skip connection between
embedding and main task layer	True	True
learning rate optimiser	Adam	Adam
learning rate	loguniform-float[1e-4, 1e-2]	1.9e-3
gradient norm	5.0	5.0
regularisation type	L2	L2
regularisation value	1e-4	1e-4
Table 18: MTL search space and best assignment using the Laptop dataset. The auxiliary task was negation detection using the Conan Doyle  $(\mathrm{NEG}_{CD})$  dataset.
Figure 2: Hyperparameter budget against expected span  $F1 / F_{1} - i$  performance for the STL and MTL models. The hyperparameter search space is stated within Tables 17 and 18 for the STL and MTL models respectively. The shaded areas represent the expected performance  $\pm 1$  standard deviation. Note the shaded area does not go beyond the maximum observed validation score as recommended Dodge et al. (2019).
F Additional Reproducibility Statistics
Embedding	Model	Number of Parameters
Including Auxiliary Task
Yes	No
Trainable	All	Trainable	All
GloVe	STL	364,042	1,785,967	364,042	1,785,967
NEGCD	385,851	2,403,876	385,122	2,403,147
NEGSSFU	385,851	7,066,176	385,122	7,065,447
SPEC	385,851	7,066,176	385,122	7,065,447
UPOS	388,413	2,952,738	385,122	2,949,447
DR	397,213	2,961,538	385,122	2,949,447
LEX	1,191,204	3,755,529	385,122	2,949,447
TL	STL	1,001,170	56,870,931	1,001,170	56,870,931
NEGCD	1,051,939	56,921,700	1,051,210	56,920,971
NEGSSFU	1,051,939	56,921,700	1,051,210	56,920,971
SPEC	1,051,939	56,921,700	1,051,210	56,920,971
UPOS	1,054,501	56,924,262	1,051,210	56,920,971
DR	1,063,301	56,933,062	1,051,210	56,920,971
LEX	1,857,292	57,727,053	1,051,210	56,920,971
Table 19: Number of parameters for each model using different embeddings ordered by number of trainable parameters. The number of parameters is different for the MTL models depending on whether the parameters from the auxiliary task are included or not. The auxiliary task specific layer is shown as the pink layer in Figure 1. The number of parameters including and not including the auxiliary task is stated as the MTL models at inference time would not use the auxiliary task parameters. There are many more trainable parameters for the MTL models ignoring the auxiliary task parameters. This is because the hyperparameter search finds a larger shared LSTM hidden dimension to be preferable for the MTL models (see Tables 17 and 18). For the GloVe MTL models the total number of parameters changes depending on the auxiliary task. This is because the GloVe embeddings contain different numbers of vocabulary words, as we filter words based on those in the auxiliary and main task datasets/corpora. The large difference in the number of trainable parameters between GloVe and TL models is due to the fact that the TL is 724 parameters larger than the 300 parameter GloVe embeddings. Lastly, the number of trainable parameters is dataset agnostic, the number of all parameters is not dataset agnostic for the GloVe models due to the vocabulary size, for clarification the model parameters reported here are for those trained on the Laptop dataset.