SlowGuess/NAACL_2022_OCR huggingface dataset:
analysis. As shown in Figures 5 and 6, GPT2 achieves higher validation accuracy, when its training losses, standard language modeling and loss corresponding to label position, have higher value
Dataset	Task	Type	Input sequence
train	inference
SemEval14	Single task	aspect term polarity prediction	<review> once we sailed, the top-notch food and live entertainment sold us on a unforgettable evening. <endofreview> <term> food positive, live entertainment positive <endofterm>	<review> once we sailed, the top-notch food and live entertainment sold us on a unforgettable evening. <endofreview> <term> food
SemEval14	Joint task	aspect term	<review> once we sailed, the top-notch food and live entertainment sold us on a unforgettable evening. <endofreview> <term> food positive, live entertainment positive <endofterm>	<review> once we sailed, the top-notch food and live entertainment sold us on a unforgettable evening. <endofreview>
SemEval14	Multi-task	aspect term & aspect category	<review> the service was attentive without being overbearing and each dish we tried was wonderful from the spring rolls to the cod with pineapple tempura. <end-ofreview> <term> service positive, dish positive, spring rolls positive, cod with pineapple tempura positive <endofterm> <category> food positive, service positive <endofcategory>	<review> the service was attentive without being overbearing and each dish we tried was wonderful from the spring rolls to the cod with pineapple tempura. <end-ofreview>
SST-2	Single-task	polarity prediction	<review> does n’t try to surprise us with plot twists, but rather seems to enjoy its own transparency <endofreview> <sentiment> positive <endofsentiment>	<review> does n’t try to surprise us with plot twists, but rather seems to enjoy its own transparency <endofreview> <sentiment>
SST-5	Single-task	polarity prediction	<review> it’s a lovely film with lovely performances by buy and accorsi . <end-ofreview> <sentiment> somewhat positive <endofsentiment>	<review> it’s a lovely film with lovely performances by buy and accorsi . <end-ofreview> <sentiment>
OOS	Single-task	intent prediction	<user> how would you say fly in italian <endofuser> <intent> translate <end-ofintent>	<user> how would you say fly in italian <endofuser> <intent>
Table 5: Examples of input sequence during training and inference of generative language model for different datasets.
Shot	Layers	Joint Accuracy	Term	Category
SB1 (F1)	SB2 (Acc)	SB3 (F1)	SB4 (Acc)
1%	12	20.75	39.26	19.69	62.82	43.4
24	20.62	37.87	18.99	61.79	41.51
5%	12	31	44.35	32.38	74.46	56.51
24	34.87	60.4	35.18	75.39	59.06
10%	12	38.37	62.47	35.98	77.43	61.32
24	41.75	65.9	40.06	79.27	62.92
20%	12	42.88	66.82	39.91	79.39	62.36
24	45	72.73	45.31	80.79	65.28
100%	12	51.63	77.43	49.71	85.34	70.57
24	55.62	81.53	57.92	82.4	70.38
Table 6: Multi-task evaluation on SemEval14 restaurant domain (SB1-4) on few-shot settings using generative language model (GPT2).
Shot	Layers	Joint Accuracy	Term	Category
SB1 (F1)	SB2 (Acc)	SB3 (F1)	SB4 (Acc)
1%	12	11.6	28.68	13.38	46.36	38.31
24	9.04	24.87	11.36	44.32	35.63
5%	12	18.43	33.81	16.74	56.85	50.06
24	20.48	34.99	18.88	61.09	54.66
10%	12	21.16	33.48	16.74	63.11	50.45
24	22.18	37.13	19.64	67.12	55.43
20%	12	25.77	37.74	20.63	69.39	62.07
24	26.96	40.6	22.15	72.9	65.39
100%	12	32.42	48.48	27.67	76.51	66.41
24	43	50.27	30.15	76.78	69.6
Table 7: Multi-task evaluation on SemEval16 restaurant domain (SB1-4) on few-shot settings using generative language model (GPT2).
than BERT and GPT2-classifier. This indicates that perhaps BERT and GPT2-classifier overfitted
to the few-shot training data. On the other hand, GPT2 language model achieves more supervision
via standard language modeling loss, which results in higher training loss, but better validation performance.
F Ablation: Model weights update during training
In order to understand models behavior during training on few-shot data, we study the weight update at each layer of GPT2 and BERT models, during training on  $1\%$  few-shot data. For each layer, the mean normalized weight update is defined as,
$$
\sum_ {i = 0} ^ {k} \frac {\left(w _ {i} ^ {l} - w _ {i - 1} ^ {l}\right)}{w _ {0} ^ {l}} \tag {10}
$$
where  $l$  indicate the layer index,  $i$  indicates training step, and  $w_0^l$  refers to initial weight value before training. The comparison between GPT2 as generative gpt2-generative, GPT2 as an ecoder for classification gpt2-classifier and BERT model when trained on 1% few shot data of SemEval14 restaurant domain are shown in Figure 7. The results indicate that Bert model has higher variance for all layers, especially for the randomly-initialized classification layer. Moreover, the mean normalized update of BERT model is larger than gpt2-generative early during training, but is smaller at the end of training, where gpt2-generative achieves higher validation performance, as shown in Figure 3. Furthermore, the mean normalized update in embedding layer of gpt2-generative is significantly larger than BERT and gpt2-classifier by one order of magnitude early at training, which increased to two orders of magnitude at the end. We conjecture that higher value in layer weights update at embedding layers, and at the end of training for other layers is perhaps due to using standard language modeling loss, which may provide more supervision signal for GPT2, compared to cross-entropy loss in BERT and gpt2-classifier models.
G Ablation: Other Sentiment Analysis Tasks
In order to extend the investigate the performance of our proposed generative language model to other sentiment analysis tasks, we also evaluate few-shot performance on SST-5 sentiment analysis dataset (Socher et al., 2013) (binary and fine-grained sentiment classification), and OOS (Larson et al., 2019) intent detection dataset. The results are shown in Figure 8, which indicate the superiority of generative model (GPT2) over discriminative
BERT. On intent detection, Figure 8(c), GPT2 also outperforms TOD-BERT (Wu et al., 2020) which exploits extra pretraining on dialogue datasets to increase its few-shot performance.
As described in section 4.2 and Table 2, aspect term extraction on restaurant domain (SemEval14) is improved in multi-task learning. To better understand model behavior, some examples are shown in Table 8. Using aspect category as supervision in multi-task learning helps the model to more accurately generate the aspect terms, reduces false positive aspect terms and wrong polarity predictions. Moreover, multi-tasking helps to better predict category polarity, using supervision from aspect term during training. Some examples of wrong prediction are shown in Table 9. It indicates that when there are negative or conflict polarity, the model struggles to correctly predict everything correctly. This often happens when there are opposite opinions for different aspect terms or categories.
(a) SemEval14 Laptop Aspect Term Polarity (SB2)
(b) SemEval14 Restaurant Aspect Category Polarity (SB4)
Figure 5: Analysis of few-shot training convergence, evaluated on SemEval14 for  $1\%$  and 1-shot training data, and few-shot performance on all settings (right). GPT2-classifier model uses a classification layer on the output of last input token without using language modeling loss for training. (best viewed in color)
Sentence	Task	Model Output
the sangria's - watered down.	aspect term	<sterm> sangria negative
aspect category	<category> food neutral
aspect term & category	<term> sangria negative <category> food negative
groundtruth	<term> sangria negative <category> food negative
everyone who works there (the host, the bartender, the servers) is so helpful.	aspect term	host positive, bartender neutral, servers positive
aspect category	<category> service positive
aspect term & category	<term> host positive, bartender positive, servers positive <category> service positive
groundtruth	<term> bartender positive, host positive, servers positive <category> service positive
in mi burrito, here was nothing but dark chicken that had that cooked last week and just warmed up in a microwave taste.	aspect term	<term> dark chicken negative, microwave taste neutral
aspect category	<category> food negative
aspect term & category	<term> chicken negative, taste negative <category> food negative
if you like seafood and or greek food you will love this place though it is not limited to just these things.	aspect term	<sterm> seafood positive, greek food positive, place negative
aspect category	<category> food positive
aspect term & category	<term> seafood positive, greek food positive <category> food positive
groundtruth	<term> greek food positive, seafood positive <category> food positive
Table 8: Examples of correct predictions in multi-task learning.
(a) SemEval16 Restaurant Aspect Term Polarity (SB2)
(b) SemEval16 Restaurant Aspect Category Polarity (SB4)
(c) SemEval16 Laptop Aspect Category Polarity (SB4)
Figure 6: Analysis of few-shot training convergence, evaluated on SemEval16 for  $1\%$  and 1-shot training data, and few-shot performance on all settings (right). GPT2-classifier model uses a classification layer on the output of last input token without using language modeling loss for training. (best viewed in color)
Sentence	Task	Model Output
certainly not the best sushi in new york, however, it is always fresh, and the place is very clean, sterile.	aspect term	<sushi negative, place positive
aspect category	<ambience positive, food positive
aspect term & category	<sushi positive, place positive <category|>food positive, ambience positive
groundtruth	<ambience positive, food conflict
while there's a decent menu, it shouldn't take ten minutes to get your drinks and 45 for a dessert pizza.	aspect term	menu positive, drinks positive, dessert pizza positive
aspect category	food conflict
aspect term & category	<ambience positive, menu positive, drinks positive, dessert pizza positive <category|>food positive
groundtruth	<ambience positive, dessert pizza neutral, drinks neutral, menu positive <category|>food positive, service negative
the portions of the food that came out were mediocre.	aspect term	portions negative, food neutral
aspect category	food negative
aspect term & category	<ambience positive, portions negative, food negative <category|>food negative
groundtruth	<ambience positive, portions of the food neutral <category|>food neutral
Table 9: Examples of wrong prediction for joint and multi-task generative language model.
(f)
Figure 7: Model Layers mean normalized update, Eq. (10), during training. Normalized update of the weight  $w$  at training step  $i$  is defined as  $(w_{i} - w_{i-1}) / w_{0}$ . Results are for training on 1% few-shot data on SemEval14 restaurant aspect term polarity (SB2) prediction task for 4 random seed. Shaded area indicates standard deviation.
(b)
Figure 8: Few-shot evaluation of GPT2 and BERT models on SST2 dev set and OOS intent detection datasets. Note: 1-shot refers to one example per class. (best viewed in color)