SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Full Results Table 7 and Table 8 present results of aligning models with RL framework. Generally, alignment with RL falls short in performance when compared to the contrastive learning (CL) framework, i.e., our proposed model. In the task of emotional support conversation, as depicted in Table 7, aligning with RL proves to be incompatible with ours and can occasionally result in a degradation in the base model's performance. As for the task of RoT generation, as shown in Table 8, the best performance when aligning each base model is nearly always achieved by our model. Alignment with RL notably increases evaluation metric values of T5 (small) and Flan-T5 (base) when employing a greedy or  $p = 0.9$  decoding strategy. However, a common issue among models aligned with RL is their tendency to generate phrases like " (something) is good/wrong" and "It is good/wrong to (something)". Although the key information (something) may be incorrect, such a pattern can exhibit significant overlap in wording with the ground truth, thereby yielding higher values in terms of some evaluation metrics. The potential reason for the suboptimal performance of RL could be the small size of ESConv and MIC. Although RL has proven potent in certain scenarios, as illustrated by
6Blender-Vanilla/Joint: https://github.com/thu-coai/Emotional-Support-Conversation; MultiESC: https://github.com/lwgkzl/MultiESC.
<sup>7</sup>https://github.com/SALT-NLP/mic
<sup>8</sup>https://huggingface.co/docs/trl/index
Table 7: Automatic evaluation results on ESConv. † represents significantly better than the corresponding base model with  $p < 0.01$ .
Model	B-1	B-2	B-3	B-4	R-L	METEOR	CIDEr	Extreme
Blender -Vanilla	Base	17.85	7.08	3.60	2.11	17.06	7.46	15.44	51.02
RL	19.31	7.01	3.19	1.70	15.64	7.54	12.85	50.10
Ours	\( 20.75^{\dagger} \)	\( 8.32^{\dagger} \)	\( 4.17^{\dagger} \)	\( 2.39^{\dagger} \)	\( 17.41^{\dagger} \)	\( 8.21^{\dagger} \)	\( 16.57^{\dagger} \)	50.38
Blender -Joint	Base	18.70	7.30	3.61	2.03	17.66	7.56	16.91	50.95
RL	21.26	8.43	4.12	2.29	17.69	8.32	16.96	51.48
Ours	\( 21.05^{\dagger} \)	\( 8.97^{\dagger} \)	\( 4.74^{\dagger} \)	\( 2.78^{\dagger} \)	\( 19.39^{\dagger} \)	\( 8.48^{\dagger} \)	\( 20.34^{\dagger} \)	\( 51.81^{\dagger} \)
MultiESC	Base	20.36	8.80	4.92	3.14	21.00	8.58	30.69	52.74
RL	21.58	8.80	4.74	2.96	20.47	8.78	28.58	51.65
Ours	\( 21.59^{\dagger} \)	\( 9.56^{\dagger} \)	\( 5.33^{\dagger} \)	\( 3.36^{\dagger} \)	\( 21.50^{\dagger} \)	\( 9.03^{\dagger} \)	\( 32.65^{\dagger} \)	\( 53.15^{\dagger} \)
Table 8: Automatic evaluation results on MIC. † represents significantly better than the corresponding base model with  $p < 0.01$ .
Model	R-1	R-2	R-L	BertScore	ScareBLEU	Avg.Len
T5 (Small)	Beam	Base	53.44	32.97	52.17	93.44	29.05	8.94
RL	52.64	32.44	51.68	93.21	29.19	7.42
Ours	53.45	32.82	52.09	\(93.49^{\dagger}\)	28.51	8.99
Greedy	Base	37.04	16.30	35.27	90.94	14.27	10.47
RL	\(40.04^{\dagger}\)	\(19.71^{\dagger}\)	\(38.86^{\dagger}\)	\(91.67^{\dagger}\)	\(18.67^{\dagger}\)	7.45
Ours	\(38.15^{\dagger}\)	\(17.22^{\dagger}\)	\(36.40^{\dagger}\)	\(91.29^{\dagger}\)	\(15.15^{\dagger}\)	9.74
p=0.9	Base	40.22	19.23	38.56	91.59	16.71	9.85
RL	\(42.84^{\dagger}\)	\(22.28^{\dagger}\)	\(41.69^{\dagger}\)	\(92.10^{\dagger}\)	\(20.74^{\dagger}\)	7.41
Ours	\(41.41^{\dagger}\)	\(20.22^{\dagger}\)	\(39.75^{\dagger}\)	\(91.90^{\dagger}\)	\(17.75^{\dagger}\)	9.38
Flan-T5 (Base)	Beam	Base	55.07	34.96	53.74	93.77	30.68	9.00
RL	50.84	30.95	49.78	93.18	27.95	7.30
Ours	\(55.18^{\dagger}\)	\(35.07^{\dagger}\)	\(53.86^{\dagger}\)	\(93.79^{\dagger}\)	30.83	9.01
Greedy	Base	37.94	17.23	36.13	91.39	15.36	9.78
RL	\(42.97^{\dagger}\)	\(22.36^{\dagger}\)	\(41.54^{\dagger}\)	\(92.27^{\dagger}\)	\(20.71^{\dagger}\)	7.30
Ours	\(38.34^{\dagger}\)	17.52	\(36.53^{\dagger}\)	\(91.44^{\dagger}\)	15.49	9.77
p=0.9	Base	41.41	20.41	39.70	92.02	18.02	9.30
RL	\(45.61^{\dagger}\)	\(25.11^{\dagger}\)	\(44.26^{\dagger}\)	\(92.64^{\dagger}\)	\(22.97^{\dagger}\)	7.27
Ours	\(41.78^{\dagger}\)	\(20.69^{\dagger}\)	\(40.09^{\dagger}\)	\(92.07^{\dagger}\)	\(18.26^{\dagger}\)	9.32
BART (Large)	Beam	Base	54.81	35.07	53.35	93.85	30.80	9.44
RL	53.02	33.22	51.58	94.26	29.91	8.39
Ours	\(55.05^{\dagger}\)	\(35.18\)	53.62	\(93.86^{\dagger}\)	30.85	9.40
Greedy	Base	54.77	34.85	53.30	93.84	30.51	9.54
RL	53.30	33.30	51.83	94.28	29.88	8.52
Ours	54.81	34.86	53.39	93.83	30.51	9.48
p=0.9	Base	54.77	34.96	53.32	93.84	30.62	9.56
RL	53.36	33.37	51.89	94.29	29.96	8.55
Ours	54.86	35.01	53.45	\(93.85^{\dagger}\)	30.63	9.48
recent research [41, 31, 16], a commonality among these successful applications is their training on expansive datasets (280K to 6M training instances). In contrast, the sizes of ESConv and MIC are significantly more modest. This hypothesis also aligns with findings from Agarwal et al. [1], which suggest that smaller datasets can potentially undermine RL's performance.
C Case Study
We present case studies for the task of emotional support conversation and integrity RoT generation in Table 9 and Table 10, respectively.
Table 9: Green text: the help-seeker's emotion and problem. Blue text: the conversation context (history). Orange text: human evaluation results.
****Case One*****
The Help-seeker's Emotion: AnxietyThe Help-seeker's Problem: Need a second job to pay off credit card bills.Problem Type: Job crisis
Supporter: Hello, how are you today?Seeker: I'm good, busy. I need another job to pay more bills .
BaseOverall	I'm sorry to hear that. Is there anything you'd like to talk about?2,1,2	Global Consensus	2,2,2
AlignedmajorOverall	I'm sorry to hear that. Would you like to talk about it?3,2,2	Global Consensus	3,3,2
AlignedsoftOverall	I'm sorry to hear that. Would you like to talk about it?3,2,2	Global Consensus	3,3,2
Alignedd-PMOverall	I'm sorry to hear that. Is there a way you can get a second job to pay off your debts?4,5,4	Global Consensus	4,4,5
********** Case Two**********
The Help-seeker's Emotion: AnxietyThe Help-seeker's Problem: Applying for work, I have been retired for 2 years and would like to return to the workforce.Problem Type: Procrastination
Seeker: I have been retired for over 2 years and would like to return to the workforce, part-time.Supporter: So you are thinking of going back to work?Seeker: Yes, it would help us greatly.Supporter: You have the right idea already.Secker: There are plenty of jobs and not sure where to start.
BaseOverall	That is a great idea. I think it would be great to get your mind off of retirement.2,3,3	Global Consensus	2,4,3
AlignedmajorOverall	Yes, there are plenty of jobs out there. You just need to know where to start.4,4,3	Global Consensus	3,4,3
AlignedsoftOverall	I see. So you are thinking about getting back into the field?4,4,4	Global Consensus	4,4,4
Alignedd-PMOverall	That is a very good question. Would you be open to looking for a new job?4,5,4	Global Consensus	4,4,4
********** Case Three**********
The Help-seeker's Emotion: SadnessThe Help-seeker's Problem: I have a friend who knew me since I was little, she is closer to my mother's age so she is kinda like my aunt growing up. She always treat me like I was little. But I'm a grown woman now and I want our relationship to go far and grow more.Problem Type: Problems with friends
Seeker: I have a friend who knew me since I was little. She is a little older than me. She still treats me like I'm still little. I tried to explain to her that I'm a grown woman and I grew out of a lot of things when I was a kid.Supporter: Is she listening to you? I know how it feels to be patronized by people who are older.Seeker: She still didn't get it. I want our relationship to move further but with her like this. I feel sad. Is there anything I can do?Supporter: Well, if she is not listening to you when you try to talk to her you have to ask yourself whether the relationship is really worth it?Seeker: That's true . I didn't think about it. I know she is a good person. So I would give her a chance and we can talk about it before.
BaseOverall	I think it's always worth talking to someone who is older than you.2,1,3	Global Consensus	2,3,3
AlignedmajorOverall	Well, it's always worth talking to someone who has been through a similar situation.3,4,2	Global Consensus	3,3,2
Alignedsoft	Well, it's always worth talking to someone who has been through a similar situation.
Overall	3,4,2	Global Consensus 3,3,2
Alignedd-PM	I think that's a great idea. I think it's great that you are willing to give it a try.
Overall	4,5,5	Global Consensus 4,5,4
Table 10: RoTs generated by different models, whose base model is BART-Large. Blue text: prompt-reply (QA) pair. Orange text: human evaluation results.
****Case One*****
Question: Is it weird that I don't want a family or a career?
Answer: No, it's not weird at all. It's perfectly normal to not want children or a family.
Base	It is okay to not want children or a family.	Well-formedness	1,1,1
Relevance	4,3,5
Global Consensus	3,3,4
Alignedsoft	It is good to have a family.	Well-formedness	0,0,1
Relevance	3,2,5
Global Consensus	1,2,3
Alignedd-PM	It is okay not to want a family or a career.	Well-formedness	1,1,1
Relevance	4,4,5
Global Consensus	5,4,4
********** Case Two**********
Question: Who was the best teacher you ever had? What qualities made them great teachers?
Answer: I don't know. I guess he had a lot of patience and he made me feel safe.
Base	It is good to have a good teacher.	Well-formedness	0,1,0
Relevance	2,2,1
Global Consensus	4,3,2
Alignedsoft	It is good to have a good teacher.	Well-formedness	0,1,0
Relevance	2,2,1
Global Consensus	4,3,2
Alignedd-PM	It is good for teachers to be patient with their students.	Well-formedness	1,1,1
Relevance	5,4,4
Global Consensus	4,4,3
********** Case Three**********
Question: HELP, a hotel sink fell into the cupboard. Am I responsible or liable? Should I try to fix it or report it to the desk? HELP
Answer: I would call the front desk and let them know. They should be able to help you.
Base	It is good to ask for help when you need it.	Well-formedness	1,1,0
Relevance	1,2,2
Global Consensus	3,4,2
Alignedsoft	It is good to report maintenance issues to the proper authorities.	Well-formedness	1,1,0
Relevance	3,3,1
Global Consensus	3,3,1
Alignedd-PM	It is good to report broken things to the hotel desk.	Well-formedness	1,1,1
Relevance	3,3,4
Global Consensus	4,4,5
D Human Rating Details
Annotation Details We engaged six annotators from diverse educational, gender, geographic, and occupational backgrounds to conduct human ratings for each task. Each sentence to be rated was assigned to three annotators, and the final result was calculated as the average of their scores. Given
that  $3 \sim 4$  sentences were generated by different models for the same input, we presented these sentences and their corresponding context or prompt-reply pair on a single page. Annotators were asked to score the generated sentences relative to each other. To avoid potential bias, the order of sentences was randomized to prevent annotators from identifying the generation model based on sentence order.
Annotation Instructions Annotators were initially briefed on the tasks to ensure they understood the objectives of the emotional support responses or RoTs. Except for the global consensus, they were asked to rate the generated responses according to their personal preferences. For the emotional support conversation task, we asked annotators to imagine themselves as help-seekers in the conversations. After providing the situational and emotional context, annotators then rated the generated responses.
Annotation Cost We remunerated annotators at a rate of  $2 \sim 3$  times their local hourly minimum wage. According to the annotators' feedback, they spent approximately 40 seconds evaluating each generated response or RoT.