SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 11: On-policy vs. off-policy scores for different variants of TruLL: when training with an off-policy loss, we remove the truncation at test time.  $\mathrm{TruLL}_{\mathrm{off},\mathrm{KL}}$  is evaluated with  $\lambda_{KL} = 0.05$ . Best values are in bold.
Algo	Score	BLEU	CIDEr ppl-e	sBLEU	sumVA
CLEVR
TruFLL (Task-LM)
TruFLL	0.56	0.17	0.06	103	0.78	N.A
TruFLLoff	0.50	0.14	0.43	104	0.88	0.96
TruFLLoff,KL	0.39	0.17	0.71	69	0.48	0.95
TruFLL (Ext-LM)
TruFLL	0.48	0.08	0.34	3.03	0.95	N.A
TruFLLoff	0.41	0.10	0.35	105	0.88	0.95
TruFLLoff,KL	0.35	0.15	0.60	20	0.55	0.96
VQAv2
TruFLL (Task-LM)
TruFLL	0.35	0.21	0.11	104	0.36	N.A
TruFLLoff	0.07	0.03	0.01	104	0.05	0.08
TruFLLoff,KL	0.12	0.24	0.25	103	0.26	0.71
TruFLL (Ext-LM)
TruFLL	0.34	0.18	0.04	24	0.83	N.A
TruFLLoff	0.09	0.04	0.01	104	0.05	0.07
TruFLLoff,KL	0.0	0.15	0.02	103	0.19	0.47
Table 12: On-policy vs. off-policy scores per decoding procedure: when training with an off-policy loss, we remove the truncation at test time. TrufLLoff,KL is evaluated with  $\lambda_{KL} = 0.05$ . Best values are in bold.
method	text-gen	score	BLEU	CIDEr	e-ppl
CLEVR
TrufLL (Task-LM)
TrufLL	greedy	0.57	0.17	0.65	39
sampling	0.55	0.17	0.66	24
lm-ranking	0.51	0.16	0.65	8.8
TrufLLoff	greedy	0.52	0.17	0.58	71
sampling	0.49	0.16	0.59	\( 10^5 \)
lm-ranking	0.48	0.17	0.58	19
TrufLLoff,KL	greedy	0.56	0.18	0.78	24
sampling	0.31	0.16	0.62	\( 10^2 \)
lm-ranking	0.31	0.18	0.74	5.8
TrufLL (Ext-LM)
TrufLL	greedy	0.48	0.09	0.34	3.1
sampling	0.48	0.10	0.35	3.1
lm-ranking	0.48	0.06	0.34	2.9
TrufLLoff	greedy	0.42	0.10	0.38	4.4
sampling	0.40	0.10	0.35	\( 10^6 \)
lm-ranking	0.40	0.10	0.34	15
TrufLLoff,KL	greedy	0.48	0.16	0.70	2.1
sampling	0.27	0.13	0.48	55
lm-ranking	0.30	0.16	0.61	2.0
VQAv2
TrufLL (Task-LM)
TrufLL	greedy	0.36	0.20	0.11	366
sampling	0.35	0.20	0.11	337
lm-ranking	0.34	0.21	0.11	95
TrufLLoff	greedy	0.09	0.04	0.02	\( 10^3 \)
sampling	0.05	0.03	0.01	\( 10^6 \)
lm-ranking	0.06	0.03	0.01	\( 10^4 \)
TrufLLoff,KL	greedy	0.16	0.29	0.46	38
sampling	0.08	0.19	0.09	\( 10^4 \)
lm-ranking	0.12	0.24	0.22	\( 10^2 \)
TrufLL (Ext-LM)
TrufLL	greedy	0.48	0.09	0.34	3.1
sampling	0.48	0.10	0.35	3.1
lm-ranking	0.48	0.06	0.34	2.9
TrufLL off	greedy	0.11	0.05	0.01	\( 10^2 \)
sampling	0.07	0.03	0.01	\( 10^5 \)
lm-ranking	0.08	0.04	0.01	\( 10^4 \)
TrufLLoff,KL	greedy	0.00	0.18	0.05	27
sampling	0.00	0.13	0.01	\( 10^3 \)
lm-ranking	0.00	0.16	0.02	\( 10^2 \)
C Human Evaluation details
For the Human Evaluation study, we designed one form per participant, with three sections evaluating respectively the language quality, language grounding and diversity criteria. Given the five evaluated models, there are ten different model pairs: each section of the form contains 10 pairwise comparison covering all the possible model pairs for the criteria. Each pairwise comparison is sampled uniformly over the 50 first question samples generated by the algorithms at test time. The evaluation of syntax errors was made within the diversity section: for each questions pair, we asked participants to tick the questions if they are grammatically incorrect.
Figure 5 displays one pairwise comparison example for the three sections, and a full form example is available at the following url: https://forms.gle/kkL38x31wF7A9YKx5.
Evaluation of question quality
Select the question that is the most syntactically and semantically correct, and is more likely to be human language.
Pair #4*
Q1: What is the cat inside of?
Q2: What color is the man's shirt?
(a) Language Quality pairwise comparison
Question Relevancy with image and answer
Select the question that is the most suitable given the image and answer.
Answer: california
Pair #2*
Q1: What is the girl wearing on his shirt?
Q2: Where could it must be appropriate found?
(b) Language Grounding pairwise comparison
Figure 5: Examples of pairwise comparison for each evaluated criteria.
Figure 5: Examples of pairwise comparison for each evaluated criteria. (cont.)
(c) Diversity/Originality with reference question. Pairwise comparison and evaluation of syntax errors.
D Additional VQG Samples
Figure 6 and Figure 7 display the 10 first dialog samples produced at test time on CLEVR, while figures 8, 9, and 10 display the 15 first dialog samples produced at test time on VQAv2.
Figure 6: Samples on CLEVR.
Human	what is the size of the other rubber cylinder that is the same color as the big cylinder?	A:small
pretrain	there is a purple object that is the same size as the purple rubber cylinder; what is its shape?
pretrain + RL	what size is the gray ball that is right of the purple sphere?
scratch	that greater tiny as shiny both are a tiny it either ball right
scratch+KL-task	there is a big thing that is the same color as the big matte cylinder; what is its shape?
scratch+KL-ext	how material is the yellow?
TruLL (Task-LM)	how big is the thing that is to the right of the big matte thing?
TruLL (Ext-LM)	what size is the object that is behind the large red thing?
Human	There is a shiny thing that is both right of the small matte thing and behind the large yellow cube; what size is it?	A:small
pretrain	there is a big thing that is the same color as the small rubber cylinder; what is its shape
pretrain + RL	there is a brown matte object to the right of the cyan object; what shape is it?
scratch	many yellow big either either that that that more that metal?
scratch+KL-task	what number of other things are the same shape as the small gray thing?
scratch+KL-ext	what is the material of that block?
TruLL (Task-LM)	what shape is the big thing that is to the right of the big cyan thing?
TruLL (Ext-LM)	what is the shape of that large thing?
Human	there is a object that is the same color as the rubber cylinder;what is its shape?	A:sphere
pretrain	there is a small thing that is the same color as the small rubber cylinder; what is its shape?
pretrain + RL	what shape is the thing that is the same color as the cylinder?
scratch	many yellow yellow and cube shape behind cubes shape less small equal shape small equal large large?
scratch+KL-task	how many other things in the color are of same material as the green shiny object?
scratch+KL-ext	how many spheres anything?
TruLL (Task-LM)	what is the shape of the small cyan thing?
TruLL (Ext-LM)	And shape?
Human	what is the color of the small thing that is the same shape asthe large gray object?	A:green
pretrain	there is another rubber object that is the same shape as the small brown object; what color is it?
pretrain + RL	what is the color of the tiny rubber thing?
scratch	many sphere less how an
scratch+KL-task	what number of other objects are the same shape as the tiny blue object?
scratch+KL-ext	what size is that cylinder?
TruLL (Task-LM)	what is the color of the tiny matte thing?
TruLL (Ext-LM)	what color is the small thing?
Human	what number of shiny objects are cyan spheres or tiny balls?	A:4
pretrain	how many other things are there of the same size as the brown rubber thing?
pretrain + RL	how many other things are there of the same size as the cyan rubber thing?
scratch	many yellow do do do either do either do balls
scratch+KL-task	what number of other things are there of the same shape as the small gray thing?
scratch+KL-ext	how many other things are of same color as ball?
TruLL (Task-LM)	how many other things are the same material as the small cyan cylinder?
TruLL (Ext-LM)	how many other things in the material of the small thing that is the same material as green thing?
Figure 7: Samples on Clevr.
Figure 8: Samples on VQA.
Human	How many trains? A:1
pretrain	How many trains are in the picture?
pretrain + RL	How many trains are shown?
scratch	AmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	How many people are in the picture?
scratch+KL-ext	The the same way of the same way of the
TrufLL (Task-LM)	How many windows are here?
TrufLL (Ext-LM)	How many different ways would we take them to reach
Human	What is the man wearing over his shirt? A:nothing
pretrain	What is in front of the person?
pretrain + RL	What is written on the person's right?
scratch	AmazingAmazingAmazingAmazingAmazing
scratch+KL-task	Is this child old or unloaded?
scratch+KL-ext	The the first thing of the world's a few
TrufLL (Task-LM)	How many different shapes have been used and used by
TrufLL (Ext-LM)	How many planes can be seen on top?
Human	Is the fence in the background metal or wood?	A:metal
pretrain	What kind of dog is on the dog?
pretrain + RL	What kind of dog is this?
scratch	AmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What is 150 seated over theware?
scratch+KL-ext	The the same way of the same way to be
TrufLL (Task-LM)	Which item has more aggressive up their legs?
TrufLL (Ext-LM)	How many different species have been captured and tagged here
Human	What is the title of the red book?	A: harry potter
pretrain	What is the cat inside of?
pretrain + RL	Is the cat inside or outside?
scratch	Amazing Amazing Amazing Amazing Amazing
scratch+KL-task	What color is the man's shirt?
scratch+KL-ext	The way of the world's a lot of the
TrufLL (Task-LM)	Which item appears higher into one and lower?
TrufLL (Ext-LM)	What was that?
Human	What color is the fire hydrant?	A:red
pretrain	What color is the hydrant?
pretrain + RL	What color is the hydrant?
scratch	AmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What color is the man's bat?
scratch+KL-ext	The first thing is a good thing that the
TruflL (Task-LM)	Which color is this fire?
TruflL (Ext-LM)	What color will your feet color look?
Human	How many wheels does the truck have? A:3
pretrain	How many people are in front of the bus?
pretrain + RL	How many slices ofists are on the plate?
scratch	AmazingAmazingAmazingAmazing
scratch+KL-task	Is summer out or cloudy next to Winchester?
scratch+KL-ext	The the most recent of the most recent years of
TrufLL (Task-LM)	How many pieces are here?
TrufLL (Ext-LM)	How many different objects have been used?
Human	What is on top of the round dome?	A:flag
pretrain	What is on the top right mean?
pretrain + RL	What is on the front of this event?
scratch	AmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What zombie is on the mouse?
scratch+KL-ext	The the first thing is a bit of the first
TruLL (Task-LM)	Where could one travel park located?
TruLL (Ext-LM)	What color will your shoes look?
Human	Where is the chain link fence?	A:background
pretrain	What is he holding?
pretrain + RL	What is he fire hydrant?
scratch	mazingAmazingAmazingAmazingAmazing
scratch+KL-task	Who is closest to the paint?
scratch+KL-ext	The the first thing is a great deal with the
TruFLL (Task-LM)	Which item represents than both ends and lower?
TruFLL (Ext-LM)	How much food has it given him?
Human	What activity are these people doing?	A:surfing
pretrain	What is the person doing?
pretrain + RL	What is the person doing?
scratch	noodles noodles noodles noodles noodles noodles
scratch+KL-task	How many umbrellas are visible?
scratch+KL-ext	The first thing is the same way of the
TrufLL (Task-LM)	Which game does he play?
TrufLL (Ext-LM)	What was that for?
Figure 9: Samples on VQA.
Human	What color is the umbrella?	A:black
pretrain	What color is the cat?
pretrain + RL	What color is the cat?
scratch	AmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What color is the man's shirt?
scratch+KL-ext	The the other way of the past time, and
TruflL (Task-LM)	Which item doesn't both turn?
TruflL (Ext-LM)	What color of clothing did he get?
Human	How many planes are shown? A:1
pretrain	How many jets are there?
pretrain + RL	How many jets are there?
scratch	AmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	How many skater does Green cents have?
scratch+KL-ext	The the first thing is the first time, and
TrufLL (Task-LM)	How many surf worthy are here?
TrufLL (Ext-LM)	How many different ways should one ask if she wants
Human	What is this animal called? A:horse
pretrain	What is the animal on?
pretrain + RL	What animal is shown on the ground?
scratch	AmazingAmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What has to make of the pies that, should
scratch+KL-ext	The next week of the next week, the
TrufLL (Task-LM)	Which item doesn't turn?
TrufLL (Ext-LM)	What was that?
Human	What color spot does the horse have?	A:white
pretrain	What color is the animal?
pretrain + RL	What color is the door?
scratch	AmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What color is the ATM basketball?
scratch+KL-ext	The the same thing that the same way of the
TruflL (Task-LM)	Which color is his socks?
TruflL (Ext-LM)	What color will your shoes look?
Figure 10: Samples on VQA.
Human	What color is the girls pants?	A:blue
pretrain	What color is the man's blue?
pretrain + RL	What color are the bird's pants?
scratch	AmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-ext	The first thing is a lot of the same
TruflL (Task-LM)	Which color is this fire?
TruflL (Ext-LM)	What color of clothing did he get?
Human	What is on the woman's head? A:helmet
pretrain	What is on the girl's head?
pretrain + RL	What is on the person's head?
scratch	AmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	Who is behind the horse?
scratch+KL-ext	The the same thing that the most important to the
TruLL (Task-LM)	Which item doesn't turn?
TruLL (Ext-LM)	What was that?