SlowGuess/ACL_2020_OCR huggingface dataset:
$N_{Rel}$  : 4
A clock with a rose on its corner sits on the wall.
$N_{Rel}$  : 3
Figure 5: Top-5 ranked predictions (relevance in parentheses) of MCA-I-H and MCA-I-VGH after both sparse annotation and curriculum fine-tuning phase.  $R_{GT}$  defines the rank of Ground Truth (GT) predicted by the model and NDCG of rankings for current question turn.  $N_{Rel}$  denotes number of candidate answer options (out of 100) with non-zero relevance (dense annotations). Here  $\spadesuit$  and  $\spadesuit$  represents predictions after sparse annotation and curriculum fine-tuning respectively.
Q Is the photo in color?
A Yes.
Q Any other people?
GT: No.
Rel: 0.8
Q What color is the apple?
A It is red and
Q What color is the orange?
A It is dark orange.
Q What is the size of the box?
A Can't tell.
Q Where is the box?
GT: Can't tell.
Rel: 1.0
Q Is it a real rose?
A I don't think so.
Q Is it an old
fashion clock?
A No, not really.
Q Is it a digital
A No.
Q Is it hanging on the wall or leaning? GT: It's hanging.
$\spadesuit$ $R_{GT}:1$  ;NDCG 83.32
(0.8) No.
(0.2) 0.
(1.0) Nope.
(0.8) No there's not.
(0.4) Just the 1.
$\spadesuit$ $R_{GT}:2$  ;NDCG 91.2
(0.8) Not that i can see.
(0.8) Not that i see.
$\text{路} _ { G T } : 9$  ;NDCG0.0
(0.0) On the table.
(0.0) In a container by the window.
(0.0) On table.
(0.0) Use no paper here.
(1.0) I cannot tell.
$\spadesuit$ $R_{GT}:5$  ;NDCG 73.59
(1.0) I can't tell.
(0.6) Not sure.
(1.0) Can't tell.
$\clubsuit$ $R_{GT}:1$  ;NDCG:81.55
(1.0) It's hanging.
(0.0) Yes, it's attached to the side of the building.
(0.0) Yes.
(0.0) It is cut out, but it is definitely sitting on something.
(0.0) It looks like.
$\spadesuit$ $R_{GT}:2$  ;NDCG 51.45
(0.0) It looks like.
(1.0) It's hanging.
(0.0) Can't tell.
(0.0) Unclear.
(0.0) I think so.
$\spadesuit$ $R_{GT}:1$  ;NDCG 74.98
(0.8) No.
(0.2) 0.
(1.0) Nope.
(0.2) 0 at all.
(0.8) Not that i can see.
$\spadesuit$ $R_{GT}:1$  ;NDCG 85.24
(0.8) No.
(0.8) Not that i can see.
(0.8) Not that i see.
(0.8) No there's not.
(1.0) Nope.
$\clubsuit$ $R_{GT}:9$  ;NDCG 0.0
(0.0) On table.
(0.0) On the table.
(0.0) In a container by the window.
(0.0) In a kitchen.
(1.0) I cannot tell.
$\spadesuit$ $R_{GT}:4$  ;NDCG 98.84
(1.0) I cannot tell.
(1.0) I can't tell.
(0.6) Not sure.
(1.0) Can't tell.
(0.0) In a container by the window.
$\clubsuit$ $R_{GT}:2$  ;NDCG 51.45
(0.0) No it is not mounted
on the wall.
(1.0) It's hanging.
(0.0) It is cut out, but it is definitely sitting on something.
(0.0) Yes, it's attached to the side of the building.
(0.0) On the rail.
$\spadesuit$ $R_{GT}:3$  NDCG 40.78
(0.0) No it is not mounted on the wall.
(0.0) Not sure.
(1.0) It's hanging.
(0.0) Can't tell.
(0.0) I can't tell.
information that the model will receive because of noise in the dataset, since these examples were already seen during the spare annotation phase. Val annotations remains unaffected for fair comparison. As expected, this simple correction increase the ground truth related metrics such as R{1,5,10} drastically.
8194
- This task is fairly simple. In this task, you will be shown a picture and a question. Your task is to specify whether the question can be answered independently just seeing the image without any other context or information about the previous conversation.
- You have to select either of the following responses:
I can confidently tell the correct answer just seeing the image
I want to know what was discussed before to answer confidently. Cannot answer with just the question and image. Need more information (context) from previous conversation.
I can answer it but by inferring using common sense
I can only guess the answer.
I can't tell the answer.
o Not relevant question for this image
Good luck and have fun! Do not forget to press submit at the end! Otherwise all your progress will be lost.
Figure 6: Instructions for the AMT task.
Question: "Are there any dogs in the image?"
You can confidently tell the correct answer just by looking at the image as at least one the dog is clearly visible present.
Question: "What is its color?"
Want to know what was discussed before to answer confidently. Do not assume anything apart from the image or question. Here you need more information on what the word 'its' is referring to, i.e. You need more information if the previous conversation was about the dog or boat.
Question: "What about the other?"
Want to know what was discussed before to answer confidently. Similarly, in this case of ambiguity, make no assumption here. We dont know if we are talking about mountain, boat or dog.
Question: "Is it a lake or sea?" / "How old is the dog?"
These are the type of questions where you can answer by common sense or just guessing. You actually dont need the previous context of the conversation.
is the plane above any bodies of water?
Figure 7: Qualification test consisting of 2 test images to allow the turkers to actually attempt the task
do you see other appliances?
8195
Figure 8: Sample task.
Figure 9: Top-20 questions in the training set. Of all the questions in the training set, only  $30\%$  questions are unique while weather related questions (like sunny, daytime, rainy) top the charts.
Figure 10: Top-20 answers in the training set. Yes/No forms a major chunk in top 20 answers.
8196
Figure 11: Top-20 answers with non-zero relevance in the dense annotations of training set. Generic and yes/no semantically equivalent answers mostly constitute the list. Percentage is calculated out of total 3652 unique answers which have non-zero relevance in train dense annotations set.
Figure 12: Number of pronouns in 10 questions of a dialog.
8197