SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Comparing  ${VQ}^{2}$  configurations on all EditBench categories
Method	Models used for assessment	Color	Count	EditBench
Material	Shape	Size	Mean
\(VQ^2 (A)\)	VQA & NLI	75.7	63.6	71.9	67.6	75.9	70.9
\(VQ^2 (B)\)	VQA & VQA	80.2	72.3	75.6	73.5	75.4	75.4
\(VQ^2 (C)\) w.o. multi-word answers	VQA	77.2	71.5	73.6	71.9	75.5	73.9
\(VQ^2 (C)\)	VQA	78.5	73.5	76.9	71.7	78.2	75.8
A.5 Evaluating Contradictions Generated by GPT4
To assess GPT4's capability in generating contradictions, we compared the outputs from a GPT4-driven ConGen approach against our PaLM-based method. We selected a set of 100 images from both COCO-Con and PickaPic-Con databases. For each image, we obtained 100 positive and 100 negative captions using PaLM. Subsequently, by utilizing the identical prompt with GPT4, we generated an additional 100 negative captions. These captions were randomly assigned labels 'A' and 'B'.
For the evaluation process, we engaged three independent evaluators from Amazon Mechanical Turk. They were presented with an image, its caption, and the two negative caption candidates. Their task was to discern whether each candidate accurately contradicted the depicted image. A snapshot of this user interface can be seen in fig. 11.
The final judgment was based on the majority vote of the workers. The results showed a close match in performance between the two models: PaLM achieved  $77\%$  while GPT4 reached  $76\%$ . This suggests our ConGen approach works effectively with various large-scale language models.
A giraffe with its neck stretched over rocks and eating grass.
Figure 11: The Amazon Mechanical Turk user interface on determining whether a contradicting caption generated by GPT4 or PaLM is correct.
A Contradiction: A giraffe with its neck coiled around rocks and eating meat.
Correctly contradicts the image  $\bigcirc$  Doesn't contradict the image
B Contradiction: A giraffe with its neck stretched over rocks and eating sand.
Correctly contradicts the image  $\bigcirc$  Doesn't contradict the image
A.6 Comparing  $VQ^2$  with PaLI vs. BLIP2
We compare PaLI and BLIP2 for the  $\mathrm{VQ^2}$  method in table 8. While PaLI yields higher scores, it's pivotal to recognize their distinct evaluation metrics. PaLI's score indicates the probability of a "yes" response, whereas BLIP2 gives a definitive answer without probability. This difference can influence the application suitability of each model.
PaLI	BLIP2
Drawbench	82.6	60.5
Winoground	63.5	58.2
COCOCON	87.1	88.5
A.7 Reproducibility
To fine-tune BLIP2, we adjust only the Q-former parameters of the model using the Adam optimizer. We train the model for two epochs and designate  $10\%$  of the training set as a validation set for early stopping and use learning rate selection between  $\{1e-5, 5e-5\}$ . A single training took 5 hours on a linux server with one A6000 GPU. All experiments took  $<2$  days.
Zero-shot  $VQ^2$ : For 10,000 text-image pairs, the inference time of every step is as follows. Answer candidate generation: when using extended answer candidates – about 1 day. Otherwise, 12 hours. Question generation and filtering: When using extended answer candidates, about 2 days, otherwise, 1 day. The last step only takes a few minutes.
Table 8: Comparison of  $\mathrm{VQ}^2$  results for BLIP2 and PaLI (ROC AUC scores). Note the difference in evaluation: PaLI's  $\mathrm{VQ}^2$  score represents the probability of a "yes" response, while BLIP2 offers a definitive answer without probability.
Aspect	PaLI	VQ2	BLIP2
Inference Time	500ms per image-text pair	40 seconds per image(full pipeline)	750ms per image-text pair(as measured in source)
Model Parameters	17B parameters	T5-XXL - 11B pa-rameters + PaLI 17B	12B parameters
Hardware Requirements	Four v4 chips [61]	16 TPU v3 cores + 4v4 chips [61]	GPU with 24GB as reportedin HuggingFace
Framework	T5X [62] on JAX[63]	T5X [62] on JAX[63]	Pytorch
Table 9: Computational Costs Summary