Table 8: Examples of answers generated by different models: For multiple-choice questions, some models output the category name directly instead of the options. Thus, we have appropriately relaxed our evaluation criteria. Answers that contain the correct category without including any other confusing categories are also considered correct. For true or false questions, we strictly evaluate the model's output.
Models	True or False Questions	Multiple Choice Questions
Prompt: Does the image contain any species belonging to the Apiales? Please respond with either "yes" or "no". GT answer: yes.	Prompt: Does the image contain any species belonging to the Fabales? Please respond with either "yes" or "no". GT answer: no.	Prompt: What does the image the species in the image belonging to? Please choose one answer from below: A: Liliopsida B: Monocots C: Insecta D: Magnoliopsida GT answer: C: Insecta.
MiniGPT4 [96]	"Yes"	"No"	"Liliopsida"
LLaVA [56]	"Yes, the image contains a species belonging to the Apiales, which is a group of flowering plants."	"Yes, the image contains a plant belonging to the Fabales, which can be identified as a small white flower with pink anthers."	"The image belongs to the category D: Magnoliopsida, which includes flowering plants and their insects."
Blip2 [50]	'' ''	'' ''	"A"
mPLUG-Owl [92]	"Yes, the image contains a species belonging to the Apiales, as it features a flower of a carrot plant."	"Yes, the image contains a species belonging to the Fabales family, which is a plant group commonly referred to as the pea or pea-related plants. The flower is a close-up shot of the plant's bud or stem, which can be identified as belonging to the Fabales family."	"The species in the image belongs to the order Insecta, more specifically to the order Lepidoptera, which includes butterflies and moths."
Otter [49]	"Yes, the image contains a species in the Apiales family, which is a flowering plant."	"Yes, the image contains a plant in the Fabaceae family."	"The image shows a specimen from the order insecta, specifically a close-up of a caterpillar on a piece of white paper or a white table."
Multimodal-GPT [25]	"### Image: ### Image: ### Image: ### Image: ### Image: ### Image: ### Image: "	"Yes, the image contains species belonging to the Apiales. \end{code} \begin{code} #! /usr/bin/env python3 #.*-coding: utf-8 *-."Created on 2017-05-10 15:05:39." import argparse, sys, json, re, requests, urllib.error, urllib.request"	"A: Liliopsida B: Monocots C: Insecta D: Magnoliopsida ### Image: ### Instruction: What does the image the species in the image"
PandaGPT [73]	"Yes."	"Yes."	"The image features a close-up of a brown, long-haired caterpillar, which is a part of the Insecta species. The caterpillar is not a plant, but rather an insect, and it does not belong to the Liliopsida, Monocots, or Magnoliopsida groups"
Visual-GLM6B [17]	"Yes, there is a plant of the Apiaceae family in this image, which includes many different species such as honeysuckle and violets. The photo shows several flowers on the plant's stem."	"Yes, there is a plant fromFabaceae family in this image. It has pink and white flowers on its stem. The plant appears to be small and green, which suggests that it might be an ornamental variety of plants like daffodils or violets."	"The image shows a brown butterfly with long wings and small eyes."
InstructBLIP [9]	"Yes"	"Yes"	"C"