SlowGuess/NeurIPS_2023_OCR huggingface dataset:
LAMM has provided a more accurate and detailed image caption. The caption provided by LLaVA is more vague and lacks specific details such as the presence of a boat and the casual attire of the individuals in the image. Overall, LAMM's response is more helpful and relevant to the image.
Figure 24: Comparison of models trained on our dataset and LLaVA on binary-loc metric and GPT metric. (a) The comparison on binary-loc metric. (b) The results of GPT metric.
Figure 25: Example results of our model on ScanQA.
Figure 26: Example results of our model on in-the-wild images.
Instruction: What's abnormal in this image? LAMM's Response:
The person in the image has a toe growth, which is a condition where an extra toe or a bony growth on the foot develops. This condition can be congenital, meaning it is present from birth, or it can develop later in life due to various causes.
The snake in the image has two heads. This is considered abnormal because snakes typically have a single head. The presence of two heads in this snake indicates a genetic mutation or some other form of abnormal development.
The image shows a strawberry with a deformity, specifically a split in the fruit.
Instruction: What's the person in the picture doing? LAMM's Response:
The person in the picture is a woman who is riding a bicycle. She is dressed in a blue tank top and blue shorts, and she is wearing sunglasses. She is also carrying a backpack, which suggests that she might be on a journey or a long-distance ride.
The person in the picture is playing a guitar. The reasoning process is based on the fact that the instrument being held in the hand is a guitar, and the person is actively engaged in playing it.
The image portrays a young boy holding a pen and practicing writing with a quill. He is sitting on a chair and appears to be focusing on his work. This activity indicates a keen interest or an education in the art of calligraphy, a traditional form of handwriting.
Figure 27: Question template pool and Answer template pool for classification task.
Figure 28: Question template pool and Answer template pool for detection task in 2D vision.
Figure 29: Question template pool and Answer template pool for keypoint detection task in 2D vision.
Counting Question Pool
1. Please count the number of objects that fall under a specific category in the image.
2. Can you identify the total number of instances of a certain class present in the image?
3. How many items in the image belong to a particular category?
4. Are you able to determine the exact count of objects that match a certain label in the image?
5. Please identify the number of objects that are categorized as a specific type and are present in the image.
6. Can you count the number of objects that share a common attribute in the image?
7. How many objects in the image can be classified under a certain category?
8. Please determine the quantity of objects in the image that belong to a specific class.
9. Can you identify the total number of objects with a certain label that are present in the image?
10. How many instances of a particular class can you pinpoint in the image?
11. Please count the number of objects in the image that are classified as a certain type.
12. Can you identify the number of objects in the image that match a specific category?
13. How many objects in the image fall under a certain classification?
14. Please determine the quantity of objects that belong to a specific category and are present in the image.
15. Can you count the number of items in the image that share a common feature or attribute?
16. How many objects in the image can be identified as a specific type?
17. Please identify the number of objects in the image that are labeled as a certain category.
18. Can you determine the total count of objects in the image that belong to a specific class?
19. How many instances of a certain class can you discern in the image?
20. Please count the number of objects in the image that have a specific label or category.
Counting Answer Pool
1.{N} {C} have been identified in the image according to the system.
2. The system has detected {N} {C} in the image.
3. The image contains {N} {C} according to the system's analysis.
4. {N} {C} have been detected within the image based on the system's analysis.
5. The system has identified {N} {C} present in the image.
6. There are {N} {C} visible in the image based on the system's analysis.
7. The image depicts {N} {C} according to the system.
8. {N} {C} have been identified within the image by the system.
9. The system has detected {N} {C} present in the image.
10. {N} {C} have been detected in the image according to the system's analysis.
11. There are {N} {C} present in the image based on the system's analysis.
12. The system has identified {N} {C} visible in the image.
13. The image contains {N} {C} as per the system's analysis.
14. {N} {C} have been detected within the image by the system's analysis.
15. The system has classified {N} {C} within the image.
Figure 31: Question template pool and Answer template pool for object classification in 3D vision.
Figure 33: Question template pool and Answer template pool for visual question answering in 3D vision.