Background
Object detection is a fundamental computer vision task with broad applications. While methods like YOLO
achieve high accuracy on standard benchmarks, they are limited to detecting only predefined object classes.
Fine-tuning a pretrained detector (e.g., YOLOv11) with a custom dataset may overcome this limitation.
However, manually annotating large datasets is impractical due to the time and effort required. An openworld object detector is a computer vision model capable of detecting objects of arbitrary classes, beyond
a fixed predefined set. In this task, we will design an auto-annotated food dataset using an open-world
object detector and try to implement a Chinese food recognizer.
Task 1: Auto-Annotated Custom Dataset Genatration (20 points)
• Objective: Generate a custom food detection dataset using any open-world object detector (the following uses Grounding DINO as an example), for automatic annotation.
(a) Prepare the environment of Grounding DINO from GroundingDINO.
(b) Use pretrained checkpoints to automatically annotate the Food-101 dataset. You can easily load
Food-101 dataset using datasets.load_dataset(’food101’) after installing datasets.
(c) Implement a pipeline using Grounding DINO to detect food objects in images. You can use the
open source models in the code repository.
1(d) Annotate Food-101 dataset following the directory format for fine-tuning YOLO. You can use just
”food” as the prompt.
(e) Visualize image samples annotated with bounding boxes and results of other steps in your report.
Task 2: Fine-tune YOLOv11n on the Generated Dataset (40 points)
• Objective: Practice how to fine-tune visual models and optimization strategies, and understand evaluation metrics.
• We recommend using YOLOv11n as base model. You can use any other method you like, the following
will use YOLOv11n as an example.
• Fine-tune the base model on your auto-annotated dataset.
• Please report the following:
(a) List the evaluation metrics for this task that you think are important and give reasons.
(b) Give the parameter settings that you think are reasonable and how model performance varies with
parameters. Give the data for your best results.
(c) Present success and failure cases in your pre- and post-fine-tuning tests.
(d) Study the architecture design of yolov11 and your chosen open-world object detector, compare the
difference between auto-annotated test images and their actual predictions, describe and briefly
analyze the differences you find.
Task 3: Fine-tune YOLOv11n on ChineseFoodNet (40 points)
• Objective: Further refine the model using ChineseFoodNet to enhance accuracy in identifying Chinese
cuisine. Construct a subset to reduce the effort required for fine-tuning.
• You can download the ChineseFoodNet dataset via ChineseFoodNet and class names.
• Please take a photo of meal that you love (web photos are allowed, photos should include at least 3
classes of food) and select the appropriate classes in the ChineseFoodNet dataset.
• Select 3 additional categories from ChineseFoodNet with low correlations to construct a subset. Design
a process that you think is reasonable for constructing the subset for fine-tuning.
• Fine-tune YOLOv11n on your new subset.
• Please report the following:
(a) Describe your subset building process in detail, and show screenshots of your subsets in detail.
(b) Show the detection result of the fine-tuned model on your prepared photo.
(c) If the previous step fails, don’t be frustrated, you can also report the results of the corresponding
class on the test set evaluation metrics as an alternative.我已经完成了task1和task2，现在请你完成一下task3的编码实现， base model用yolov8