아래 방법에 대해서 요약 설명 해줘.
3.1. Semi-automatic image-text data labeling pipeline
This section describes a streamlined workflow to generate a highquality,
multi-modal and instruction-following dataset with key-object
coordinates for developing a domain-tailored VLM toward construction
safety compliance monitoring. A semi-automatic and self-instructed
annotation pipeline driven by an in-context learning framework is
designed, as illustrated in Fig. 2.
The data generation pipeline utilizes the pre-trained capability of
GPT-4 V and GroundingDINO, i.e., open-set object recognition of
GroundingDINO as well as multimodal understanding and prompting of
GPT-4 V. Specifically, the semi-automatic labeling pipeline performs the
following five steps:
1) Automatic Label and Bounding Box Generation: GPT-4 V is utilized to
generate a key object list (pseudo labels) of each image in batches.
The pseudo labels are then passed to GroundingDINO as text
prompts, in order to generate candidate bounding boxes for the
specified kinds of objects.
2) Pseudo Label Correction: While the combination of pre-trained GPT-4
V and GroundingDINO demonstrates competitive zero-shot image
processing accuracy, their inferred pseudo-labels and bounding
boxes are manually inspected to ensure accuracy and reliability.
3) Seed Image Selection and Hand-Crafting: A seed image is chosen from
the annotated images of each safety scenario (e.g. confined space as
shown in Fig. 3), and the description of the seed image is handcrafted
according to the verified bounding boxes either as grounded
captioning (detailed descriptions) or question answering of
safety compliance analysis. This forms an example of the instructionresponse
dataset that we want to acquire.
4) In-context Learning for Dataset Generation: The seed image with the
example instruction-response set is passed to an in-context learning
prompt, for GPT-4 V to generate candidate instruction-response sets
for remaining images. The in-context learning prompt designed for
acquiring instruction-response sets for grounded captioning in this
paper is summarized below:
System Message: You are a professional construction safety inspector
generating labeled data for fine-tuning a safety domain-specific Vision
Large Language Model.
You will be given the accurate data of bounding boxes of some workers
and objects in the given annotated image. The coordinates of the BBoxes
are in ‘x1,y1,x2,y2’ format with the origin at the top left corner, the xaxis
to the right, and the y-axis downward, (x1,y1) and (x2,y2) as topleft
and bottom-right corners of the box respectively and values in relative
coordinates multiplied by 1000 (prefixed with zeros to three digits).
Please help me draft an instruction for a brief description of the safety
condition shown in the image and the corresponding response, and the
response should include reference to the given bounding boxes of workers
and objects. Please count the number of workers and give a number ID to
each worker (ignore the BBox IDs of the workers and start the counting
and ID numbering from 1, from left to right of the image). Also, include
each worker’s BBox and any of his appearance (like vest color or helmet
color) when you mention his safety condition. The given BBox data is
sorted in ascending order of x1 coordinate (from left to right). Please
include request for grounding with coordinates in the instruction part, and
keep the response concise within 200 words, step-by-step and specific to
the content of the image. Only include things that you are sure about. Now
you will be given an example for reference:
Example:
{
Image: # An image of a construction site with a few workers and a tower
crane.
Objects: [.
{“value”: 1, “label”: “worker”, “box”: “140,320,240,720”};
… # List of candidate objects with corresponding label and bounding box.
{“value”: 8, “label”: “red safety helmet”, “box”: “814,294,890,360”}.
]
Instruction: Please describe the work environment and safety condition in
the image, and include the worker IDs (count from left to right) and coordinates
for crucial objects in format [object] [[x0,y0,x1,y1]].
Response: The image depicts a construction site with a tower crane in the
background and five workers visible:
- [Worker 1] [[000,330,100,740]] is not wearing a safety helmet or vest.
- [Worker 2] [[140,320,240,720]] has a white [safety helmet]
[[164,325,202,368]], but lacks a safety vest.
- [Worker 3] [[370,230,520,980]] is wearing a red [safety helmet]
[[410,230,490,300]], yet lacks a safety vest.
- [Worker 4] [[480,260,600,910]] is without any safety helmet or vest.
- [Worker 5] [[800,290,970,999]] sports a red [safety helmet]
[[814,294,890,360]], but lacks a safety vest.
Immediate action is required to ensure all workers are equipped with
safety helmets and high-visibility vests.
}
You need to rephrase the instruction in a natural way. In the response, the
grounded objects with BBoxes should be marked in format [object] followed
by coordinates in format [[x1,y1,x2,y2]]. Use the format [[x1,y1,x2,y2;x1,
y1,x2,y2;x1,y1,x2,y2;…]] (No spacing between sets of coordinates) for
multiple BBoxes of the same object type. You should also pay attention to
whether each worker wears sufficient PPE, especially a safety helmet and a
safety vest. Now, you will be given the image and key objects for your output
of the corresponding instruction and response:
Image: #The image to be used in the dataset.
Objects: #The key object list of the image with the IDs and coordinates.
Output:
The candidate instruction-response sets generated by the above
prompt may contain certain faults and require further polishing.
5) Polishing Candidate Datasets: The candidate instruction-response sets
generated by GPT-4 V are further optimized with human supervision
to create more desired samples in the dataset, each forms an imageinstruction-
response triplet. A sample output is illustrated in Fig. 4,
following the conversational format adopted by VLM-related
research [13]. The instruction across different images is slightly
different to inject textual diversity, which fosters the VLM to learn
more robust and generalizable features from the input instruction.