SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct the data collection and deploy the interfaces on AMT. Our approach entails a meticulous design of the entire process to streamline the procedure and enhance its efficiency. To facilitate workers understanding and proper execution of the data annotation, we provide them with an elaborate tutorial contained in a 5-page document (https://shorturl.at/bpBUw), along with a supplementary video demonstration (https://www.youtube.com/watch?v=husejlhNyfo). These links remain accessible at all times for reference purposes.
In the tutorial, we ensure that each step of the interface is accompanied by detailed instructions, making it self-contained and easy to follow. Figure 9 displays the interfaces used in our crowdsourcing task for data collection, offering a visual representation of the user experience.
The annotation is divided into three phases: Preparation, Initial Editing, and Follow-up Editing. In the Preparation phase, we provide clear instructions on how to access the source image, log in to DALL-E 2, and upload the source image to prepare for editing.
During the Initial Editing phase, we clarify the terms "Edit Instruction" and "Global Description", ensuring workers understand their respective purposes.
- "Edit Instruction" is a directive that describes the suggested edits and how workers wish to alter the image. We encourage workers to phrase their instructions as if they are speaking to a helper in a simple and colloquial manner, such as 'Let the dog drink the wine'.
- "Global Description" provides a comprehensive description of the image after the suggested edit has been applied, e.g., 'A dog lying down is holding a bottle of wine between its paws'. This description is input into DALL-E 2 to generate the desired image. We also specify the expected outcomes of this initial edit to guarantee all steps are covered and prevent any omissions.
In the Follow-up Editing phase, users are free to carry out follow-up edit turns on the image generated in the first turn. The process remains similar to the second phase, facilitating a smooth continuation of the annotation process.
E.2 Monitoring the Annotation Process
Throughout the task, workers are encouraged to provide comments and feedback after each session. Also, during the entire annotation process, we continuously check the data to ensure the quality. Specifically, in the trial period, we checked all annotated examples in a batch with 10 sessions to provide prompt feedback to each worker on data quality (both in image and instruction). Only workers that can deliver satisfactory results will be advanced to the next stage, where they will be asked to do more tasks on AMT. Then, we spot checked on 5 of each 100 sessions in the rest of the annotation process. In checking, the sessions containing subpar images, with issues relating to image quality and instruction consistency, are eliminated. Additionally, we maintained frequent communication with the workers, providing timely guidance and requesting certain turns to be redone if the quality is unsatisfactory. As time progresses, we observe a significant decrease in the frequency of communication, and we find that all workers consistently pass the checks in the later batches of data annotations. This indicates a notable improvement in the quality of the annotated data as the process advances.
Figure 9: Data collection interface on AMT.
E.3 Human Evaluation
We conduct multiple human evaluation tasks on AMT to assess the quality of our dataset (Section 3.3) and evaluate the generated images from different models (Section 4.4). For these tasks, we design three different types of interfaces. The first type (Figure 10) involves individual evaluation using a 5-point Likert scale to measure the quality of the images. The second type (Figure 11) is a multi-choice comparison task, where evaluators compare four top-performing methods in Table 2 and Table 3, including Text2LIVE, GLIDE, InstructPix2Pix, and fine-tuned InstructPix2Pix on MAGICBRUSH. The last type (Figure 12) is a one-on-one comparison task, providing a more nuanced evaluation between fine-tuned InstructPix2Pix and other strong baselines as well as the ground truth. Both consistency and image quality are assessed in each human evaluation task, with the original image and the textual instruction provided at the beginning.
Edit suggestion: Have the cat have big, green eyes
Figure 10: The interface of individual evaluation on AMT to assess the dataset quality as well as generated images by different models.
Q1: Compare the edit instruction with the actual changes made in the edited image. Assign a score from 1 to 5 to assess how accurately/consistently the edit instruction has been implemented in the edited image, where:
Score 1: The edited image is not consistent with the edit instruction at all.
$\bigcirc$  Score 2: The edited image is mostly inconsistent with the edit instruction.
Score 3: The edited image is somewhat inconsistent with the edit instruction.
Score 4: The edited image is mostly consistent with the edit instruction.
$\bigcirc$  Score 5: The edited image is consistent with the edit instruction.
Q2: Assign a score from 1 to 5 to assess the overall quality of the edited image (Some aspects you may consider, such as the overall appearance, color and tone, and the overall brightness), and assign a score from 0 to 10 for cleanliness blending of edited elements with the original image, and the overall natural appearance of the modifications, etc.).
Score 1: The edited image is Very Poor.
Score 2: The edited image is Poo
Score 3: The edited image is Fair.
Score 4: The edited image is Good.
Score 5: The edited image is Very good
Q1: Compare the edit instruction with the actual changes made in the edited images. Select one edit image that most accurately/consistently implements the edit instruction.
$\bigcirc$  Image 1.
$\bigcirc$  Image 2.
$\bigcirc$  Image 3.
$\bigcirc$  Image 4.
Q2: Select one edited image that exhibits the best image quality. (Some aspects you may consider, such as the preservation of visual fidelity from the original image, seamless blending of edited elements with the original image, and the overall natural appearance of the modifications, etc.)
Figure 11: The interface of multi-choice comparison on AMT to evaluate generated images by different models.
Image 1
Image 2
Figure 12: The interface of one-on-one comparison on AMT to assess generated images by different models.
$\bigcirc$  Image 1.
$\bigcirc$  Image 2.
The two images above are equal in terms of implementing the edit suggestion.
Q2: Select one edited image that exhibits the best image quality. (Some aspects you may consider, such as the preservation of the original image, seamless blending of edited elements with the original image, and the overall natural appearance of the modifications, etc.)
$\bigcirc$  Image 1.
$\bigcirc$  Image 2.
The two images above exhibit equal image quality.