Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This paper introduces two creative sketch datasets of birds and creatures, segmented into parts, each with ~10k doodles collected from Amazon MTurk workers. In a user study, people tend to favor sketches from their dataset over the similar Gogole QuickDraw sketches. Additionally, the authors propose a GAN architecture for generating novel sketches in an incremental fashion, one part at a time. They provide many qualitative results as well as human studies to validate their approach.

### Explanation of Rating
While the new datasets look nice, I'm not sure that they sufficiently different from or better than existing sketch datasets. Overall, the scale, complexity, and diversity of the sketches are roughly the same as some of the other datasets mentioned in the paper. It isn't obvious to me which novel applications or approaches are made possible with this dataset that were not possible before.
With respect to the proposed generative model, I think a user-in-the-loop interface is a reasonable approach, but though the model seems compatible with such a system, the authors do not actually implement it. It would be nice to see this interface in practice, since without it, the results do look a bit better but are not particularly more useful than those from other GAN models.

### Pros
- The UI and methodology for data collection, in which a user is asked to first draw an eye, followed by semantic parts of their choosing is interesting and appears to be conducive to better sketch quality.
- The paper is well written and contains many qualitative results and figures.
- The labeling of sketches by semantic parts presents an advantage over previous datasets.
- The iterative/incremental design of the generative model is a nice step towards computer-aided user-in-the-loop creative design.

### Cons
- While the authors claim that Figure 2 shows that their dataset is more diverse and creative, I'm not sure if that's objectively the case and if the marginal improvement is sufficiently important for downstream tasks. 
- I understand the authors' argument that part-level granularity is more reasonable than stroke-based granularity, but I think that one major insight that can be gained from studying sketch datasets is with respect to how humans draw. For this reason, it would be useful to have information about the order of individual strokes, which the dataset does not appear to include.
- I'm not sure about the argument that raster images are a better format for a generative model than vector. It seems like the long-range relationships and connectivity as well as the sparsity inherent in a vector representation are pretty important in a sketch. Consequently, the some of the results produced by the model appear to suffer from topological artifacts and poorly-defined line segments.
- It would be useful to provide a citation for the doodling process used for data collection.
- How were the 100 sketches from Creative Birds and QuickDraw chosen for the user study?
- The paper claims the importance of certain architecture choices, e..g, that part channels help the model better predict the part locations. Some ablation study to verify this would be appropriate.
- Page 4: "subjetcs" (typo)

---

Thank you to the authors for their comments and clarifications.  I still am skeptical that the proposed dataset is a fundamental improvement over Quick Draw---ultimately, both datasets contain compact sketches from a single category containing relatively few strokes. But given that your updates and clarifications have addressed many of my questions, I am raising my score.
review_title: Introduces two novel doodle sketch datasets as well as a generative model for producing new sketches but does not seem to provide a substantial advantage over existing work

final_score: rating: 7: Good paper, accept
review_content: This paper put forward two Creative datasets - Creative Birds and Creative Creatures, which have corresponding part annotations. Based on Part selector and Part generator. The authors demonstrate a multi-stage sketch generation approach - DoodlerGAN, and achieve the STOA compared with other methods in several metrics.

The topic of creative generation is interesting and gradually becomes the future research direction in the image generation field. There are some pros:
- The datasets provide a novel perspective to decouple the basic elements of sketch images with fine edge semantic labels and text descriptions. These annotations could be used in the fine-grained domain adaption task, text-to-sketch generation or some reverse tasks. The reviewer thinks this contribution is meaningful and solid.
- The sketch generation fashion is enlightening. Although the idea of multi-stage generation has been used in many synthesis tasks, it's mainly for different resolutions or granularities (local and global). The partial strokes combine the overall sketch images is a domain-specific technique. The authors reimplement in deep learning way. The reviewer has to say that it may have some limitations on other fields, it's appreciated if the authors can share new possibilities, but that's not the point.
- The experiments are solid. Quantitative evaluation and human evaluation both outperform other baseline methods. It's convincing.

Some concerns are also proposed:
- How are the predicted labels added to the generator? It doesn't seem to show up on Figure 3(a). The reviewer wants to know more details.
- The reviewer has some confusion about the noise range in Figure 3(b), which is N(0,1). However, there is different descriptions in sec 4.2 - N(0,2). Which one is the true sampling distribution?
- The full loss functions are lack in the mainly body, which should be depicted more clearly.
review_title: A good paper

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors collect a large dataset of sketches of imaginative birds and other creatures, then train a GAN to produce similarly creative sketches. They use both quantitative and qualitative methods to evaluate their work and find that the generated sketches are of higher quality than both other models and human sketches, and are novel.

This topic is delightful and your paper is quite well written. My only concern is fit for the venue. This isn't a paper about a learned representation and doesn't encounter any theoretical issues regarding representations, why should it be published at ICLR rather than another venue, e.g. ICCC (https://computationalcreativity.net/iccc21/)? There are plenty of papers at ICLR that aren't about representation learning these days, so I'm still inclined to accept, but I'd like to hear an argument.

More specifically, I appreciate the level of detail regarding the dataset collection study and your model training process. I would also like to see more discussion of your "conditioning perturbation" trick. Why does it work? Can you characterize the effect the perturbations have on the generated sketches?

I'm a little confused about Figure 5: were the images from each method compared against the same set of images? If not, you may need to do some more sophisticated statistical analysis than the Bernoulli confidence interval to judge the probability that each method is chosen above each competitor. I also find it hard to believe that the human drawings perform significantly worse than the GAN drawings across the board. Does that indicate a problem with the dataset you constructed or some bias among your participants?

Very minor issue, but I think you also have a typo in Table 1, the DS column should probably be labeled GD to match the table caption.
review_title: Delightful, well written paper! I have concerns about its fit here.

final_score: rating: 7: Good paper, accept
review_content: The contribution of this paper is twofold. First, the authors introduce a pair of manually collected (via AMT) datasets of creative sketches (birds and general creatures) each along with part annotations. Second, they propose a part-based Generative Adversarial Network for the generation of unseen compositions/configurations of novel parts (legs, body, etc.) for creative sketches. The latter is quantitatively and qualitatively (via human studies) evaluated. 

Strengths

- Generally, the paper is well introduced and easily understood. Especially, the provided dataset is very useful in promoting the development of the problem addressed: creative sketch generation. Although not being a really “novel” idea/data, providing new benchmarks/datasets/competitions for the AI community is always refreshing.

- The authors also include a generative architecture which is then quantitatively evaluated against several working implementations of the pretrained SOTA baselines. They also provide easy-to-understand evaluation metrics allowing the community to keep on working on the task at hand. The experimental results demonstrated the effectiveness of the proposed method.

- The illustrative experimental setting shows the usefulness of the datasets, the potential of their generative approach and its easiness of use.

Weaknesses

- Of course, this is a general concern any time a diagnostic dataset/tool is introduced. Rarely is an immediately surprising insight offered in the same paper. The value of such datasets/tools is often clear only in hindsight with the benefit of time. If they are useful, they see organic adoption. If they are not, they organically fall to the wayside. 

- Furthermore, one can argue that there is no particularly novel insights in this paper (no science behind the collection of the datasets and the proposed GAN is a logic incremental evolution of the SOTA). However, the paper is very well written and structured, the methodology followed is correct and the experimental setting is comprehensive (with promising results), so I'm happy to let the noisy process of science (and reviewing process) figure out the value here.

review_title: An improved approach to automatic sketching 

Meta-review: While much of generative modeling is tasked with the goal of generating content within the data distribution, the motivation of this work is to examine whether ML techniques can generate creative content. This work has 2 core contributions:

1) Two new datasets of creative sketches: birds and creatures, that have part annotations (size ~ 10K samples for each set). The way the datasets are structured with the body part annotations will facilitate the creativity aspect of the approach later described in the paper.

2) This paper propose a GAN model that is part-based, which they call DoodlerGAN. It is inspired partly by the human's creative process of sequential drawing. Here, the trained model determines the appropriate order of parts to generate, which makes the model well suited for human-in-the-loop interactive interfaces in creative applications where it can make suggestions based on user drawn partial sketches.

They show that the proposed model, trained on the part-annotated datasets, are able to generate unseen compositions of birds and creatures with novel body part configurations for creative sketches. They conduct human evaluation and also quantitative metrics to show the superiority of their approach (for human preference, and also FID score).

Many reviewers, including R1 and myself observe that the datasets provided, along with the parts-based labeling and modeling approach are a clear advantage over existing datasets and methodology. With ever growing importance of generative models used in real world applications, including the creative industry, I believe this paper provides a much needed fresh take on creative ways of using our generative models besides making them larger, or achieving better log-likelihood scores. Many reviewers, including R3, would think that this work is indeed a "Delightful, well written paper! I have concerns about its fit here." I strongly believe such works in fact definitely *do* belong at ICLR, and I think this work has the potential to get researchers in the generative modeling community to rethink what they are really optimizing for.

I believe this paper will be a great addition to ICLR2021, and I look forward to see their presentation to the community to spark more creativity in our research endeavors. For this reason, I'm strongly recommending an acceptance (Poster).
Decision: Accept (Poster)