Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper attempts to formalize a notion of creativity in generative models. The idea is to see if a generative model trained on one dataset can be used to generate novel samples that resemble elements of another dataset. In this case, it is examined whether a generative model trained on digits could be used to generate samples that look like alphabetical characters. Several metrics for determining the alphabetical nature of the generated samples are given; this is used as a proxy for novelty. It is shown that these can be useful in choosing models that generate novel samples outside of the classes the model was initially trained on.

I can agree with the premise that when it comes to out-of-class novelty, likelihood is probably not a good measure since it will penalize models that generate samples that are too far outside of the data distribution. However, I'm not  yet convinced that the conclusions drawn here would generalize beyond the specific examples given in the paper. It would be good in a future iteration to see this same analysis on another dataset, or perhaps even to reverse the existing experiment (train on alphabetical characters, evaluate on digits). Another possibility would be to test on several different alphabets, like those found in Omniglot.

Although I think this particular analysis is limited (it is a workshop submission), I do think it proposes an interesting direction for measuring the novelty of samples from a generative model. I could see this being a potentially useful direction for measuring interesting properties of generative models in terms of creativity.

How are the panagrams (a)-(d) generated? Are letters chosen based on Euclidean distance to some reference characters?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: 
This paper attempts to formalize the notion of 'computational creativity' from a machine learning perspective, in order for machine learning researchers to make better progress on this problem. In particular, the authors propose measuring the 'computational creativity' of a model by several metrics intending to capture whether the model can generate new objects from classes unseen during training.

I think this is an interesting paper and a good first step in this area. Indeed, absent proper definitions and metrics for vague concepts such as 'creativity', it is difficult to make progress on related computational problems. While the proposed metrics are not perfect,* they seem reasonable enough to warrant future investigation, and thus I think this paper is worthy of acceptance as an ICLR workshop paper.

*Further thoughts: I'm not convinced that these metrics are selecting for the "right" models from a creativity point of view. If Figure 1 is really a random sample of digits generated by one of the 'most creative' models according to the proposed metrics, it seems like it is mostly just good at capturing lower-level correlations in the data, while generating random high-level details. Thus it seems like a 'creative' model is one that has been artificially limited in order to poorly model high-level features of the data. This seems intuitively to contrast with creativity as we perceive it in humans -- creative humans are still capable of modeling the world around them, they are just able to combine what they've learned in new and interesting ways. Perhaps 'true creativity' is out of the reach of current generative models? (Or, perhaps the word 'creativity' is not really meaningful from a computational perspective?)  However, I'm not an expert in this area, and I still think the idea is worthwhile presenting, as it may generate interesting discussions. In future work, I'd like to see a more thorough analysis of what model settings lead to  the most 'creative behaviour' according to these metrics.
review_title: Review

Decision: Accept