Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a framework for interactive language learning, called Mechanical Turker Descent (MTD). Over multiple iterations, Turkers provide training examples for a language grounding task, and they are incentivized to provide new training examples that quickly improve generalization. The framework is straightforward and makes few assumptions about the task, making it applicable to potentially more than grounded language. Unlike recent works on "grounded language" using synthetic templates, this work operates over real language while maintaining interactivity.
 
Result show that the interactive learning outperforms the static learning baseline, but there are potential problems with the way the test set is collected. In MTD, the same users inherently provide both training and test examples. In the collaborative-only baseline, it is possible to ensure that the train and test sets are never annotated by the same user (which is ideal for testing generalization). If train and test sets are split this way, it would give an unfair advantage to MTD. Additionally, there is potentially a different distribution of language for gamified and non-gamified settings. By aggregating the test set over 3 MTD scenarios and 1 static scenario, the test set could be skewed towards gamified language, again making it unfair to the baseline. I would like to see the results over the different test subsets, allowing us to verify whether MTD outperforms the baseline for the baseline's test data.
review_title: Well-executed gamification of grounded language learning

final_score: rating: 7: Good paper, accept
review_content: TL;DR of paper: Improved human-in-the-loop data collection using crowdsourcing. The basic gist is that on every round, N mechanical turkers will create their own dataset. Each turker gets a copy of a base model which is trained on their own dataset, and each trained model is evaluated on all the other turker datasets. The top-performing models get a cash bonus, incentivizing turkers to provide high quality training data. A new base model is trained on the pooled-together data of all the turkers, and a new round begins. The results indicate an improvement over static data collection.

This idea of HITL dataset creation is interesting, because the competitive aspect incentivizes turkers to produce high quality data. Judging by the feedback given by turkers in the appendix, the workers seem to enjoy the competitive aspect, which would hopefully lead to better data. The results seem to suggest that MTD provides an improvement over non-HITL methods.

The authors repeatedly emphasize the "collaborative" aspect of MTD, saying that the turkers have to collaborate to produce similar dataset distributions, but this is misleading because the turkers don't get to see other datasets. MTD is mostly competitive, and the authors should reduce the emphasis on a stretched definition of collaboration.

One questionable aspect of MTD is that the turkers somehow have to anticipate what are the best examples for the model to train with. That is, the turkers have to essentially perform the example selection process in active learning with relatively little interaction with the training model. While the turkers are provided immediate feedback when the model already correctly classifies the proposed training example, it seems difficult for turkers to anticipate when an example is too hard, because they have no idea about the learning process.

My biggest criticism is that MTD seems more like an NLP paper rather than an ICLR paper. I gave a 7 because I like the idea, but I wouldn't be upset if the AC recommends submitting to an NLP conference instead.
review_title: Cool idea, but better suited for an NLP conference

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper provides an interesting data collection scheme that improves upon standard collection of static databases that have multiple shortcomings -- End of Section 3 clearly summarizes the advantages of the proposed algorithm. The paper is easy to follow and the evaluation is meaningful.

In MTD, both data collection and training the model are intertwined and so, the quality of the data can be limited by the learning capacity of the model. It is possible that after some iterations, the data distribution is similar to previous rounds in which case, the dataset becomes similar to static data collection (albeit at a much higher cost and effort). Is this observed ? Further, is it possible to construct MTD variants that lead to constantly improving datasets by being agnostic to the actual model choice ? For example, utilizing only the priors of the D_{train_all}, mixing model and other humans' predictions, etc.




review_title: Interesting data collection scheme

Meta-review: This paper provides a game-based interface to have Turkers compete to analyze data for a learning task over multiple rounds. Reviewers found the work interesting and clear written, saying "the paper is easy to follow and the evaluation is meaningful." They also note that there is clear empirical benefit "the results seem to suggest that MTD provides an improvement over non-HITL methods." They also like the task compared to synthetic grounding experiments. There was some concern about the methodology of the experiments but the authors provide reasonable explanations and clarification.

One final concern that I hope the readers take into account. While the reviewers were convinced by the work and did not require it, I feel like the work does not engage enough with the literature of crowd-sourcing in other disciplines. While there are likely some unique aspects to ML use of crowdsourcing, there are many papers about encouraging crowd-workers to produce more useful data. 
Decision: Accept (Poster)