Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a new evaluation platform for what they define 'useful' general AI and the desired characteristics for this kind of system.

Pro:
(Attempts to) Tackle a very important problem, that has yet to be properly formalized or agreed on by the general community.  
It's in line with other efforts, such as openai.com/blog/universe, gvgai.net, github.com/deepmind/lab, to bring forward new and diverse tasks for the community to play with. This ultimately pushes us to develop more general learning algorithms that indeed need to "learn to learn" or learn to adapt to different, but related tasks. I think that's something that has been more and more important. 

Cons:
The major problem I have with paper and framework, stems not necessarily from the tasks themselves, but from the identified desiderata. It seems to be <<very>> natural language/text focused. This is an on-going debate whether or not that is a crucial component in the development of general AI and how we will interact with AI. It seems to me, that most of the effort -- at least computationally -- would be spent modelling the particular structure present in text-like inputs/data and that automatically shifts the focus away from what we should be doing, or what the AIs should be trying to figure out which is more complex tasks, more planning and optimization challenging scenarios.
Which brings me to the second point. Say you believe in the desiderata outline, the tasks seem to match what was highlighted in the agenda, but the level of complexity it's relatively low. That's not to say that these are simple tasks for our learning algorithms to pick up, just the complexity doesn't lie in the task per se, but trying to model the language/syntax. I fail to see what succeeding on these tasks, say, in general, about now taking this system and applying it to optimizing energy consumption, or recognizing emotions, or even something domain-related like dialogue systems.

To sum: I think the paper addresses a very real problem and, as I said previously, I don't think we don't have the/a right answer or even a satisfactory answer at this point in time. I do think though the framework proposed is too limited in scope to claim generality. That being said, it might still be 'useful' -- if you agree with the proposed desiderata, it seems like a sensible set of tasks to try out.  

 
  
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method for evaluating the capacity of a learning algorithm to function as a "general AI". The proposal consists of two pieces:

- high-level desiderata for a general AI: namely, the ability to efficiently learn multiple tasks with shared structure from natural language guidance provided as a generic bit stream.

- a concrete task for investigating these desiderata: namely, membership and sampling queries for regular languages specified by regular expressions.

I'm honestly not sure what to make of this paper. It's very much in the same spirit as the earlier bAbI tasks, which, while well-intentioned, I think have done active harm to the AI research community by making it socially acceptable to develop methods for toy problems without ever verifying that they actually scale-up to real-world tasks. To use the dichotomy the authors introduce in section 3, the problem is that essentially all meaningful progress in the field has come from "bottom-up" approaches: "top-down" approaches have a poor track record of scaling up, while the set of challenging reasoning problems solved by "bottom-up" approaches continues to grow.

On the other hand, I recognize that mine may no longer be a mainstream position, and that in any case it's unfair to downgrade a position paper because I disagree with the position. It's probably healthy for the community to have this discussion, and the ICLR crowd perhaps needs it most of all.

But I do think the position could be better defended. In particular: we already know that:

1. It's easy for RNNs to sample from and query regular languages (learned by example---I don't actually know of work on starting from symbolic REs as done here)

2. It's certainly possible to learn from this mixed RL / text-based supervision condition (e.g. Weston 2016).

These two things together make up the whole task. So it's not obvious that we can't solve it by throwing generic RNN machinery at it, and the burden is on the authors of a new task to show that it can't already be solved using state-of-the-art methods. The paper's claim that "We hope the CommAI-mini challenge is at the right level of complexity to stimulate researchers to develop genuinely new models" would be much stronger if it already demonstrated that genuinely new models are required; right now that demonstration is missing.

Now suppose we do solve this task (with whatever model). What have we learned? Just that it's possible to quickly learn how to work with regular languages? What notion of compositionality is present in this task and not, e.g. natural language parsing? What have we learned about "learning to learn" that's different from learning good gradient descent algorithms, initializers, RL with natural language instructions, etc.?
review_title: Review

Decision: Accept