Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes two Interactive fiction tasks in which the learning of commonsense knowledge and multi-hop reasoning are important. The authors claim that in these tasks a model would need to learn "various types" of commonsense in the form of world model learning. They propose several baselines based on BiLSTM and BERT, and show the gap between human performance.
strengths: The proposed problem of learning commonsense knowledge and multi-hop reasoning together from interactive fiction is interesting. The experimental results that "triple modeling" is consistently better than "textual entailment" exhibits the importance of reasoning, but it is unclear how it is related to "multi-hop".
weaknesses: The authors formulate their task as POMDP, but it is unclear how the baselines reflect the partially observable nature. It is also not clear how the “multi-hop” reasoning is modeled in these baselines. Why the comparison with human subjects is fair? Humans may not need to learn these commonsense from the IF tasks. Others see Correctness and Clarity. 


Post-rebuttal

After reading the author's response, I still stand by my initial rating. The paper is vaguely written, using too many terms without careful definition. The POMDP issue is not well-justified and performance results are not clearly explained. I believe a well-done dataset paper should clearly demonstrate uniqueness, clarity, and a sufficient benchmark.

(modified: 20 Jul 2021)
correctness: The authors do not provide a justification of whether the training data is sufficient to learn the "commonsense knowledge" and the "multi-hop reasoning". I am curious why the performance on pretrained BERT does not surpass trained LSTM too much does not illustrate an insufficiency? 
clarity: The clarity of this paper can be largely improved. There are lots of vague terms that require some educated guess. For example, the authors keep mentioning "reasoning types" and "knowledge types", using these concepts to highlight the uniqueness of the proposed dataset. But it is unclear to readers what these types are. L61-65 of P2, it is written "[d]ue to the textual observation’s narrative nature, a large portion of the textual observations are not a sole statement of the action effect, but an extended narrates about what happens because of the effect", which also occurs vague to me the different of "a sole statement of the action effect" and "an extended narrates about what happens because of the effect". 

In the model and the experiment sections, the authors mention they employ pretrained BERT. But it is not verbose if the pretrained model is finetuned or frozen. 

relation to prior work: I am not quite familiar with this literature. As mentioned in weakness, the evaluation of relation to prior work is substantially hindered by the vagueness in some key concepts.
additional feedback: I think this paper can be largely improve if the authors could find an elegant way to illustrate more details in the dataset. Currently, there are too many vague high-level concepts that are ungrounded to specific examples or statistics.
review_title: Official Review for JECC

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents two datasets based on human interactive gameplays from the Jericho environment and Zork Universe, where the goal is to predict the next observation based on the current observation (roughly a description of the scene the player is playing at that moment) and the action that the player took.

Among the main contributions of the paper the authors argue it is a useful dataset for developing models intended for commonsense and multi-hop reasoning. 

strengths: 1. Creating reasoning frameworks with natural language sentences is a relevant to evaluate models and what they can do. Although this dataset is restricted to game scripts, it’s an useful testbed for this purpose.
weaknesses: 1. The authors say that in comparison to much of previous work, their dataset requires different types of reasoning to solve the task correctly, and support this claim on examples like the ones shown in Figure 1. Yet, I feel it would be good to clearly specify the *different* types of reasoning that are in the dataset and which types are required to solve each sample, maybe using human evaluation of all samples, but if not possible at least to provide some quantitative evidence. I feel that for a dataset like this it would be interesting to be able to know what ‘kinds of reasoning’ the models are not able to perform.

2. Related to the previous point, the authors discuss multi-hop reasoning and how this dataset should be useful for multi-hop systems. However, it is not clear to me how the baselines presented in S4 are multi-hop, since they seem to be roughly an LSTM+attention and BERT, and if they are not, why a multi-hop model has not been considered?

3. I’m not specially familiar with these video games, so I would appreciate it if the authors could expand on the next question. To my understanding, some of the next observations might not be driven only by commonsense or by ‘what is expected’, i.e. I would expect that the player's actions might turn out sometimes into surprise observations, traps, etc, is that correct? And if so, how would this affect the evaluation of the dataset?

4. Could the authors comment more on how this adds research value to the original Jericho environment (https://arxiv.org/pdf/1909.05398.pdf) which they consider as the starting point to create their dataset? 

correctness: The methodology to create the corpus is correct, the motivation is valid and makes sense, the proposed baselines are valid and the evaluation metrics are valid as well.
clarity: Overall, the paper is presented in a clear way. There are some minor aspects that can be improved such as the layout of the Figures, and a few details about the models description and the evaluation (I included my specific comments in the Additional feedback).
relation to prior work: The authors compare against different types of work, however, it is not totally clear to me in the current version why the JECC and ZUCC dataset present additional challenges to some of the previous work (for instance, the next sentence prediction datasets). Would it be possible to elaborate more on this? (see also Figure 1).
additional feedback: 1. Text boxes from Figure 1 are too small, IMO.
2. Line 201 mentions that ‘narratives form a whole story’ which is True, however, doesn’t this vanish with the proposed format of the dataset, since each triplet (observation, action, next_observation) seems to be handled and predicted independently from the others, is this correct?
3. In lines 169-170, 177-180 the authors discuss the problems with the drop actions and actions that have the same effects as a drop action. However, it is not clear to me how these latter actions are identified to down-sample them. Would it be possible to expand on this?
4. The Limitations  paragraph is a bit vague and little informative in its current version, so it is not really useful for the readers as it is. I suggest expanding it and discussing specific limitations.
5. In Figure 3, we see the action as a_{t+1}, but I wonder whether it should be a_{t}, as specified in other parts of the paper (e.g. line 150)?
6. In lines 278-282 the authors mention that the co-matching models perform better than the match models and say that the ‘co-matching mechanism better addressed our datasets’, but there is no explanation or hypothesis about why, would it be possible to expand on this?
7. The batch size various -> The batch size varies?
8. I had some problems understanding the evaluation of the walkthrough tuples vs all tuples. I think the source of the problem for me is that these concepts are scattered through different and non-consecutive parts of the paper (e.g. thin red lines in Figure 1, what is a walkthrough and a triplet belonging to a walkthrough, different corpora sizes for JECC and ZUCC, triplet transformation of the corpora, ...). I think it would be helpful to briefly re-collect these concepts when discussing the evaluation of walkthough vs all tuples.
9. In equation (1) (and similarly in equivalent models) H^{o,a} would correspond to the last hidden vector of the LSTM?
review_title: Predict the next observation of an interactive gameplay given the current observation and the selected action

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a completely new approach to creating tasks for evaluating common-sense reasoning. 

The contributions are as follows:

* new common-sense reasoning data sets (ZUCC, JECC) based on interactive fictions,
* strong neural baselines for the data sets,
* human evaluation for samples of the data sets.
strengths: The main strength is that the authors are proposing a completely new approach for creating data sets for common-sense reasoning tasks. The domain of common-sense reasoning benchmarks really needs new approaches and the idea presented by the authors (to extract tasks from interactive fictions) is very interesting. The data sets created in this way do bring new quality to the problem of evaluation of common-sense reasoning.

Another strength is that strong baselines are given along with human baselines. Interesting insights are given with baselines.


weaknesses: The authors are using, at least for JECC, real walk-throughs. It is not clear whether they are really needed. Why not just use all the tuples extracted from the game? (I assume it is possible, as games are deterministic, though I'm not sure). Actually, this approach is used by the authors themselves for the ZUCC data set. The authors should show that data sets based on real walk-throughs are really superior to the ones just extracted from the games (as the latter approach is much easier).

The section on baselines is in general of good quality. I have, however, some doubts:

* more state-of-the-art Transformer models, such as RoBERTa, should be evaluated,
* it is not a weakness per se, but good results for a simple BiLSTM model (I assume it is not a pre-trained model, right?) are quite surprising and suspicious (so maybe the task is not as difficult as it seems?), i think more discussion should be devoted to this, for instance, maybe even simpler models (e.g. based on tf-idf) would obtain good results? 
* it is not clear to which extent the results for BERT are affected by the fact the texts are too long, this should be assessed quantitatively (e.g. what would be the score if such cases were excluded?)

It is not clear how many annotators were employed, one or more. One is of course too few... more annotators should be used to obtain more reliable estimation of human performance.







correctness: I did not find any issue here. The evaluation methods and experiment design are OK, in my opinion.
clarity: The paper has some unclear moments. The discussion about reasoning beyond verification/validation (196-204) is quite unclear.

I don't think the discussion of IF games in terms of POMDPs is really needed (section 3.1). In my opinion, it could be skipped or shortened.

The paper might be sometimes hard to follow for readers without a background in reinforcement learning, for instance a short description of the Jericho environment would be helpful.

It is not clear whether human evaluation was done under the same conditions as the evaluation of neural models: in 302-303 it seems the humans saw all the possible actions, whereas 155-163 suggest that just one action $a_t^j$ is selected.
relation to prior work: No comments here.
additional feedback: Minor remarks:
* 26: unnecessary space before the footnote mark
* 175: "walkthough"
* 127: "langauge"
* some words are lower-cased in references, e.g. "winograd", "Bert" (instead of "BERT")
review_title: Common-sense Reasoning Tasks from Interactive Fictions

Meta-review: The proposed dataset interesting and unique. However, there are also still significant limitations rased by the reviewers that should be addressed before accepting the paper. Also, the presentation of the paper should be substantially improved, and the performance results should be better explained. After discussion with multiple area chairs, we feel that the paper is not yet ready for publication. Perhaps the remaining issues could be addressed before the second round of this track.
Decision: Reject