Abstract: We explore the challenge of action prediction from textual descriptions of scenes, a testbed to approximate whether text inference can be used to predict upcoming actions. As a case of study, we consider the world of the Harry Potter fantasy novels and inferring what spell will be cast next given a fragment of a story. Spells act as keywords that abstract actions (e.g. 'Alohomora' to open a door) and denote a response to the environment. This idea is used to automatically build HPAC, a corpus containing 82 836 samples and 85 actions. We then evaluate different baselines. Among the tested models, an LSTM-based approach obtains the best performance for frequent actions and large scene descriptions, but approaches such as logistic regression behave well on infrequent actions.

Introduction: Natural language processing (nlp) has achieved significant advances in reading comprehension tasks BIBREF0 , BIBREF1 . These are partially due to embedding methods BIBREF2 , BIBREF3 and neural networks BIBREF4 , BIBREF5 , BIBREF6 , but also to the availability of new resources and challenges. For instance, in cloze-form tasks BIBREF7 , BIBREF8 , the goal is to predict the missing word given a short context. weston2015towards presented baBI, a set of proxy tasks for reading comprenhension. In the SQuAD corpus BIBREF9 , the aim is to answer questions given a Wikipedia passage. 2017arXiv171207040K introduce NarrativeQA, where answering the questions requires to process entire stories. In a related line, 2017arXiv171011601F use fictional crime scene investigation data, from the CSI series, to define a task where the models try to answer the question: ‘who committed the crime?’. In an alternative line of work, script induction BIBREF10 has been also a useful approach to evaluate inference and semantic capabilities of nlp systems. Here, a model processes a document to infer new sequences that reflect events that are statistically probable (e.g. go to a restaurant, be seated, check the menu, ...). For example, chambers2008unsupervised introduce narrative event chains, a representation of structured knowledge of a set of events occurring around a protagonist. They then propose a method to learn statistical scripts, and also introduce two different evaluation strategies. With a related aim, Pichotta2014Statistical propose a multi-event representation of statistical scripts to be able to consider multiple entities. These same authors BIBREF11 have also studied the abilities of recurrent neural networks for learning scripts, generating upcoming events given a raw sequence of tokens, using bleu BIBREF12 for evaluation. This paper explores instead a new task: action prediction from natural language descriptions of scenes. The challenge is addressed as follows: given a natural language input sequence describing the scene, such as a piece of a story coming from a transcript, the goal is to infer which action is most likely to happen next.

Question: Why do they think this task is hard?  What is the baseline performance?
Answer: 1. there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty.
2. Macro F1 = 14.6 (MLR, length 96 snippet)
Weighted F1 = 31.1 (LSTM, length 128 snippet)