Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The statement that the framework you propose is `modeling the behavior of animals' is quite general.
In particular you have considered the case of one insect, the fruit fly, whose motion and action states (behavior) are tightly correlated.
The motion of the pen and the hand-written digit is also correlated.
Can you comment on how you would expect your model to perform if other animals were considered in which the same action/behavior such as (`opening a door' - `starting a car'), (`drinking' - `eating') can have very similar motions? 

In other words, is your model restricted to modeling behavior when it's correlated with the agent's motion patterns?
review_title: On the correlation of behaviour and motion

final_score: rating: null
review_content: In Figure 4b, BESNet and BENet have very comparable performance. In particular, BENet is better both with and without filter in the per-bout score. Do you happen to have any intuition or thoughts on why that is the case? Would that be related to the fact that in Fig 5b, there is almost no difference between the LL's associated with RNN and BESNet?

review_title: performance detail

final_score: rating: 7: Good paper, accept
review_content: While my above review title is too verbose, it would be a more accurate title for the paper than the current one (an overall better title would probably be somewhere in between). 

The overall approach is interesting: all three of the key techniques (aux. tasks, skip/diagonal connections, and the use of internal labels for the kind of data available) make a lot of sense.

I found some of the results hard to understand/interpret. Some of the explanation in the discussion below has been helpful (e.g. see my earlier questions about Fig 4 and 5); the paper would benefit from including more such explanations. 

It may be worthwhile very briefly mentioning the relationship of "diagonal" connections to other emerging terms for similar ideas (e.g. skip connections, etc). "Skip" seems to me to be accurate regardless of how you draw the network, whereas "diagonal" only makes sense for certain visual layouts.

In response to comment in the discussion below: "leading to less over-segmentation of action bouts" (and corresponding discussion in section 5.1 of the paper): I would be like to have a bit more about this in the paper. I have assumed that "per-bout" refers to "per-action event", but now I am not certain that I have understood this correctly (i.e. can a "bout" last for a few minutes?): given the readership, I think it would not be inappropriate to define some of these things explicitly.

In response to comment about fly behaviours that last minutes vs milliseconds: This is interesting, and I would be curious to know how classification accuracy relates to the time-scale of the behaviour (e.g. are most of the mistakes on long-term behaviours? i realize that this would only tell part of the story, e.g. if you have a behaviour that has both a long-term duration, but that also has very different short-term characteristics than many other behaviours, it should be easy to classify accuractely despite being "long-term"). If easy to investigate this, I would add a comment about it; if this is hard to investigate, it's probably not worth it at this point, although it's something you might want to look at in future.

In response to comment about scaling to human behavior: I agree that in principle, adding conv layers directly above the sensory input would be the right thing to try, but seriously: there is usually a pretty big gap between what "should" work and what actually works, as I am sure the authors are aware. (Indeed, I am sure the authors have a much more experiential and detailed understanding of the limitations of their work than I do). What I see presented is a nice system that has been demonstrated to handle spatiotemporal trajectories. The claims made should correspond to this. 

I would consider adjusting my rating to a 7 depending on future revisions.

review_title: using auxiliary tasks, skip connections and "internal labels" to classify spatiotemporal trajectories with an RNN

final_score: rating: null
review_content: "Animal behavior is nondeterministic": I think that this is an overstatement, since it depends on your frame of reference. At the very least I'd rephrase it by saying that we do not have access to the underlying cognitive processes, which makes the behavior look like a non-determinist one given the information we have.

My actual question is: given that the model of the behavior is non-deterministic, would a GAN be more natural fit for the predictive part of this task?

review_title: Non-Deterministic nature

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a method for joint motion prediction and activity classification from sequences with two different applications: motion of fruit flies and online handwriting recognition.

The method uses a classical encoder-decoder pipeline, with skip connections allowing direct communication between the encoder and the decoder on respective levels of abstraction.
Motion is discretized and predicted using classification. The model is trained on classification loss combined with a loss on motion prediction. The goal is to leverage latter loss in a semi-supervised setting from parts of the data which do not contain action labels.

The idea of leveraging predictions to train feature representations for discrimination is not new. However, the paper presents a couple of interesting ideas, partially inspired from other work in other areas.

My biggest concern is with the experimental evaluation. The experimental section contains a large amount of figures, which visualize what the model has learned in a qualitative way. However, quantitative evaluation is rarer.

- On the fly application, the authors compare the classification performance with another method previously published by the first author.
- Again on the fly application, the performance gain on motion prediction in figure 5c looks small compared to the baseline. I am not sure it is significant.
- I did not see any recognition results on the handwriting application. Has this part not been evaluated?

Figure 5a is difficult to understand and to interpret. The term "BesNet" is used here without any introduction.

Figure 4 seems to tell multiple and different stories. I'd suggest splitting it into at least two different figures.

review_title: Interesting paper with mostly qualitative evaluation 

final_score: rating: null
review_content: The authors mention several times, that actions are not outputs, but states. I am not sure to see the difference. Given the equations (and figures) y looks like a network output to me. It depends on the hidden state and not other variable depends on it. Moreover, as labels are not always provided, in this case it's relationship to the actual actions is unclear.
review_title: Actions as states

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a recurrent architecture for simultaneously predicting motion and action states of agents.
The paper is well written, clear in its presentation and backed up by good experiments.
They demonstrate that by forcing the network to predict motion has beneficial consequences on the classification of actions states,
allowing more accurate classification with less training data.
They also show how the information learned by the network is interpretable and organised in a hierarchy.

Weaknesses:
- a critical discussion on the interplay between motion an behaviour that is needed to experience the benefits of their proposed model is missing from the paper.
- moreover, a discussion on how this approach could scale to more challenging scenarios "involving animals" and visual input for instance and more general "behaviours" is also missing;
The criticism here is pointed at the fact that the title/abstract claim general behaviour modelling, whilst the experiments are focused on two very specific and relatively simple scenarios,
making the original claim a little bit far fetched unless its backed up by additional evidence.
Using "Insects", or "fruit flies" would be more appropriate than "animals".
review_title: Review

final_score: rating: null
review_content: I would like to follow on the original question of AnonReviewer2.

If you were to rename your title to be :
”Learning Recurrent Representations for Hierarchical Modeling of xxyy Behaviors”, 
what adjective might you use for xxyy and why? For example, the strongest statement would be that xxyy=“” implying that the work applies to all behaviors, while xxyy=“Spatiotemporal” would be a stronger qualifier requiring less justification.

Consider that human behaviour, for example, is often measured in motion capture data where each joint angle is associated with one or more degrees of freedom, thus implying a significantly higher-dimensional input vector than that used to describe the behavior of a fruit fly (let alone the larger visual input space required to differentiate between eating and drinking as mentioned below). Do you have any evidence that your system scales successfully in these ways? (Note that sparsity of labelled data, individual idiosyncrasies, longer time scales, etc might all be relevant issues with such behaviors).


review_title: extent of behavior modeling

Meta-review: Originality and Significance:
  The paper develops a recurrently coupled discriminative / generative hierarchical model, as applied to fruit-fly behavior and online handwriting. Qualitative evaluation is provided by generating motions, in addition to quantitative results. Writer identity and fly gender are learned in an unsupervised fashion. While the individual components of the solution are not particularly novel, their combination together with the detailed experimental validation makes the method potentially interesting to a broad audience. Some reviewers have concerns about the broad claims that are implied by the title and abstract, and thus it is recommended that these be refined to be more specific about the method and the applications.
 
 Quality and Clarity:
  The paper is well written.
 
 Pros:
 - interesting problem and application domains; will be of broad interest
 - useful ideas and architecture: shows that forcing the network to predictions about motion leads to improved classification
 - well written paper backed up by good experiments
 
 Cons:
 - the individual architectural features have for the most part been proposed before
Decision: Accept (Poster)