Title: Learning Recurrent Representations for Hierarchical Behavior Modeling
Authors: Eyrun Eyjolfsdottir
Kristin Branson
Yisong Yue
Pietro Perona
Venue: ICLR 2017
Keywords: Unsupervised Learning
Semi-Supervised Learning
Reinforcement Learning
Applications

Abstract: We propose a framework for detecting action patterns from motion sequences and modeling the sensory-motor relationship of animals, using a generative recurrent neural network. The network has a discriminative part (classifying actions) and a generative part (predicting motion), whose recurrent cells are laterally connected, allowing higher levels of the network to represent high level behavioral phenomena. We test our framework on two types of tracking data, fruit fly behavior and online handwriting. Our results show that 1) taking advantage of unlabeled sequences, by predicting future motion, significantly improves action detection performance when training labels are scarce, 2) the network learns to represent high level phenomena such as writer identity and fly gender, without supervision, and 3) simulated motion trajectories, generated by treating motion prediction as input to the network, look realistic and may be used to qualitatively evaluate whether the model has learnt generative control rules.  


Review discussion:
The statement that the framework you propose is `modeling the behavior of animals' is quite general.
In particular you have considered the case of one insect, the fruit fly, whose motion and action states (behavior) are tightly correlated.
The motion of the pen and the hand-written digit is also correlated.
Can you comment on how you would expect your model to perform if other animals were considered in which the same action/behavior such as (`opening a door' - `starting a car'), (`drinking' - `eating') can have very similar motions?
In other words, is your model restricted to modeling behavior when it's correlated with the agent's motion patterns?
Dear reviewer,
Thank you for your question. The system is not restricted to modeling behavior when it is correlated with the agent’s motion patterns, it can disambiguate between action classes with similar motion patterns by two means:
1) Visual context. The input to the network is not only the motion sequence of an animal but also a sequence of sensory inputs. In the case of a fruit fly, sensory inputs represent the position of other agents and the chamber walls relative to the fly. If the system were applied to humans, the motion sequence might be the output of wearable motion sensors or a tracked skeleton and the sensory input the output of a 1st person camera or an overhead camera.
While the motion pattern for ‘drinking’ and ‘eating’ might be similar, the visual context could disambiguate between the two.
2) Temporal context. Action labels are embedded in the recurrent cells, in our case the gated recurrent units, which gives the system a long short term memory of actions. ‘Starting a car’ would thus be more likely than ‘opening a door’ after a sequence of ‘opening a door’ and ‘sitting down’ has been observed (given that the sequence ‘opening a door’ --> ‘sitting down’ --> ‘starting a car’ appears more often in the training labels than ‘opening a door’ --> ‘sitting down’ --> ‘opening a door’).
Sincerely,
Eyrun
The authors mention several times, that actions are not outputs, but states. I am not sure to see the difference. Given the equations (and figures) y looks like a network output to me. It depends on the hidden state and not other variable depends on it.
Moreover, as labels are not always provided, in this case it's relationship to the actual actions is unclear.
Dear reviewer,
Thank you for your question. In Figure 1 we show that classification vector \hat{y} is embedded in hidden state h^2 and in the equations on page 4 we show that \hat{y} is h^L(1:N), rescaled to be in the range [0 1] rather than [-1 1] (making it suitable for cross entropy loss computation). Rewriting the relationship between \hat{y} and h^L as h^L(1:N) = \hat{y}*2 - 1 perhaps makes it more apparent that classification \hat{y}_{i+1} and motion prediction \hat{x}_{i+1} depend on classification \hat{y}_i.
When no labels are provided during training that means that no restriction is placed on h^L.
When labels are provided, for all or any of the frames, the labels force the first N units of h^L to be classification units.
When labels are given in training, the network classifies each frame but cross entropy loss is computed only on frames for which labels are given.
Best,
Eyrun
I also found the way this was presented to be quite hard to understand. I think that now I get it.
"Animal behavior is nondeterministic": I think that this is an overstatement, since it depends on your frame of reference. At the very least I'd rephrase it by saying that we do not have access to the underlying cognitive processes, which makes the behavior look like a non-determinist one given the information we have.
My actual question is: given that the model of the behavior is non-deterministic, would a GAN be more natural fit for the predictive part of this task?
Thank you for your comment.
Indeed, we do not claim that behavior is nondeterministic but rather point at evidence that suggests it. It may or may not be deterministic in nature but, as you mention, within our approximations it is more likely to be. We will emphasize this in the text. In general, we would like our system to be able to model nondeterministic behavior.
GANs are not directly relevant when considering output being nondeterministic given the input. The adversarial part of GANs effectively serves as a loss function for the generative model, pushing it to produce more realistic outputs, but it does not make the generative model non-deterministic. The generative model is commonly deterministic, conditioned on random noise. In our setting we explicitly model the distribution over possible outputs given an input.
The training signal we use for motion prediction is the ground truth future motion of an agent. An interesting future direction would be to use adversarial training to criticize multi-step predictions of our network, with the goal of generating more realistic simulations.
I would like to follow on the original question of AnonReviewer2.
If you were to rename your title to be :
”Learning Recurrent Representations for Hierarchical Modeling of xxyy Behaviors”,
what adjective might you use for xxyy and why?
For example, the strongest statement would be that xxyy=“” implying that the work applies to all behaviors, while xxyy=“Spatiotemporal” would be a stronger qualifier requiring less justification.
Consider that human behaviour, for example, is often measured in motion capture data where each joint angle is associated with one or more degrees of freedom, thus implying a significantly higher-dimensional input vector than that used to describe the behavior of a fruit fly (let alone the larger visual input space required to differentiate between eating and drinking as mentioned below). Do you have any evidence that your system scales successfully in these ways? (Note that sparsity of labelled data, individual idiosyncrasies, longer time scales, etc might all be relevant issues with such behaviors).
Dear reviewer,
Thank you for your question. I apologize for the late response which is due to my travels.
We state in the introduction that this work focuses on going from trajectories to actions, with emphasis on modeling the dynamics of behavior rather than feature extraction. We take advantage of the fact that we can obtain such data for flies behaving naturally in a controlled environment, which cannot be easily obtained for humans.
Thus, the fruit fly can be used as a model organism, not only for science but also for AI research. Modeling human behavior is not our only end goal, the modeling of laboratory-animal behavior is very useful for the behavior science community.
We do test our model on a subset of human behavior, online handwriting, that fits easily into our framework. Should we apply the framework to human motion capture with video as sensory input we would not directly apply the framework but, for instance, put a convolutional neural net between the sensory input and our network to extract a higher level sensory representation before feeding it to the dynamical system.
I would also like to point out that the data on which we evaluate our framework also has some of the complexities you mention:
**Sparsity of labeled data. This can mean that:
a) Data is partially and intermittently labeled. Our framework handles this by computing loss only on frames where labels are provided (and that loss gets propagated back to preceding input frames that may be themselves not be labeled).
b) In a fully labeled dataset, behaviors of interest are rare and the *null* category takes up majority of the frames. This is the case with our fly behavior data where majority of actions take up less than 1% of the frames.
**Individual idiosyncrasies. This is the main challenge of the handwriting data.
**Long time scales. Fly behaviors are performed at various time scales, some actions are as short as a millisecond while others can take up several minutes (this is the case within and across action classes).
Best,
Eyrun
In Figure 4b, BESNet and BENet have very comparable performance.
In particular, BENet is better both with and without filter in the per-bout score. Do you happen to have any intuition or thoughts on why that is the case? Would that be related to the fact that in Fig 5b, there is almost no difference between the LL's associated with RNN and BESNet?
Thank you for your comment. On Fly-vs-Fly performance of BESNet and BENet is on par when data is fully labeled, but we show (in Figure 4a) that with partial labels the generative network performs significantly better than its discriminative counterpart, on all three datasets.
A better bout-wise performance and worse frame-wise performance indicates that classification is smoother in time. This could happen if the learnt bias for an action results in a lower threshold on the classification score, leading to less over-segmentation of action bouts. It is possible that motion prediction benefits from more accurate frame-wise classification, thus improving the frame-wise performance (which is the classification target we optimize for) at the expense of bout-wise performance (which we do not explicitly optimize for). This would be an interesting point to analyze further.
Motion prediction may equally good with or without diagonal connections, but we show that the diagonal connections enable higher level phenomena to be represented in higher states of the model, making them suitable states for action classification.
To summarize, a standard RNN may have good performance on both action detection and motion prediction tasks, but the key advantages of our network are: 1) training on the two tasks simultaneously leads to better performance on the action detection task, especially when labels are scarce, 2) embedding action labels in hidden states allows us to induce actions during simulation, and 3) diagonal connections allow higher level phenomena to be represented in higher states of the network, making them suitable states from which to classify or discover behavioral phenomena.
This paper proposes a recurrent architecture for simultaneously predicting motion and action states of agents.
The paper is well written, clear in its presentation and backed up by good experiments.
They demonstrate that by forcing the network to predict motion has beneficial consequences on the classification of actions states,
allowing more accurate classification with less training data.
They also show how the information learned by the network is interpretable and organised in a hierarchy.
Weaknesses:
- a critical discussion on the interplay between motion an behaviour that is needed to experience the benefits of their proposed model is missing from the paper.
- moreover, a discussion on how this approach could scale to more challenging scenarios "involving animals" and visual input for instance and more general "behaviours" is also missing;
The criticism here is pointed at the fact that the title/abstract claim general behaviour modelling, whilst the experiments are focused on two very specific and relatively simple scenarios,
making the original claim a little bit far fetched unless its backed up by additional evidence.
Using "Insects", or "fruit flies" would be more appropriate than "animals".
Thank you for your review. We have added a discussion at the end of the paper about scaling to more complex data.
Regarding a critical discussion on the interplay between motion and behavior, in the introduction we explain that motion trajectories and actions represent behavior at different levels of timescales, and that actions can be detected from motion trajectories. If your point refers to a more biological explanation, such as the one we discuss in our global response, I am also happy to add that to the introduction of the paper.
The paper presents a method for joint motion prediction and activity classification from sequences with two different applications: motion of fruit flies and online handwriting recognition.
The method uses a classical encoder-decoder pipeline, with skip connections allowing direct communication between the encoder and the decoder on respective levels of abstraction.
Motion is discretized and predicted using classification. The model is trained on classification loss combined with a loss on motion prediction. The goal is to leverage latter loss in a semi-supervised setting from parts of the data which do not contain action labels.
The idea of leveraging predictions to train feature representations for discrimination is not new.
However, the paper presents a couple of interesting ideas, partially inspired from other work in other areas.
My biggest concern is with the experimental evaluation. The experimental section contains a large amount of figures, which visualize what the model has learned in a qualitative way.
However, quantitative evaluation is rarer.
- On the fly application, the authors compare the classification performance with another method previously published by the first author.
- Again on the fly application, the performance gain on motion prediction in figure 5c looks small compared to the baseline. I am not sure it is significant.
- I did not see any recognition results on the handwriting application. Has this part not been evaluated?
Figure 5a is difficult to understand and to interpret. The term "BesNet" is used here without any introduction.
Figure 4 seems to tell multiple and different stories. I'd suggest splitting it into at least two different figures.
Thank you for your review. To answer your questions:
** I did not see any recognition results on the handwriting application. Has this part not been evaluated?
- Figure 4 has quantitative results for all 3 labeled datasets, including handwriting, for varying number of training labels.
** Figure 4 seems to tell multiple and different stories. I'd suggest splitting it into at least two different figures.
- The 3 plots in Figure 4 are grouped because they all refer to classification performance, in order to make best use of space.
- We have added more details to the caption of figure 4 to connect the stories.
** Figure 5a is difficult to understand and to interpret. The term "BesNet" is used here without any introduction.
- BESNet and BENet are described on page 7 paragraph 2, before figure 5 on page 8.
- We have added more details to the caption of figure 5.
** The experimental section contains a large amount of figures, which visualize what the model has learned in a qualitative way.
However, quantitative evaluation is rarer.
- Figure 5 shows quantitative 1-step motion prediction performance, and figure 4 quantitatively evaluates classification performance.
- For multi-step prediction (i.e. simulation), qualitative results can be more informative than quantitative, as the objective function may not capture what looks realistic to humans. This is comparable to image generation where qualitative results and human scoring are standard methods of evaluation, e.g. https://arxiv.org/pdf/1506.05751v1.pdf.
- The paper does indeed have a lot of qualitative results, which are better demonstrated on our project website as videos: http://www.vision.caltech.edu/~eeyjolfs/behavior_modeling/
While my above review title is too verbose, it would be a more accurate title for the paper than the current one (an overall better title would probably be somewhere in between).
The overall approach is interesting: all three of the key techniques (aux. tasks, skip/diagonal connections, and the use of internal labels for the kind of data available) make a lot of sense.
I found some of the results hard to understand/interpret. Some of the explanation in the discussion below has been helpful (e.g. see my earlier questions about Fig 4 and 5); the paper would benefit from including more such explanations.
It may be worthwhile very briefly mentioning the relationship of "diagonal" connections to other emerging terms for similar ideas (e.g. skip connections, etc). "Skip" seems to me to be accurate regardless of how you draw the network, whereas "diagonal" only makes sense for certain visual layouts.
In response to comment in the discussion below: "leading to less over-segmentation of action bouts" (and corresponding discussion in section 5.1 of the paper): I would be like to have a bit more about this in the paper. I have assumed that "per-bout" refers to "per-action event", but now I am not certain that I have understood this correctly (i.e. can a "bout" last for a few minutes?): given the readership, I think it would not be inappropriate to define some of these things explicitly.
In response to comment about fly behaviours that last minutes vs milliseconds: This is interesting, and I would be curious to know how classification accuracy relates to the time-scale of the behaviour (e.g. are most of the mistakes on long-term behaviours? i realize that this would only tell part of the story, e.g. if you have a behaviour that has both a long-term duration, but that also has very different short-term characteristics than many other behaviours, it should be easy to classify accuractely despite being "long-term"). If easy to investigate this, I would add a comment about it; if this is hard to investigate, it's probably not worth it at this point, although it's something you might want to look at in future.
In response to comment about scaling to human behavior: I agree that in principle, adding conv layers directly above the sensory input would be the right thing to try, but seriously: there is usually a pretty big gap between what "should" work and what actually works, as I am sure the authors are aware. (Indeed, I am sure the authors have a much more experiential and detailed understanding of the limitations of their work than I do). What I see presented is a nice system that has been demonstrated to handle spatiotemporal trajectories. The claims made should correspond to this.
I would consider adjusting my rating to a 7 depending on future revisions.
Thank you for your review. Majority of your comments are addressed in the global response.
Regarding how classification accuracy relates to the time-scale of the behavior, I agree that it would be interesting to further analyze the types of errors we get with respect to timescale and we plan to explore this for future development.
Thanks for the responses.
A few more final comments:
1) re: terminology in title/abstract, "behaviour" vs "spatiotemporal behaviour". At this point, I'll leave it to the committee/area chairs-- I think the key points have been made in our comments/responses, & I don't have any further thoughts/opinions about it either way.
2) re: def'n of "bout": good, thanks.
3) re: diagonal/skip connections: I appreciate the small revision. If they are no different from what is/has becoming fairly standard to refer to as "skip" connections, i am not sure i see the benefit of using a different term.
4) re: relationship of classification accuracy to time-scale (see my question in my above review). The more I think about this, the more I am curious about it, and the more relevant it seems, but I realize that I previously said that "if it is hard to investigate, it is probably not worth it".
I do think the paper describes interesting and well-done work, and my previous comments have indeed been addressed to various extents; my "hidden real-valued state"  for the review would be to partially increase the rating (e.g. to 6.5). In the review above, I'm simply changing it to 7.
Thanks again for your feedback, I have enjoyed our discussion.
Dear reviewers,
Thank you all for your feedback and suggestions.
** We have added the following changes to the paper:
- better explanation of figures 4 and 5 in their captions
- clearer definition a bout (Section 5.1, first paragraph)
- analogy between diagonal connections and skip connections added (Section 3.1, first paragraph)
** Regarding scaling to other data
In Section 4, paragraph 1, we describe how data should be represented to fit our framework, and we have added a discussion at the end of the paper about scaling to more complex scenarios (Section 6, paragraph 2).
** In response to modeling “behavior” being too strong of a claim and that it should be replaced with “spatiotemporal behavior” or “fruit fly behavior”:
1) We qualify our statement in the abstract, stating both that we focus on motion sequences (i.e. trajectories) and that we test it on fruit flies and handwriting, which we further elaborate on in the introduction.
2) I would argue that behavior is spatiotemporal.
People from different fields may think of behavior differently, ranging from generating to observing behavior. A roboticist may think of it in terms of moving joints, an animator in terms of moving vertices, and a computer vision researcher in terms of moving pixels.
As noted by Levitis et al. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2760923/), “Behavioural biologists don’t agree on what constitutes behaviour”, but the authors offer a definition that agrees with most: “Behaviour is the internally coordinated responses (actions or inactions) of whole living organisms (individuals or groups) to internal and/or external stimuli, excluding responses more easily understood as developmental changes.” Some other definitions listed in the paper are that behavior is:
“The total movements made by the intact animal.” Tinbergen (1955)
“Externally visible activity of an animal, in which a coordinated pattern of sensory, motor and associated neural activity responds to changing external or internal conditions.” Beck et al (1991)
The sequence of motor responses as a function of external and internal stimuli, which results in the body moving in space over time, is what we aim to model.
3) We do not only test our model on fruit fly behavior but also on handwriting, thus prepending behavior with “fruit fly” or “insects” would be misleading.
To conclude: If behavior can be defined as spatiotemporal, and we test our model on data other than insects or laboratory animals, I find the title to be fitting as is. I am willing to continue that conversation if the reviewers still strongly disagree.
** Remaining questions/concerns are addressed in individual responses.
Best,
Eyrun

Meta-review:
role: PC
data: comment: Originality and Significance:
  The paper develops a recurrently coupled discriminative / generative hierarchical model, as applied to fruit-fly behavior and online handwriting. Qualitative evaluation is provided by generating motions, in addition to quantitative results. Writer identity and fly gender are learned in an unsupervised fashion. While the individual components of the solution are not particularly novel, their combination together with the detailed experimental validation makes the method potentially interesting to a broad audience. Some reviewers have concerns about the broad claims that are implied by the title and abstract, and thus it is recommended that these be refined to be more specific about the method and the applications.
 
 Quality and Clarity:
  The paper is well written.
 
 Pros:
 - interesting problem and application domains; will be of broad interest
 - useful ideas and architecture: shows that forcing the network to predictions about motion leads to improved classification
 - well written paper backed up by good experiments
 
 Cons:
 - the individual architectural features have for the most part been proposed before

Decision: Accept (Poster)