Title: End-to-End Learnable Histogram Filters
Authors: Rico Jonschkowski
Oliver Brock
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Problem-specific algorithms and generic machine learning approaches have complementary strengths and weaknesses, trading-off data efficiency and generality. To find the right balance between these, we propose to use problem-specific information encoded in algorithms together with the ability to learn details about the problem-instance from data. We demonstrate this approach in the context of state estimation in robotics, where we propose end-to-end learnable histogram filters---a differentiable implementation of histogram filters that encodes the structure of recursive state estimation using prediction and measurement update but allows the specific models to be learned end-to-end, i.e. in such a way that they optimize the performance of the filter, using either supervised or unsupervised learning.

TL;DR: a way to combine the algorithmic structure of Bayes filters with the end-to-end learnability of neural networks

Review discussion:
1. A strong prior on the motion model is assumed -- linear motion with Gaussian noise. This is applicable only in simple scenarios: such motion model is not directly applicable in settings, where either: (1) the actions relate to the resulting motion in a more complicated way, or (2) the environment effects the transition -- which is the case for most practical settings.
How can this approach be scaled to such more complex settings?
2. How does the proposed method perform in more realistic settings, e.g. drone navigation in a complex environment (with obstacles) with inertial measurements and fan-velocity actions?
Thank you very much for your questions. We will answer each question first concisely and then give more detailed explanations.
1. Linear motion models are not a limitation of end-to-end learnable histogram filters (E2E-HFs). a) Linear motion models were only needed to make unsupervised learning work. Supervised learning worked well with generic motion models in all our experiments. b) Locally linear or other simple motion models have proven useful in practice and are widely applied in robotics. c) More complex/generic motion models can be used instead.
Overall, E2E-HFs present an opportunity, not a requirement, to insert prior knowledge about robot motion.
However, as mentioned in the paper, the assumption that motion is independent of the state is a restriction of E2E-HFs. We are currently addressing this problem to scale the approach to more complex settings as described below (see 2.b).
a) In our experiments, we found that the restrictions on the motion model were only required in the unsupervised setting, while training the E2E-HF using supervised learning worked well even when we used a generic feed forward network as motion model (we omitted these experiments due to space constraints).
b) Practical experience in robotics shows that it is often sufficient to ignore the environment in the motion model (Thrun et al.: Probabilistic Robotics), especially when the robot has access to odometry/inertial measurements, which already include the result of actual physical interactions.
When robots are blocked by obstacles, their wheel odometers or inertial sensors will show no movement, which will be the action-input to the filter. The only remaining issues are wrong predictions of movements through obstacles.
However, this can typically be ignored because faulty motion predictions lead to wrong observation predictions in the future (and are thus discarded in the measurement updates).
c) More complex motion models can be used as long as the motion is not affected by the state, which our implementation of the measurement update cannot handle due to our reformulation of the prediction using convolution, as mentioned in our paper. Below, we describe how we will address this issue in future work (see 2.b).
2. The scenario you are describing is in fact very similar to the one we used in the paper, and even if we make it harder still, our method can be adapted to it (a). We agree that we must, ultimately, test our approach in more realistic, complex settings.
However, we are currently addressing shortcomings of our method that are required before it can be scaled to really complex high-dimensional settings (b). Knowing that the presented work is only an intermediate step, we want to share it with the community such that other people can pick up this idea and work on it simultaneously in order to reach the goal of solving complex real world tasks.
a) In the scenario of Question 2, we would use the inertial measurements (i. e. robot velocities) instead of fan velocities as actions for the filter (as described in 1. b), which brings us to the scenario used in the paper. The existence of obstacles and the fan-velocity actions would only affect motion generation, not navigation. Now, if you remove the inertial sensor from the drone so that fan-velocities are the only action-related information that we have, then there are two options: (i) if we have knowledge about the kinematics of the drone, we can define a suitably structured motion model or (ii) if we don't have this knowledge, we can use a generic neural network for the motion model.
In the latter case, however, we will have to balance this loss of information, e. g. by providing more data, by adding a few labels to otherwise unsupervised data, or by adding additional information elsewhere (e. g. in the form of suitable loss functions as described in our work: "Learning State Representations with Robotic Priors").
b) The main bottlenecks of our method are that computation scales exponentially with number of state dimensions and the fact that motion must be independent of the state. We point out both issues in the paper and we are currently working on an extension of this work that addresses both of these concerns by going from the histogram filter to a particle filter. This extension will allow to use the state as input for the motion model and to represent the belief with non-uniform resolution, which are important next steps to scale to more complex settings.
How large were the LSTM baseline models? Did the authors perform hyper-parameter optimization (eg on model size & learning rate) to a reasonable degree for the baselines?
Thank you for your questions.
We used a stacked (two-layer) LSTM with 32 hidden units per layer.
In a previous workshop-version of the paper, we compared against different architectures (LSTMs and vanilla RNNs with one or two layers), see Fig. 3 in that preliminary paper (http://www.robotics.tu-berlin.de/fileadmin/fg170/Publikationen_pdf/Jonschkowski-16-IROS_WS.pdf). All baseline models performed about the same with two-layer LSTMs having a slightly better performance than the other baselines. We also performed a rough search over the number of hidden units and 32 seemed to lead to a reasonable underfitting/overfitting trade-off for the tasks and the given amounts of data. To further reduce the amount of hyper-parameter tuning, we used an adaptive learning rate method for gradient descent: ADAM with default parameters for all methods (including ours), and we performed early stopping with patience of 100 episodes to automatically adjust the complexity of the learned models to the given amounts of data.
Having said that, we did not perform an exhaustive search over all possible baseline architectures. We don't think that the qualitative results would be different. It should not be surprising that the histogram filter outperforms LSTMs in a setting with little data since there is so much information that the LSTM would have to learn from data which is hard-coded into the histogram filter (that is the very point of our paper).
But I definitely see your point and I am interested in a fair comparison. If you think that it is crucial, I could perform another search over the LSTM hyper-parameters. Which ones would you vary and in what ranges?
I have just rerun a few experiments with larger LSTMs (64 hidden units / 128 hidden units per layer) and it did not substantially affect the learning curves. I will continue investigating this and update the learning curves in the paper using the best performing baseline (64 hidden units might be a little bit better than 32, but with 128 units performance is already decreasing again). And what I did not mention before: We are using gradient clipping (clipping parameter = 5) to ensure stability for training the LSTMs.
The authors propose a time-series model with discrete states for robotics applications. I think the proposed method is too simplistic to be useful in the presented form, eg. 1) the state space (dimensionality & topology) is exactly matched to the experiments 2) displacements in the transition model are linear in the actions 3) observations are one-dimensional. This seems to be quite behind the current state of the art, eg “Embed to Control” by Watter et al 2015, where a state representation is learned directly from pixels.
Furthermore the authors do not compare to any other method except for an out-of-the-box LSTM model.
Also, I feel like there must be a lot of prior work for combining HMMs + NNs out there, I think it would be necessary for the authors to relate their work to this literature.
We think that it is difficult to compare to E2C (Watter et al. 2015) because they solve a different problem.
While E2C estimates the state from a single observation (assuming it to be Markov), we are concerned with partially observable problems where a single observation is insufficient to estimate the state---which is the case for most robotic problems. There are a number of approaches that learn state representations from pixels assuming Markov observations, including our own work (Jonschkowski and Brock, 2015), but these are unable to learn recursive state estimation loops.
Hence, this paper.
We compare to LSTMs because LSTMs are the state of the art in sequence-based learning. We are unsure which other models we should compare against. We are also not aware of prior work that combines HMM/MDP structure with neural networks that would applicable for learning state estimation in partially observable problems. Could you please provide references to these works?
Summary:
The authors propose a histogram based state representation with differentiable motion models and observation updates for state tracking from observations. Linear model with Gaussian noise is used as the motion model, while a neural network is used to learn the measurement model. They track robot states in: (1) 1-D hallway, and (2) a 2D arena.
Positives:
1. Show how to encode prior knowledge about state-transitions in the architecture.
2. No assumptions about the observation model, which is learned purely from data.
3. Better accuracy than baselines with limited training data.
Negatives:
1. The motion model is too simplistic. The authors in their response to earlier questions say that a generic feed-forward neural network could be used to model more complicated motions.
However, then the novelty of their framework is not clear -- as then the proposed model would just be a couple of neural networks to learn the motion and observation models.
2. The observation model again is too simplistic (e.g., one dimensional observations), and is proposed to be a generic feed-forward network. Here again, the technical novelty is not clear.
3. The histogram based representation is not scalable as also highlighted by the authors.
Hence, the proposed approach as it is, cannot be applied to more complicated settings.
4. In Figure 5(a,b), where they compare the state-estimation accuracy with other baselines (i.e., LSTMs), it is clear that the accuracy of the LSTM has not saturated, while that of their model has. They should do larger scale experiments with more training data (e.g., 10k,100k,500k samples).
Note that while sample efficiency is a desirable property (also discussed in Section 6.2), we do expect models with prior knowledge to work better for small number of samples than models which do not assume any structure. Experiments with larger number of samples would be insightful.
Thank you very much for your review. We would like to comment on the negative points 1-4 and clarify the contribution and the aim of our paper.
Points 1 and 2 question the novelty/contribution of the motion model and the measurement model, but neither the motion model nor the measurement model are the core contributions of our paper. The main technical contribution is the differentiable implementation of the histogram filter algorithm to enable end-to-end learning (where any model can be plugged in). The---equally important---conceptual contribution is the idea of using existing algorithms as priors in deep learning, or put differently, using the differentiable implementation of existing algorithms as a principled approach to explore new deep learning models.
The main objection in points 1-3 of this review (and the main objection of AnonReviewer3) is: *This model is too simplistic to be useful in practice.* This statement is true---for this paper and for the majority of basic research. The end-to-end histogram filter is not supposed to be the final answer to learning state estimation, it is only supposed to demonstrate an explorative step in a new direction. The results are not meant to prove practicality of the model but show that there is a gradient that might be worthwhile following. We absolutely agree that more work needs to be done to make this practical.
But only by sharing new ideas and experimental findings with the community can we enable such future work, which is why we are writing this paper.
TL;DR: We agree that the presented method is not practical yet, but we are surprised that this should be the most important measure of scientific value.
More detailed comments:
1.
Even when the motion model is implemented as a generic feed-forward network, the resulting E2E-HF still uses structural assumptions of the Bayes filter algorithm: (a) It forces the hidden activations to represent a belief over states (rather than any combinations of features over the history). (b) It assumes conditional independence of observations and actions given the state by separating the update and prediction steps. (c) It constrains the measurement update to multiplicatively influence the belief. (d) It constrains the prediction step to "move" beliefs in the state space. All of these constraints are optimal for state estimation when we consider the problem to be an MDP, a model that has proven very useful. (We will add this explanation to the paper.)
2. See above.
3.
While the histogram representation of the belief does not scale well, the aforementioned structure does. Following the conceptual approach presented in the paper, the histogram filter can be extended naturally to a particle filter (by enabling the histogram bins to move in the state space). Particle filters have been applied to very high-dimensional problems (e.g. SLAM). This extension of the presented work has the potential to scale to highly complex settings.
4. This is an excellent suggestion. We will perform additional experiments to find out if/with how much data LSTMs start outperforming E2E-HFs.
Summary: This paper presents a differentiable histogram filter for state estimation/tracking. The proposed histogram filter is a particular Bayesian filter that represents the discretized states using beliefs. The prediction step is parameterized by a locally linear and translation-invariant motion model while the measurement model is represented by a multi-layered neural network. The whole system is learned with both supervised and unsupervised objectives and experiments are carried out on two synthetic robot localization tasks (1D and 2D). The major claim of this paper is that the problem-specific model structure (Bayesian filter for state estimation) should improve pure deep learning approach in data-efficiency and generalization ability.
+This paper has nice arguments about the importance of prior knowledge to deep learning approach for specific tasks.
+An end-to-end histogram filter is derived for state estimation and unsupervised learning is possible in this model.
-This paper seems to have a hidden assumption that deep learning (RNN) is a natural choice for recursive state estimation and the rest of paper is built upon this assumption including LSTM baselines.
However, this assumption itself may not be true, because Bayesian filter is a first-established approach for this classic problem, so it it more important to justify if deep learning is even necessary for solving the tasks presented. This requests pure Bayesian filter baselines in the experiments.
-The derived histogram filter seems to be particularly designed for discretized state space. It is not clear how well it can be generalized to continuous state space using the notation "x". More interestingly, the observation is discrete (binary) as well, which eventually makes it possible to derive a closed-form measurement update model. This setup might be too constrained. Generalizing to continuous observations is not a trivial task, not even to mention using images as observations like Haarnoja et al 2016. These design choices overall narrow down the scope of applicability.
Thank you very much for your review. We would like to comment on the two negative points.
1) The paper does not assume that deep learning is the natural choice for learning state estimation. It assumes that deep learning with generic networks marks one end of a spectrum of approaches while Bayes filters mark the other end of this spectrum. Our work demonstrates by example how to construct intermediate approaches that balance the benefits of both extremes.
All our experiments do have a "pure Bayesian filter baseline", a pure histogram filter (HF), whose models are fit to the data of the task to have a fair comparison (these models are virtually identical to the "correct" models for the given tasks).
2) We would like to clarify that the state space in the task is continuous in all our experiments.
As pointed out in the paper, the discretization employed by the filter is a bottle neck, as is the use of discrete (although not necessarily binary) observations. We are currently working on both issues using different representations of the belief.
We are retracting our paper "End-to-End Learnable Histogram Filters" from ICLR to submit a revised version to another venue.

Meta-review:
role: PC
data: comment: In many respects, this is a strong paper, in my opinion better than the reviews thus far in the system suggest. The idea of learning the parameters of a state estimation system, even if it is a simple example like a histogram filter, is an interesting idea from both the ML and robotics perspectives.
 
 However, I think it's also fairly clear (from the reviews if nothing else), that substantial additional work needs to be done if this paper is to convey these ideas clearly and impactfully to the ICLR community. The value of the histogram filter may be obvious to the robotics community (though even there it seems like it would be worth pursuing the extension of these ideas to the case e.g. of particle filters), as these have historically been an important conceptual milestone in robotic perception, but the examples shown in this paper are extremely simplistic. Given that histogram filters inherently scale poorly with dimension, it's not clear from this paper itself why the techniques show particular promise for scaling up to realistic domains in the future.
 
 Pros:
 + Interesting idea of using training a state estimator end-to-end for robotics tasks
 
 Cons:
 - Histogram filters, while well-motivated historically from the robotics standpoint, are really toy examples at this point except in a very small number of settings.
 - The results aren't all that compelling, showing modest improvement (over seemingly not-very-tuned alternatively approaches), on fairly simple domains.

Decision: Reject