Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a novel approach for learning visual servoing based on Q-iteration. The main contributions of the paper are:

1. Bilinear dynamics model for predicting next frame (features) based on action and current frame
2. Formulation of servoing with a Q-function that learns weights for different feature channels
3. An elegant method for optimizing the Bellman error to learn the Q-function

Pros:
+ The paper does a good job of exploring different ways to connect the action (u_t) and frame representation (y_t) to predict next frame features (y_{t+1}). They argue in favour of a locally connected bilinear model which strikes the balance between computation and expressive ability. 

Cons:
- While, sec. 4 makes good arguments for different choices, I would have liked to see more experimental results comparing the 3 approaches: fully connected, convolutional and locally connected dynamics.

Pros: 
+ The idea of weighting different channels to capture the importance of obejcts in different channels seems more effective than treating errors across all channels equally. This is also validated experimentally, where unweighted performance suffers consistently.
+ Solving the Bellman error is a difficult problem in Q-learning approaches. The current paper presents a solid optimization scheme based on the key-observation that scaling Q-function parameters does not affect the best policy chosen. This enables a more elegant FQI approach as opposed to typical optimization schemes which (c_t + \gamma min_u Q_{t+1}) fixed. 

Cons:
- However, I would have liked to see the difference between FQI and such an iterative approach which holds the second term in Eq. 5 fixed.

Experimental results:
- Overall, I find the experimental results unsatisfying given the small scale and toy simulations. However, the lack of benchmarks in this domain needs to be recognized.
- Also, as pointed out in pre-review section, the idea of modifying the VGG needs to be experimentally validated. In its current form, it is not clear whether the modified VGG would perform better than the original version.

Overall, the contribution of the paper is solid in terms of technical novelty and problem formulations. However, the paper could use stronger experiments as suggested to earlier to bolster its claims.

review_title: Good approach for visual servoing, but could use better experimentation

final_score: rating: null
review_content: 1. The VGG used in the paper retains the pre-trained weights of VGG learned from imagenet but modifies the architecture to remove max-pooling and replaces conv with dilated conv. Since the conv layers in original VGG were trained with pooled input from preceding layers, I am not sure that they would transfer well to the new setting. Have you tried retaining the original VGG with 224 x 224 input, max-pooling and 1-dilated conv?

2. Given that the original VGG was intended for object classification, wouldn't it be better to fine-tune the entire network with the new multi-scale loss for predicting features in the next time-step. This will no longer be a ridge-regression problem as stated in Sec.4.3. However, it will ensure that the network is adapted to the new setting.

3. Is Eq. 6 in FQI similar to line-search for temporal difference updates? Is it significantly better than simple temporal difference updates with fixed step size?

4. The experimental setup seems fairly limited: the synthetic car examples in the test set are not visually different from those in the training set. The authors claim that one of the contributions of the paper is to develop a method which can handle "extensive visual variation", "moving objects, occlusions and other complex phenomena". Are there better benchmarks for visual servoing which reflect these challenges?
review_title: Clarifications on modified VGG used for feature dynamics

final_score: rating: null
review_content: Following the ICLR reviewing guidelines, here are a few short pre-review questions:

1) Why is visual tracking not considered as related work or baseline? This is one of the core problems in computer vision, straightforward to apply to the task of servoing (which is much simpler if you know where the object is located), and has a vast literature tackling the challenges mentioned by the authors (and more). It seems to me hard to defend completely ignoring this large part of the literature, and not comparing to, say, the top trackers from the VOT challenge (http://votchallenge.net). Note that although some of them use handcrafted features, many use deep learning and have very fast open source implementations.

2) Have you compared to trajectory prediction methods? Instead of modeling visual feature dynamics, directly modeling the actual dynamics is a standard approach. Furthermore, it does not have the problems that the authors have to work around (too strong invariance of pre-trained deep features, high dimensionality, etc). Beyond the motivation in the introduction, this is needed, in my humble opinion, to quantitatively justify the need for all these different/additional difficulties and their associated necessary approximations.

3) Have you tried using hand-crafted features (e.g., SIFT, SURF, HOG) as a baseline? It is much better than pixels, and often better than pre-trained (or even fine-tuned) deep features for instance-level modeling tasks like tracking or matching, because image-level classification deep networks are (by design) too invariant (not just because of max pooling). Several papers in retrieval or matching have made this point already, and this is confirmed by your experiments that require selecting the appropriate feature layers depending on the test scenarios. Adding a SIFT or HOG baseline is very simple to do (cf. official examples part of OpenCV), and in my humble opinion, mandatory to substantiate the claim made by the authors that they outperform hand-crafted features (raw pixels are not hand-crafted features).

4) Have you run any real-world experiments? The toy synthetic experiments seem overly simplistic to me in terms of visual challenges (appearance variation, occlusions, clutter, lighting conditions, variety, etc). In addition, I think they do not allow to explore the main challenge of visual servoing that the authors mention: the fact that different objects might be very close in appearance. This is very often the case in practice (e.g., for pedestrians or cars), and one of the main focus of research on tracking (especially multi-object tracking, cf. the KITTI or MOT benchmarks). Is there no existing real-world benchmark for visual servoing? One can also imagine using real-world videos (e.g., from an aerial drone, several such public datasets exist) and simulated realistic dynamics coupled (for instance) with an artificially restricted field-of-view.

5) Can you clarify a bit what is the main novelty and contribution? Using pre-trained VGG features is not a significant one in my opinion (especially in light of the aforementioned points), and the most interesting part of the paper (learning the weighting of the features) is only very succinctly described in 5.3.
review_title: Pre-review questions

final_score: rating: null
review_content: 1. In order to avoid the need for "large amounts of experience", you propose to learn a predictive model. To my knowledge, there is still no consensus in RL on whether model-based  is faster than model-free in general. I would like if you could substantiate your point. Is it that robotics domains admit models which are "nicer" to learn (smooth dynamics due to structured physics and so on) and thus lead to sample efficient model-based algorithms ?

Along with the final scores in table 1, I think that it would be great if you could report on the sample efficiency. 

2. The way that you phrase your problem on page 3 is rather interesting "[...] since it weights the errors of distractors objects [...]". Have you considered posing your problem as an "attention mechanism" in the same vein as the recent wave of attention-based DL methods ? To which extent could the existing attention methods be used for your problem ? And from the opposite: could you use your approach to devise better attention models in classical supervised learning ? 

3.1 [Problem statement] I find this aspect confusing: "distance metric that encodes the goal for particular task". What you might be trying to say is that by minimizing (1) (which I assume is the L2 in feature space) your policy is implicitly learning a mapping from its action space to a representation in which the goal can easily be expressed. In general, I would find useful if you could be more explicit in your problem formulation: what is the action space precisely, what would the cost function be in general, do you assume an MDP ? 

3.2 I'm not sure why the optimization step described in (6) is useful. Also why is the update $\alpha^(k - \frac{1}{2}) \theta^{k-1}$ a valid minimization step ? Is it some kind of line search ? 

review_title: Sample complexity, relation to attention, formulation

final_score: rating: 7: Good paper, accept
review_content: 1) Summary

This paper proposes to tackle visual servoing (specifically target following) using spatial feature maps from convolutional networks pre-trained on general image classification tasks. The authors combine bilinear models of one-step dynamics of visual feature maps at multiple scales with a reinforcement learning algorithm to learn a servoing policy. This policy is learned by minimizing a regularized weighted average of distances to features predicted by the aforementioned model of visual dynamics.

2) Contributions

+ Controlled experiments in simulation quantifying the usefulness of pre-trained deep features for visual servoing.
+ Clear performance benefits with respect to many sensible baselines, including ones using ground truth bounding boxes.
+ Principled learning of multi-scale visual feature weights with an efficient trust-region fitted Q-iteration algorithm to handle the problem of distractors.
+ Good sample efficiency thanks to the choice of Q-function approximator and the model-based one-step visual feature dynamics.
+ Open source virtual city environment to benchmark visual servoing.

3) Suggestions for improvement

- More complex benchmark:
Although the environment is not just a toy synthetic one, the experiments would benefit greatly from more complex visual conditions (clutter, distractors, appearance and motion variety, environment richness and diversity, etc). At least, the realism and diversity of object appearances could be vastly improved by using a larger number of 3D car models, including more realistic and diverse ones that can be obtained from Google SketchUp for instance, and populating the environment with more distractor cars (in traffic or parked). This is important as the main desired quality of the approach is robustness to visual variations.

- End-to-end and representation learning:
Although the improvements are already significant in the current synthetic experiments, it would be interesting to measure the impact of end-to-end training (i.e. also fine-tuning the convnet), as it is possibly needed for better generalization in more challenging visual conditions. It would also allow to measure the benefit of deep representation learning for visual servoing, which would be relevant to ICLR (there is no representation learning so far, although the method can be straightforwardly adapted as the authors mention briefly).

- Reproducibility:
The formalism and algorithms are clearly explained, but there is a slightly overwhelming mass of practical tricks and implementation details described with varying levels of details throughout the paper and appendix. Grouping, simplifying, or reorganizing the exposition of these implementation details would help, but a better way would probably consist in only summarizing the most important ones in section and link to an open source implementation of the method for completeness.

- Typos:
p.2: "learning is a relative[ly] recent addition"
p.2: "be applied [to] directly learn"

4) Conclusion

In spite of the aforementioned limits of the experiments, this paper is interesting and solid, in part thanks to the excellent reply to the pre-review questions and the subsequent improved revision. This leads me to believe the authors are more than capable of following to a significant extent the aforementioned suggestions for improvement, thus leading to an even better paper.
review_title: Interesting reinforcement learning paper needing maybe a little bit more work on improving the benchmark and exposition

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper investigates the benefits of visual servoing using a learned
visual representation. The authors  propose to first learn an action-conditional
bilinear model of the visual features (obtained from a pre-trained VGG net) from
which a policy can be derived using a linearization of the dynamics. A multi-scale,
multi-channel and locally-connected variant of the bilinear model is presented.
Since the bilinear model only predicts the dynamics one step ahead, the paper
proposes a weighted objective which incorporates the long-term values of the
current policy. The evaluation problem is addressed using a fitted-value approach.

The paper is well written, mathematically solid, and conceptually exhaustive.
The experiments also demonstrate the benefits of using a value-weighted objective
and is an important contribution of this paper. This paper also seems to be the
first to outline a trust-region fitted-q iteration algorithm. The use of
pre-trained visual features is also shown to help, empirically, for generalization.

Overall, I recommend this paper as it would benefit many researchers in robotics.
However, in the context of this conference, I find the contribution specifically on
the "representation" problem to be limited. It shows that a pre-trained VGG
representation is useful, but does not consider learning it end-to-end. This is not
to say that it should be end-to-end, but proportionally speaking, the paper
spends more time on the control problem than the representation learning one.
Also, the policy representation is fixed and the values are approximated
in linear form using problem-specific features. This doesn't make the paper
less valuable, but perhaps less aligned with what I think ICLR should be about.

review_title: Excellent technical paper, but emphasis on control rather than representation

Meta-review: All reviewers viewed the paper favourably, although there were some common criticisms. In particular, the demonstration would be more convincing on a more difficult task, and this seems like an intermediate step on the way to an end-to-end solution. There were also questions of being able to reproduce the results. I would strongly recommend that the authors take this suggestions into account.
Decision: Accept (Poster)