Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a deep learning based approach for estimating contact information from proprioceptive sensory data. First, the authors collected the training data by deploying locomotion controllers on various terrains. The ground-truth data is estimated with the normal component of foot positions in the body frame (line 206-207) or captured with a motion capture system (line 207-208). Then the paper learns neural networks that consist of several convolutional and fully connected layers. Once the contact states are estimated, it can be fed into Contact-Aided Invariant Extended Kalman Filtering to estimate the global location. The authors have demonstrated that the proposed method can estimate the contact status at the accuracy of 80%~95%. And they also have demonstrated that the estimated contacts are helpful for localization.
strengths_and_weaknesses: + The paper applies a machine learning technique for state estimation.
+ The proposed method is validated in two folds: local level (contact state estimation) and global level (path reconstruction).
- The ground-truth data needs some clarification
- The learned model may be overfitted to the robot and controller.

summary_of_recommendation: I am positive about this paper, but I would save my full support because of some drawbacks. I would like to see the authors’ responses before making the final decision. Until then, my support is lukewarm.

The first confusion is about the ground-truth contact information. The paper seems to say that they are *estimated* from the foot height in the body frame (except for the grass scenario). Then why do we need to estimate it using machine learning? We can simply apply the same procedure to estimate the ground-truth contact information from the foot height. This issue is particularly odd because this can nullify all the motivation of the paper. And this is also connected to the evaluation. The results presented in Table 3 are also compared against the *estimated* ground-truth? If yes, I don’t think it is a fair comparison.

My other concern is about overfitting to the motion. The given dataset seems to be generated with a fixed controller (model-based controller? deep RL controller? maybe better to describe more details). However, we are not sure whether the proposed method can be generalized to different motions or gaits . For instance, can we use the same model for bounding gaits, or manually created jumping motions? I would suggest the authors discuss this generalization issue in the text.

As stated in the paper, I am not sure we can easily assume perfect contacts without foot slips. I guess it happens quite often during data collection, particularly on certain terrains such as pebbles or grass. This would limit the impact of the proposed work. But it seems not clear what the authors can do further on top of the current discussion in Section 6.

I am not too familiar with the Contact-Aided Invariant Extended Kalman Filter technique, but the validation with this seems reasonable. It will be better if Figure 4 illustrates the locations where the model fails to predict. 

issues: My major concerns are 1) data collection and 2) overfitting. Particularly, the data collection issue can make the entire work significantly less impactful. Hope to get some clarification during the rebuttal phase.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a supervised deep learning method for contact detection for quadrupedal robots, a critical subroutine for many state estimation and control methods for legged robots. The authors integrate their learned detector in one such state estimator, which is shown to perform nearly as well as ground-truth contact state.
strengths_and_weaknesses: Strengths:
1) The performance of the method is particularly impressive given the limited sensing modalities (no contact sensors or parallel/series spring deflections) that it leverages.
2) Not only was the detection approach clearly explained, but also the authors provided a great high-level summary of the invariant EKF used in the paper.

Resolved Weaknesses:
1) The authors do not provide a performance comparison to any of the other cited contact detection methods.

Weaknesses:
1) A critically important characteristic of any state estimator is that it does not excite instability when used in real-time as the input to a controller. This may be of particular issue in this approach if the associated controller requires the contact state. However, the authors only use the method to reconstruct trajectories offline, which does not provide insight into in-the-loop performance.


summary_of_recommendation: The authors propose a great concept for learned contact estimation in the absence of contact sensing. However, without in-the-loop experiments or a comparative study, it is difficult to assess the suitability or performance of the method for state estimation. Furthermore, the reason for needing the learned module for localization and mapping tasks is not sufficiently motivated in the text. Thus the paper does not seem ready for publication.

issues: Resolved Major Issues:
1) Authors should either conduct a comparative study with other contact detection methods, or convincingly argue why the ones discussed in the paper are not suitable for comparison.

Remaining Major Issues:
2) Authors should conduct a control experiment with the contact detector and state estimator in the control loop.


Nitpick:

There appears to be some issue in the calculation of RMSE of the relative pose error. In the Appendix, the “Sidewalk+Asphalt Road” example is claimed to only have 3 [cm] of error per frame, yet the provided figure (Fig. 4) seems to imply that the estimation is off by nearly 20 [m] by the end of the trajectory.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper addresses contact estimation of a quadruped robot that is not equipped with force/torque sensors on its feet. It fuses the inertial and kinematic motion data of the robot over a time window by a deep convolutional neural network, which outputs a set of binary values (contact/non-contact) with respect to each foot. The training data was collected in several outdoor environments with a help of a motion capture system, and is provided as Open-source quadruped contact data sets recorded using an MIT Mini Cheetah. Using the proposed method, the invariant extended Kalman filter for localization was built and evaluated. It showed a competitive performance with ORB SLAM2.

strengths_and_weaknesses: It is remarkable that the same-level accuracy with the state-of-the-art SLAM technique was achieved only with proprioceptive information even in challenging environments. It certainly shows the efficacy of the proposed method.
On the other hand, it seems to be arguable that the method requires huge amount of data recorded over many frames only for detecting if feet are in contact or not. Since the computation resource that a robot can carry is still limited, the availability of the method is a bit questionable for the reviewer. A question which naturally arises in him/her is if it is possible to estimate the position and orientation of the robot without explicitly estimating the contact state. This idea seems preferable as it exploits the computation resource more than the contact detection and might be a shortcut for the purpose. It is probably related with the "quality" of contacts as the authors note.

The reviewer's bigger question is about the generality of the detector, namely, whether the trained detector works for robots other than the MIT Cheetah. Although he/she guesses it does to some extent at least for quadruped robots, he/she wants to know if it is invariant with respect to size, weights and body proportions.

summary_of_recommendation: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
issues: To investigate if the trained contact detector works for robots other than the MIT Cheetah by using, for example, a physics simulator, if possible.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper addresses the problem of legged robot state estimation. Since joint angles can be directly measured using encoders, the central challenge of this problem has been the *global* pose and velocity of the robot (root). Previous work “InEKF” [5] built an extended Kalman Filter incorporating IMU and binary contact sensor measurements to address this problem, utilizing insights such as stance leg being static during contact. And this paper adopts the same method.

The contribution of this paper, is that, since some robots do not have binary contact sensors as the Cassie robot in [5], a neural-network contact classifier is therefore built to predict this info. As standard network architecture and training procedure seem to be good enough for this purpose, the key challenge here is thus how to obtain ground truth contact label in the training data.

For contact labeling, the paper mostly identify contact points by calculating minima height values of feet from motion cycles kinematics. In addition, a bit of manual data cleaning is also needed. (L208)
strengths_and_weaknesses: Strength:
The idea of generating contact labels offline when we have future info of the trajectory and use them to train an online network for contact prediction is interesting.

Weakness:
The main weakness, in my opinion, is also about the contact prediction network and whether we really need that. Since the Ground Truth contact labels are basically generated by calculating the lowest point of each gait cycle from joint kinematics (L208-209), what the network essentially does is to memorize the kinematics of its own gait.

This raises the immediate question of can we use some simple heuristic rule to replace the network? After all the network will not work when the gait pattern changes, unless we collect new data using the new controller (which seems not easy). But since that would imply we roughly already know the gait pattern beforehand, in my mind heuristic rules would be very easy to implement. 

Minor points:
The introduction can be improved by emphasizing what is hard about state estimation in the beginning. I only realized the key is root trajectory estimation (not estimating all joint q’s, they are given) when I skimmed the paper and saw figure 4. Previous work [5] did a better job in clearing that out at first.

The whole section 3 seems to come from [5] — the only change seems to be different robot morphology. It can be made more concise as well.
summary_of_recommendation: Building such a contact prediction network does not seem difficult if we went thought the data collection. But, as discussed before, I am not sure if machine learning is justified here if we already have the controller ready, and I would love the authors to take some serious re-thinking or let me know what I am missing fundamentally. Again, we should keep in mind that training such a network requires us to already have a working controller, and neural networks in general would not "extrapolate" to other motions/controllers without special treatment.
issues: Discussed above -- why learning, as well as using gait kinematics to generate ground-truth labels, is justified when we already know the gait pattern.
review_title: null

Meta-review: The authors consider a deep-learning based approach to estimate contacts for legged robots.  Physical contact sensors on legged robots typically fail due to repeated impacts and the proposed contact estimator using multi-modal proprioceptive sensory data is important and critically needed.  The authors should clarify how the ground-truth contact information is obtained - if no additional sensors were used, then why use this learning-based approach?  The authors should also discuss what fraction of their data has feet slipping.  Please provide more information on the computation time / induced delays in using this estimator.  This will be important in using the estimator stably in a feedback loop.  Finally, any thoughts on what part of the data is robot specific? Can the collected data be used to estimate contact on other legged robots?

I have taken the author responses and discussion with the reviewers into consideration. I commend the authors on making a sincere attempt to address all the reviewer comments and feedback.  I think accurate contact detection is a critical component of legged robot research.  The authors provide information in Table in Sec. IV showing the improved performance using their method.  I am recommending an acceptance with poster.

It will be interesting to find out ratio of false positives vs false negatives - I presume one of this is more important for control.
Decision: Accept (Poster)