Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper develops a soft finger and leverages learning to estimate force/displacement from visual observation of deformations during interaction with external objects.
strengths_and_weaknesses: Strengths:
1. The paper is leveraging learning to enable force estimation from deformations of a 3d printed finger leveraging a camera. This could enable low cost fingertips for use in the industry where the fingertips can be swapped out with minimal cost. 
2. Designing these fingertips to have 6d force/torque estimation is very interesting and leveraging learning to go around modeling the deformations has great benefits as shown by the paper.
3. The teleop application shown with the proposed method is very cool and shows a direct application of the approach.


Weakness:
1. The paper learns a mapping from finger deformation to 6D force/torque. However the accuracy of the predictions is not validated. Plotting the error in predictions both in magnitude and direction would be helpful. Previous force estimation methods have reported errors on test dataset to help compare across methods [biotac_force].
2. Does the learned model transfer to a new sensor of the same design? Discussing this question with some experiments is needed. 
3. The paper does not cite any existing work on tactile force estimation. Specifically, the idea of collecting data leveraging robots and learning a NN is not novel, it has been used to estimate force and deformation fields in previous works[biotac_force, biotac_physics].
4. Why are the predictions very noisy? Is there any filtering that is done post prediction to enable smooth teleop? 
5. The paper doesn’t show what happens when force is applied in non-normal directions, i.e. between parallel (shear) and normal angles. How does the accuracy change when force is applied in these directions? While estimating shear force is not necessary, having accurate force angle prediction is important for many manipulation tasks[biotac_directionality].  Showing some plots when force is applied in these directions is required to better understand the limitations of the work.
6. Can the force estimated be used for manipulation tasks such as gentle object placement? [biotac_force]


References:

[biotac_force] Sundaralingam, Balakumar, Alexander Sasha Lambert, Ankur Handa, Byron Boots, Tucker Hermans, Stan Birchfield, Nathan Ratliff, and Dieter Fox. "Robust learning of tactile force estimation through robot interaction." In 2019 International Conference on Robotics and Automation (ICRA), pp. 9035-9042. IEEE, 2019.

[biotac_physics] Narang, Yashraj S., Karl Van Wyk, Arsalan Mousavian, and Dieter Fox. "Interpreting and Predicting Tactile Signals via a Physics-Based and Data-Driven Framework."

[biotac_directionality] Gutierrez K, Santos VJ. Perception of Tactile Directionality via Artificial Fingerpad Deformation and Convolutional Neural Networks. IEEE Trans Haptics. 2020 Oct-Dec;13(4):831-839. doi: 10.1109/TOH.2020.2975555. Epub 2020 Dec 25. PMID: 32092014.

summary_of_recommendation: The paper does not introduce any novel learning/data collection methods. But leverages learning to enable force estimation from hard to model deformations of a fingertip. There could be several applications of the proposed solution. Addressing the following is required to improve the recommendation: 
1. Reporting magnitude and directional accuracy.
2. Analyzing transfer across other fingertips of the same design.
3. Analyzing accuracy of force estimation in non-normal directions.
4. Showing a force controlled task such as grasped object placement based on detection of object contact with table.
issues: Issues are listed in recommendation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper explored the use of soft material for robot force sensing and control. The main contribution is the introduction and analysis of a cone-shaped metamaterial, and two vision-based approaches to infer the applied force on the material based on deformation. The proposed system is evaluated on a soft robot force control task.

strengths_and_weaknesses: Strengths:
1. The idea of leveraging soft materials and computer vision for robot force control is interesting and reasonable.
2. The proposed metamaterial structure is concrete and carefully analyzed.
3. Two different methods were evaluated to learn the force profile of the soft material under different forces and are both shown to be effective, though one has some generalization issue.
4. Preliminary real-world experiments were conducted to demonstrate the usefulness of the proposed idea.

Weaknesses:
1. Limitations of the proposed method are not well discussed. For example, what’s the highest frequency that the proposed system can run at? A force-torque sensor can usually run at very high frequency but a camera may have limited frequency. Another potential limitation is that the system would be more sensitive to lighting conditions and also introduce a higher cost of computation in exchange for lower cost in hardware. Some discussion along these lines would be helpful.
2. The proposed system didn’t demonstrate superior task performance in the evaluated task: in Figure 6, it seems that with force-torque sensors the resulting trajectory is smoother.
3. For the AR-tag approach, is it able to infer pulling or push forces on the material? For example if the robot hits a wall vertically, it would be important to properly detect that.

summary_of_recommendation: I think the idea presented in this paper is interesting and promising in my knowledge. The design of the meta-material and the presented learning approach to infer the force and torque readings are solid. On the other hand, some more discussions are needed and a better experiment setup would make the paper even stronger and more interesting. For example, if one can build a two-finger gripper out of the proposed method, it’ll be able to sense forces per-finger. This is something difficult to achieve with existing methods and can potentially improve manipulation performance, which would better support the value of the paper.
issues: Please see weaknesses.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper applies a deep learning approach utilizing convolutional neural networks to the problem of extracting torque measurements from a soft structure. This structure, having a pyramid-like form, is meant to be attached to the end-effector of a robot. Images and videos show an example where it is mounted on a UR 10 robot arm. The general idea is to record camera images of the internal deformation of the structure from its base and then using a supervised learning approach to learn a neural network predicting the external force. The motivation for this is the idea that the structure can be used as a 'soft finger' on a robot to gently guide it for demonstrating/recording movement trajectories.
strengths_and_weaknesses: Although not entirely novel (see below), the idea of extracting force measurements from visual observations is a good line of enquiry for this paper. Furthermore, the general setup of the experimental testbed seems rigorous. The video is of very good quality.

However, I have major concerns regarding the general learning setup and its motivation as it is described in the paper. 
- The general learning problem is not very well formalized: What exactly is the learning problem the paper tries to tackle? Just as an example, the paper does not mention that the learning setup is supervised at least once. The greater question is, why is the learning problem supervised? Why not unsupervised or using reinforcement learning? Furthermore, the ATI Nano25 sensor appears smaller than the camera box at the bottom of the finger. Why not simply attach the ATINano25 sensor instead of a camera? What extra information would the camera provide, and why would we be unable to use an F/T sensor at the base?
- The estimated force/torque data is utilized to be fed back into the controller of the UR10 in task-space, controlling the position of its end-effector. Why can this not be solved with said additional force sensor at the base of the finger or a more sensitive (robot) controller?  This setup would not require deep learning at all.
- Why is deep learning the algorithm of choice? Why not use the information from the tag extracted with OpenCV to train an SVM or another regression algorithm? Figure 5 shows only the predictions of the used neural network, some other learning algorithm as baseline would be expected. Or an explanation of why other algorithms are not applicable.
- I cannot entirely agree with the conclusion that the proposed setup would allow other researchers a more versatile or low-cost setup: Assuming reasonable variance in manufacturing or 3D-printing the proposed structure, every lab would have to carry out the calibration method described, i.e. collecting a supervised data set with force measurements and images showing the deformation. This process would also be required to be repeated every time a new finger/structure is produced. 
- The paper introduced the soft structure as a 3D metamaterial. As the introduction of the paper already mentions, a meta-material is usually a composite material, i.e. a material made up of at least two or more materials. It is not clear if this is the case here, or if simply the structure of the elastic material (in the form of a pyramide like tubing) determines alone its properties. 
- For me, the results presented in Figure 6 (b) (UR 10 force sensor) look considerably more natural and better than (d) (proposed camera-based approach).

A more interesting approach would have been to learn the required force/torque estimated in a reinforcement learning setup, in the absence of any additional sensory equipment, by letting the robot arm interact in some fashion with the environment. This would also strengthen the applicability of the paper for a robot learning conference.
summary_of_recommendation: From the point of robot learning, it is not clear what the exact contribution of the paper is. The described learning setup is rather weak, and the principal problem could be solved by exchanging the camera with a force/torque sensor and a more sensitive robot control algorithm, which would not require learning at all. Furthermore, the approach is limited to setups where it is possible to acquire a supervised data set by the use of force/torque sensors. A more exciting approach could be an autonomous learning process in which the robot arm learns the direct mapping from captured structural deformation to motor commands. This would also allow the operator to apply the proposed finger structure to any robot, not just ones with accurate kinematic modelling in the absence of any force/torque sensors.
issues: - Figure 5: The graphs should show a grid-pattern to allow the viewer to evaluate the deviation between predicted and actual force. Furthermore, both x- and y-axis should have the same range.
- Section 3.2 should be restructured and introduce the exact learning problem in a formal way. 
- Explain what the exact problem the paper is trying to solve is. Why do we need deep learning for it? How can it be made applicable to the outside of your lab and to other researchers who might want to use the proposed finger structure but do not have access to the testbed used to collect the supervised data?
- Is the proposed approach using CNN networks really the only approach capable of solving this problem? For example, how does the proposed algorithm using a camera compares against directly attaching a force/torque sensor to the base of the finger structure?
- Fig. 6 should show in (a-d) the reference trajectory to follow.
- Lastly, why use a supervised learning approach + a controller in task-space? Why not directly learn the motor commands given the deformation in the finger structure? 
- The paper could discuss further camera-based approaches for manipulation such as "OmniTact: A Multi-Directional High-Resolution Touch Sensor"
- The paper would also benefit from a better discussion about the potential applications of this structure, especially regarding the prospect of using it for human demonstrations in the robot learning community.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper explores the feasibility of learning robot force control and interaction using soft metamaterial and machine vision.

The paper explores two methods: directly learning the force and torque response from raw images of the metamaterial’s internal deformation, and indirectly estimating the 6D force and torque using a neural network by visually tracking the 6D pose of a marker fixed inside the 3D metamaterial. The paper finally integrates the two methods and achieves similar force feedback and control interactions in simple tasks.

The results show that the learning method holds the potential to support the concept of soft robot force control, providing an intuitive interface at a low cost for robotic systems, generating comparable and capable performances against classical force and torque sensors.
strengths_and_weaknesses: I have no knowledge in this domain before. I cannot evaluate the novelty and technical contribution of the paper correctly. I only make sure the claims in the paper are correct.

Strength:

The paper has several contributions: 
-- The investigation of a soft, 3D metamaterial with differential stiffness for passive form adaptation in omni-directions in the radial plane;
-- A direct vision-based sensory integration of the 3D metamaterial for estimating the force and torque using a modified resnet;
-- The design integration of a marker inside the omni-adaptive metamaterial to facilitate efficient, accurate, and real-time 6D force and torque using vision-based pose tracking;
-- System integration towards soft robot force control with comparable performance against commercial force and torque sensors with much-reduced hardware cost and complexity.

All the contributions are clearly stated and empirically demonstrated in the paper.

Weakness:

I do not see explicit weakness in the paper.
summary_of_recommendation: I cannot evaluate the paper correctly so I give a rough evaluation of the paper. I will discuss with other reviewer about the final recommendation for this paper.
issues: No issue.
review_title: null

Meta-review: The paper proposes achieving soft robot force control using a 3D flexible structure and learning estimates of force-torques by using vision, potentially decreasing the cost of robotic systems. In general, the paper has been considered of having a good impact, originality, and technical quality.

Amongst others, reviewers have raised issues regarding deficiencies in the related work, confusion concerning effectiveness in the estimation of forces in certain directions (vertically hitting a wall), and whether the performance against the conventional force-torque sensor is truly comparable. Additionally, it seems that a bit more motivation and more details for the CNN have been missed. The authors are encouraged to discuss the raised points with the reviewers during the rebuttal.

Post rebuttal ==============

The paper has been gone through extensive discussions between authors and reviewers. The overall scores have been increased during the rebuttal as the authors clarified most of the concerns. As a final result, we observe an improved final manuscript.


Decision: Accept (Poster)