Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for generating tactile data for objects using a conditional GAN, with training data coming from NeRF trained on  comparatively small amounts of RGB-D data but separately for each object. The motivation comes from the fact that tactile data from sensors is hard to collect and generalize across hardwares. Related approaches use GANs directly from visual data, but this requires a lot of images. 

Approach: NeRF is trained separately for each object, then used to generate various RGB-D images. The cGAN is trained on a set of NeRF-generated RGB-D + ground truth tactile images, then is used to generate tactile images from the NeRF-generated RGB-D (in turn generated from a desired pose) and a predetermined background image.

On holdout object experiments, the proposed approach reaches seemingly high SSIM wrt ground truth. MSE is more varied, but low for novel views at least. Using this method for fine-tuning shows some performance improvement as well. 
strengths_and_weaknesses: **Strengths**
- Paper is exceptionally clear and well-written. An interesting but easy read, which is ideal. Congrats to the authors on a very well-presented work!
- The approach is intuitive and has a clear value proposition. The advantages with respect to training data efficiency are evident, and the paper as a whole is well-motivated given both the baseline approach of actual tactile sensors as well as more related approaches using cGANs.
- While the experimental section isn't particularly thorough, it's well designed. It's good to see holdout experiments and comparison between novel view and novel object

**Weaknesses**
- There doesn't seem to be any baseline in this paper. I realize that the value isn't exactly degree of improvement on some metric, but this is ultimately a new approach to an existing problem that should make its value evident. This can include even a brief analysis on comparison of data efficiency, training time, and generality between many single-object NeRFs vs. the prior cGAN work. Even if performance from other methods is better, an argument can be made for why this one is worthwhile/worth iterating on
- Related to this, we are essentially viewing the results in a vacuum. This makes it difficult to judge whether a user should try this method
- Figure 3 is helpful, but ultimately readers without a lot of experience with tactile methods won't know whether the tactile comparison is particularly good or not. Perhaps this paper isn't for such readers, but the venue certainly is. The fake images look promising but with room for improvement compared to the real ones, but ultimately I don't know how to judge whether this is a step forward. It's even more unclear given that there may be increased data efficiency and increased compute requirements to consider.
- This has no bearing on my score, but I find the title somewhat off-putting even if it's a good pun.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak accept because I don't see any major flaws in this paper and it seems useful, but I also am not convinced of its significance - not because I have a reason to believe it isn't significant, but because 1) the results aren't deeply surprising and 2) I have nothing to compare to. Someone with more domain knowledge will be able to provide a better assessment. 
issues: - Lack of baselines, comparisons, or analysis of value/potential impact
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a framework for generating synthetic tactile data, since collecting data on a physical system is tedious and simulators lack of accurate models. The method leverages a trained NeRF model to generate novel views of a scene, and a cGAN to generate the tactile data given a rendered NeRF view. To evaluate the quality of the synthetic tactile images they use SSIM and MSE, and use the generated data to train a classifier on a physical robot.
strengths_and_weaknesses: Strengths:
1) The paper is clear, easy to follow and the problem is well motivated.  

2) This paper addresses an important and challenging problem in a practical way, offering a clever solution. It proposes a way to scale data collection for tactile sensors without modelling the sensor directly, potentially saving time and effort compared to physical data collection.

3) The paper demonstrates generalization properties by transferring learning from a specific sensor to a different one through fine-tuning, which is important since it shows we can leverage tactile datasets for different sensors.   

Weaknesses:
1) In figure 3 it’s hard to understand the difference between the synthetic and real tactile images, perhaps the authors can add a disparity/difference image between the two to emphasize it.

2) In 5.1.1 the authors mention that the models get lower SSIM on novel objects due to artifacts in the synthetic images that can disappear with additional fine-tuning data. If these results exist, then adding them to the table would strengthen this argument.

3) The results in tables 1 and 2 could be made clearer by computing normalized/intuitive metrics (% for example).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend accepting the paper. Not only that it addresses an important problem, but it also presents and interesting use of NeRF for tactile data generation.
issues: The weaknesses.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes to train a generative 3D visual model (NeRF) of objects that it then queries at specific viewing angles to produce inputs to a conditional GAN with the intent to produce tactile signatures as would be experienced by applying a GelSlim sensor from that selected viewing angle. The premise is that this process would be effective in generating tactile data that would otherwise be expensive to collect on a real-world robotic system.
strengths_and_weaknesses: Merits and Strengths:

The paper does indeed address an important challenge in manipulation: how do we generate tactile data for downstream robot learning tasks, given that it is so expensive to collect data? There are a lot of existing learning approaches that we can draw from given that the GelSlim’s and similar vision-based tactile sensors produce images and this paper recognizes that effectively. The paper does a good job of evaluating the proposed approach on a real-world robot experiment implemented with a Franka Emika Panda. 

Weaknesses and Limitations:

The biggest concern with the paper is: What is the core technical contribution/novelty? The paper uses existing and well-established learning techniques (NeRFs + Conditional GANs) to generate realistic image-based tactile signatures. There was no need to adapt the NeRF or cGAN architectures as they both directly operate on images (and the conditioning vector can be any vector in R^n). The losses are standard, as is the training procedure and the data generation. The application (generating tactile signatures within distribution of real-world signatures) is interesting. However, without a technical nugget (e.g., novel learning/inference algorithm, novel architecture, novel simulator, novel functionality) there is not a sufficient advancement in our knowledge or state-of-the-art in robot learning to warrant publication in the manuscript's current state. To illustrate this point further, the methods section (where the core novelty of any paper should shine) is simply an overview of NeRFs and cGANs. Assuming the author is familiar with both, the entire section can be boiled down to 1 paragraph, suggesting only a very incremental contribution.

Beyond the above concern, the performance evaluation and benchmarking are lacking. It is true that evaluating the quality of tactile signatures is difficult. This is because there is a strong task dependence. Some tasks require ultra-fine detail while others only need very coarse detail. As such, it is difficult to draw any conclusions from the metrics used to evaluate the tactile signatures. The units for both metrics are missing and there is no effort in contextualizing beyond the idea that small numbers are better. We get no intuitive feel for how much more of an improvement the numbers suggest. The improvement in the downstream task is also relatively modest. Finally, there are many generative approaches the paper could benchmark against and those would have made fine benchmarks but there is no attempt in the manuscript. 

The success rate in the Franka Kitchen environment is low for all models, and there is a modest (about 15% based on the graph) improvement but nothing to suggest that R3M is the solution. The Meta Worlds results are inconclusive, the performances are quite similar. Adroit is between the two, R3M does better but it's within 10% of the next baseline. We cannot conclude that R3M is a significant advance on the state-of-the-art based on these results.

The limitations section is, by far, the worst I’ve read in the several papers I have reviewed for CoRL and is certainly not up to par with the intent of the prompt provided by the conference organizers. This section is 6 sentences of which the first is a paper summary and the second is a filler. The second stated limitation is particularly sticking out. The purpose of the paper is to mitigate the requirements for data collection since it is expensive; however, the model needs to be retrained per new object. This is an important limitation that requires much more discussion than a throw away citation. I don’t really see the conclusion section adding any value to the paper, there is no new information presented. One approach could be to remove it and have a proper discussion and limitations section. Finally, I would want to see a clear statement on the biggest assumption: rigidity of the object. The tactile signature is only really predictable from the image because the object does not comply. This should receive far more attention and be clearly stated + implications noted.

General Comments:

The manuscript provides some encouraging preliminary results; however, without a core novel contribution (whether it is algorithmic, architecture, simulation/modeling or something to this effect) it is not ready for publication.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The manuscript provides some encouraging preliminary results; however, without a core novel contribution (whether it is algorithmic, architecture, simulation/modeling or something to this effect) it is not ready for publication. Simply put, concatenating two existing approaches with essentially no modification for the purpose of an application is not a sufficient contribution. 
issues: Please see the weaknesses section, issues are clearly discussed there.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a technique for synthesizing tactile images from RGB scans of objects which are subsequently converted to NeRFs. The authors argue that simulators for generating tactile images are cumbersome to use because they rely on accurate object and sensor models, and are often specific to a single sensor type. To remedy this, they propose a method which uses cGANs to generate tactile readings from (1) a baseline tactile reading, and (2) an RGBD rendering from a trained NeRF of an object. Their experiments show good generation quality of the tactile images, and experiments on object classification suggest that using additional cGAN-generated data improves performance by around 10% compared to just using raw data.
strengths_and_weaknesses: Strengths

The paper is well-written, clear, and thorough, and I think the strategy of data-driven tactile synthesis is an approach worthwhile sharing with the community as an alternative to purely simulated approaches. The experiments are well constructed to illustrate the points argued in the text, and fairly presented. 

Suggestions
- Though authors report that NeRF training time is no longer prohibitive, which is true, I feel a central weakness of the approach is its time-consuming real data collection process, requiring human involvement to gather and position training objects and initiate the captures, followed by 100s of sampled touches. Reporting time to collect data would be helpful to include, as well as some avenues to speed the process and reduce human involvement in the future.
- I feel it should be made more clear in the results tables and experiment descriptions that there are a different number of test objects for sim classification and real classification (2 vs 3), or ideally an additional test object should be added to the real tactile classification experiment to allow more meaningful comparison between sim and real results. The discussion states that "the higher absolute accuracy [of physical results] compared to the simulated results is likely due to the smaller number of objects for the real-world dataset.", which I agree with, but think this could be easily remedied by adding an additional test object to the real-world experiments.
- I would appreciate more detail in the classification experiments; namely, what specific objects were used for sim and real experiments, and what are some of the failure modes exhibited in both sim and real classification? 
- Showing transfer to a different sensor in the real would be much more compelling than the simulated omnitact sensor results, if the hardware is available. Even showing transfer between different DIGITS would be interesting to show ability to capture small variations in sensor construction.
- It would be interesting to investigate sim2real performance to test the importance of real data. I suggest the following 2 comparisons to add to table 3, right-hand side: (1) sim data tested on real object test set, (2) sim+gen data tested on real object test set. Although real data is presumably more accurate, the simulated set has more data (500 touches per objects vs 132, 24 objects vs 6), so I would be curious to see its performance in the real world.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I recommend accepting the paper, and make some suggestions to improve clarity and enhance the experiments in the previous section.
issues: The writing suggestions should be addressed during revisions, and I think adding an additional real test object and comparing sim2real results would be reasonable in the time frame available, though understand that showing real-world transfer to a new sensor may be infeasible.
review_title: null

Meta-review: metareview: The paper proposes a combination of NeRFs and cGANs as a means to generate synthetic but realistic tactile data, as would be seen from optical tactile sensors such as GelSight.  This is in order to save time on data collection for learning downstream tasks that operate on optical tactile data.

The reviewers agree that the paper is very clear and well-written, and that the method offers real value in easing a difficult data collection task in robotics. Most of the reviewers' original concerns have been well-addressed by the revised paper and the authors' rebuttal.  The paper will likely have value for the CoRL audience.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)