Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation
Authors: Shaohong Zhong
Alessandro Albini
Oiwi Parker Jones
Perla Maiolino
Ingmar Posner
Venue: CoRL 2022 Poster (2022)
Keywords: Camera-based tactile sensing
cross-modal tactile data generation

Abstract: Tactile perception is key for robotics applications such as manipulation. However, tactile data collection is time-consuming, especially when compared to vision. This limits the use of the tactile modality in machine learning solutions in robotics. In this paper, we propose a generative model to simulate realistic tactile sensory data for use in downstream tasks. Starting with easily-obtained camera images, we train Neural Radiance Fields (NeRF) for objects of interest. We then use NeRF-rendered RGB-D images as inputs to a conditional Generative Adversarial Network model (cGAN) to generate tactile images from desired orientations. We evaluate the generated data quantitatively using the Structural Similarity Index and Mean Squared Error metrics, and also using a tactile classification task both in simulation and in the real world. Results show that by augmenting a manually collected dataset, the generated data is able to increase classification accuracy by around 10\%. In addition, we demonstrate that our model is able to transfer from one tactile sensor to another with a small fine-tuning dataset.

Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation
Shaohong Zhong, Alessandro Albini, Oiwi Parker Jones, Perla Maiolino, Ingmar Posner
Oxford Robotics Institute
University of Oxford, United Kingdom
{shaohong, alessandro, oiwi, , ingmar @robots. ox. ac uk
Tactile perception is key for robotics applications such as manipulation. However, tactile data collection is time-consuming, especially when compared to vision. This limits the use of the tactile modality in machine learning solutions in robotics. this paper, we propose a generative model to simulate realistic tactile sensory data for use in downstream tasks. Starting with easily-obtained camera images, we train Neural Radiance Fields (NeRF) for objects of interest. We then use NeRF-rendered RGB-D images as inputs a conditional Generative Adversarial Network model (cGAN) to generate tactile images from desired ori- entations. We evaluate the generated data quantitatively using the Structural Simi- larity Index and Mean Squared Error metrics, and also using a tactile classification task both in simulation and in the real world. Results show that by augmenting a manually collected dataset, the generated data is able to increase classification accuracy by around 10%. In addition, we demonstrate that our model is able to transfer from one tactile sensor to another with a small fine-tuning dataset.
Abstract:
Keywords: Camera-based tactile sensing, cross-modal tactile data generation
Introduction
Humans rely heavily on tactile sensing for tasks such as identifying and grasping objects (e.g. pick- ing keys from pocket) [1, 2]. In this context, tactile sensing is fundamental to retrieve contact information or properties of the object such as roughness or stiffness, and is also able to comple- ment vision in occluded scenarios [1]. Tactile sensing is also critical for robotics applications such as manipulation and control [3] and object or texture recognition [4]. These tasks, especially those related tactile-based object recognition, are usually tackled with machine learning methods that typically require large amounts of data for training [1. 5, 6]. However, collecting tactile data is challenging as the robot needs to physically interact with the environment and the object. While cameras can capture the global shape of an object, tactile sensors can only capture local features, and long and tonecuinugnngg exploration procedure is usually required to capture the whole shape [7]. Beyond the problem relating to tactile exploration, tactile sensing is also still lacking standards the hardware level [8]. For the same physical stimulus, the output of different tactile systems can differ significantly, thus limiting the validity of the collected data to a specific sensing technology.
Given the difficulties of tactile data collection, the problem of generating synthetic tactile sensor re- sponses from data acquired using different modalities (which are easier to collect) becomes relevant. In particular, recent works show that vision data (RGB-D images) contain rich sensory information that can be used to generate tactile data [9, 10, 11]. Given camera or depth image of the object sur- face as input, these approaches can generate the corresponding tactile sensor output. One limitation of these vision- based generative approaches is that they require the collection of visual samples at given positions and orientations to generate the corresponding synthetic tactile data [9]. However, with the development of neural volume-rendering techniques such as NeRF [12], it is now possible to synthesise high-quality RGB-D images for novel view orientations of a scene, given only sample 2D images and their associated camera poses [12]. In this way, NeRF provides additional informa- tion on the structure of the scene that we leverage for generating tactile images for 3D objects.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

NeRF training
training
Tactile generation
RGB image collaction
NeRF RGBD rendering
Tactle image collaction
NoRF
cGAN
New tactlie Image
Figure I: Overview of the framework. We first capture camera images around the object to train a NeRF model. We then collect a tactile image dataset to train a conditional GAN model that takes NeRF-rendered RGB-D images input and outputs tactile images. Once the cGAN model is trained, we can query NeRF models with arbitrary poses, and pass the RGB-D output to the cGAN model obtain the desired tactile image from the same orientation as the query pose.
The main contribution of this paper is to introduce a novel framework for leveraging the RGB-D images that NeRFs render to generate tactile sensory data via a deep generative model. As far as the authors are aware, ours is the first work that learns a generative model conditioned on RGB-D inputs for the generation of tactile data. Compared to simulation -based approaches for tactile data generation, our framework removes the need for accurate hand-engineered modelling and calibration of the tactile sensor well as the objects of interest. Compared to other learning- based approaches, our method is capable of handling 3D objects of different geometries, and is able to generalise to novel views of the object through the use of NeRF models. The use of NeRF models also removes the need for an RGB-D camera to manually obtain a new input image each time a new object reading desired. In addition, our framework is capable of transferring to new sensor with a small fine- tuning dataset through the use of a deep generative model conditioned on sensor background image. In this way, we can overcome the domain gap between different tactile sensors, and open up the potential to leverage different tactile datasets for even better performance.
2 Literature Review
This section reviews the approaches that simulate generate data for camera- based tactile sensors [13, 14]. These devices use camera to capture the deformation of a soft medium and output a tactile image. One class of approaches are based the development of accurate simulators for camera- based tactile sensors, which aim to make the robot learn tactile features in a simulated environment and transfer the knowledge to the real world [15, 16, 17, 18). For example, the TACTO simulator presented in [15] allows one to emulate both Digit and the OmniTact sensors by simulating contacts an off-the-shed physics engine [14, 19]. Different simulators, such as Taxim aim to simulate the responses of a GelSight sensor by using polynomial look-up table [16]. Another approach to leverage methods such as physics-based rendering [17] and mepth--aps from physics simula- tors [18]. Additional works attempt to address the sim2real issue in these tactile simulators through the use of CyclyGANs [20] and texture generation networks [21]. Even though tactile simulators present an attractive option for data synthesis, these tactile simulators require accurate modelling and
calibration of the specific tactile sensor via additional calibration equipment [15, 16). An accurate object model is also needed for simulating contacts [16]. In contrast, our proposed learning-based bbsed framework removes the need for modelling the sensor or the object.
A separate class of work explores learning-based approaches. A majority addresses the problem of generating tactile data from vision, using camera-based sensors, both modalities encode informa- tion using the same data structure [10, 22, 9, 11]. For example. Li et al. [10] estimate the response of a GelSight sensor using cGAN model [23] conditioned on a vision sequence that captures the robot touching the object. However, their method requires a robot to perform the touching action to generate the tactile response. Patel al. [22] leverage depth sensors and the object mesh to gener- ate tactile images, similarly building tactile images of individual objects directly using a NeRF-like model. However, their approach is incapable of generalising to new objects because new NeRF model needs to be trained on tactile images for the new object. Lee et al. [9] employ cGAN to generate tactile images from vision, and vere-versa, dataset containing top-down views of flat clothes our work is most similar in spirit
cGAN model. Gao et al. [24, 25] also attempt to render

---

to this. However, we differ significantly in our use of RGB-D images rendered from NeRF models. This removes the need retake a camera image for each desired tactile image. Our problem setting is also harder, as it focuses on 3D objects with complex geometries. In addition, we demonstrate the transfer of the trained cGAN model to generate data for a new tactile sensor, which, as far as the authors are aware, is the first that demonstrates such generalisation capability across tactile sensors.
3 Methods
As seen from Figure our approach involves first training individual NeRF models for objects of interest. Although they only require RGB images for training, the of NeRFs enables to learn the 3D structure of the scene and render RGB-D images from arbitrary viewpoints. Then, propose train a cGAN model to generate tactile data conditioned both the RGB-D images rendered by NeRF models and a reference background image for the tactile sensor. To generate tactile data from novel poses. we simply render corresponding RGB-D images from an object's NeRF model and pass the results as inputs to the cGAN model.
3.1 Neural radiance fields
A Neural Radiance Field (NeRF) is model for synthesising novel views of scenes [12]. It requires a set of RGB images and their associated camera poses for training. A trained NeRF is able to synthesise high-quality and novel views of the modelled scene. Formally, NeRF model represents continuous scene as a function F, which takes the 3D location of point m = (Ir y. and a 2D viewing direction d (8, Φ) input, and predicts the RGB colour c (r. g, b) and volume density of the point from the particular viewing direction. The learned function is parameterised using multi-layered perceptron Fe. Using Fe, images of the scene can be rendered from arbitrary camera poses. Given the camera center the color of each pixel is estimated by summing N samples along the camera ray C(r), where C(r) = o + td, using a numerical quadrature approximation [26]:
N
C(r)
Σ n=1
7T exp( exp(-onōx))cen
where T, = exp( En=1 on) and 8, bn+1 - ln- By predicting the volume density of sampled points in this process, NeRF can trivially render the depth image of the scene [12], capacity that we take advantage of this study. After rendering the required RGB-D image, we perform simple preprocessing by centre cropping of the RGB-D image and normalisation of the depth image before passing the RGB-D image as input to the cGAN model.
Tactile Image Generator G
Tactile Image Discriminator D
Pose
RGB
RGB
Enc
NeRF
Depth
Tactile
Depth
real
Enc
fake
Background
Tactile
Enc
Figure 2: Proposed cGAN architecture. Given a desired pose, an RGB-D image is rendered from the NeRF model. The RGB, depth, and reference sensor background images are then encoded through separate encoders. The outputs of these encoders are concatenated before being fed to the generator network to generate the desired tactile image. During training, classifier is used differentiate whether tactile image is generated by the generator (fake) or is ground-truth tactile image (real). The classifier loss and an L. image loss are used to train the generator during training.
3

---

3.2 Conditional generative adversarial network
After using the NeRF model render the RGB-D image of an object from a target camera pose (o, de), the goal is to generate the corresponding tactile image. We define the matching criteria to be where the tactile sensor orientation is the same as the camera orientation, and the tactile sensor position is at a point along the centre camera ray (s, de), where = o + 1,d. and ig is dependent on the geometry of the object.
We employ a conditional Generative Adversarial Network to learn the mapping G : {x, z} + between a rendered RGB-D image x and noise z, on one hand, and the corresponding tactile image y. on the other [27]. Taking inspiration from prior work [10], we modify the cGAN model to condition the generative model on the RGB image, the depth image, and reference background image b using separate encoders, with skip-connections added between the encoder for the reference background image and the generator (see Figure 2 for detailed architecture). Following the eGAN literature [27], the noise z can be implemented using Dropout [28] in the generator network G. The objective function of the cGAN model is
G arg min max LoGAN(G,)) GLLI
(2)
G D
where
LouAN(G.)) = Ex Ex,y~p(x.y) [xg y)] Exxxp(xxx) [log(1 D(x, G(x,
(3)
and
GG) -
(4)
Experiments
Experiments are first performed in simulation as proof-of-concept and then in the real world to test the hypothesis that generated tactile data can improve performance on downstream robotics tasks. To collect tactile images, we employ Digit tactile sensors [14]. These sensors have two main components: an RGB camera and a compliant layer made of a soft gel. As working principle, the camera captures the deformations of the soft gel caused by contact forces. For the simulation experiments, we use TACTO to simulate the Digit tactile readings [15]. We note that the TACTO simulator uses PyBullet [29] as its physies engine and PyRender [30] as the rendering engine.
4.1 Dataset generation
Simulation For simulation experiments, we select 27 common household objects from the scanned YCB dataset [31]. To train a NeRF model for each object, we collect images taken from evenly sampled poses around sphere centred on the object. To perform realistic collection of tactile readings, we randomly sample initial positions on a sphere centred on the object and move the tactile sensor from the initial position towards the centre of the object. To obtain tactile readings, constant force is applied to the simulated Digit sensor, with the objects fixed. We discard cases when no reading is obtained, and collect 500 touches per object. Correspondingly. 500 RGB-D images are synthesised from the NeRF model for each object, using the sensor orientation as the view orientation. The rendered RGB-D images are then preprocessed and paired with their corresponding tactile images for training the cGAN model. For evaluation, we hold out 3 distinct objects as the novel object test set and use 24 objects for training. We also collect validation and a novel view test set of 50 paired tactile and RGB-D images from novel orientations for each training object.
Real-world For experiments in the real world, we select common houschold objects from both the YCB dataset and other objects belonging to similar categories [31]. For training the NeRF models, we collect 1 18 images taken from camera poses that are uniformly sampled within a range in the hemisphere around the object using the set-up in Figure 1.
To automate data collection, we use 7-DoF Franka Emika Panda arm with an Intel RealSense D415 camera for taking images, and select the camera pose range based on the workspace of the robot. For collecting the tactile reading, as seen in Figure 1, we attach Digit sensor on the end effector flange of the Panda arm, and control the arm to move towards the centre of the object. For training. perform 132 touches per object and collect 50 frames per touch. We also discard the cases when

---

no reading is obtained. We then synthesise and preprocess the corresponding RGB-D images using the NeRF model and pair them with the tactile images for training the cGAN model. For testing, we hold out 3 distinct objects as the novel object test set, and we hold out 12.5% of the training dataset as the novel view test set. It should the touch and not the frame. The resultant training dataset contains 398 touches and 19, 19,900 frames. For each object, collecting images for the NeRF models takes approximately 20 minutes, whereas the collection of tactile data takes over 2 hours. The details of the objects given in the Appendix.
noted that the test set is randomly chosen using the index of
All the objects in the simulated and real-world datasets are assumed to be rigid. For soft objects, their deformation when subjected to an external force. which is challenging to model, needs to be taken into account. By considering rigid objects only, the tactile sensor response mainly depends on the deformation of the soft layer of the sensor and on the contact force applied.
4.2 Evaluation details
As losses in cGAN are not directly indicative of the quality of the training results, it is difficult to use them to evaluate the realism of the generated tactile images. It is also difficult to manually evaluate the realism of such tactile images. Thus, we employ set of approximate metrics and an example task to evaluate the quality of the generated images. We first evaluate the Structural Similarity Index (SSIM) and the Mean Squared Error (MSE) between the generated image and the ground truth tactile image on the hold-out test sets. Note that the average SSIM across RGB channels is taken, and that in simulation, ground-truth images refer to the simulated tactile images from the TACTO simulator. The SSIM is reference metric with a range of [0, 1]. whereas the MSE measures the difference in tactile
pixel values with range of [0, 65025), since classification as an example task to evaluate the usefulness of the generated tactile datasets. The same evaluation pipelines are applied to both simulation and real-world experiments. In evaluation, we scale all images to 128 x 128 which is also the size of the output of the cGAN model.
consider 8-bit images [32]. Next,
For both simulation and real- wall-woldd experiments, fication task, using a simple convolutional neural network to perform object classification based on one tactile image. For comparison, we prepare two training datasets, one consisting of only ground- truth tactile images, the other consisting of both ground-truth and generated images. The test dataset consists of only ground -ruth images. This is performed for both simulation and real- real-world data.
evaluate the generated images in a tactile classi-
We also implement two benchmarks for comparison: 1) conditional GAN network for tactile image generation proposed by Lee et al. [9] 2) CycleGAN network [33]. In our experiments, both networks take RGB images as inputs and output tactile images at the corresponding pose. We use the default configurations proposed by the authors for our experiments.
Results and Discussion
We evaluate the results of tactile image generation in both a simulated setting and a real-world setting through a set of quantitative metrics and a tactile classification task. We also evaluate the results of transferring the learned cGAN model to a different sensor.
5.1 Simulation results
From Table 1, we can see that our approach able to achieve high SSIM and low MSE values, indicating that the generated tactile images are structurally similar to the ground-truth (simulated) tactile images for both novel views of training objects and novel objects. This is corroborated by the qualitative results shown in Figure 3. On the novel view dataset, tactile images very similar to ground-trutt ones, without having to explicitly model the object or the tactile sensor. The higher MSE value for novel objects could be due to difference in the position of the predicted indentation, which leads to more differences with the ground- pixel-wise, as seen from Example and 3 difference images in Figure which indicate position mismatch.
approach is able to generate
To evaluate the quality of the generated tactile images, we use tactile object classification as an example task. By augmenting given tactile dataset with generated tactile images, we are able to improve the success rate of the tactile classifier by a large margin. This shows that the proposed generative model is able to generate tactile images useful for the tactile classification task. Through
5

---

Depth
Tartlle
Roal Tactll
Difference
Depth
Fake Tactle
Real Tactlio
Difference
Example
Example 2
Exomple
Example
Simulation Results
Real-World Results
Figure 3: Qualitative results on novel object test sets. Our approach generates faithful tactile images in both simulated (left) and real-world (right) domains on objects it has not seen before in training. The columns show RGB images, depth images, fake tactile images, real tactile images, and the difference images between the real and the fake images. The difference images are thresholded to remove noise. The differences are mostly due to a mismatch in position or amount of indentation and exist mostly around the edges of the indentation. The fake tactile images are model generated; real tactile images provide ground truth.
Test set
Method
SSIM
MSEI
T-N
0.980 0.001
14.5 + 0.8
Novel
Lee
0.678 + 0.002
1040 + 10
view
Cycle
0.649 + 0.002
1660 + 10
Dataset
Accuracy/$5
T-N
0.961 + 0.000
84 +3
Sim
85 =
Novel
Lee
0.687 0.001
990 +
Sim + T-N
96 +
object
Cycle
0.660 + 0.002
1610 + 10
Sim Lee
86 + 4
BG
0.921 + 0.000
67.4 3
Sim + Cycle
80 + 0
(a) SSIM values between generated tactile images and ground-trun tttnh (simulated) tactile images.
(b) Classification results for simu-
lated dataset.
Table 1: Evaluation results for generated tactile images in simulation. Results shown average
values with standard errors. T-N: Touch NeRF (Ours). BG: Reference sensor background image.
leveraging NeRF models, novel view angles and generate additional tactile images to further augment the dataset at a low cost. Compared to the benchmarks, approach outperforms by large margin in both quantitative metrics and the classification task. This is likely due to the conditioning on depth, which enables approach to predict tactile responses 3D objects.
also able to render additional RGB-D images of the object from
5.1.1 Transferring to a different sensor
By using a much smaller fine-tuning dataset than the original RGB-D and tactile training dataset, we are able to fine-tune a trained cGAN model to adapt to a different sensor with different charac- teristics. To demonstrate this, select the OmniTact sensor
example target tactile sensor for fine-tuning [19]. We collect simulated OmniTact images using a similar procedure as described in Section 4.1 and pair the tactile image with the corresponding RGB-D image rendered from NeRF models. We collect tactile images for each training object for training and 50 each as the novel view test set. We hold out the same 3 objects as the novel object test set. We then continue train the cGAN model that has been pre-trained on the simulated Digit dataset with the new OmniTact dataset, and evaluate the results using similar metrics described in Section 4.2.
Using much smaller fine-tuning dataset, the generated OmniTact tactile images still resemble the ground-truth (simulated) images, shown in the SSIM and MSE results in Table 2a, and qualita-

---

Fake Tactle
Real Tactle
Figure 4: Example of generated tactile images for the simulated Omni'Tct sensor. In this case, the real tactile images are generated by the simulator.
Test Method
SSIM
MSEJ
Novel
T-N
0.865 + 0.002
550 + 10
view
FS
0.829 + 0.002
950 +
T-N
0.839 + 0.003
760 + 30
Dataset
Accuracy/%
Novel
T-N+
0.847 + 0.003
760 30
Sim
86 + 0
object
FS
0.811 + 0.005
1300 = 60
Sim T-N
94 +
BG
0.856 + 0.001
1390 = 20
Sim + FS
82 = 1
SSIM values hetween generated tactile images and ground-truth (simulated) tactile images.
(b) Classification results for using fine-tuning dataset.
Table 2: Evaluation results for generated tactile images for a new sensor in simulation. Results
shown are average values with standard errors. T-N: Touch NeRF (Ours) with fine-tuning. T-N+: Touch NeRF with additional fine-tuning data. BG: Reference sensor background image. FS: Touch NeRF model trained from scratch.
tively in Figure 4. The lower SSIM value for novel objects is likely due to the presence of artefacts the generated images. This could be remedied by adding an additional amount fine-tuning data, seen in the T-N+ results in Table2a, which include additional 45 training images (~1/3 of the size of the original fine-tuning dataset). From Figure 4, can also be seen that the generated tactile images able to capture the shape of the indentation. Additionally, the background of the tactile image is accurately replicated, pointing to the benefit of conditioning the generative model on a ref- erence sensor background image. Classification results in Table 2b demonstrate that the generated tactile images for the new sensor are still useful for increasing the accuracy of the tactile classifier by augmenting the tactile dataset. The fine-tuned cGAN model also outperforms the benchmark cGAN model trained from scratch, demonstrating the usefulness of pre-training and pointing to the potential of the proposed approach in leveraging available tactile datasets for different sensors to improve performance. This offers a potential avenue for further improving the data efficiency in tactile data generation using our approach.
5.2
Real-world results
From Figure 3 it can be seen that the proposed framework is able to generate realistic tactile images in the real world on both novel views of training objects as well objects that it has not seen before during training. It is also able to generate the change in lighting caused by the object pressing against the soft gel. The difference in the amount of indentation between the generated image and the ground-truth image could be due to the variation of the force applied, as we only require the force to be within a range during tactile data collection.
Table 3 shows that the generated tactile images are able to capture the key properties of ground-trutt tactile images collected in the real world. The SSIM values are lower than those in simulation in Table 1, which is expected due to the presence of noise the real world. Additionally, classification results in Table 3b indicate that the generated tactile images useful in increasing the accuracy of the downstream tactile classification task. The failure modes exhibited the classification experi- ments are found to be mostly due the similar tactile features exhibited in some objects.
can be seen that our approach outperforms both benchmarks quantitative metrics and the clas- sification task. It should be noted that the RGB-D images rendered by real-world NeRF models
7

---

contain more artefacts than those in simulation, as seen in Figure 3. However, even using the noisier RGB-D renderings, our approach is still able to generate high-quality tactile images for 3D objects. Additionallyy, our approach offers more generalisability on two fronts: firstly, as seen from the results in Table 3a, our approach outperforms in the novel object test set by a large margin; secondly, by leveraging NeRF model, our approach is able to generate high-quality tactile images from novel viewpoints. This further removes the need to manually retake images for every new tactile image, as required by the baseline approaches.
Test set
Method
SSIM f
MSE
T-N
0.887 = 0.001
56 =
Novel view
Lee
0.708 = 0.006
960 = 50
Dataset
Accuracy/%
Cycle
0.781 + 0.006
420 + 60
Real
74
T-N
0.838 = 0.001
3
Real + T-N
83 - 2
Novel
Lee
0.723 = 0.003
850 = 30
Real + Lee
76 -: 2
object
Cycle
0.765 = 0.002
390 - 10
Real+ Cycle
70 = 0
BG
0.814 + 0.000
239 + 3
T-N*
70 1: 6
(a) SSIM values between generated tactile images and ground-truth (real) tactile images.
(b) Classification results for real- world dataset.
Table 3: Evaluation results for generated tactile images in the real world. Results shown are average values with standard errors. T-N: Touch NeRF (Ours). BG: Reference sensor background image. T-N*; Touch NeRF deployed 'in the wild' (training dataset contains no real data).
To illustrate the proposed approach in deployment, we collect RGB image data (48 images per object) for three additional objects on a different background using a mobile phone and trained corresponding NeRF models (with a lower resolution setting for faster training). We then generate tactile images for each object using our trained real-world cGAN model, and evaluate the quality of the generated tactile images in similar classification task. The classifier is trained only generated data, and is tested on manually-collecter real tactile data. The T-N* result in Table 3b shows that the proposed approach is still able to achieve high accuracy in this more challenging setting. In this case, the data collection took around minutes per object with casually taken camera images, further demonstrating the ease of deployment of our approach and the potential for reducing human efforts in tactile data collection.
Conclusion and Limitations
In this paper, we present a novel framework for the generation of tactile data for camera-based tactile sensors. The framework leverages NeRF models to render RGB-D images of an object from desired poses, and passes the RGB-D images to a cGAN model to generate the desired tactile images. Com- pared to state-of-the-ats this approach allows for the generation of tactile images for 3D objects from arbitrary viewpoints, without the need for accurate sensor or object models. Results demon- strate that the generated tactile images are structurally similar to ground-truth images, and are useful in downstream roboties tasks such as tactile classification. We further demonstrate the potential of novel capability to transfer from one tactile modality another with small fine-tuning dataset.
One potential limit of the approach is that a NeRF model needs be trained for each object. How- ever, it should be noted that the collection of image data for training NeRF models requires much less human effort than the collection of tactile data, as illustrated by our deployment experiment. This is also arguably simpler than building a scanned object model. Additionally, recent advances in accelerating NeRF training also significantly mitigate the computation time needed [34].
We also assumed to operate only on rigid objects, with a fixed force range and orientation (pitch and yaw) of the tactile sensor. Soft objects might require modelling the non-linear deformation of the object when in contact, which adds additional complexity in predicting the tactile response. This is currently out of the scope of the paper. must also be noted that even under the rigid object as- sumption, generating tactile images for 3D objects is challenging and is still an open problem for the tactile-sensing community. The generalisation to different forces and orientations, and potentially soft objects, is left to future work.
8

---

Acknowledgments
We thank Jun Yamada for help with the experimental set-up, and Jack Collins and Alexander Mitchell for suggestions on the paper draft, and Yizhe Wu and other members of the Applied Ar- tificial Intelligence Lab and Soft Robotics Lab for helpful discussions. We thank the anonymous reviewers for their valuable feedback in revising the paper. This work was supported part by the UKRVEPSRC Programme under Grant EP/V000748/1, and by a CSC-PAG Oxford Scholarship. We would like to acknowledge the use the University of Oxford Advanced Research Computing http://xx.di..rr///...///// /ttp://0522///moooo22222.. We would also
(ARC) facility in carrying out this work. like to acknowledge the use of the SCAN facility.
References
[1] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, Malik, E. H. Adelson, and Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3307, 2018. 101910..10// 2882279777
[2] Y. Narang, B. Sundaralingam, M. Macklin, A. Mousavian, and D. Fox. Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6444-6451, 2021. doi:
10. 10.1109/CR44450062220 9599
[3] Z. Kappassov, J.-A. Corrales, and Perdereau. Tactile sensing dexterous robot hands review. Robotics and Autonomous Systems, 74:195-220, 2015. ISSN 0921-8890. doi:htips://
URL https: htttps:/www.scienneee wwweecedieeecccee.oeeeoe.oeoe accine/eeeee////
atricll/ppi//59218890000001
[4] S. Luo, Bimbo, R. Dahiya, and H. Liu. Robotic tactile perception of object properties:
A review. Mechatronics, 48:54-67, 2017. ISSN 0957-4158. dot.htpps//0oi.or///1.00016 meccaatronies.2117.11.1 p:1/S099744181170155755
URL https: //ww:////wwww.. scieecedrrccc...ccc..o http:///ww..caaaaaaa.occooo/ttaa
[5] J. M. Gandarias, J. M. Gómez-de Gabriel, and A. J. García-Cerezo. Enhancing perception with tactile object recognition in adaptive grippers for human-robot interaction. Sensors, 18(3), 2018. ISSN 1424-8220. 1424- 09244-8200/1833/22
0339///0000009999 URL attps://www.ndpi. mdpi. com/
[6] L. Cao, R. Kotagiri. F. Sun, H. Li, W. Huang, and Z. M. M. Aye. Efficient tactile object recognition with randomized tiling convolutional networks in a hierarchical fu- sion strategy. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Mar. 2016. doi:10.1609/aaii. htt.:/aaiivvi...000.. 10012.
URL https://oor aaai. wwai.org//aaaaaaaaa....aa.oaaaaa itliiidexx.ppp phpp AAAI/
article/ http//veew////// 10412.
[7] Z. Pezzementi, E. Plaku, C. Reyda, and G. D. Hager. Tactile-object recognition from appear- ance information. IEEE Transactions on Robotics, 27(3):473-487 2011. doi:10.1 109/TTO.. 2011.2125350.
[8] R. S. Dahiya, Metta, M. Valle, and G. Sandini. Tactile sensing- humans humanoids. IEEE Transactions on Robotics, 26(1):1-20, 2010. doi: 109//TRO022009920336222
[9] J.TT. Lee, D. Bollegala, and S. Luo. "Touching to see" and "seeing to feel": Robotic cross- modal sensory data generation for visual-tactile perception. In 2019 International Confer- on Robotics and Automation (ICRA), pages 4276-4282, 2019. doi:10.1 119:10.119//C 8793763.
[10] Y. Li, J.-Y. 7hu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10609-10618, 2019.
[11] S. Cai, K. Zhu, Y. Ban, and T. Narumi. Visual-tactile cross-modal data generation using residue-fusion gan with feature-matching and perceptual losses. IEEE Robotics and Automa- tion Letters, 6(4):7525-7532. 2021. doi:10. 110.0..110//LRA.22..330000

---

[12] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1): 99-106, dec 2021. ISSN 0001-0782. doi: 10::0.114//35035300 URL hwtphstt////.....oo httpp:://do.oor/10.. 1145/3503250.
[13] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for esti- mating geometry and force. Sensors, 17(12), 2017. ISSN 1424-8220. doi: dbc:10.3390/s171227766 URI. https:: //wp:////wwwwwe// wwwp:.ddp..... cwwmom2//// 10200/17/12/22
[14] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, Jayaraman, and R. Calandra. Digit: novel design for a low- cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838 3845, 2020. doi: 10..100/LRA.22220222711.
[15] S. Wang, M. Lambeta, P.WW. Chou, and R. Calandra. Tacto: fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930-3937, 2022. doi: 10. 10.0
[16] Z. Si and W. Yuan. Taxim: An example-based simulation model for gelsight tactile sen- sors. IEEE Robotics and Automation Letters, 7(2):2361-22) 1-366, 2022. 1oi:10.1109/1RAA.1 3142412.
[17] A. Agarwal, T. Man, and W. Yuan. Simulation of vision-based tactile sensors using physics based rendering. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1-7, 2021. doi:10.1 1099/CCRA4450022221..66666
[18] D. Gomes, Paoletti, and Luo. Generation of gelsight tactile images for sim2real learn- ing. IEEE Robotics and Automation Letters, 6(2):4177-4188 2021. 1b1:100.109/RRA.20000 3063925.
[19] A. Padmanabha, F. Ebert, S. Tian, R. Calandra, C. Finn, and S. Levine. Omnitact: A multi- directional high-resolution touch sensor. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pages 618 624,,, 2020.
[20] W. Chen, Y. Xu, Z. Chen, P. Zeng, R. Dang, R. Chen, and J. Xu. Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan. IEEE Robotics and Automation Letters, (3):6187-6194 2022. doi:10. 109.1199....222.1167777
[21] T. Jianu, D. F. Gomes, and S. Luo. Reducing tactile sim2real domain gaps via deep texture generation networks. In 2022 International Conference on Robotics and Automation (ICRA), pages 8305 3305-8311, 2022. doi: 109/C//4/9// Tt..:4466669222..999 10122991110 1801.
[22] K. Patel, S. Iba, and N. Jamali. Deep tactile experience: Estimating tactile sensor output from depth sensor data. In 2020 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9846-9853, 2020. 105000.1000/1055443332020000000 202.934156.
[23] M. Mirza and S. Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1784, 2014. URL http://arxix wwtprii.orrgab///eee
1784.
[24] R. Gao, Y.-Y. Chang, S. Mall, L.. Fei-Fei, and J. Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In 5th Annual Conference Robot Learning, 2021. URL
[25) R. Gao, Z. Si, Y.-Y. Chang, S. Clarke, J. Bohg. L. Fei-Fei, W. Yuan, and J. Wu. ObjectFolder 2.0: multisensory object dataset for sim2real transfer. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] N. Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, (2):99-1008 1995.
[27] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967-5976, 2017. 11.11...:...111.11111.. 10w.10000/VVPR.211..000
10

---

[28] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: A sim- ple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958 2014. URL. http:///:// org/ ppapers/wtb/sriivastaaaaaa html.
[29] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[30] M. Matl.
/ttp::///gthbb httpe///ttpppppppp/oo.ooo//////s 2019. 2019.
[31] B. Calli, A. Singh, Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M.
Dollar. Yale-emu-berkeley dataset for robotic manipulation research. The International Jour- nal of Robotics Research, 36(3):261-268, 2017. 10f:10.1177007733441100000 URL
https://doi.org//u 1177 11111777770277331177000111
[32] Wang. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEFE Transactions on Image Processing, 13(4):600-612, 2004. doi:10. 109/TIP:2003.8 119861.
[33] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-imagee translation using cycle- consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017. 1tt110.119/11CC2201...0
[34] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102: July 2022. doi: 10. 00.1145/52822233331222 URL htt.tt/.../aaacccaaaaee///trrrrr httpr//ooooooooooe///////oo/ooor
35300177
[35] L. Yen-Chen. Nerf-pytorch. https: gttgghtthhu..e.ee
2020..
[36] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In N. Navab, Hornegger, M. Wells, and A. F, Frangi, edi- tors, Medical Image Computing and Computer-ssisted Intervention - MICCAI 2015, pages 234-241, Cham, 2015. Springer International Publishing. ISBN 478-3-319-245744. doi:


Reviews:
[{"review_id": "UXU58IMPqVm_", "reviewer": "Reviewer_Vqs5", "summary": "This paper proposes a technique for synthesizing tactile images from RGB scans of objects which are subsequently converted to NeRFs. The authors argue that simulators for generating tactile images are cumbersome to use because they rely on accurate object and sensor models, and are often specific to a single sensor type. To remedy this, they propose a method which uses cGANs to generate tactile readings from (1) a baseline tactile reading, and (2) an RGBD rendering from a trained NeRF of an object. Their experiments show good generation quality of the tactile images, and experiments on object classification suggest that using additional cGAN-generated data improves performance by around 10% compared to just using raw data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well-written, clear, and thorough, and I think the strategy of data-driven tactile synthesis is an approach worthwhile sharing with the community as an alternative to purely simulated approaches. The experiments are well constructed to illustrate the points argued in the text, and fairly presented. \n\nSuggestions\n- Though authors report that NeRF training time is no longer prohibitive, which is true, I feel a central weakness of the approach is its time-consuming real data collection process, requiring human involvement to gather and position training objects and initiate the captures, followed by 100s of sampled touches. Reporting time to collect data would be helpful to include, as well as some avenues to speed the process and reduce human involvement in the future.\n- I feel it should be made more clear in the results tables and experiment descriptions that there are a different number of test objects for sim classification and real classification (2 vs 3), or ideally an additional test object should be added to the real tactile classification experiment to allow more meaningful comparison between sim and real results. The discussion states that \"the higher absolute accuracy [of physical results] compared to the simulated results is likely due to the smaller number of objects for the real-world dataset.\", which I agree with, but think this could be easily remedied by adding an additional test object to the real-world experiments.\n- I would appreciate more detail in the classification experiments; namely, what specific objects were used for sim and real experiments, and what are some of the failure modes exhibited in both sim and real classification? \n- Showing transfer to a different sensor in the real would be much more compelling than the simulated omnitact sensor results, if the hardware is available. Even showing transfer between different DIGITS would be interesting to show ability to capture small variations in sensor construction.\n- It would be interesting to investigate sim2real performance to test the importance of real data. I suggest the following 2 comparisons to add to table 3, right-hand side: (1) sim data tested on real object test set, (2) sim+gen data tested on real object test set. Although real data is presumably more accurate, the simulated set has more data (500 touches per objects vs 132, 24 objects vs 6), so I would be curious to see its performance in the real world.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting the paper, and make some suggestions to improve clarity and enhance the experiments in the previous section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The writing suggestions should be addressed during revisions, and I think adding an additional real test object and comparing sim2real results would be reasonable in the time frame available, though understand that showing real-world transfer to a new sensor may be infeasible."}}, {"review_id": "PIYD3icJ32x", "reviewer": "Reviewer_Fy6z", "summary": "This paper presents a method for generating tactile data for objects using a conditional GAN, with training data coming from NeRF trained on  comparatively small amounts of RGB-D data but separately for each object. The motivation comes from the fact that tactile data from sensors is hard to collect and generalize across hardwares. Related approaches use GANs directly from visual data, but this requires a lot of images. \n\nApproach: NeRF is trained separately for each object, then used to generate various RGB-D images. The cGAN is trained on a set of NeRF-generated RGB-D + ground truth tactile images, then is used to generate tactile images from the NeRF-generated RGB-D (in turn generated from a desired pose) and a predetermined background image.\n\nOn holdout object experiments, the proposed approach reaches seemingly high SSIM wrt ground truth. MSE is more varied, but low for novel views at least. Using this method for fine-tuning shows some performance improvement as well. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Paper is exceptionally clear and well-written. An interesting but easy read, which is ideal. Congrats to the authors on a very well-presented work!\n- The approach is intuitive and has a clear value proposition. The advantages with respect to training data efficiency are evident, and the paper as a whole is well-motivated given both the baseline approach of actual tactile sensors as well as more related approaches using cGANs.\n- While the experimental section isn't particularly thorough, it's well designed. It's good to see holdout experiments and comparison between novel view and novel object\n\n**Weaknesses**\n- There doesn't seem to be any baseline in this paper. I realize that the value isn't exactly degree of improvement on some metric, but this is ultimately a new approach to an existing problem that should make its value evident. This can include even a brief analysis on comparison of data efficiency, training time, and generality between many single-object NeRFs vs. the prior cGAN work. Even if performance from other methods is better, an argument can be made for why this one is worthwhile/worth iterating on\n- Related to this, we are essentially viewing the results in a vacuum. This makes it difficult to judge whether a user should try this method\n- Figure 3 is helpful, but ultimately readers without a lot of experience with tactile methods won't know whether the tactile comparison is particularly good or not. Perhaps this paper isn't for such readers, but the venue certainly is. The fake images look promising but with room for improvement compared to the real ones, but ultimately I don't know how to judge whether this is a step forward. It's even more unclear given that there may be increased data efficiency and increased compute requirements to consider.\n- This has no bearing on my score, but I find the title somewhat off-putting even if it's a good pun.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak accept because I don't see any major flaws in this paper and it seems useful, but I also am not convinced of its significance - not because I have a reason to believe it isn't significant, but because 1) the results aren't deeply surprising and 2) I have nothing to compare to. Someone with more domain knowledge will be able to provide a better assessment. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Lack of baselines, comparisons, or analysis of value/potential impact\n", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "K9YWgsI77Fm", "reviewer": "Reviewer_ozxa", "summary": "This paper proposes a framework for generating synthetic tactile data, since collecting data on a physical system is tedious and simulators lack of accurate models. The method leverages a trained NeRF model to generate novel views of a scene, and a cGAN to generate the tactile data given a rendered NeRF view. To evaluate the quality of the synthetic tactile images they use SSIM and MSE, and use the generated data to train a classifier on a physical robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is clear, easy to follow and the problem is well motivated.  \n\n2) This paper addresses an important and challenging problem in a practical way, offering a clever solution. It proposes a way to scale data collection for tactile sensors without modelling the sensor directly, potentially saving time and effort compared to physical data collection.\n\n3) The paper demonstrates generalization properties by transferring learning from a specific sensor to a different one through fine-tuning, which is important since it shows we can leverage tactile datasets for different sensors.   \n\nWeaknesses:\n1) In figure 3 it’s hard to understand the difference between the synthetic and real tactile images, perhaps the authors can add a disparity/difference image between the two to emphasize it.\n\n2) In 5.1.1 the authors mention that the models get lower SSIM on novel objects due to artifacts in the synthetic images that can disappear with additional fine-tuning data. If these results exist, then adding them to the table would strengthen this argument.\n\n3) The results in tables 1 and 2 could be made clearer by computing normalized/intuitive metrics (% for example).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting the paper. Not only that it addresses an important problem, but it also presents and interesting use of NeRF for tactile data generation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "EgQpWuX3G_f", "reviewer": "Reviewer_gjCP", "summary": "The paper proposes to train a generative 3D visual model (NeRF) of objects that it then queries at specific viewing angles to produce inputs to a conditional GAN with the intent to produce tactile signatures as would be experienced by applying a GelSlim sensor from that selected viewing angle. The premise is that this process would be effective in generating tactile data that would otherwise be expensive to collect on a real-world robotic system.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Merits and Strengths:\n\nThe paper does indeed address an important challenge in manipulation: how do we generate tactile data for downstream robot learning tasks, given that it is so expensive to collect data? There are a lot of existing learning approaches that we can draw from given that the GelSlim’s and similar vision-based tactile sensors produce images and this paper recognizes that effectively. The paper does a good job of evaluating the proposed approach on a real-world robot experiment implemented with a Franka Emika Panda. \n\nWeaknesses and Limitations:\n\nThe biggest concern with the paper is: What is the core technical contribution/novelty? The paper uses existing and well-established learning techniques (NeRFs + Conditional GANs) to generate realistic image-based tactile signatures. There was no need to adapt the NeRF or cGAN architectures as they both directly operate on images (and the conditioning vector can be any vector in R^n). The losses are standard, as is the training procedure and the data generation. The application (generating tactile signatures within distribution of real-world signatures) is interesting. However, without a technical nugget (e.g., novel learning/inference algorithm, novel architecture, novel simulator, novel functionality) there is not a sufficient advancement in our knowledge or state-of-the-art in robot learning to warrant publication in the manuscript's current state. To illustrate this point further, the methods section (where the core novelty of any paper should shine) is simply an overview of NeRFs and cGANs. Assuming the author is familiar with both, the entire section can be boiled down to 1 paragraph, suggesting only a very incremental contribution.\n\nBeyond the above concern, the performance evaluation and benchmarking are lacking. It is true that evaluating the quality of tactile signatures is difficult. This is because there is a strong task dependence. Some tasks require ultra-fine detail while others only need very coarse detail. As such, it is difficult to draw any conclusions from the metrics used to evaluate the tactile signatures. The units for both metrics are missing and there is no effort in contextualizing beyond the idea that small numbers are better. We get no intuitive feel for how much more of an improvement the numbers suggest. The improvement in the downstream task is also relatively modest. Finally, there are many generative approaches the paper could benchmark against and those would have made fine benchmarks but there is no attempt in the manuscript. \n\nThe success rate in the Franka Kitchen environment is low for all models, and there is a modest (about 15% based on the graph) improvement but nothing to suggest that R3M is the solution. The Meta Worlds results are inconclusive, the performances are quite similar. Adroit is between the two, R3M does better but it's within 10% of the next baseline. We cannot conclude that R3M is a significant advance on the state-of-the-art based on these results.\n\nThe limitations section is, by far, the worst I’ve read in the several papers I have reviewed for CoRL and is certainly not up to par with the intent of the prompt provided by the conference organizers. This section is 6 sentences of which the first is a paper summary and the second is a filler. The second stated limitation is particularly sticking out. The purpose of the paper is to mitigate the requirements for data collection since it is expensive; however, the model needs to be retrained per new object. This is an important limitation that requires much more discussion than a throw away citation. I don’t really see the conclusion section adding any value to the paper, there is no new information presented. One approach could be to remove it and have a proper discussion and limitations section. Finally, I would want to see a clear statement on the biggest assumption: rigidity of the object. The tactile signature is only really predictable from the image because the object does not comply. This should receive far more attention and be clearly stated + implications noted.\n\nGeneral Comments:\n\nThe manuscript provides some encouraging preliminary results; however, without a core novel contribution (whether it is algorithmic, architecture, simulation/modeling or something to this effect) it is not ready for publication.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The manuscript provides some encouraging preliminary results; however, without a core novel contribution (whether it is algorithmic, architecture, simulation/modeling or something to this effect) it is not ready for publication. Simply put, concatenating two existing approaches with essentially no modification for the purpose of an application is not a sufficient contribution. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the weaknesses section, issues are clearly discussed there.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)