Haptics-based Curiosity for Sparse-reward Tasks
Authors: Sai Rajeswar
Cyril Ibrahim
Nitin Surya
Florian Golemo
David Vazquez
Aaron Courville
Pedro O. Pinheiro
Venue: CoRL2021 Poster (2021)
Keywords: Intrinsic Motivation
Touch
Curiosity
Manipulation

Abstract: Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in haptics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (\method{}), learns what visible objects interactions are supposed to ``feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test our approach on a range of haptics-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (haptics- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.

Haptics-based Curiosity for Sparse-Reward Tasks
Sai Rajeswar" Cyril Ibrahim²; Nitin Surya', Florian Golemo 1.2,3 David Vazquez', Aaron Courville 1.2.4 Pedro O. Pinheiro Montrral Institute for Learning Algorithms, Universitéé de Montréal, Element AI, Montréal, *CIFAR Fellow
Robots in many real-world settings have access to force/torque sen- sors their gripper and tactile sensing is often necessary for tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in hap- tics feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Haptics-based Curiosity (HaC), learns what visible objects interactions are supposed to "feel" like. We encourage exploration by rewarding interactions where the expectation and the experience do not match. We test approach on a range of haptics-intensin robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple ex- periments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-mddalle (haptics- or vision-only approaches as well as other curiosity-basee based methods and find that our method performs better and is more sample-efficien
Abstract:
Introduction
Most successes in reinforcement learning (RL) come from games [1, or scenarios where the reward is strongly shaped [3, 4]. In the former, the environment is often fully observable, and the reward is dense and well-defined. reward functions. While it may be possible to hand-craft dense reward signals for many real-world tasks, we believe that is a worthwhile endeavor to investigate learning methods that do not require dense rewards.
the latter, large amount work is required to design useful
Closely related to the sparse rewards problem is the issue of exploration. One reason that traditional RL agents struggle with sparse-reward problems is a lack of exploration. An agent may not obtain useful rewards without an intuitive exploration strategy when rewards are sparse. Exploration based on intrinsic curiosity comes naturally to many animals and infants (who start crawling and exploring the environment at around months [5] and oftentimes even before they can crawl by using their hands and mouth to touch and probe objects). Touch is local experience and encodes accurate geometrical information while handling objects. Experimental studies in infants has suggested that tactile and visual sensory modalities play a central role in systematic learning of tasks related to object understanding, interaction and manipulation [6. 7].
Ideally, we would like our RL agents to explore the environment in an analogous self- fashion learn the dynamics and object properties, and use this knowledge to solve downstream tasks. Just as how humans utilize different sensory modalities to explore and understand the world around them, exploration in robots should be more embodied and related potentially other sensor modalities. We believe that building autonomous agents that are self-driven and seek to explore via multi-modal interaction are crucial to address key issues developmental robotics.
combination of vision and touch and
Recent works in RL have focused a curiosity-driven exploration through mediction-bbsed sur- prise [8, 9, 10]. In this formulation, forward dynamics model predicts the future, and if its pre- diction incorrect when compared to the real future, the agent surprised and is thus rewarded. This encourages the agent to look for novel states while improving its visual forward model in re- turn. However, this formulation can be practically challenging to optimize since there are many
"Equal contribution. Correspondence to: Sai
Cacihaaammmmnnn umontreal.ca>>
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

states that are visually dissimilar but practically irrelevant (e.g. for a pushing task, moving a robotic end-effector without touching the object creates visual novelty but contributes little to task-related knowledge). One way to constrain this search space over curious behaviors is by involving another modality like haptics.
In this work, we demonstrate that a self-guided cross-modal exploration policy can help solve sparse- reward downstream tasks that existing methods without this curiosity struggle solve. Our method uses cross-modal consistency (mismatch between visual and haptic signal) to guide this exploration. To use self-play knowledge in downstream tasks, we relabel past experiences, providing a dense reward signal that allows modern off- off-policy RL methods to solve the tasks. While there are many existing methods that use artificial curiosity/inrrnsic motivation, the majority of these methods ei- ther rely on strong domain knowledge (e.g. labels of state dimensions in Forestier et al. [4], a goal- picking strategy in Andrychowicz et al. [11]) or are prone to get stuck in local optima when a single meaningless stimulus creates enough surprise to capture the attention of the agent (e.g. noisy-TV experiment from [12]). Other approaches depend on unrealistic assumptions and goal condition- ing [11]. Our method presents a novel approach in the family of prediction-based models [8, 13, 9] and yields better performance on a wide range of robotic manipulation tasks than purely vision- based and haptics-based approaches [8, 9]. The tasks are chosen with careful consideration- comprise of preliminary robotic manipulations such as grasping, pushing, and pulling. In this work, present the following contributions:
A new curiosity method to help solve sparse-reward tasks that cross- modal consistency (predicting one modality from another) to guide exploration. We implement in this work as vision and touch modalities, but the formulation of our method does not require any knowledge about the underlying modalities and can thus be applied to other settings.
We create and maintain a manipulation benchmark of simulated tasks, MiniTouch, inspired by Chen al. [14), Andrychowicz et al. [11], where the robotic is equipped with force/torque sensor. This allows evaluation of models' performance on different manipula- tion tasks that can leverage cross-modal learning.
We validate the performance of our method on MinITouch environment comprising of four downstream tasks. We compare purely vision-based curiosity approaches and standard off- policy RL algorithms. Our method improves both performance and sample efficiency.
2 Related Work
Intrinsic Motivation
Intrinsic motivation is an inherent spontaneous tendency to be curious or to seek something novel in order to further enhance one's skill and knowledge [15, 16, 17]. This prin- ciple is shown to work well even in the absence of well-defined goal objective. In reinforcement learning, intrinsic motivation has been a well-researched topic [18, 19, 20, 21, 22, 23). An intuitive way to perform intrinsic motivation is through the use of "novelty discovery". For example, incen- tivize the RL agent to visit unusual states or states with substantial information gain [24]. In its sim- plest form, this can be achieved with up- ueieetttting mechanisms such state visitation counts [25]. Count-based methods have also been extended to high sigh-dimensional state spaces [26, 12, 27]. Alter- native forms of intrinsic motivation include disagreement [28], empowerment [29, 30].
Exploratory intrinsic motivation can also be achieved through "curiosity" [18, 31]. In this setting, an agent is encouraged to visit states with high predictive errors [10, 8, 9] by training forward dynamics model that predicts the future state given the current state and action. Instead of making predictions in the raw visual space, Pathak et al. [9] mapped images to feature space where relevant information is represented via an inverse dynamics model. Burda et al. [8] demonstrate that random features sufficient for many popular RL game benchmarks. This approach may work well with tasks that require navigation to find a reward because each unseen position of the agent in the world leads to high intrinsic reward when unseen. However, in the case of manipulative tasks, we are less interested in the robot visiting all the possible states and more interested in states where the robot interacts with other objects. In this work, we leverage multimodal inputs that encourage the agent to find novel combinations of visual and force/torque modalities.
Projjcct website:

---

Forward model
Visual encoder
Haptics decoder
MSE loss
2++1
Visual input
Zz
Ze+1
(a) Haptics Control module
(b) Forward module
Figure 1: Haptics-based Curiosity Model. (a) The input image x; time / transformed into a 256-dimensional feature vector 2 = enc(as using a CNN encoder. The haptics decoder network predicts corresponding force/torque vector h, = dec(z;). The L2 norm between predicted haptics used as exploration reward. (b) To stabilize training, an additional network is used to predict the forward dynamics, and the difference between predicted next latent state 2 and actual next latent state 2t+1 used weighted additional term in the reward.
hy and observed haptics
Self-Supervised Learning via Cross-modality Exploiting multimodality to learn unsupervised representations dates back at least 1993 [32]. Multimodal signals are naturally suitable for self- supervised learning, as information from one modality can be used to supervise learning for another modality [33, 34). Different modalities typically carry different information, e-g., visual and touch sensory modalities emerge concurrently and often in an interrelated manner during contact- manipulation tasks [35]. Specifically, force/torque motor signal has always been major component in the literature of perception and control [36, 37, 38].
The most common ways leverage multimodal signals to learn representations through vision and language [39, 40]. Gao et al. [41], Li et al. [42] demonstrated a unified approach to learning representations for prediction tasks using visual and touch data. Chen al. [43] learn world-models from multimodal data via shared latent space. In robotics and interactive settings, the use of modalities such tactile sensing [44, 45, 46] is increasingly popular for grasping. manipulation and other externally-speciffed tasks [47, 48]. Lee et al. [49] showed the effectiveness of self-supervised training of tactile and visual representations by demonstrating its use on peg insertion task.
While the mentioned approaches have used multiple sensory modalities for learning better represen- tations, in this work demonstrate its utility for allowing agents explore. Similar ours, Dean et al. [50] use multimodal sensory association (i.e. audio and visual) to compute the intrinsic re- ward. Their curiosity-basee bassed formulation allows agent to efficiently explore the environment in settings where audio and visual signals are governed by the same physical processes. In addition to the different nature of sensory signals, they a discriminator that determines whether observed multimodal pair is novel. This might not work in our case as touch is a more sparse signal and using discriminator could lead to ambiguous outcomes.
Proposed Approach
The goal of our method, Haptics-Based Curiosity (HaC), is to encourage the agent to interact with objects. HaC provides reward signal for an RI. agent to explore the state space of task that involves interacting with objects. The exploration phase is independent of the downstream task, i.e., relying solely on visual and force/torque signals, without a reward signal from the downstream task.
Similar to how people spend more time exploring stimuli that are more incongruous [7]. HaC guides the agent to focus its experience on different novel moos-modal associations. We augment this intrinsic objective with a future visual state prediction objective similar to the one in Pathak et al. [9] avoid getting stuck in undesired inactive configurations. Note that we sometimes refer the future state prediction objective in the text as forward dynamics objective. In this work, we focus on the cross-modality between vision and touch, but the same idea could applied other pairs of sensory domains, such as vision and sound, or touch and acoustics.
3.1 Problem Formulation
The learning problem formalized as Markov decision process (MDP) defined by tuple (S,A, p,7) of states, actions, transition probability, reward, initial state distribution, and

---

discount factor. The goal is to find the optimal policy that maximizes the discounted sum of re- wards, π E Ex 3/r(s,, ar]]. . In our case, each state 8 the trajectory is composed of both visual and corresponding touch features as detailed in the following section. We use soft actor- critic policy gradients (SAC) [51] to train our policies, but in principle, our proposed approach algorithm-agnostic. The policy π is evaluated with an estimation of the soft Q-value:
((5 ax) _ r(st, a.)
- (V(+1))
(1)
where V(s:) (QQss a.) log (T (arls))) is the soft value function.
3.2 Haptics-based Prediction
Our core prediction framework consists of two modules: (i) a haptics control module, which learns to predict expected haptic signal from the visual input, and (ii) forward dynamics model, which predicts the next latent state from the current latent state and the current action (see Figure 1). Let the state of the environment s; = (ze, h2) at time t be composed of visual feature 2 (the encoded visual input) and haptic signal ht. The touch prediction model consists of a convolutional encoder 2, - enc(x:) that transforms the image xe into latent representation ze and fully-connected decoder h, dec(z;) that transforms the latent into predicted haptic signal ht. The encoder- decoder is trained with an L2 reconstruction loss, i.e. for every image Hy and force/torque sensor h4:
Lhaptics = [he h1
-
(2)
A high prediction error on a given image indicates that the agent has had few interactions like this. Therefore, to harness this "surprise" to guide exploration, we define the intrinsic reward at time t during exploration to be proportional to this reconstruction loss. This essentially allows the agent policy to visit under-explored configurations of the state space by encouraging interactions where the system does not know what the target object "feels" like. In addition to efficient exploration, being aware of such incongruity via touch prediction aids learning local regularities in the visual input. This in turn could assist better generalization to unseen states. An overall pipeline of the framework is shown in Figure la.
3.3 Regularization Through Forward Dynamics Model
We found empirically (and we demonstrate in the experiments section below) that the surprise stem- ming from haptic novelty was not enough to cause object-centric interaction. We postulate that by incorporating visual surprise (i.e. the mismatch between predicted forward dynamics and observed dynamics) [9], we can create an agent that seeks out visual novelty as well as haptic one and thus leads to better state space coverage. To this end, model is augmented with forward dynamics model (see Figure- 1b) 2 =
10:1=-ffmmi fdm(2,, a,) that learns to map the latent state 2, (obtained from the visual encoder enc) and action a at time to the predicted latent state 2 1 at the next timestep. This model is also trained with I.2 loss:
Lidm= = |1 2441l12 [lfdm(enc(z:), a.) enc(#1+1)ll2
(3)
The intrinsic reward is defined the convex combination of the cross modal prediction loss (Eq. 2) and the forward dynamics model loss (Eq. 3):
Ti (1 A) Lhaptics + X L Lfam
where À E [0, 1] is a balancing factor. The effect of the factor on overall performance is studied in the ablation experiments described in Section 5.1.
3.4 Training
Learning is divided into two stages: (i) an exploratory step, where the agent performs free explo- ration following HaC, and (ii) adaptation step, where the agent is tasked to solve downstream problem, given a sparse reward.
During the exploratory step, each trajectory consists of pairs of image and force/torque features, (=1, h1 ), (22, h2).... (zn, h.)) These trajectories are used for two purposes: (i) updating the parameters of the prediction model to help shape the representations and (ii updating the exploration policy

---

Pushing
Pickeup
Opening
Playing
Figure 2: 'MiniTouch'' benchmark tasks include opening door, pushing object to target, grasping and lifting an object and a toy task in which object interactions are counted.
based on the intrinsic reward rs. Note that we use the touch features only to craft the intrinsic rewards and the input to RL agent consists of visual features alone to be comparable the baselines. For vision-based curiosity models, Burda et al. [8] observed that encoding visual features via a random network constitute simple and effective strategy compared learned features. In Section 5.1, we investigate the performance of our model in both scenarios, i.e., when the features are learned vs random. The overall optimization problem this step consists of the policy learning (driven by intrinsic reward), the touch reconstruction loss (Eq. 2), and the forward dynamics loss (Eq. 3). During the downstream adaptation step, the parameters of the policy network, the Q network and the replay buffer are retained from the exploratory phase. The objective of the down-stream task is computed as:
min [- ri]]
(5)
where rf in this phase is task-specific external sparse reward. In both steps, the objectives optimized with Adam [52].
Implementation Details: The encoder is four-layered strided CNN followed by a fully connected network. We use LeakyRece ReLU [53] as non-lincar activation in all the layers. The decoder network is two-layered MLP that maps 256-dimensional visual features to touch vectors. For more detailed description of the networks (SAC policy network and HaC networks, including the forward model) and syper-parametent please refer to the supplementary material Section 7.3.
Experimental Setting
Our experiments focus on a tabletop robot manipulation from raw image observations and raw force/torque sensory values, which we refer as "touch vector". For our experiments, 7-DoF Franka Emika Panda arm with two-finger parallel gripper. Each of the fingers is equipped with a simulated force/torque sensor that measures the joint reaction force applied We utilize PyBullet [54] to simulate the robot arm and haptic sensor'.
4.1 TiniTouch Benchmark
Our proposed benchmark, MoncTouch, consists of four manipulation tasks: Playing, Pushing, Open- ing, Pick-up. Each of the tasks along with corresponding actions, observations, and rewards described in detail in the Supplementary Material Section 7.1 and further illustrated in Figure 2. MiniTouch is an active repository" and we expect to update the benchmark with new tasks and datasets. The tasks are inspired by Yu al. [55] but are not based proprictary simulator, feature an arm that we have access to for real-world experimentation (in follow-up work), and where the arm is equipped with haptic sensor.
4.2 Baseline Comparisons and Metrics
For the task evaluation, study two versions of our model: (i) HaC-Pure considering the touch tor reconstruction intrinsic reward alone, and (ii) HaC, considering the full intrinsic reward (Eq. 4). We compare with several well-known intrinsic exploration baselines based visual features:
SAC: The unmodified Soft Actor-Crit Critic algorithm from Haarnoja et al. [51].
ICM: SAC augmented with the state-of-the oft visual curiosity approach Intrinsic Curiosity
Module (ICM) [12], which uses a visual prediction model guide exploration.
Based on code from official Franka Emika repo http::/ggttubb.cm//eeeemnn////
5

---

Orject Movement
Truch interaciion
020
410
03
220
Sep
0.3
05
3.0
3.5
2.
(a)
(h)
(curs)
HoC (ours)
Disagroement
Figure 3: Object Interaction To quantitatively evaluate agent's object interaction, consider both haptics interaction and object displacement. (a) shows the object movement evaluation (avg displacement in mm from the starting point) of our method compared with the baseline methods over training steps; (b) depicts the number of touch interactions, evaluated on the "playing" task. (c) Heat maps showing the object location the beginning (left) and end (right) of the HaC training. The object locations are spread-out towards the end indicating interactive movement.
Disagreement: It uses model disagreement as objective for exploration [56, 28). It lever- ages variance in the prediction of an ensemble of latent dynamics models the reward.
RND: Random Network Distillation [57] utilizes a randomly initialized neural network to specify an intrinsic reward for visiting unexplored states in hard exploration problems.
We built a Pytorch [58] version of these baselines based on their open source code (details in sup- plementary material). We use the following metrics to evaluate method and the baseline models:
Exploration success: measures the percentage of times that the agent attained the goal state exploratory phase, i.e. with no external reward. Higher is better. Success: denotes the percentage of times that the agent attained the goal state during the down- stream task phase.
Episode steps: The number of steps required for each episode to succeed. This metric an indicator of sample efficiency. The lesser the number of steps, the faster the agent's ability to succeed.
Touch-interaction: Amount of interaction the agent's fingers have with the underlying object. We measure this by computing the variance of force/torque sensory signal across the whole episode. Object movement: The agent can resort to constantly engaging with the object unnecessarily to satisfy the objective. We, therefore, monitor the variance of door angle (for the Opening task) and the variance of object position (for the remaining tasks) over the course of training. A higher movement indicates diverse state space in addition physical interaction.
Results and Discussion
HaC and baselines were trained on a Panda robot agent [54] for one million steps. In the exploratory phase of the training. pre-train method only with the curiosity-biie -based intrinsic reward. We then progress to the adaptation phase. Also, note that across all tasks, HaC-Pure is based purely on cross-modal prediction, while HaC includes visual forward prediction reward in addition.
Figure 3a and 3b shows results on the basic task of playing with a single object. Since single object interaction does not have explicit goal states to evaluate, we instead measure the agent's ability to constantly engage and play with the underlying object. HaC-Pure displays four times better interaction with the underlying object when compared to SAC (see Figure 3b). Note from the plots that there is a trade-off when using HaC and HaC-Pure between constant interaction (i.e. touch interaction performance) and object movement dynamics. Collecting variety of such interesting data during the exploratory phase helps the agent in terms of sample efficiency while solving the downstream tasks. Figure 8 shows similar comparison for Open-door task. In the following section, we examine the role the forward objective term on touch interaction and possible ways to encode agent's is observation in ablation studies.
We compare HaC and HaC-Pure with SAC and sfate-of-the-eee vision-basee curiosity baselines on the remaining downstream tasks in MiniTouch. From Figure 4 it is evident that HaC and HaC- Pure perform better than SAC in all the tasks and better than the vision-based curiosity models in the majority of the tasks. Using SAC alone hinders the performance and is often unable to solve any of the three tasks. This not surprising since the model not motivated enough to collect diverse and useful data through interaction. We hypothesize that HaC-Pure without the visual

---

Expiiaation Succesa
Epecoda Staps
Reach intracees
200
100
Pushing
04
22
0.1
000
2.0
2.5
1.0
slap
230
2.
Expleration Success
9,00095
Ephoode Strpp
Touch interctioon
Opening
0n
0.
022
200
2.0
skep
Exploration Success
Sacowes
Eypeode
Touch Intercdioo
1000
Rick
0.11
500
0000
165
165
HaC- (ours)
HaC (ours)
ICM
Disagreement
RND
SAC
Figure 4: MiniTooch Evaluations. Each row in the figure outlines the performance of the HaC
variants and the baselines on a MiniTouchh task. Each column marks performance on the four spec- ified evaluation metries are averaged across 5 random seeds and shaded areas represent mean + one standard deviation while darker line represents the mean. In the majority of the tasks, HaC agents attain success in the ploratory phase with no external reward (see text). Note: We exclude the single object playing task success in this task is equivalent object interaction as depicted in Figure. 3.
number of training steps the x-axis expressed in 1c5, The results
Oiject Novement
620
Truch Interaction
Object
Trutt Interaction
400
600
0.3
400
3002
200
0.0
165 0.5
145
135
185
(a)
(b)
0.5 0.7
0.9
Figure 5: Forward Prediction weight. Touch and Object movement evaluated on the (a) playing task and (b) Open-door task for different forward prediction weightings sampled [1, Oj. Large weight (darker plot) favors object-movement towards the end of training, where as smaller weight improves touch-interaction. Middle value in the sweep range (e.g. 0.5) balances the trade-off. Note that movement is measured as distance(mm) for the Playing and angular distance for Open-door.
prediction objective Lydm could potentially bias the solutions towards more physical interaction, which is not necessary for every task. For instance, constant interaction in the pushing task could be seen as a hindrance as does not the object slide easily towards the target. Observe that ICM performs better than HaC-Pure in the pushing task, however, HaC dominates in performance by about 15%. Recall that the goal is not just to succeed but to help attain success in sample efficient manner in fewer steps. The results support our hypothesis that cross-modal curiosity enables an RI. agent to succeed at an early stage in training and often without any external reward. Similarly, our model outperforms success compared to RND but surpasses RND towards the end. Although HaC and HaC-Pure attain similar success in the pick-up task towards the end of the training, it is compelling note that HaC- Pure attains faster convergence. This is because the picking task requires constant touch interaction (where HaC-Pure has advantage), as opposed to diverse object movement.
the opening task without external reward. However, HaC initially has lower
7

---

Metric
Pushing
Open Door
Pick-up
Playing
HaC
ICM+hapticr
ICM
HaC
ICM+hoptics
ICM
HaC
ICM-hapticr ICM
HaC
ICM+hapiics ICM
Exploration 1
0.403
0.291
0.187
0.669
0.355
0.083
0.063
0.051
0.013
Success 1 0.733
0.678
0.597
0.983
0.571
0.114
0.891
0.825
0.780
Episode steps J.
$7.84
87.61
95.24
23.34
97.10
199.3
30.54
33.77
42.19
Touch-interaction 247.79
210.11
202.66
600.1
287.97
43.56
980.7
984.2
358.15
267.021
Table I: Haptics-based future prediction Table compares the mean evaluations for HaC and
ICM+haptics on all the four tasks emphasizing the importance of cross-modal association(see text).
Figure 6: Comparing different latent space performance pushing and grasping tasks. Features learned via haptics prediction per- form better than those learned using IDF or Random.
Liker Nallang
HM2
03
HCC E
Inc
5.1 Ablations
Forward objective and tuning Lambda Visual forward prediction , plays important role when is used in the right proportion. Our intrinsic reward is a weighted combination of cross- modal prediction and forward prediction as defined in Eq.4. Figure 5 illustrates the model behavior with different levels of emphasis on the forward loss term, with À uniformly sampled between 0 and 1. Higher weights indicate that the future prediction dominates force/torque prediction. This leads to more object movement but lesser robot's 's constant touch-interaction with the underlying objects. This is useful in tasks such as Pushing. A smaller À value leads better inactive behavior which is handy in tasks such as grasping. In our experiments we choose an intermediate value (e.g. 0.5) that works best for all of our tasks.
Haptics in future prediction. The goal of this experiment is to strengthen the argument of cross- modal association. While conducting experiments, it is important to deduce information of one modality from another modality in a related manner than simply adding another modality on top of visual information. We created an additional baseline, ICM+haptics, where in addition to the visual prediction model we include the haptics-based future prediction model. We hypothesize ICM+haptics to perform better than ICM as it has additional information (haptics). The haptics- based future prediction model takes touch vector as input and predicts the touch vector for the next time step. Table I compares HaC and ICM+haptics on MiniTouch tasks and we observe that ICM+haptics is better than ICM but compares below HaC. Table 4 in the supplementary compares RND based baseline.
Latent features for forward dynamics. Choosing ideal embedding space for decoding the touch vector (Figure la) and for predicting future state (Figure. 1b) is important. Existing approaches rely on a pretext inverse dynamics (ID) task of predicting the agent's action given its current and next states [9]. Another simple yet strong method is use features from random but fixed initialization of the encoder [12]. In our work, we learn the features by leveraging the self-supervised pretext task of predicting one modality from the other. Figure compares (1) encoder, i.e. learned through cross- modal prediction(HaC) (2) random feature encoder(HaC-RF)) (3) encoder learned through ID task (HaC-IDF). In each case, the decoder network is optimized through touch prediction. We observe that the random features variant is stable and effective on both HaC and ICM models.
6 Conclusion
We formulated Haptics-based Curiosity (HaC) aimed at encouraging exploration via haptic interac- tion with the environment. We demonstrated in this work, that by involving additional modalities, the performance of curiosity-based systems on downstream tasks can be increased. We observed increased interaction with target objects, and presented evidence that HaC learns to solve the Mini- Touch benchmark tasks in an efficient manner while vanilla RL algorithms and vision-based curios- ity formulations struggled. Force/torque sensing is widespread in the lab and industrial robots but while there are plenty of robotic benchmarks, we believe that tactile feedback is an under-explored modality and by releasing our benchmark, we hope enable future research in this exciting area.
8

---

Acknowledgements
We would like to sincerely thank Glen Berseth, Deepak Pathak, Edward Smith and Krishna Murthy for valuable feedback and insightful comments on the paper.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. CoRR, 2013.
[2] D. Silver, A. Huang. C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, Schrittwieser,
I. Antonoglou, V. Panneershelvam, and M. Lanctot. Mastering the game of go with deep neural networks and tree search. nature, 2016.
[3] Y. Zhu, Z. Wang, J. Merel, A. A. Rusu, T. Erez, S. Cabi, T. Saran, Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, 2018.
[4] S. Forestier, Y. Mollard, and P. Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. CoRR, 2017.
[5] D. Vernon, C. Hofsten, and L. Fadiga. A roadmap for cognitive development in humanoid robots. 978-3-642-16903--, 11, 2011.
[6] A. E and R. CH. The importance of touch in development. Paediatr Child Health, 2010;
[7] M. R. Connolly and L.. Harris. Effects of stimulus incongruity on children's curiosity as mea- sured by looking time and expression change. Psychonomic Science, 1971.
[8] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Largescale study of curiosity- learning. ICLR, 2019.
[9] D. Pathak, Agrawal, A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. ICML, 2017.
[10] R. Raileanu and T. Rocktäschel. RIDE: Rewarding impact-driven exploration for procedural- lygenerated environments. ICLR, 2020.
[11] M. Andrychowicz, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, McGrew, J. Tobin,
O, Pieter Abbecl, and W. Zaremba. Hindsight experience replay. In NeurIPS, 2017.
[12] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. ICML, 2019.
[13] S. H. Huang, M. Zambelli, J. Kay, M. F. Martins, Y. Tassa, P. M. Pilarski, and R. Hadsell. Learning gentle object manipulation with curiosity-driven deep reinforcement learning. CoRR, 2019.
[14] A. S. Chen, Nam, S. Nair, and C. Finn. Batch exploration with examples for scalable robotic reinforcement learning. arXiv, 2020.
[15] R. White. Motivation reconsidered: the concept of competence. Psychological review, 1959.
[16] A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In International Conference Development and Learning[CCDL), 2004.
[17] S. Di Domenico and R. M. Ryan. The emerging neuroscience of intrinsic motivation; A new frontier in self-determination research. Frontiers in Human Neuroscience, 2017.
[18] J. Schmidhuber. A possibility for implementing curiosity and boredom in model- model-building neural controllers. ICSAB: From animals animats, 1991.
[19] P-Y. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions evolutionary computation, 2007.

---

[20] P.-Y. Oudeyer and F. Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in neurorobotics, 2009.
[21] N. Lair, Colas, R. P. J.-.M Dussoux, P. F. Dominey, and P.-Y. Oudeyer. Language grounding through social interactions and curiosity-driven multi-goal learni. arXiv, 2019.
[22] K. Marino, A. Gupta, R. Fergus, and A. Szlam. Hierarchical rl using an ensemble of proprio- ceptive periodic policies. ICLR, 2019.
[23] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly. Episodic curiosity through reachability. aarXiv preprint arXiv: /8 1810.0227, 2018.
[24] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel. Curiosity-driven exploration in deep reinforcement learning via bayesian neural networks. CoRR, 2016.
[25] A. L. Strehl and M. L. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 2008.
[26] M. Bellemare. S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. NeurIPS, 2016.
[27] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. JCML, 2017.
[28] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via self-supervised world models. In ICML, 2020.
[29] A. S. Klyubin, D. Polani, and D. L. Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, 2005.
[30] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. CoRR, 2016.
[31] R. Dubey and T. L. Griffiths. Understanding exploration in humans and machines by formal- izing the function of curiosity. Current Opinion in Behavioral Sciences, 2020.
[32] V. R. de Sa. Learning classification with unlabeled data. In NeurlPS, 1993.
[33] R. Gao, R. Feris, and K. Grauman. Learning to separate object sounds by watching unlabeled video. ECCV, 2018.
[34] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. ECCV, 2018.
[35] R. Blake, K. V. Sobel, and T. W. James. Neural synergy between kinetic vision and touch. Psychological Science, 15:397-402, 2004.
[36] M. Kalakrishnan, L. Righetti, P. Pastor, and Schaal. Learning force control policies for compliant manipulation. In IROS, 2011.
[37] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[38] G.-H. Liu, A. Siravuru, S. Prabhakar, M. Veloso, and G. Kantor. Learning end-to-end multi- modal sensor policies for autonomous navigation. In CoRL, 2017.
[39] N. Srivastava and R. R. Salakhutdinov. Multimodal learning with deep boltzmann machines. NeurIPS, 2012.
[40] Z. Gan, Y.-. Chen, L. Li, C. Zhu, Y. Cheng. and J. J. Liu. Large-scale adversarial training for vision-andd-lagguage representation learning. In NewrIPS, 2020.
[41] Y. Gao, L. A. Hendricks, K. J. Kuchenhecker, and T. Darrell. Deep learning for tactile under- standing from visual and haptic data. ICRA, 2016.
[42] Y. Li, Y. 7hu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In CVPR, 2019.
10

---

[43] K. Chen, Y. Lee, and H. Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. ICRA, 2021.
[44] H. van Hoof, N. Chen, M. Karl, P. van der Smagt, and Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In IROS, 2016.
[45] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 2018.
[46] A. Murali, Y. Li, D. Gandhi, and A. Gupta. Learning grasp without seeing. International Symposium on Experimental Robotics, 2018.
[47] L. Pape, C. M. Oddo, M. Controzzi. C. Cipriani, A. Förster, M. C. Carrozza, and J. Schmid- huber. Learning tactile skills through curious exploration. Frontiers in Neurorobotics, 6:6, 2012.
[48] J.-T. Lee, D. Bollegala, and S. Luo. "touching to see" and "seeing to feel": Robotic cross- modal sensory data generation for visual-tactile perception. ICRA, 2019.
[49] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. ICRA, 2019.
[50] V. Dean, S. Tulsiani, and A. Gupta. See, hear, explore: Curiosity via audio-visual association. In NeurlPS, 2020.
[51] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 1O0I:0/800.022000 2018.
[52] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and
Y. LeCun, editors, ICLR, 2015.
[53] B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolu- tional network. arxiv, 2015.
[54] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016-2019.
[55] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CORR, 2019.
[56] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In ICML, 2019.
[57] Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov. Exploration by random network distilla- tion. ICLR, 2019.
[58] A. Paszke, S. Gross, Massa, and et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
[59] V. Mnih, A. P. Badia, M. Mirza, A. Graves, Lillicrap. T. Harley, D. Silver, and
K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.


Reviews:
[{"review_id": "vqzQ6XHOSiO", "reviewer": "Reviewer_5q2G", "summary": "This paper develops a touch-based curiosity approach for touch-intensive robot manipulation tasks with sparse rewards. The paper describes a two-stage learning approach where in the first stage, exploration is performed based on cross-modal (visual and touch) consistency and mismatch that inherently encourages robot-object physical interactions and in the second stage, the parameters from the exploration stage are re-used to achieve a downstream task with a task-specific sparse reward.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Exploration based on cross-modal consistency that encourages physical interaction based on haptic modality mismatch and augmenting it with visual surprise based on a forward dynamics model\n- Consideration of a variety of metrics including object movement to avoid passive physical interactions\n- A benchmark with four simulated manipulation tasks with F/T sensor data: playing, pushing, opening, and picking up\n- Baseline comparison with single-modality approach\n\nQuestions / Suggestions for Improvement:\n\n- In haptics, sensing inherently depends on action. What this means is, forces during a physical interaction are not only dependent on object states (object physical characteristics) but also on robot states (what was the impact velocity etc.). The touch-control module, which maps visual signals to haptic signals for checking haptic modality mismatch does not consider robot states explicitly for this mapping. Can the paper comment on how will the cross-modal consistency generalize across different robot control behaviors (for example: different robot velocities for pushing etc.)?\n\n- What happens if the task success reward during the second phase is actually simultaneously used with the touch interaction reward in a single stage itself? Can a comparison be done with the two-stage approach? \n\n- Can the parameters from the exploratory phase be reused across multiple tasks if all it’s getting is touch experience from visual inputs? Insights would be useful. For example, exploratory tasks would just focus on playing with the object and learning more about the visual-haptic mapping. And then, you use that to perform various tasks with the object such as pushing, opening, pick-and-place etc.?\n\n- Can the paper discuss how the methods would deal with physical interactions that may require repeated make and break of contacts, such as in some in-hand manipulation scenarios?\n\n- In a related way, do the methods bias the solutions towards more physical interaction even if they are not necessary? For example, a pushing task can also be achieved by increasing the robot’s impact velocity and letting the object slide on a table-top once the model has a good understanding of the physics. Can the paper discuss this and provide some insights?\n\n- Some more insights about the results would make the paper stronger such as: 1) Why is TOC-future worse than TOC? 2) Why does the visual-curiosity based approach (ICM) perform better than TOC-pure (touch-based curiosity) for the pushing task? Is it because for the pushing task, object movement could probably be more crucial?\n\n- “Each of the fingers is equipped with a simulated force/torque sensor that measures the joint reaction force applied to it” - ? Why is there a 6D F/T sensor on each finger instead of one at the wrist or a simple tactile sensor on each finger?\n\n- Fig.1 - caption: \\hat{z_t} -> \\hat{z^_{t+1}}\n\n- It would be very helpful to the readers if the graphs are made more legible with bigger fonts, larger frame sizes etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper describes an interesting two-stage learning approach which encourages physical interaction by considering cross-modal consistency and forward dynamics in the exploratory stage and sparse task rewards in the adaptation stage. As long as the limitations of the approach are clearly identified and the points raised above are discussed in the revised manuscript, I would be in favor of publication of this manuscript.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See “Suggestions for Improvement” section above.\n", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "VmNzux7Gsj", "reviewer": "Reviewer_2Equ", "summary": "The work presents an approach that considers visuo-tactile representations for addressing tasks such as pushing objects and opening door, etc. (in simulation). In the proposed method, cross-modal consistency between visual and touch signals is used to guide exploration in reinforcement learning tasks, i.e., mismatches between visual features and touch representations will lead to surprises and more exploration. Two variants of the method are studied: Touch-based Curiosity (ToC) and Pure ToC, i.e., with the intrinsic reward being derived both from the visuo-tactile representations and vision-based predictive model, and visuo-tactile representations only. The experiments show that this approach can obtain better results than the standard (vision-based) Soft Actor-Critic algorithm and the state-of-the-art vision-based curiosity method ICM (SAC +  Intrinsic Curiosity Module).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The idea of cross-modal curiosity for reinforcement learning tasks is interesting.\n2. The experiments show that this approach can obtain better results than the standard (vision-based) Soft Actor-Critic algorithm and the state-of-the-art vision-based curiosity method ICM (SAC +  Intrinsic Curiosity Module).\n\nWeaknesses:\n1. Only simulation experiments are presented and no real robot data have been used in the experiments.\n2. The presentation of the paper can be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work presents an interesting idea of cross-modal curiosity for reinforcement learning tasks but it lacks proper justification of the motivation of cross-modal curiosity and needs further validation on the real robots.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "1. Title: Touch-based isn't entirely correct as the work doesn't consider any touch sensors per se and rather a force/torque vector.\n2. A few settings of the methods are not clear. Why are the touch features only used to craft the intrinsic rewards? Why does the input to RL agent consists of visual features alone? If both touch features and visual features are taken as the input to the RL agent, the performance of the RL agent may be better. As for the loss functions, why was the L2 reconstruction loss selected instead of L1 loss? \n3. Is there a better way to \u0000get optimal lamda (Eq. 4) than uniformly sampling it between 0 and 1? Also, it may not be optimal to use a fixed lamda, has a changeable been attempted to maximize the performance?\n4. Introduction / Paragraph 2: It would be interesting to find an argument for exploration from a computation standpoint. What's the practical advantage of animals and infants being curious? Why is it preferable to random exploration?\n5. Figure 1:  the representation of the loss function should be consistent between figure (a) and (b), to ease the paper reading.\n6. In the caption of Figure 1, \"actual next latent state zˆ_t\" should be \"actual next latent state z_{t+1}\".\n7. Related Work / Paragraph 4 (101-107) should be revised, other works have studied visuo-tactile representations.\nLee, J.T., Bollegala, D. and Luo, S., 2019, May. “Touching to see” and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 4276-4282). IEEE.\n8. Related Work / Paragraph 5 (114-115) \"This might not work in our case as touch is a more sparse signal and using a discriminator could lead to ambiguous outcomes.\" This sentence seems to be ambiguous, as it is dependent on the used sensor, and not backed by any evidence.\n9. Problem Formulation (132) is not correct, pi_star is the optimal policy that maximizes the discounted sum of rewards.\n10. Figure 2: It is not clear what (c) and (d) mean.\n11. Figure 4: Which one is better? More touch information and more object movement? Why? Why are there 4 plots (either angle and mm)?\n12. Figure 5(b) shows good generalization to novel objects, but there is no details about the seen dataset and unseen dataset. \n13. Table 1: The column refers to the method as ToC and ToC-fut, the caption as ToC and ToC Pure. In other parts of the paper, the same ambiguity persists e.g. 274-281.\n14. Lines: 290-298: What are random features? (e.g. reference the section) and what are IDF?\n15. Figure 5: It would be better to add extra spacing between figures (a) and (b) to help the interpretation, and use one single legend in figure (b)  to be consistent with figure (a).\n16. Some grammar issues:\nline 118: a RL agent -> an RL agent\n line 145: a L2 -> an L2"}}, {"review_id": "ILQWKHG_t3S", "reviewer": "Reviewer_vFAA", "summary": "This paper computes intrinsic motivation (i.e., artificial curiosity) by a vision and tactile sensing for improving the exploration in sparse reward function. Integrating tactile sensing is expected to guide the robot exploration in touch-intensive tasks better as the haptic information is more relevant to the rewards of these tasks. The proposed method trains an image encoder to transform an image input (x_t, where t denotes the timestep) to a latent vector z_t and then a decoder to reconstruct the haptic sensor inputs \\hat{h}_t by the latent vector z_t. Furthermore, the author finds that incorporating forward dynamics model prediction objectives in vision inputs can improve exploration performance. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The idea of cross-modality association is still underexplored in curiosity-driven exploration.\n- The author compares with plenty of baselines of curiosity-driven exploration algorithm.\n\nWeakness:\n- The experiment setup is unclear. For example, are the state inputs in both exploration and adaptation step the same? Do the baselines take touch and vision as state inputs? \n- Some descriptions are vague. For example, L242,  \"Figure 2a indicates that ToC is slightly more dynamic in the interaction\n243 in contrast to ToC-Pure. \". What is \"more dynamic\"?\n- Weak connection between the experiments and the proposed method. For example, wow does the proposed method ToC improve generalization to novel objects? Generalization is important but it seems to me ToC is not expected to improve generalization over novel objects. Also, it would be better to see the generalization ability of other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the idea is well-motivated, and the cross-modality curiosity is promising. Most curiosity algorithms only use single modality information. I think this paper did a good job of bringing up the attention on cross-modality association in curiosity-driven algorithms. That being said, the reasoning for the experimental results is unclear to me in the current version of the manuscript. I would like to adjust my rating after rebuttal.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "1. What are the inputs of the SAC, ICM, Disagreement, and RND? The text seems that they only take images as inputs for both exploration and adaptation steps. If so, the comparison might be unfair. The author tries to argue that cross-modality (touch and vision) “association” is beneficial for exploration and, therefore, improves adaptation. To claim this, the author should compare with the baselines that take both vision and tactile sensing as inputs but don’t encourage the association between tactile and visual inputs. It is necessary to compare with these baselines because the performance gain of ToC could be from using multi-modality inputs.\n\n2.  As the introduction said, you constrain the search space by touch because the task prior tells us that haptic novelty is more relevant to the tasks. That could be true in most touch-intensive tasks, but it might be misleading in some cases. For example, visually similar but materially different objects could constantly generate novelty to the robot and introduce noises in the reconstruction loss. I would like to see more discussion about this point.\nI’m not sure of the meaning of the heatmaps in Figure 2. I can’t find the reference in the text, and the caption is confusing. Could you elaborate more on these heatmaps?\n\n3. Did you train ICM, RND, and Disagreement with haptic features? That would be an important baseline for ToC. If ToC outperforms the baselines trained with haptic features or haptic + vision features, the importance of cross-modality association will be strengthened. The current experiments are not convincing to me that cross-modality association is the key for the performance gain in the experiments.\n\n4. In the adaptation step, is each agent trained by images only or images+touch?"}}, {"review_id": "85axhb2AoaY", "reviewer": "Reviewer_6jo7", "summary": "This paper presents a curiosity-based approach that leverages both vision and \"touch\" as a self-supervised pretext task. The task is to predict the observed haptics signal from the visual input (Fig 2).  Experiments show the approach leads to better performance on downstream sparse-reward RL tasks. The authors also contribute a benchmark (minitouch) can that be used with their method to pre-train networks and doesn't require a proprietary simulator. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Major comments\nIn general, this paper is clear and easy-to-follow. The key idea (cross-modal learning) is simple and easy to implement, and the experiments validate the approach to a good degree (up to the points below).  \n\nAs for weaknesses, the technical advance appears to be incremental given that previous work (e.g., those in Sec 2 and [47]) have shown the benefits of cross-modal learning for learning better representations. The paper also has many minor errors that should be corrected before it can be accepted. Please see below for specific questions/concerns:\n\n- line 147: Eq (2) and statement that \"A high prediction error on a given image indicates that the agent has had few interactions like this.\" This is not necessarily true as the image may not be informative of the tactile sensation, e.g., consider real textured surfaces v.s. a flat printed texture. Essentially, the underlying predictive distribution may be multi-modal (as in multiple peaks and not multiple sensors) and the prediction error may not be informative. \n- Can the authors confirm that the touch information is sparse, i.e., only occurs when there is contact? If so, then the decoder would have zero output for a large amount of the state space when there is nothing in contact with the arm? \n- In the experiments and Fig 5(b), the authors claim \"robustness to diverse shapes\": how different are the unseen objects? The performance variance on the unseen tasks appears larger.  \n- A short discussion of how the approach can be used in the real-world would help to increase the paper's impact. From the experiments, it seems a large number of steps are needed (20k)? Also, it's unclear if each \"training step\" is an episode or a time-step? The paper also mentions \"episode steps\" and it is unclear what is an episode is in the setup.\n- Fig 2 (c) and (d) are unclear. Are the x,y-axis coordinates? which coordinate frame? I don't understand what is meant by \"Robot’s ﬁngers are spread everywhere at starting while learn to focus near the object towards the end of training.\" Also, where is the object? at the top left? \n- Fig 3: the pushing task in the \"Success\" \"Episode steps\" and \"Touch interaction\"  seem to be over 1e6 steps rather than 1e5?\n- Fig 3. The episode steps appear to start at a different number of steps for ToC and ToC-pure? Are these episode steps collected from the adaptation phase? \n\n### Minor issues: \n- Tuning $\\lambda$ for the forward objective weight may be an issue and some discussion on how to pick it would help.\n- There is related work in learning tactile skills through exploration (Pape et al, 2012) and multimodal learning (Chen et al, 2021) that attempts to learn consistent representations from multiple modalities for RL.  \n- 188: \"two-layerd\"->\"two-layered\"\n- 214: \"Network Distiallation\" -> \"Network Distillation\"\n- Fig 3: \"darker line represents mean\" -> \"darker line represents the mean\"\n- Table 1 caption: \"Toc-Pure\" is \"ToC-future\"? \n- 257: \"to succeeds\" -> \"to succeed\"\n- 259: \"surpasses towards\" -> \"surpasses RND towards\" \n- 295: \"figure.5a\" -> \"Figure 5a\"\n- pg. 5 . Two footnotes with the same number (1). \n\n### References\n\n- Pape et al, 2012, Learning tactile skills through curious exploration, Frontiers on Neurorobotics. \n- Chen et al, 2021, Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning, ICRA 2021.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I like the idea of doing cross-modal learning. However, the paper is borderline and falls slightly short due to the limited technical novelty and the numerous minor errors.\n\n### Post Response \nThank you for responding to my comments. I have amended my recommendation accordingly. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above for specific issues. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a multi-modal algorithm to improve exploration in presence of torque/force readings

Quality:

(+) The experiments seem to support the hypothesis of the manuscript

(-) Reviewer vFAA highlighted several clarifications and additional experiments that would be nice to perform.

(-) Experiments are only performed in simulation

Clarity:

(+) The paper is overall well written

(-) All reviewers suggested several changes to the text to make it more accurate and clear.

(-) As remarked by Reviewer 2Equ the paper does not indeed make use of touch sensors, but only of torque/force sensors. The  title and a large part of the nomenclature in the paper is thus misleading and should be modified to accurately reflect the content of the paper

Originality:

(+) The mixture of curiosity-based systems and multi-modal sensing (vision and force/torque) is novel

Significance:

(+) The reviewers agree that the topic tackled in this manuscript is interesting and potentially impactful.

(-) Reviewer 6jo7 had concerns regarding the significance of the contribution compared to existing work 


In summary, while the paper deal with an interesting topic, the current manuscript seems to have several limitations, from experiments that should be improved, to clarity, and overall contribution of the paper. 


Minor comment:
Recently, several simulators for tactile sensors have been open-sourced (e.g., https://github.com/facebookresearch/tacto, https://github.com/robotology/gazebo-yarp-plugins/issues/55) why not using one of them instead of force/torque sensors?

---

After rebuttal:
The authors improved the paper and addressed most of the concerns from the reviewers.
There now seems to be agreement from the reviewers that the paper is sufficiently solid.
Although the title has not been changed in the manuscript, I do expect the authors to change it as agreed in the rebuttal, and remove all mentions of "touch sensing" from the manuscript.

Minor comment: Fig 4, 5, 6 are hardly readable, and completely unreadable when printed in B&W. You should consider making the figures larger and more readable.
Decision: Accept (Poster)