user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Classification in the dark using tactile exploration

Paper content:
Combining information from different sensory modalities to execute goal directed actions is a key aspect of human intelligence.
Specifically, human agents are very easily able to translate the task communicated in one sensory domain (say vision) into a representation that enables them to complete this task when they can only sense their environment using a separate sensory modality (say touch).
In order to build agents with similar capabilities, in this work we consider the problem of a retrieving a target object from a drawer.
The agent is provided with an image of a previously unseen object and it explores objects in the drawer using only tactile sensing to retrieve the object that was shown in the image without receiving any visual feedback.
Success at this task requires close integration of visual and tactile sensing.
We present a method for performing this task in a simulated environment using an anthropomorphic hand.
We hope that future research in the direction of combining sensory signals for acting will find the object retrieval from a drawer to be a useful benchmark problem
A core aspect of human intelligence is the ability to integrate and translate information between multiple sensory modalities to achieve an end goal.
For example, we have no trouble discriminating between a set of keys, a wallet or a watch kept in our pocket by simply feeling them with our hands.
Similarly, we can easily retrieve a desired object present inside a dark drawer even if we can't see the objects using touch sensation from our hands.
Not only can we retrieve a previously known object, but if we are shown an image of a previously unseen object, we would still have no trouble retrieving this object using only tactile exploration inside the drawer even in absence of any visual feedback.
Such translation of information between sensory modalities is not specific to tactile and vision, but is noticed between other modalities as well.
For instance, it is easy to imagine someone walking down the stairs and opening the door by simply hearing the sound that was generated.
These examples demonstrate how easily humans can translate information between sensory modalities.
Different sensory modalities provide a different view of the same underlying reality.
The ability to transform between sensory modalities therefore provides an interesting way to learn useful representations of sensory inputs.
Recent work in self-supervised learning has made extensive use of this observation and shown that useful visual features can be learned by predicting, from images, corresponding sounds BID18 , ego-motion BID1 BID11 , depth or even predicting color values from grayscale images BID23 .In
addition to learning feature representations, another and possibly more critical use of sensing from multiple modalities is performing goal directed actions in partially observable settings. In
the running example of retrieving objects from a drawer, the agent receives only the image of the object as input and in absence of any light source in the drawer, the agent solely relies on its tactile sensing to find the object. Other
examples are a pedestrian getting alerted when she hears the sound of a car coming from the back or animals in the jungle being alerted of a tiger behind the bushes by the sound of the movement. Yet another
example showing close integration of two modalities (vision and touch) is a study that found it became almost impossible for human participants to perform the seemingly trivial task of picking up a matchstick and lighting it when their hands were anesthetized BID12 ). Figure 1 :
(Left) Shows our experimental setup. p objects
are in a drawer and a dexterous hand equipped with tactile sensing can explore novel objects using deterministic routines. In this case
, p = 3 but we compared performance by varying the number of objects (Middle) We are presented with a query image as seen by the inset in the top right of the image. We explore
the objects in the drawer using tactile sensing only to identify the object (Right) We then retrieve the object by applying a grasping routineIn this work we use the task of retrieving objects from a drawer as an experimental setup to investigate joint learning from two sensory modalities of vision and touch. Because the
agent is provided only with a visual image of the object to be retrieved, it must translate into the representation space of tactile sensing to retrieve objects only by touching them. In the general
case of retrieving the object, the agent must first explore spatially to locate where the objects are. Once it finds
the object, it must move its fingers in an exploratory manner to collect information required to determine if the object is the one that needs to be retrieved. Solving this
problem in its full generality requires not only good goal directed exploration strategies and also a method for translating between different sensory signals. We therefore
think that object retrieval from a drawer is a good challenge problem for investigating different models that combine visual and tactile information for a target end task.In our setup the agent learns a mapping from visual to tactile signals using unsupervised exploration. This mapping
enables the agent to determine the representation of the image in the representation space of tactile sensing (i.e. expected tactile response). The agent explores
each object present in the drawer by touching it and compares the result of its exploration with the expected tactile response. Performing this comparisons
requires a good representation of raw tactile signals. For learning such a representation
, we leverage the results in image classification, where it was found that a network pre-trained to classify images from the Imagenet dataset into one thousand image categories learns features useful for many other visual tasks. Similar to image classification,
we pose the task of classifying objects from tactile signals collected by touching eleven objects. We show that tactile representation
learned by performing the task of classification, generalize and can be used to retrieve novel objects. We present results in a simulated environment
and the agent explores the objects using an anthropomorphic hand.
We present a model that when presented with a query image can identify a novel object from a set of p objects using tactile sensing only.
As the number of novel objects presented in a single trial increased, this task quickly became more challenging.We show that a model with pre-determined exploration routine can identify the object but a richer exploration of the objects could allow us to answer more challenging inferences such as pose, texture, etc.
One could imagine doing this by training a RL agent that uses the performance of { 2 as a reward while generating trajectories that explore an object for identification.Presently, we train our two networks F 1 and F 2 independently.
Since, our larger goal is to identify objects it would be interesting to jointly optimize the network to maximize object identification.
This can help smooth prediction errors that are not consequential to classification or identification.While the MuJoCo simulator provides fast physics solvers that can compute realistic contact forces they only compute normal contact forces.
Research in neuroscience has shown BID15 ) that a variety of forces are applied and measured while employing tactile sensing.
It would be interesting to perform similar experiments on robot grippers equipped with tactile sensors.Our current setup still trains network F 2 in a supervised fashion.
This is biologically implausible and tedious for practical scalability on real robots.
It would be interesting if we posed this problem as a self-supervised problem and explored learning to identify novel objects using tactile sensing.

assistant: text: In this work, we study the problem of learning representations to identify novel objects by exploring objects using tactile sensing. Key point here is that the query is provided in image domain.