Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain
Authors: Trinity Chung
Yuchen Shen
Nathan Kong
Aran Nayebi
Venue: NeurIPS 2025 oral ()
Keywords: NeuroAI
Temporal Neural Networks
Tactile Perception
Somatosensory Cortex
Primary area: neuroscience_and_cognitive_science
TL;DR: Task-optimized convolutional recurrent neural networks trained on realistic tactile inputs align with rodent somatosensory data, suggesting brain tactile processing uses temporally-precise representations shaped by categorization-driven optimization.

Abstract: Tactile sensing remains far less understood in neuroscience and less effective in artificial systems compared to more mature modalities such as vision and language.
We bridge these gaps by introducing a novel Encoder-Attender-Decoder (EAD) framework to systematically explore the space of task-optimized temporal neural networks trained on realistic tactile input sequences from a customized rodent whisker-array simulator. 
We identify convolutional recurrent neural networks (ConvRNNs) as superior encoders to purely feedforward and state-space architectures for tactile categorization. 
Crucially, these ConvRNN-encoder-based EAD models achieve neural representations closely matching rodent somatosensory cortex, saturating the explainable neural variability and revealing a clear linear relationship between supervised categorization performance and neural alignment.
Furthermore, contrastive self-supervised ConvRNN-encoder-based EADs, trained with tactile-specific augmentations, match supervised neural fits, serving as an ethologically-relevant, label-free proxy.

For neuroscience, our findings highlight nonlinear recurrent processing as important for general-purpose tactile representations in somatosensory cortex, providing the first quantitative characterization of the underlying inductive biases in this system. 
For embodied AI, our results emphasize the importance of recurrent EAD architectures to handle realistic tactile inputs, along with tailored self-supervised learning methods for achieving robust tactile perception with the same type of sensors animals use to sense in unstructured environments.

Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain
Trinity Chung"! 'J Yuchen Shen'?, Nathan C. L. Kong', and Aran Nayebi
'Robotics Institute, Carnegie Mellon University; Pittsburgh, PA 15213 Machine Learning Department, Carnegie Mellon University; Pittsburgh, PA 15213 Neurosciencc Institute, Carnegie Mellon University; Pittsburgh, PA 15213
*Department of Psychology, University of Pennsylvania; Philadelphia, PA 19104 Equal contribution.
ftrinityc, yuchens3, anayebi Hacs Jacs. cmu. cmu.edu; nclkongdsas upenn. udu edu
Abstract
Tactile sensing remains far less understood in neuroscience and less effective in artificial systems compared to more mature modalities such as vision and lan- guage. We bridge these gaps by introducing framework to systematically explore the space of mnsk-optimized temporal neural networks trained on realistic tactile input sequences from customized rodent whisker-array simulator. We identify convolutional recurrent neural networks (ConvRNNs) as superior encoders to purely feedforward and state-space architec- tures for tactile categorization. Crucially, these ConyRRNN-encoder-baeee models achieve neural representations closely matching rodent somatosensory cortex, saturating the explainable neural variability and revealing a clear linear relationship between supervised categorization performance and neural alignment. Furthermore, contrastive self self-supervised SanyRNN-encoder-baaee EADs, trained with tactile-specific augmentations, match supervised neural fits, serving ethologically--eleeann, label- free proxy.
(EAD)
EAD
For neuroscience, our findings highlight nonlinear recurrent processing as important for general-puue purppol- tactile representations in somatosensory cortex. providing the first quantitative characterization of the underlying inductive biases in this system. For embodied AI, our results emphasize the importance of recurrent EAD architectures to handle realistic tactile inputs, along with tailored self- supervvee learning methods for achieving robust tactile perception with the same type of sensors animals use to sense in unstructured environments.
Introduction
Tactile perception plays an essential role in the manipulation and interpretation of complex en- vironments through active sensing [Lederman and Klatzky 2009). Animals effectively leverage tactile sensors, such as whiskers, to precisely navigate, forage, and identify objects even in noisy and unstructured conditions [Grant] 2025]; specifically, contact whiskers deliver critical sensory input for navigation, foraging, and hunting low-light environments [Ahl [9866, while water-flow whiskers enable seals to discriminate prey movements from general water currents in similar con- ditions [Dehnhardt al. 2001], and specialized insect hairs respond to wind stimuli, providing information essential for flight stability and agility [Sterbing-D Angelo et al." 2011].
Rodents are common model organism for studying tactile perception in the brain due to the fine experimental control they offer and their ability to discriminate object location, shape, and texture
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

using only their whiskers [Hires et al., 2015 Cheung et al. 2019]. These whiskers, or vibrissae, transmit rich mechanical signals to mechanoreceptors at their base, enabling nuanced environmental understanding without direct sensory receptors along their lengths. In fact, rodent whisking behavior known to parallel how humans touch objects with their fingertips [Staiger and Petersen 2021], J, such as in the Lateral Motion Exploratory Procedure [Lederman and Klatzky 2009]. Despite significant interest in translating such biological capabilities into robots, artificial systems still struggle to match the tactile perceptual prowess of animals, limiting their functionality in realistic, unstructured environments Navarro- et al. 2023].
There are two primary reasons for this gap: one from the robotics hardware side, and the other from the neuroscience side. On the hardware side, current bio-inspired whisker sensors for robots face several critical limitations, including substantial hardware complexity when scaling sensor arrays beyond approximately 18-20 whiskers, each requiring individual transducers, significantly increasing wiring and computational demands [Pearson et al. 2011] Assaf et al. [0T6]. These sensors also struggle to accurately discriminate multiple simul ttnuouaneous stimuli such as airflow, direct contact, and inertia, unlike biological whiskers (Simon et al. 2023]. Additionally, mechanical discrepancies from biological whiskers-such as reduced sensitivity, limited flexibility, and constrained bending angles- impede precise tactile sensing in dynamic environments [Kent et al., 2023 Kent and Bergbreiter 2024].
These limitations also apply to anthropomorphic robot hands, which remain in a relatively early stage of development. For example, besides not being able to sense temperature, good mechanical skin has remained an open challenge in haptics for approximately four decades, primarily due to difficulties in creating artificial skins that maintain realistic deformation during object contact, with current pneumatic approaches proving inadequate to retain shape (Shimonomura [209 Dahiya et al., 2009]. While visuotactile camera-based sensors such as GelSight [Yuan et al., 2017 offer a limited haptic solution by providing high-resolution localized surface deformations, they fundamentally differ from human tactile sensing, which actively integrates diverse mechanoreceptor inputs over larger surfaces and multiple contacts [Ward- et al., 2018). Other issues persist with magnetic Bhirangi et al. 2021]. [Bhirangi et 20024a) which have low spatial resolution and is prone to error from electromagnetic interference [Bhirangi al. 2024b]. These hardware limitations thus make it difficult at the moment to identify robust algorithms for tactile processing that operate on realistic sensory inputs.
On the neuroscience side, despite extensive experimental characterization of rodent somatosensory pathways (e.g. [Armstrong-James et al. 1992, Kerr et al. 2007]), the neural computations underlying precise tactile perception remain poorly understood, primarily due to a scarcity of computational models of this system. Rodent whisker sensing involves hierarchical processing, analogous to vision, where sensory signals propagate from primary neurons in the trigeminal ganglion through parallel thalamic pathways, eventually reaching the primary and secondary somatosensory cortices (SI and S2) [Bosman et al. 2011] Moore et al. 20155. However, relatively few computational approaches have explicitly modeled these neural transtormations [Zhuang al. 2017]. and none have assessed if any of these models accurately match neural population responses in somatosensory cortex.
To bridge these gaps, we build and systematically evaluate temporal neural networks explicitly trained on biomechanically-reaiiitt force and torque tactile sequences that mice receive, customized from the first complete 3D simulation of the rat whisker array by Zweifel et al. [2021]. We identify convolutional recurrent neural networks (ConvRNNs), particularly intersectionRNNG as superior in tactile categorization and neural alignment compared to feedforward (ResNet) and attention- state-space models (S4 and Mamba). Leveraging supervised and contrastive self-supervised learning adapted specifically for tactile data, we demonstrate direct linear relationship between tactile categorization performance and neural fit, saturating the currently explainable neural variability and surpassing inter-animal consistency benchmarks, with contrastive self-supervised learning serving as an equally neurally predictive, label-free proxy. Thus, we provide the first quantitative characterization of inductive biases required for tactile algorithms to match brain processing.
2 Related Work
Task-optimized neural network models have emerged as currently the most quantitatively accurate framework for understanding brain function. Such goal-driven modeling first successfully explained

---

neural responses in primate visual areas across hierarchical stages Yamins et al., Khaligh-Razze
and Kriegeskorte, 2014), and subsequently. auditory [Kell al., 2018 Feather al.] 2019]] mo- tor Sussillo et al. Michaels et al.). memory [Nayebi et al. 2021 and language [Schrimmf et al. 2021] brain areas. While many these results are in humans and non-human primates, goal-driven modeling approaches have more recently been extended to mouse visual cortex, demonstrating that shallow architectures trained via contrastive self-supervised learning best capture mouse visual cortical representations [Nayebi* et al. 2023 Bakhtiari et al. 2021].
Despite these advances, goal-driven computational modeling has not yet been extensively applied to tactile processing, even though tactile sensory systems share bierarchical and recurrent processing features with vision [Felleman and Van Essen [991]. Zhuang et al. [2017) provided foundational goal-driven modeling approach for the rodent whisker- trigeminl system, yet their exploration was limited to relatively simple recurrent architectures and solely supervised categorization loss functions, and they did not compare their models to brain data. Here, we significantly broaden the architectural Seeerr-Atteeeeeeeeeeeeee
exploration via
parameterization. We incorporate sophisticated Convolu- tional RNN ("ConvRNN"") architectures previously developed by |ayebii al. [2018 8) which were shown to match primate visual cortex dynamics. Additionally, we explore advanced temporal models such as State Space Models (SSMs) (e.g. S4 [Gu et al.) [021] and Mamba [Gu and Dao 2023) )
and Transformers Halldr et al. 2024). We also employ self-supervised loss functions validated in mouse [Nayebi* et al.) [2023] Bakhtiari al. [021] and primate visual cortex [Zhuang et al. [202]],
adapted to specifically deal with force and torque inputs. Finally, we provide the first direct neural comparison rodent somatosensory cortical responses across all 64 models.
Methods
All our code for whisker simulation, model training, and neural data analysis is available on GitHub: https://githi:
High-variation factile dataset generation with a biomechanical whisker simulator. Given
ongoing development of tactile sensor hardware to match the flexibility of biological tactile
sensors such as fingertips and whiskers, we trained our models on synthetic data, focusing on object categorization and self-supervised learning using high-variation dataset like ShapeNet [Chang al. 2015). This involves the simulated objects interacting with biomechanically realistic rodent whisker model, WHISKiT Physics [Zweifel et al.) 2021], the first 3D simulation of the rodent's complete whisker system, provide high- nigh-dimensional force and torque inputs. Each of the 30 33 whiskers of an average whisker array (Belli et al. [2017] 2018) is modeled as a chain of rigid conical segments interconnected by torsional springs and actuated according to established equations of motion [Knutsen et al.) 2008).
(a)
Classification Whisk Datasets
Sweaping Speed
Object Haight
Object Rotation
Object Distance
Classiiicaton Accuracy
Lownor Highndd
Rancom Chence
1.00
Highterr Low
0.50
0.00
Number Classes
Figure 1: ShapeNet Whisking Dataset. (a) (I) With average mouse whisker array measurements from Bresee et al. [2023,, (II) objects are whisked simulation using WHISKiT [Zweifel et al., al.] 2021 resulting in (III) force and torque data for sweeping 9981 ShapeNet objects of 117 categories with various sweep augmentations. The augmentations vary the (1) speed, (2) height, (3), rotation, and (4) distance of the objects relative to the whisker array. We constructed two datasets: large. low-fidelity set with more sweep augmentations, and a small, high-fidelit set with fewer augmentations (sce appendix). (b) An SVM classification up to 4 different classes of objects (cups. microwaves, chairs, and trains) for the datasets show that the classes are distinguishable (above chance).
3

---

Since our objective is to compare models with currently available mouse somatosensory data [Rodgers 2022], we adapted this array to the 30 whiskers of the mouse [Bresee et al. 2023], arranged array with zero padding. Furthermore, adult mice can exhibit maximal bite forces of approximately 8-10 Newtons [Freeman and Lemen [008] Table I1, and since facial tolerance to applied forces would realistically be a fraction of this maximal bite force, our chosen simulation clipping range of +1000 milli-Newtons 1#1 N) remains comfortably within biologically plausible limits.
x
We created a whisking dataset for the shape categorization task, generated by passively brushing objects against the whisker array. Our primary consideration here is to generate a high-variation dataset by which to strongly constrain the representations that are learned, allowing us to more effectively simulate evolutionary pressures, in line with the "Contravariance Principle" (Cao and Yamins, 2024] of goal-driven modeling. On each of the 9981 different objects from ShapeNet, we apply several combinations of sweep augmentations, including adjusting the object sweeping speed, height, rotation, and distance relative to the whisker arry (Fig. [. These augmentations on our passive whisker sweeps can be considered to produce data that is isomorphic to performing active whisking in a systematic way. The 1 177 object categories are selected based on the work of Zhuunng ct al. [0177] to ensure reasonably distinguishable classes. We constructed two datasets: one large dataset replicating the Zhuang et al. [017]] data with 288 randomized sweep augmentations and physics simulation step rate of 110 Hz (High-Vann Variation/Low- Soann//oww-idelyty) and another with 16 sweep augmentations and higher simulation step rate of 1000 Hz (Low- Variation/High -iidelyytdllty...... See the Appendix[AI] [A] for detailed sweep augmentation procedures. Although higher temporal resolution is available from the simulation, we extract 22 timesteps for both datasets, corresponding to the average whisking frequency of 20 Hz. in rodents (Sofroniew and Svoboda 2015).
wew--Attteeee-Deooeee (EAD) temporal model search parameterization. Tactile recognition is performed by the somatosensory cortex, which, though less understood than vision, exhibits hierarchical processing and long-range feedback connections (Lederman and Klatzky 2009 Navarro Guerrero et al., 2023]. These basic anatomical insights motivate our exploration of hierarchical, temporal neural network (TNN) models such as ConvRNNs shown previously to match primate
visual cortex dynamics [Nayebi* et al. 2018 Nayebi et al. 2022]. J. SSMs [Gu et al. 2021]. Trans- formers Hallar et al.] 2224). and ResNets He al. [0166] feedforward control. We provide library PyTorchTNN which enables large-scale exploration of integrating multiple TNN modules that also combined by long-range feedback with feedforward connections.
These considerations produce a rather large search space of model architectures. Therefore, to be able to effectively search the space of TNN models systematically, we came up with an Encoder-Attender Decoder (EAD) parameterization, schematized in Fig. 2a Convolutional recurrent and state-space layers (such as ConvRNNs and S4) are particularly effective for encoding smooth and compressible temporal signals, given their recurrent smoothing properties and linear-time complexity. In contrast, transformers and other attention-lik, like architectures (e.g., Mamba) excel at processing temporally sparse or irregularly informative inputs, as they independently weight each timestep. Given that tactile inputs from force and torque sensors provide temporally smooth signals, encoder (E) layers closer to these inputs naturally benefit from convolutional and recurrent mechanisms that integrate information locally over time, effectively reducing redundancy and noise. In contrast, higher-level temporal aggregation (A), which must selectively integrate meaningful signals across longer and potentially irregular timescales, is better served by attention-based architectures like Transformers or Mamba, which can dynamically weigh distinct timesteps based on their informational content. Finally, the decoder (D) is either the classification layer for supervised learning, or similarly producing self- supervised features for contrastive learning or autoencoding. The EAD paradigm flexibly combines these complementary modules, enabling an effective and structured exploration of the temporal model space for tactile processing.
Specifically, we explore the following EAD combinations (visualized in Fig. 2a
Encoder:
Zhuang's ' recurrent model (Zhuang et al., 2017]. ResNet (He et al. 2016), S4 Gu et al.
2021 : 2)
Attender: Transformer (Haldar al.. [0244] (e.g., GPT), Mamba Gu and Dao 2023
None (i.e.,
attender-free); 3) Decoder: MLP. We further investigate different variants of Zhuang's 's model by replacing the recurrent layers in the ConvRNNs with UGRNN (Collins et al.) 2017 Intersection- RNN [Collins et al., 2017). LSTM [Hochreiter and Schmidhuber) 1997]. and GRU [Cho et al., 2014), and we present the update rules for the variants in Appendix A3 In the PyTorchTNN library, each time step in Conv RNNs corresponds to a single feedforward layer processing its input and passing it to the next layer, in contrast to treating each entire feedforward pass from input to output as a single

---

(b)
categorization (supervised) contrastive (SimCLR, SimSiam) autoencoding (AE)
Input
Decoder
Attender
augmentation
vertical
horizontal
rotation
Mamba
temporal
Encoder
c0oV
cony
Cony
Image augmentation
ConvRNN
ResNet
S4
Gaussian blur
color jitter
grayscale
(EAD) architecture, with task objectives being supervised categorization, supervuuee learning (SimCLR, SimSiam, autoencoding). The ConvRNN encoder includes recurrecceee at each layer where we vary different RNNs.
Figure 2: (a) Seeedder-Attnner-Deeee
(b) Types of data augmentations applied SSL models. Given temporal tactile input time T, our tactile augmentation vertically, horizontally, temporally flips, and rotates the features, while traditional image augmentation introduces Gaussian noise, color jitter, and grayscale.
integral time step, as is normally done with RNNs [Spoerer et al., 2017]. This implementation of temporal unrolling parallels biological systems, where stimuli are sequentially processed from one cortical layer to the next.
Model Training and Tactile Augmentations. Besides the supervised categorization objective, also consider self-supervised learning (SSL) losses: SimCLR [Chen et al. 2020]. which learns robust representations via distinguishing the embeddings of augmentations of one image from other images, SimSiam [Chen and Hc 2021]. which maximizes the similarity between the embeddings of two augmentations of the same image, and autoencoding (AE) [Olshausen and Field 1996). where use a 3-layer deconvolution network to reconstruct an image from its sparse latent representation.
We use a train/validation/test split of 80/10/10% and report the top-5 test accuracy. For supervised learning, we train models for 100 epochs and test on the checkpoint saved with the highest validation accuracy. For SSL objectives, we train for 100 epochs and save the model with the lowest validation loss, then continue training for another 100 epochs with checkpoint frozen and a learnable linear layer top of it. To stabilize training, we also consider adding layer norm [Ba et al. 2016) to the ConvRNNs as an alternative when training was unstable. All experiments are conducted on NVIDIA A6000 GPUs, and one set of model search experiment takes 8~-16 hours.
In addition the various sweep augmentations used generate the initial dataset, we apply tactile augmentations at training time as a cheap way to generate more training data. We illustrate the tactile input, our tactile augmentations, and traditional image augmentations for SSL in Fig. 2b Given temporal tactile input over time T, the proposed augmentation either vertically flips, horizontally flips, rotates, or reverses the input over time. Although the variable whisker lengths mean that these operations are not completely physically accurate, our tactile augmentations still meaningfully mimic flipping/rotating the whisker array or reversing temporal masking as an additional augmentation, but found that this did not significantly improve task performance (Appendix A5 Tab. 7). Applying traditional image augmentations like color jitter or gray scale cause the models to fail to train, which provides evidence that the choice of augmentations should represent operations relevant to the given modality.
direction of motion. We considered
Neural Data Comparison. We use the neural dataset from Rodgers 2022 which recorded neural population activity across superficial (L.2/3) deep (L6) layers the barrel cortex as the mice used their whiskers to perform a simple 2-object (convex/concave) classification task. Note that this low number of distinct stimuli is critical limitation which we address the Discussion section. After filtering for valid trials, the neural data we used contains total of 999 neural units across 11 mice. The time window of each trial is clipped to the time of first whisker contact with the object until the time the mouse makes its decision (about 1-2 seconds). The neural response is binned into intervals
5

---

of approximately 45-50 ms, corresponding neatly with the 15 15-20 Hz whisking frequency typical of rodents (50 ms per eycle at 20 Hz [Sofroniew and Svobodai 2015), and aligning specifically with our model's 's integration scheme of five sub-timesteps (-9 ms each) per timestep. This choice provides a temporal resolution optimal for capturing detailed neural activity during whisker-object contact periods.
We evaluate neural alignment using Representational Similarity Analysis (RSA), parameter-free approach that compares the pairwise dissimilarity structure of model and neural population responses across the same set of stimuli. RSA compares the pairwise dissimilarity structure of neural responses and model activations to common set of stimuli, allowing direct comparison of representational geometry without requiring additional parameter fitting (Kriegeskorte et 2008).
To ensure only reliable neurons are included, we compute split-half internal consistency of the neural responses across trials and retain only neurons with greater than 0.5. All RSA evaluations are thus conducted on this filtered subset of self-consistent neurons, and corrected by this internal consistency. Further details about this procedure are included in Appendix A4
reliability
The noise-corrected RSA Pearson's score is also computed between one mouse to every other mouse, and then averaged this score across all mice to obtain the mean animal score which will serve the baseline for model-brain evaluation, to account for inherent commal-to-animae variability (which we denote as "a2a" in the barplots). Thus, we want our models to match the brain, at least as well as animals do each other.
Next, we replicated the experimental setup of the barrel cortex dataset Rodgers 2022 in silico. Fig. shows the 6 different stimuli used in the experiment, which are concave/convex objects at three different distances (near, medium, far) from the whisker array. With 3D model reproduction of the concave/convex objects in the experiment, active whisking is simulated under the 6 stimuli using a real recorded whisking trajectory [Bresee et al. 2023) to generate the model input.
We then obtain the neural alignment score for models by computing the maximum, across all layers, of the median RSA score between each layer's representation and the neural responses, averaged across mice. The standard error is calculated across the RSA Pearson's 's r value from the model to each mouse.
4 Results
Validation of Sensor and Dataset via SVM Decoding. We first verified the basic discriminability of the whisker array dataset using support vector machine (SV classifiers on the High- Variation/Loww Fidelity and Low- VariattionnHigh--iiellii datasets [Fig. Both showed strong decoding perfor- mance, exceeding chance and remaining robust even with increased task difficulty from more object categories. The comparable results across sampling rates confirm that the smaller Low-Vat Varr/a//.H-/iiiii dataset retains sufficient discriminative tactile information, allowing for efficient model training without compromising performance.
ConyRNN Encoders Outperform Feedforward and Attention Architectures for Tactile Categorization. We next systematically evaluated the task performance of the EAD architectures trained on tactile force/torque sequences. Overall, we found that across EAD architectures, the
Top-5 Test Cat. ACC.
Task Performance per Model (Calored by Encoder, Labeled by Loss)
Encoder
Atzender get mamba none
0.G
vantia
gru
resnot
lotm zhoang
0.4-
0.0
simsiam
simclr
30
supervised
Model
Figure 3: Tactile Categorization Accuracy. The lighter-colored left bar represents the randomly ini- tialized version for every model. The best-performing model is Zhuang-GPPTSSpperiise (rightmost cyan bar). Models with the encoder being S4 are excluded as the training losses explode before the first epoch is finished.

---

Real
Sim
medium
near
concave
conveX
concave
convex
(b)
Neural Fit
inter
apt
Neural Fit (RSA Pearson's f
1.36
intor aeter-nimaalnn max
IColored by Encoder, Labeled by Loss)
resnet
gru
2huang
mombo
ugme
none
1.0
0.8-
0.6-
0.4-
0.2-
simisian
Ainiciam
sinisiam
simisiam
sinicl
Similam
Ainisiam
suparnsion
simsiam
Simisiam
sinielr
simci
simsile
simsiam
Simssia
simisiam
simisiam
simasiam
sinusiam
simulam
simicl
siniclr-
Sinisiam.
supeniosd
BuDanasod
0.0
Model
Figure 4: Model Neural Evaluation. (a) We use six different stimuli (concave/convex x
near/medium/ffrr replicating the conditions in the mouse neural dataset in simulation. (Real images were taken from video recordings in neural dataset [Rodgers] 2022).. ). (b) Comparison neural fit (noise-corrected RSA Pearson's s r) across models. The mean animal-to-animal score is 0.18 and the maximum between all pairs of animals is 1.34. The leftmost "a2a" bar represents the mean animal-to-animal neural consistency score. The lighter-colored left bar represents the randomly initialized version for every model.
choice of encoder (E) was quite important for tactile recognition. Specifically, ConvRNN encoders, especially the IntersectionRNN [Collins et al. 2017], surpassed the purely feedforward ResNet18 and SSM (S4) encoders, in supervised tactile categorization tasks (Fig. 3). Additionally, models trained with our custom force-and-torque touudttrqu--ppeiffii contrastive self-supervised learning (SSL) augmentations (Fig. 2b) outperformed untrained networks of the same architecture, and those trained with standard image-based augmentations that involve Gaussian blur and color jittering [Chen al., 2020) did not train with the best architecture despite hyperparameter tuning, demonstrating the importance of tailored tactile augmentations for enhancing task performance (Fig. 3
Importance of Architectural Inductive Bias. We note that the raw model input (the tactile force/torque sequences obtained from actively whisking on the 6 stimuli objects) achieves correla- tion of 0.46 (Fig. 4b)) and untrained randomly initialized models also achieve moderately positive correlation to the mouse neural data. Randomly initialized networks are highly non-random functions as they are architectures selected to perform the task well when trained. Therefore, architecture matters lot, and this is a well-noted phenomenon in NeuroAI across brain regions (Yamins et al.) Naycbi et al. 2023 Schrimpl's et al. 2020]. By doing this comparison, it allows us to isolate the contributions of the architecture vs. loss function for every pair of (architecture, loss function) tuples. The models that saturate our noise ceiling (bars on far right in Fig. 4b) are noticeably improved when trained.
ConyRNN Encoders Saturate Explainable Neural Variance in Rodent Somatosensory Cortex. To assess biological realism, we compared internal model representations to neural recordings from rodent whisker somatosensory cortex. All trained EAD models outperformed raw sensor inputs in neural alignment, underscoring the importance of nonlinear temporal processing modeling brain-like tactile representations (Fig. 44)). In fact, the best models saturated all of the held-out explainable neural variance-without fitting any additional parameters-even when tested on entirely novel objects under substantially different experimental conditions (active whisker sensing rather than passive contact, as used in training). Remarkably, the neural predictivity exceeded the average inter-animal neural consistency (leftmost black "a2a" bar in Fig. 4b), thereby robustly passing the Al Turing Test on this dataset [Feather* et al. 2025). Among these and consistent with the categorization results, EAD models with ConvRNN encoders consistently provided better neural fits compared to feedforward (ResNet) and SSM-based (S4) encoder architectures, underscoring the biological plausibility of recurrence in modeling tactile processing. In fact, saw strong
7

---

(a)
(b)
Task Performance Neural Fit: SSL
Task Performance vs Neural Ri:: Supervised
0.0
Taa.5
Tast Cat. Acc.
0.5
1.0
Neural Fit (RSA Pearson's r)
Loss
Neural Fit (RSA Pearson's f)
Encoder
Attender
InterSSmCLA Zhuang GPTSSuP.
1.0
simdr
1.0
vanilla
gru
gpt
anddm init.
resnet
lmtm
mamba
0.9
ugrnnn
zhuang
x
image sug.
0.8
0.8
0.60
pactile sug
0.6-
0.7
0.6-
random init
0.5
image aug.
lacllle aug
0.05
0.15
0.20
0.45
0.50
0.55
Top-5 Test Categorization Accuracy
Top-5 Test Categorization Accuracy
0.5
RSA Pearson's
Figure 5: Comparing Task Performance and Neural Fit. (a) The task performance of SSL models are about one order of magnitude below the performance of supervised models, yet are able to achieve comparable neural fit. (b) For supervised models, we observe a trend of better task performance leading to increased neural correspondence. Plotting best fit line, we find the correlation (c) The tactile augmentations were effective in improving both the neural fit and task performance. The models were unable to be trained with image augmentations.
0.59.
linear trend between tactile supervised categorization performance and neural fit (r = 0.59, Fig. 5b ). with the model layers that best predict the tactile neural responses being closest to the decoder layer (Fig. A22. Although the number of tast-opti parameters matters for both categorization test set performance and neural fit generalization (Fig. AT), they are not the whole story, as the SimCLR- EAD with the IntersectionRNN encoder best matches the neural data with far fewer parameters (~ 3.80 x 10? parameters) than its supervised counterpart with a GPT-based attender (~> 6.38 x 107 parameters).
GPT-based Attenders Provide Modest Improvements in Task Performance and Neural Align- ment. In addition to varying the encoder (E) layer of the EADs. we investigated different temporal aggregation ("Attender") schemas. We observed that GPT-based Attenders modestly outperformed Mamba-based and no-attention controls in both supervised task categorization and neural alignment (Fig. 5b). Although these improvements were subtle, the consistency of the result suggests that incorporating some form of attention downstream of the ConvRNN encoder is beneficial, particularly for our
neural models. This result suggests the prediction that attention like mechanisms may be implemented in somatosensory cortex through selective modulation of hierar- chical processing pathways, such as those from primary sensory neurons in the trigeminal ganglion through the thalamus and subsequently into primary and secondary somatosensory cortical areas (S) and S2), which could be validated by experiments involving targeted perturbations or optogenetic manipulation of these specific pathways during tactile discrimination tasks.
Self-Supervised SimCLR Training Matches Supervised Models and Serves as an Ethologically- Relevant Label-Free Proxy. We compared neural alignment achieved by supervised training against SSL methods, adapted with tactile-specific force-and- and-torque augmentations. This comparison is necessary because discriminating among 117 human- numan-recognizable shape classes is not directly ethologically relevant for rodents. Nevertheless, ShapeNet's extensive diversity of 3D objects provides strong structural constraints for modeling whisker-trigeminal processing at biological scales, constraints which smaller, mouse-relevant object sets might fail to impose. Paralleling findings in vision YYamins et al. "Chaligh-Razan and Kriegeskorte] 2014], where training on large-scale object categorization leads to generalizable representations beyond specific categories, we similarly suggest that ShapeNet's 't's diversity provides meaningful constraints network structure irrespective of exact object identity. Additionally, by demonstrating that self-supervised learning (SSL) methods yield neural representations comparable (if not marginally better) to supervised approaches, we directly address an open direction highlighted by Zhuang et al. [2017), who emphasized the need for developing more ethologically relevant yet practically scalable tasks for studying tactile processing.
In fact, we found specifically that contrustive SSL loss functions consistently reached neural align- ment scores similar to the best supervised models, confirming its utility as an ethologically-reeeve but label-free approach to tactile representation learning (Fig. (5a), and outperforming other non- contrastive self-supervised approaches such as autoencoding, The SimCLR-optimiin Intersection-
8

---

RNN EAD achieved the best absolute neural alignment score across all models overall, and was comparable its neural alignment its supervised (and more parameter-dense) variant (rightmost green bars in Fig. 4b). This result mirrors findings in primate visual cortex (Zhuang et al. [021]. where contrastive SSI. methods equally predicted neural response patterns compared to supervised alternatives, suggesting supervised learning serves as proxy for this more ethologically-relevan loss function. Interestingly, the parity between contrastive SSL and supervised models observed here differs from mouse visual cortex, where contrastive SSL methods substantially exceeded supervised methods in neural predictivity [Nayebi* et al. 2023]. Notably, the SSL methods, which are best aligned with tactile neural representations overall, achieve only moderate linear probe categoriza- tion performance (Fig. 5a). implying that somatosensory cortex representations might prioritize broader, task-agnostic sensory encoding rather than purely specialized, integorization-driveer features. Furthermore, just as we found for downstream task performance in Fig. 3 we also observed it was critical have tactile-specific SSL. augmentations for developing biologically accurate tactile representations, compared to both standard image-based SimCLR augmentations and randomly initialized architecture-fixx fixed controls (Fig. (Fig.[cc). We further provide evidence for the design of our tactile augmentation in Appendix A5
5 Discussion
Implications for Somatosensory Cortical Processing. We developed a novel
Decoder (EAD) parameterization of the space of temporal neural network models (TNN) trained to perform tactile recognition on biomechanically realistic force and torque sequences, greatly extending and answering the open question posed by ZZhang et al.] [2077] of characterizing the ethologically relevant constraints of rodent whisker- based tactile computations. Our results establish ConvRNN encoders, particularly the IntersectionRNN, as currently superior in tactile categorization performance and neural representational alignment compared to feedforward (ResNet) and state-space models (SSM), suggesting recurrent processing is more relevant overall to the rodent somatosensory sys- tem. Furthermore, we demonstrate that contrastive self-super learning (SimCLR), particularly when trained with tactile-specific augmentations, yields neural alignments comparable to supervised methods, highlighting supervised training as a proxy for a more ethologically relevant, label-free rep- resentation. The modest yet consistent benefits observed from GPT- Attenders indicate potential attention-lin mechanisms operating within hicrarchical tactile processing pathways, suggesting a fruitful avenue for experimental validation.
Taken together, our findings indicate that nonlinear recurrent processing plays an essential role in the rodent somatosensory cortex, reflecting neural encoding strategies that prioritize broad, general- purpose tactile representations. This work provides the first quantitative characterization of the inductive biases required for tactile algorithms match brain processing, opening new opportunities for deeper insights into sensory representation learning and somatosensory cortical dynamics.
Implications for Embodied AI and Robotics. Current artificial tactile sensors and models fall short of animal-like capabilities, limiting the functional use of robots in real-world, unstructured scenarios. Our results underscore the importance of biomechanically realistic inputs and temporally recurrent architectures for developing tactile perception models that perceive touch similarly as animals do. Our demonstration that mattiee-specin (sequential force/torque array) SSL augmentations significantly enhance performance underscores this point, emphasizing the necessity of tailored training methods for robotic tactile systems. Future work in embodied robotic systems that leverage these insights could potentially overcome existing sensor limitations, including scaling complexity, stimulus discrimination challenges, and restricted sensing surfaces, thus achieving more robust and nuanced environmental interactions similar to biological organisms.
Limitations and Future Directions. Our findings do not represent the final word, but rather the beginning of an improved understanding of tactile sensory processing in both animals and machines. Most importantly, currently available tactile neural datasets remain severely limited in stimulus diversity with the number of object conditions tested, restricting the captured neural variability, and could be the reason why inter-animal consistency values are lower for the statistical average between animals than the current pointwise empirical maximum of 1.3 (Fig. 4bb), indicated by the NeuroAI Turing Test. Expanding neural datasets to include broader sets of tactile stimuli and additional animals will thus be crucial for future work, enabling models to approach and potentially surpass this theoretical ceiling of neural predictivity.

---

In the longer term, incorporating multimodal sensory integration will be ieportant-particulin combining tactile inputs with other modalities, such as proprioception or vision, through biologically- inspired fusion operations that integrate signals extensively across intermediate layers, as is observed the brain [Navarro- Guerrero et al. 2203], rather than relying solely on last-layer concatenation, as commonly done now Yang et al. [024]. Exploring precisely where and how these multimodal signals are best fused, using diverse fusion operations (e.g., concatenation, attention, routing), is already supported by our PyTorchTM) library and will yield stronger constraints models, potentially enabling more robust tactile-driven decision-making in genuinely unstructured scenarios as well as animals do.
Broader Impacts. Improving robotic tactile sensing to not only match human capabilities, but also quantitatively validating, as we explicitly do, shared processing of object properties with human collaborators the level of internal representations, could enhance human safety and efficiency in environments such as healthcare, assistive technologies, and manufacturing in the future. However, advancements in tactile-enabled robotics might also disrupt employment in sectors relying heavily on manual labor. It will therefore be important to thoughtfully manage these technological transitions through proactive policy-making and workforce training.
10

---

Acknowledgements
We thank Katerina Fragkiadaki, Albert Gu, and Roberta "Bobby" Klatzky for helpful discussions. A.N. thanks the Burroughs Wellcome Fund and Google Robotics Award for funding.
References
A. S. Ahl. The role of vibrissae in behavior: a status review. Veterinary research communications, 10(1): 245-268, 1986.
M. Armstrong-James, Fox, and A. Gap-Gupta. Flow of excitation within rat barrel cortex on striking a single vibrissa. Journal of neurophysiology, 68(4):1345-1358 1992.
Assaf, D. Wilson, S, Anderson, Dean, Porrill, and M. Pearson. Visual-tactile sensory map calibration of a biomimetic whiskered robot. In 2016 JEEE International Conference on Robotics and Automation (ICRA), pages 967-972 IEEE, 2016.
Ba, J. R. Kiros, and G. E, Hinton. Layer normalization, 2016. URL htt.:ap:///axxi.orr////aa// 06450
Bakhtiari, P. Mineault, Lillicrap, C. Pack, and B. Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances Neural Information Processing Systems, 14-25164-25175 2021.
H. M. Belli, A. E. Yang, C. Bresee, and M. Hartmann. Variations in vibrissal geometry across the rat mystacial pad: base diameter, medulla, and taper. Journal of Neurophysiology, 11 117(4))1807-182) 2017.
H. M. Belli, C. S. Bresee, M. M. Graff, and M. J. Hartmann. Quantifying the three-dimensional facial morphology of laboratory rat with a focus on the vibrissae. PLoS One, (3(4)=00149811 2018.
R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. arXiv preprint arXiv:2111.0077. 2021.
R. Bhirangi, V. Pattabiraman, E. Erciyes, Y. Cao, T. Hellebrekers, and L. Pinto. Anyskin: Plug-and-play skin sensing for robotic touch. arXiv preprint arXiv:2409.08276 2024a.
R. Bhirangi, C. Wang, Pattahiraman, C. Majidi, A. Gupta, T. Hellebrekers, and L.. Pinto. Hicrarchical state space models for continuous eqquence-to-sequenee modeling. arXiv preprint arXiv:2402. 10211, 2024b.
W. Bosman, R. Houweling, B. Owens, Tanke, Shevehouk. N. Rahmati, H. Teunissen, Ju, W. Gong, S. K. Koekkoek, et al. Anatomical pathways involved in generating and sensing rhythmic whisker movements. Frontiers in integrative neuroscience, 5:53, 2011.
Bottou. Large-scale machine learning with stochastic gradient descent. Proceedings of COMPSTAT 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177-186. Springer, 2010.
C. Bresee, H. M. Belli, Luo, and M. J. Z. Hartmann. Comparative morphology of whiskers and faces of mice (mus musculus) and rats (rattus norvegicus). Journal of Experimental Biology, 226 19)jeb2445997, 10 2023. ISSN 0022-0949. doi: 2..2242/jbb2445597 URL pattps://dol .r.. prrg/00/r 10. 1242/ jeb. 245597
R. Cao and D. Yamins. Explanatory models neuroscience, part 2: Functional intelligibility and the contravari- ance principle. Cognitive Systems Research, 85: 101200, 2024.
A. Chang, T. Funkhouser. L. Guibas, Hanrahan, Q. Huang, Li. Savarese, M. Savva, S. Song. H. Su, ct al. Shapenet: An information-rich 3d model repository. arXiv preprint 112.0215/ 2015.
Chen, S. Korbblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597 1607. PmLR, 2020.
X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758, 2021.
J. Cheung, P. Maire, J. Kim, J. Sy, and S. A. Hires. The sensorimotor basis of whisker-guided anteroposterior object localization in head-fixed mice. Current Biology, 29( 10):303)0--000 2019.
K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau. F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation, 2014. URL https //axxiv , org/orr///.......... abs/ 1406. 1078

---

J. Collins, J. Sohl-Dickstein, and D. Sussillo. Capacity and trainability in recurrent neural networks. In ICLR, 2017.
E. Coumans and Y. Bai. Pybuilet, python module for physics simulation for games, robotics and machine learning. http::/pybuulett. org 2016 2021.
R. Dahiya, G. Metta, M. Valle, and G. Sandini. Tactile sensing from humans to humanoids. transactions on roborics, 26(1):1-20, 2009.
IEEE
G. Dehuhardt, B. Mauck, W. Hanke, and H. Bleckmann. Hydrodynamic trail-following in harbor seals (phoca vitulina). Science, 29365527):002-004, 2001.
J. Feather, A. Durango, R. Gonzalez, and J. McDermott, Metamers of neural networks reveal divergence from human perceptual systems. In Advances in Neural Information Processing Systems, pages 10078-10089, 2019.
J. Feather", M. Khosla*, N. Murty", and A. Nayebi*. Brain-mode evaluations need the neuroai turing test. arXiv preprint arXiv:2502.1143 2025.
Essen. Distributed hierarchical processing the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1-47, 1991.
Felleman and D.
W. Freeman and C. A. Lemen. A simple morphological predictor of bite force in rodents. Journal of Zoology. 275(4):418-422, 2008.
R. A. Grant. Can we study whisker movements to gain insights into the natural sensory behaviours of mammals? The Journal of Physiology, 2025.
A. Gu and T. Dao. Mamha: Linear-time sequence modeling with selective state spaces. arXiv preprint arXlv:331.00752, 2023.
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured spaces. arXiv preprint arXiv:2111.00%% 2021.
Haldar, Z. Peng, and Pinto. Baku: An efficient transformer multi-task policy learning, arXiv preprint arXiv:2406.07555 2024.
K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 770-778, 2016.
S. A. Hires, D. A. Gutnisky, J. Yu, D. H. O' O'Connor, and K. Svoboda. Low-noise encoding of active touch by layer 4 in the somatosensory cortex. elife, 4:e06619, 198666999000 2015.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 98)::1335-1780, 1997.
A. J. Kell*, D. L. Yamins*, E. N. Shook, S. V. Norman-Haignere, and J. H. McDermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals contical processing hierarchy. Neuron, 98(3):630 644, 2018
A. Kent Bergbreiter. Flow shadowing: method to detect multiple flow headings using an array of densely packed whisker-inspired sensors. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2024.
T. A. Kent, H. Emnett, M. Babaei, M. J. Hartmann, and S. Bergbreiter. Identifying contact distance uncertainty in whisker sensing with tapered, flexible whiskers. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 607-613. IEFE, 2023.
J. N. Ketr, C. De Kock, D. S. Greenberg, R. M. Bruno, Sakmann, and F. Helmchen. Spatial organization of neuronal population responses in layer 2/3 of rat barrel cortex. Journal of neuroscience, 1366333333 2007.
S.-M Khaligh-Raan Razavi and N. Kriegeskorte. Deep supervised, but not unsupervised, models may explain cortical representation. PLoS computational biology. 10 lee100915 2014.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
P. M. Knutsen. A. Biess, and E. Ahissar. Vibrissal kinematics in 3d: tight coupling of azimuth, elevation, and torsion across different whisking modes. Neuron, 59(1):35-4) 2008.
12

---

N. Kriegeskorte. M. Mur. and P. A. Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiery in systems neuroscience, 2:249, 2008
S. J. Lederman and R. L. Klatzky. Haptic perception: A tutorial. Attention, Perception, & Psychophysics, 71(7): 1439 1459, 2009.
I. Loshchilov and Hutter. Decoupled weight decay regularization, 2019. URL URL http::://aaxvvvoog//aaaa 1711.05101
K. Mamou and F. Ghorbel. simple and efficient approach for 3d mesh approximate convex decomposition. pages 3501-3504, 11 2009. doi: 10. 109/CCTP:0009 40668.
J. A. Michaels, S. Schaffelhofer, A. Agudelo- Toro, and Н. Scherberger. A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping. 117/50):32124-321355
J. D. Moore, N. Mercer Lindsay, M. Deschênes, and D. Kleinfeld. Vibrissa self-motion and touch are reliably encoded along the same somatosensory pathway from brainstem through thalamus. PLoS biology. 13(9): c1002253, 2015.
N. Navarro-Guerrero, S. Toprak, J. Josifovski, and L. Jamone. Visuo-haptic object perception for robots: an overview. Autonomous Robots, 47(4):377-403, 2023.
A. Nayebi*, Bear". Kubilius", K. Kar, Ganguli, Sussillo, DiCarlo, L. Yamins. Task-driven convolutional recurrent models of the visual system. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information P Processing Systems, volume 31. Curran Associates, Inc., 2018.
A. Nayebi, A. Attinger, M. Campbell, K. Hardcastle, I. Low, C. Mallory, G. Mel, B. Sorscher, A. Williams, S. Ganguli, L. M. Giocomo, and D. L. Yamins. Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. Advances Neural Information Processing Systems, 34, 2021.
A. Nayebi, Sagastuy-Brent
M. Bear, Kar, J. Kubilius, S. Ganguli. Sussillo, J. J. DiCarlo, and D. L. Yamins. Recurrent connections the primate ventral visual stream mediate a tradeoff between task performance and network size during core object recognition. Neural Computation, 34: 14:1152-1675, 2022.
A. Nayebi", N. C. Kong", C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. Mouse visual cortex as limited resource system that self-learns ecologically-gent general representation. PLOS Computational Biology. 19, 2023.
B. A. Olshausen and J. Field. Emergence of simple-cell receptive field properties learning sparse code natural images. Nature, 38166583)607-600, 1996.
M. Pearson. Mitchinson, Sullivan, A. Pipe, and T. Prescott. Biomimetic vibrissal sensing for robots. Philosophical Transactions of the Royal Society B: Biological Sciences, (66(1581)30885-300 3096, 2011.
C. C. Rodgers. detailed behavioral, videographic, and neural dataset on object recognition mice. Scientific Data, 91))620,, 2022.
M. Schrimpf, J. Kubilius, M. Lee, N. A. R. Murty, R. Ajemian, and J. DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Newron, 108(3):433-223, 2020.
M. Schrimpf. I. A. Blank, G. Tuckute, C. Kauf, E. A. Hosseini, N. Kanwisher, J. B. Tenenbaum, and E. Fedorenko. The neural architecture of language: Integrative modeling converges predictive processing. Proceedings National Academy of Sciences, 118(45)) 18, 2021.
Shimonomura. Tactile image sensors employing camera: A review. Sensors, 19(18):3933, 2019.
N. Simon, A. Z. Ren, Piqué, Snyder, D. Barretto, M. Hultmark, and A. Majumdar. Flowdrone: wind estimation and gust rejection on uavs using fast-response hot-wire flow sensors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5393-5399, IEEE, 2023.
N. J. Sofroniew and K. Svoboda. Whisking, Current Biology, 25(4):R137-R140 2015.
C. J. Spoerer, P. McClure, and N. Kriegeskorte. Recurrent convolutional neural networks: a better model of biological object recognition. Front. Psychol., 8:1-14, 2017.
J. Staiger and C, C. Petersen. Neuronal circuits in barrel cortex for whisker sensory perception. Physiological reviews, 101(1):353 415, 2021.
13

---

S. Sterbing-D) Angelo, M. Chadha, C. Chiu. B. Falk, W. Xian, J. Barcelo, J. M. Zook, and C. F. Moss. Bat wing sensors support flight control. Proceedings of the National Academy Sciences, 108(27):11211-11296 2011.
D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy. A neural network that finds naturalistic solution production muscle activity. 18(7):1025-1033
B. Ward-Cherrier, N. Pestell, L. Cramphorn, Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora. The tactip family: Soft optical tactile sensors with 3d 3d-printed biomimetic morphologies. Soft roboties, 5(2): 216-20 227, 2018.
Yamins, Hong. C. F. Cadieu, E. Solomon, Seibert, hierarchical models predict neural responses in higher visual cortex. 111(23):811-8624
DiCarlo. Performance-optimized
Yang, C. Feng, Z. Chen, Park, D. Wang, Dou. Zeng. Chen, Gangopadhyay, A. Owens, et Binding touch to everything: Learning unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26340-263553 2024.
Y. You, Gitman, and B. Ginsburg. Large batch training of convolutional networks.
W. Yuan, S. Dong, and E. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
Zhuang, J. Kubilius, M. J. Hartmann, and D. Yamins. Toward goal-driven neural network models rodent whisker-trigeminall system. Advances in Neural Information Processing Systems, 30, 2017.
Zhuang. S. Yan, A. Nayebi, Schrimpf. C. Frank, J. DiCarlo, and Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), 2021.
O. Zweifel, N. Bush, Abraham, Murphey, M. J. Hartmann. A dynamical model for generating synthetic data to quantify active tactile sensing behavior the rat. Proceedings of the National Academy of Sciences. 118(27):e201 1905118, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Please refer to the Abstract Introduction and Discussion
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations in the [iscusssionsccction (Limitaaions and Future Directions.)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well spec--fffccitinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core
paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide our seen-soureed code for TorchTNN and reference other software used in Methods The details of creating the whisking dataset with exact parameters used is provided in the A Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
Open access to data and code
the data and code, with sufficient instructions faithfully reproduce the main experimental results, described supplemental material?
Question: Does the paper provide open
Answer: [Yes]
Justification: As mentioned in Methods our PyTorchTNN library is open-sourced. The whisking dataset is also provided.
Guidelincs:
The answer means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines settps://nips.cc/put guides/Codesiee https///ooooouuueiiiioooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttss://niss.cc/ppun for more details.
Buzdee//DodeSuumiissioooPPo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
The 80/10/10 data split, epochs, and other training details are provided Methods
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The process of calculating these values is described Methods SEM values shown figures where applicable, which is referenced in Results
Guidelines:
answer means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Methods we state the GPUs used (Nvidia A6000s) and approximate time to train per model (8~166 hours).
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full rescarch project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question:
Does the research conducted the paper conform. every respect, with the NeurlPS Code Ethics
Answer:
[Yes]
Justification:
We pledge that this research conforms with the NeurIPS Code of Fthies.
Guidelincs:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We mention broader impacts as an explicit subsection in our Discussion secttion
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datascts)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original creators of ShapeNet dataset, WHISKIT simulator, and others in the Methods section and respect their licenses in usc/adaptation of the data/software.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are introduced in the paper well documented and is the documentation provided alongside assets?
Answer: [Yes]
Justification: Setup and usage documentation is provided PyTorchNNN whisking dataset.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMSs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
htttpp///hoiiisss.c///ofrrreeerr Please refer to our LLM policy (https: petttp://neuriiii cc//Confeeences/222555 MLMM for what should or should not be described.
20

---

Appendix
Table of Contents
We detail the contents in the appendix below.
A1. Whisk Dataset Generation details the parameters and the modifications of the simulator for generating our whisking dataset for training.
A2.
Model Definitions present present the mathematical definitions of the RNNs used in the ConvRNNs (i.e, Zhuang's 's model (Zhuang et al., 2017) in our experiments.
A3.
Model Training includes the optimizer, scheduler, and learning rate configurations for training different models for both supervised and self-supervised learning.
A4. Neural Evaluation explains how the neural fit scores are calculated, and presents visual- ization of model parameters vs. neural fit, per layer neural fit scores, and representational dissimilarity matrices for neural evaluation.
A5.
Additional Experiments show results on stimuli decoding based on learned representations and self-supervised learning performance with temporal masking as the augmentation.
A1 Whisk Dataset Generation
Our whisking dataset uses the same 9981 ShapeNet objects and 117 category labels in Zhuang et al. [017]. but using an improved whisker model and various sweep augmentations, which are
listed Table [
Sim. Freq.
Speed
Height
Rotation
Distance
Size
Total Sweeps
(1) 1000 Hz
30 mm/s
-5, 0 mm
0, 30, 90, 120
5, mm
40 mm [20~60]
316,192
(2)
110 Hz
[30~60]
-,,03
[0~359]
2,076,048
Table 1: Sweep Augmentations used for the two whisking datasets, which we refer as (1) Low- Low-Fidelity. Simulation Frequency refers to the frequency corresponding to the physics timestep used Bullet physics engine [Coumans and Bai 2016-2021J. the backend for WHISKiT simulator [Zweifel et al. [202]] For dataset (2), the speed, rotation, and size is each sampled from the range 26 times. The total number of sweeps equals the number of sweep augmentation combinations x 9981 objects.
Variation High-Fidelity and (2)
WHISKIT Simulator Modifications. We make a few enhancements to the original WHISKIT simulator [Zweifel al. 2021]
Number of whisker links (where whiskers are modeled chain of springs (Zhuang et al.
2017]) are dynamically adjusted by the length of the whisker instead of fixed number.
Allow for loading objects with or without convex hull. In the whisk datasets, objects are
loaded with convex hull (using V-HACD [Mamou and Ghorbel 2009] ) to keep collisions more stable. In generating the simulated stimuli for model input neural evaluation, convex hull was not used in order to preserve the concavity of the "concave" object. The object was geometrically simple enough to not affect collision stability.
Add camera settings for viewing in orthographic or perspective projection.
Load multiple mice/rats at time.
All our code for the simulation, model training, and neural data analysis is available on GitHub: wwt.c////euurraaetts-aaaaa/////a www.ile-whiiiingg
https://g' htt//ggttuu....// thub. www..
21

---

A2 Model Definitions
We provide the update rules for UGRNN and IntersectionRNN in our experiments as follows. For other RNN variants, please refer to Nayebi et al. [2022]
input at time and layer
state at time and layer
W UC: learnable weight matrices at layer {
b bias terms at layer
or element-wise multiplication
a: sigmoid function
tanh: hyperbolic tangent function
ReLU: rectified linear unit function
linear transformation or convolution (depending on context)
A2.1 UGRNN
G Lanh(W * of Ue s/-1 + 1/:)
g o (We * o + Ug s + by 1)
(2)
of oe St (1 9 oe
hy s
A2.2 IntersectionRNN
m tanh(W' of + Uh s blm)
(5)
n ReLU(W, n of + U. of + b%))
F a (WP- * of + Ug * i + 1p + 1) E
Vt (W of + +U, o by
(8)
of = pt o s + (1 - p!) o m
(9)
A of o o (1 v n
(10)
22

---

A3 Model Training
All of our experiments are conducted on NVIDIA A6000 GPUs. For supervised learning, use a batch size of 256 for all the models and train for 100 epochs. For SSL, during the pre-training stage, we use batch size of 256 for SimCLR and autoencoding (AE), and batch size of 1024 for SimSiam, following Nayebi* et al. 2023], and train for 100 epochs. During the linear probing stage, we freeze the checkpoint saved with the lowest validation loss and add a trainable linear layer. We further train such model with labels for 100 epochs, with a batch size of 256, an initial learning rate
of 0.1, with the StepLR scheduler, and the SGD optimizer with momentum [Bottou 2010).
We detail the optimizers [Bottou 2010 Kingma and Ba 2013 Loshchilov and Hutter 2019 You al. and schedulers, and their configurations used in supervised learning and SSL Table 2
Optimizer
Configuration
SGD
momentum 0.9, weight-decay 10 weight-decay = 10-1
Adam
Adam W
weight-decay = 10-1
LARS
momentum 0.9, weight-decay 10-1
Scheduler
Configuration
StepLR
step_size = 30
ConstantLR
fixed learning rate
CosineLR
carmup_epocke 10
ConaneAnnealing
min_lr 0.0, warmup_epoch 10, warmup_ 10-1
Table 2:
Optimizers and Schedulers with default configurations of supervised learning and SSL when training different model architectures.
For supervised learning, we present the model training configurations Table 3 where we detail the choices of optimizer, scheduler, learning rate, the encoder and attender of our EAD architecture, and we omit the decoder due to space limits, as it is always a linear layer/MLP. As Zhuang+GPT the best performing supervised model in terms of classification accuracy, explore different variants of Zhuang's encoder with attender being GPT.
Encoder
Attender
Optimizer
Scheduler
Learning Rate
ResNet
None
SGD
StepLR
10(-1,-2.3]
SGD
StepLR 101-1,-2;-3,-0
Zhuang
None
SGD
ConstantLR 10-2,5 -0
7huang- (UGRN, IntersectionRNN, GRU, LSTM)
{SGD, Adam, AdamW
Stepl.R
10(-1,-2,-3,-4)
None
(LayerNorm) AdamW
Stepl.R
10t-1,-2.-3-44
ResNet, Zhuang, Zhuang- Beeanng(UGRNYE (UGRNN IntersettioRNNNLLN, GRU, LSTM), S4
AdamW
CosineLR 10-4
GPT
(SGD, AdamW)
StepLR
101-1.-2,-1)
AdamW
CosineI.R
10-4
ResNet, Zhuang. $4
Mamba
(SGD, AdamW)
Stepl.R
101-1,-2,-33
Table 3: Model Training Configurations for Supervised Learning, We use (} to indicate different choices of a specific component (i.e., encoder, optimizer, learning rate) the search space. We consider adding layer norm variant when searching the best configuration for Zhuang's variants (i.e., the second row), which is denoted as "LLN".
For SSL, we follow Nayebi* et al. [2023] and use specific configurations of optimizer and scheduler for different losses, which are shown in Table 4 For SimSiam, we try the Aonealln scheduler both with and without 10 epochs of warmup to search for better model performance. For AE, we use a 3-layer deconvolution network decode the sparse latent representation from our EAD framework into the original tactile input, regardless of the choice of model architectures.
23

---

Loss
Optimizer
Scheduler
Learning Rate
SimCLR
LARS
Anneaann
10 2,-,33,,
SimSiam
SGD
CosineAnnealing (with & w/o warmup)
10(-11-2,-3,-1
AE
SGD
Stepl.R
10f-1,-2,-8,-4
Table 4: Optimizer, Scheduler, and Learning Rate Configurations for SSL when training different model architectures.
We present the model architectures explored for SSL in Table where we present the encoder and attender of our EAD architecture, during the pre-training stage, only the encoder and attender used, and during the linear probing stage, the decoder is always one-layer linear classification head.
Encoder
Attender
Zhuang,
InterssettoonRNN-LNN, GRU, LSTM)
None
ResNet, Zhuang-IntersctionRRNNNNL Zhuang,
GPT
ResNet, Zhuang, S4
Mamba
Table 5: Model Architecture Configurations for SSL. We () indicate the different variants of encoders. We consider adding layer norm (LN) as a variant when searching the best configuration for Zhuang's variants.
24

---

A4 Neural Evaluation
We use the NeuroAl Turing Test [Feather* al. 2025 evaluate the neural similarity of mice whisking to models performing tactile categorization.
RSA Correlation. Due to the low number of stimuli, correlation metric. RSA is computed over stimuli and neurons, where the average is over source animals/subsampled source neurons, bootstrapped trials, and train/test splits. This yields vector of these average values, which we can take median and s.e.m. over, across animals.
RSA [Kriegeskorte et al. 2008 as
For the neurons of animal A to animal B in the set of animals A we estimate the RSA correlation:
RSA (s $2)
(RSA (1", (9,1)) REAIABBJAAA
(11)
RSA (sf, > RSA (sf, AEAIAB)EAAA
Each neuron in our analysis is associated with value for when it was target animal (B), averaged over subsampled source neurons and 1000 bootstrapped trials. This yields a vector of these average values, which take median and standard error of the mean (s.e.m.) over, as we do with standard explained variance metrics.
Spearman-Browe Correction. The Spearman- Brown correction be applied to each of the terms the denominator individually, as they are each correlations observations from half the trials of the same underlying process to itself (unlike the numerator).
RSA (X = (RD(() RDM(y))
2RSA Y)
1 + RSA Y)
Inter-Animal Consistency. To estimate the inter- consistency, evaluate the pooled animal consistency for each animal. One animal is held out at time, then compared to the aggregated across units from the remaining animals. We found the mean pooled animal score 0.175 with a s.e.m. of 0.161 and maximum score of 1.34.
(a)
(b)
Model Parameters vs Neural
Model Parameters vs Task Performance
gpt
Neural Fit (RSA Pearson's f)
Top-5 5 Test Categorization Acc.
0.75
Encoder
vanills
resnet
ugrris
0.8
0.25
supervized
simer
0.4
stmslam
0.00
Total Parameters
1e7
Total Parameters
1e7
Figure A1: Model Parameters compared with categorization performance and neural fit. (a) Legends for (b) (Encoder colors) and (c) (Loss colors). (b) Models with GPT as attender have higher correlation score slightly higher than those without, but high neural fit still achievable without more parameters demonstrated by the Inter+SimCI.R model (high green "x"). (c) Models with GPT as the attender has more parameters and, when trained with supervised learning, tends to have higher top-5 categorization accuracy.
25

---

(a)
Neural Fit per Layer in Inter+ GPT+SL
Neural Fit per Layer Inter+ SimCLR
RSA Pearson's
1.0
1.0
RSA Pearson's
0.5
0.5
0.0
conv1
conv2
conv3
conv4
convs
conve
fc7
Ica
got
emm
tad
60
gat
gpt
b2
got
b3
gat bg
gpt b5
gpt, 66
gpt, b7
got Im head mip 10
gpt In
0.0
conv2
conv2
conv3
conva
conv5
conv6
tc7
fca
Figure A2: Neural Fit Score per Model Layers colored by encoder, attender, and decoder. No is the model that
bar means the score is NaN for that layer. (a) Saar=--PT--Suerrvisll.camm scored the highest neural fit out of the supervised models. We observe that later GPT layers perform increasingly better. (b) The last fully-comnected layer of Inter+SimCLR achieved the highest r value.
(a)
(b) Inter+GPT+SL:
Simulated Stimuli
Inter+SimCLR, Layer FC-8
Data
0.4
0.000
0.005
0.010
0.0
(e)
Dissimilarity (1 Corr)
Flattened RDMs
0.75
(a) Simulated Stimuli (scaled 2x)
(b) Inter Mite++GP++L SL (scaled 50x)
0.50
(c) Inter+ (scaled 1.5x)
(d) Mouse Neural Data
0.25
0.00
Pairwise Stimulus Index
Figure A3: Representational Dissimilarity Matrices (RDM) for tactile data and neural evaluation. (a) RDM of the 6 simulated stimuli which is used the model input in neural evaluation. (b) The GPT LM Head layer of www-GG---Supeviieee.ee..
the supervised model with the highest neural fit score. (c) The last Fully-Connected layer in Inter+SimCLR achieves the highest neural (d) RDM performed on the 6 stimuli in the mice neural data. (e) A visualization of the flattened RDMs, scaled approximately fit (RSA correlation does not take scale into account).
fit score out of SSL models.
26

---

A5 Additional Experiments
As basic way to probe the distinguishability of the models' learned representations of stimuli, we performed decoding test with results shown in 6 Note that Inter+SimCLR is the model with the best neural score; Zhuang+GPT-SSperviice has the best task score; Gitrr-GTTSSupevisse has best neural score out of supervised models; Resnet-Mamba+SiiCLL. has best task score out of SSL models.
Rodgers' task variables (convex/concave) are behavioral probes, and are not necessarily the natural computational goals of somatosensory cortex. Our RSA analysis shows that cortical representational geometry is best matched by tactile-optimized models, especially with self-supervised losses, indicat- ing that the cortex builds general substrate for tactile recognition and discrimination across diverse objects. When reduced to six-condition probe, this broad representational space may not project well onto the labels, yielding poor decoding despite underlying ethologically aligned representations.
Model
Shape (chance=0.5)
Distance (chance=0.33)
Inter+
0
0
0.33
0.5
Inter+CiPT-Superiised
0
0.5
Maanett-Mama+SiinnLL. 0
0.67
Animal Neural Data
0.44
0
Table 6: Stimuli decoding. For each of the 6 stimuli, we train a logistic regression model on the other stimuli and test on held-out stimulus to decode either the shape (convex/concave) or the distance (far/medium/near). The stimuli decoding score for Animal Neural Data was obtained by decoding per mouse, then averaging across mice.
We have also tried adding temporal masking. where we randomly mask 75% the as tactile augmentation, but it did not significantly improve the task performance or neural fit score (Table 7).
Model
Top-5 Cat. Accuracy
Neural Fit
Inter+SimCLR
0.15
0.96
Inter+SimCLR with Temporal Masking
0.18
0.42
Resnet-Mamma++SimCLLL
0.23
0.70
Masnett+Mamaa++iinLL. with T. Masking
0.28
0.67
Table 7: Temporal Masking. Results from retraining two models with temporal masking as an
additional tactile augmentation. Top-5 categorization task performance improved slightly (-5%), but neural fit scores decreased.
27


Reviews:
[{"review_id": "HjWO15BVu5", "reviewer": "Reviewer_Lgyo", "summary": "This paper systematically explores neural network architectures for tactile perception inspired by the rodent whisker system. The authors propose an \"Encoder-Attender-Decoder\" (EAD) framework and train various models using realistic simulated tactile data from virtual whisker interactions with 3D objects. They demonstrate that convolutional recurrent neural networks (ConvRNNs), outperform other architectures on tactile object categorization tasks. These ConvRNN models exhibit remarkable alignment with neural recordings from rodent somatosensory cortex, surpassing average inter-animal consistency. This research quantitatively identifies nonlinear recurrence as a key inductive bias in rodent tactile processing, guiding future development in embodied tactile AI systems.", "questions": "See Weaknesses.\n\nAdditional question: Are there any models that perform lower than the a2a baseline in neural fit? For example, untrained different architectures", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n[S1] The proposed Encoder-Attender-Decoder framework enables systematic exploration of neural architecture space, moving beyond ad-hoc architecture selection. It facilitates comprehensive comparisons of multiple models against biological data, highlighting essential inductive biases relevant to rodent tactile processing.\n\n[S2] The paper provides valuable insights and findings without explicitly fitting models to neural data. The ConvRNN encoders effectively saturate explainable neural variance in rodent somatosensory cortex, and demonstrate a clear correlation between task performance and neural fit.\n\nWeaknesses:\n\n[W1] In Figure 4, nearly all models—including randomly initialized versions—outperform the a2a baseline regarding neural fit. Additionally, certain untrained models outperform their trained counterparts. The method used to evaluate neural fit and the impact of model training need further investigation.\n\n[W2] The neural data task involves only six distinct stimuli (concave/convex at three distances), potentially simplifying variance saturation compared to scenarios with more diverse stimulus sets.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.\n\n### Low number of stimuli\nThe mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that we could find. \n\n### Model neural fit scores compared to a2a baseline\nAll models are indeed above the mean animal-to-animal (a2a) fit, but below the *maximum* a2a baseline. This is noted in the caption but we will update the figure to make this clearer. (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.)\n\nWe noted both of these limitations in the “Limitations and Future Directions” paragraph in the Discussion section (lines 351-354): *“Most importantly, current tactile neural datasets remain limited in stimulus diversity and the number of object conditions tested, restricting the captured neural variability and could be the reason why inter-animal consistency values are lower for the statistical average between animals than the current pointwise empirical maximum of 1.3.”*"}, {"review_id": "0nlSAa5CcL", "reviewer": "Reviewer_6uQw", "summary": "This paper introduces an Encoder-Attender-Decoder (EAD) framework for modeling tactile perception in rodents by training temporal neural networks on realistic tactile sequences simulated from a rodent whisker array. The authors build the ShapeNet Whisking Dataset which is a synthetic tactile dataset generated by simulating a rodent’s whisker array interacting with 9,981 3D objects from ShapeNet across 117 categories. Using the WHISKiT Physics simulator, the authors modeled forces and torques at the bases of 30 whiskers during object sweeps with varied speeds, heights, rotations, and distances, creating rich temporal tactile signals reflective of real whisker sensing. The authors systematically compare convolutional recurrent neural networks (ConvRNNs), feedforward ResNets, and attention/state-space models. The results emphasize the importance of recurrent processing and self-supervision for capturing tactile representations, advancing both scientific understanding and robust tactile perception in embodied AI.", "questions": "1. Did you explore other recurrent variants (e.g. vanilla RNNs, bi-directional RNNs), and how did they perform?\n2. ShapeNet is a human-centric dataset. How do you think the neural alignment results would change if the dataset contained ecologically relevant rodent objects (e.g. burrow structures, food items)?\n3. How transferable are your ConvRNN-based models to real robotic tactile sensors such as GelSight, given mechanical differences?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The paper is easy to follow, with major contributions mentioned clearly in the Introduction.\n2. The motivations behind this paper are of significant importance, in that they build a systematic study of encoder, attention, and decoder architectures for inferring sensory semantics from simulated rodent whiskers, which can pave the way for architecture design in robotic tactile sensing.\n3. The authors present the ShapeNet Whisking Dataset which is a synthetic tactile dataset generated by simulating a rodent’s whisker array interacting with 9,981 3D objects from ShapeNet across 117 categories.\n4. The author’s insights around contrasting usefulness of recurrent and attention-based architectures, in that the temporally smooth signals from tactile inputs benefit from convolutional and recurrent mechanisms that integrate information locally over time, hold in the presented results.\n\nWeaknesses:\n\n1. While this study itself is meaningful, the connection to a real-life application, say, in robotics, is far-fetched. Is there any evidence that the takeaways may hold for any available sensing modalities such as force/torque or vision-based touch sensing from GelSight?\n2. Although the simulations are biologically realistic, models are trained on synthetic data rather than real tactile sensor readings, leaving a gap between simulation and potential real-world heterogeneity in tactile data.", "was_revised": true, "final_justification": "The authors addressed all the questions I raise in the rebuttal, specifically around exploring alternate RNN architectures. I am happy to recommend this paper for acceptance and will keep my score of 5 (Accept).", "rebuttal": "Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.\n\n### Transferability to robotic tactile sensors\nOur motivation for this work is to better understand a biological tactile sensing system so that, down the line, we can try to replicate biological tactile capabilities in robots using the inductive biases discovered in this paper. As such, the primary result of this work is in a neuroscientific scope and not deployed on robots. Still, our work (1) provides new tactile augmentations which may also be used to train self-supervised policies on robot tactile data, and (2) serves as a validation of low-dimensional force sensors as sufficient for object discrimination, as opposed to requiring high resolution images as tactile data (visuotactile sensors like GelSight).\n\n### Simulation-to-reality gap in tactile modeling\nWe used the most biologically accurate simulation model of whiskers available based on real mouse morphology in order to reduce the gap, but without real whisker force sensors, we are unable to evaluate the extent of the gap ourselves. Especially with tactile sensors, the wide variety in sensor implementation (optical, capacitance, mechanical, magnetic, pneumatic, etc.) and high amount of noise and variability even within sensor types means that collecting quality real data is very difficult. A discussion of the limitations of current hardware is included in the Introduction section– current bio-inspired robotic whisker sensors are limited by hardware complexity, poor multi-stimuli discrimination, and mechanical shortcomings that hinder accurate tactile sensing compared to biological whiskers. While the creation and validation of a real whisker array force sensor is outside the scope of our current neuroscientific evaluation, we definitely agree that this would be beneficial to evaluate our models on real whisker-like hardware once available.\n\n### Exploration of alternative recurrent architectures\nThank you for your suggestion! Currently, the RNN architectures explored in our paper include: time-decay (the original Zhuang’s model), GRU, LSTM, UGRNN, IntersectionRNN. \nWe didn’t consider any bi-directional architectures for supervised learning as it breaks the biologically plausible temporal unrolling mechanism, where each layer is sequentially activated as time increases. For self-supervised learning, we incorporate the idea of time reversing in our proposed tactile augmentation (see Fig. 2b, middle, temporal), which flips the input temporally and is similar to the high-level idea of bi-diretional RNNs.\n\nWe further replace the recurrent cell with vanilla RNN, and conduct experiments on the following architectures:\n1. Zhuang+GPT (best task-performant and neural-fit architecture for supervised learning)\n2. Zhuang+IntersectionRNN+SimCLR (best neural-fit architecture for self-supervised learning)\n\nThe best task-performant architecture for self-supervised learning is ResNet+Mamba+SimCLR, which doesn’t include any RNNs; hence, it’s not included in the additional experiments. We present the task performance (top-5 classification accuracy) and neural fits of the vanilla RNN variants below.\n\n|       model             | Zhuang-vanilla-RNN+GPT | Zhuang-vanilla-RNN+SimCLR | supervised best | self-supervised best |\n|-------------------------|-------------------------|-----------------------------|------------------|------------------------|\n| top-5 acc.              |       64.11              |           14.48              |       69.53      |         22.68           |\n| neural fit              |             0.90          |             0.69           |       0.95        |          0.96              |\n\nWe can observe from above that vanilla RNN doesn’t provide further improvement in either task performance or neural fit compared to existing RNN architectures.\n\n\n### Ecological relevance of ShapeNet for neural alignment\nAlthough ShapeNet is not ecologically relevant for mice, we note that the untrained models are worse than the trained models at neural predictivity, suggesting our choice is beneficial. There are no suitable rodent-relevant datasets, so we would have to hand-craft one by selecting a few objects. Arguably, the dataset diversity in ShapeNet should prepare the model for more downstream tasks, while a smaller but rodent-specific dataset may not be large or diverse enough to do well. Finally, in NeuroAI more broadly, it has been shown that having dataset diversity is critical for model-brain alignment, not just task performance– this is known as the “Contravariance Principle” [1, 2, 3].\n\n[1] R. Cao and D. Yamins. “Explanatory Models in Neuroscience, Part 1: Taking Mechanistic Abstraction Seriously.” Cognitive Systems Research, vol. 87, Sep. 2024, p. 101244. ScienceDirect, https://doi.org/10.1016/j.cogsys.2024.101244.\n\n[2] D. Yamins and J. DiCarlo. “Using Goal-Driven Deep Learning Models to Understand Sensory Cortex.” Nature Neuroscience, vol. 19, no. 3, Mar. 2016, pp. 356–65. PubMed, https://doi.org/10.1038/nn.4244.\n\n[3] A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. “Mouse Visual Cortex as a Limited Resource System That Self-Learns an Ecologically-General Representation.” PLoS Computational Biology, vol. 19, no. 10, Oct. 2023, p. e1011506. PubMed, https://doi.org/10.1371/journal.pcbi.1011506."}, {"review_id": "GC3XebIDc6", "reviewer": "Reviewer_qP7w", "summary": "This work focuses on rodent tactile processing, and evaluates neural networks trained on biomechanically-realistic force / torque tactile sequences. The results show that recurrent ConvRNNs (IntersectionRNNs) are better at tactile categorization and neural alignment than feedforward ResNet and attention-based models. The models show impressive neural alignment, saturating the explainable neural variability and surpassing inter-animal consistency benchmarks with both supervised and contrastive self-supervised objectives.", "questions": "Results Figure 4. How can the animal-to-animal scores be so low (<.20) and the model-to-animal scores so high (approaching 1.0, noise adjusted)? How could one model RDM be almost perfectly correlated (noise adjusted) with each individual animal RDM (without any tuning to individuals), but those animal rdms be so poorly correlated with each other?\n\nResults Figure 4. Can a model score well just by distinguishing convex from concave? Is there reliable variation between near-medium-far (within convex only; or within concave only), and if so, can the models capture this variation as well? This would at least suggest that the models are capable of capturing “fine-grained differences in real neural data” (within the constraints of this particular real-neural dataset). Relatedly, since the RDMs are so small, it would be useful to depict them in a Figure, to clarify what the “neural target” is for these analyses\n\nFor task-performance, is it fair to use “top-5 classification” as your measure of task-performance for the self-supervised models, since this is not in fact the task that the model was trained on? I suppose the more neutral measure of task-performance would be some version of the task loss (maybe somehow normalized to allow comparison across tasks), or even a complementary “instance-recognition” measure (how well individual samples are discriminated from the rest, which is more in-line with the objective of the self-supervised models.\n\nRegarding the strong neural alignment of the self-supervised models and its parity with the supervised models, I would find these results more compelling if it could be clearly demonstrated that there’s some fine-grained signal here (not just getting the convex vs. concave distinction, assuming that’s PC1 for these neural data, but also getting the near-medium-far variation within convex, or separately within concave).\n\nWere the rodent neural data combined across brain regions? Why not do separate analyses for S1, S2? Or are the RDMS too similar between these regions to bother with the breakdown?\n\nMinor  \n- How was the explainable variance ceiling estimated?\n-\tDid you identify the best-fitting layer in separate split of data than what you used to provide the final score for that layer? If so, this could be clarified in the final paragraph of the methods (pg. 6, just before Results).\n-\tCan the authors include an explanation for what the IntersectionRNN is?\n-\tHow ethologically valid do you think the simclr variant is? Could you have had the simulated rodent actually “sample the object twice” and still get strong learning?\n-\tAre there any other “precision measurements” or “fine-grained” analyses afforded by the rodent neural dataset, e.g., tuning of individual neurons?\n-\tIsn’t the importance of an inductive bias of “non-linear recurrent hierarchical network” expected from prior work on rodent tactile processing?\n-\tI didn’t really understand how this work might impact robotics exactly. How might these insights be integrated into robotic systems? Are tactile robotics not already using recurrent CNNs? Or is the idea that recurrent EAD systems are new and should be used?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths  \n- First application of model-brain-alignment methods to tactile processing across a broad set of model architectures and both supervised and self-supervised training objectives\n-\tNovel Encoder-Attender-Decoder parameterization\n-\tDirect neural comparison between a large set of models and rodent somatosensory cortical responses\n-\tTraining models on a synthetic dataset (simulated objects interacting with biomechanically-realistic rodent whisker model) is creative and provides a strong validation of the bio-mechanical model (given the success of the resulting convRNN models in predicting real neural data) \n\nWeaknesses  \n- The alignment with real rodent neural data appears to be based on RDMs computed from just 6 stimuli. This might be the best dataset available, but this dataset would seem to provide a rather coarse measure and should temper claims about the strength of the alignment between models and real neural data. Reading the abstract I was excited by the finding that the neural alignment scores saturated the noise ceiling, but less surprised that this was the case after reading that the analysis focuses on 6 stimuli. The results are still promising, but I would not necessarily expect “saturating the noise ceiling” to hold up under a larger, more varied dataset.", "was_revised": true, "final_justification": "I'm convinced that the main limitations of this work are inherited from limitations in the behavioral dataset, but I would encourage the authors to raise these limitations more clearly to make sure they are more clear to readers. In my view the methods and approach are \"out in front\" of the dataset, and so therefore the claims are a bit \"out in front\" of the data as well. But I think the strengths of the methods introduced, and advances made here in neural modeling in this domain (which basically expose the need for better neural data) are important and likely will have a broad impact.", "rebuttal": "Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.\n\n### Low number of stimuli\nThe mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that was available. We agree this is an unfortunate limitation that reduces the generalizability of our findings; though we note that low numbers of stimuli are a common occurrence in neuroscience. Our task-optimized modeling approach, along with our inter-animal consistency analysis, highlights the need for future neural datasets to move forward in this concrete direction of collecting more stimuli. This was noted in the “Limitations and Future Directions” paragraph in the Discussion section (lines 351-354). (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.) Finally, since we train the models on the large-scale tactile whisking task rather than fit to the neural data directly, we note that our models can be evaluated *without change* on future neural datasets that collect more stimuli.\n\n### Discrepancy between model-to-animal and animal-to-animal alignment\nThe model-to-animal neural score is obtained by using the maximum alignment score across the model to each animal. As a result, the models indeed achieve a score above the mean animal-to-animal (a2a) fit, but all are also below the maximum a2a baseline. (This is noted in the caption but we will update the figure to make this clearer.) In other words, the variability between animals is very high but there is always an animal pair that do very closely match. The inter-animal consistency score calculation is included in the Appendix (A3, Eq. 1). It is the RSA score between the response patterns for each pair of animals.\n\n### Distinguishing different types of stimulus and variations\nNo, we find that matching the neural data is not simply explained by distinguishing convex from concave; but rather, the models more reliably decode near-medium-far, suggesting that there is more variability along this axis for neural predictivity, than merely convex vs. concave. Below, we show that the decoding performance of the best models is above chance for near-medium-far. For each of the 6 stimuli, we train a logistic regression model on the 5 other stimuli and test on the 1 held out stimuli. While the number of stimuli is too low to make a definitive claim, interestingly our most task performant self-supervised model had the highest stimuli decoding score.\n| Model                 | Note                                                          | Shape (chance=0.5) | Distance (chance=0.33) |\n| --------------------- | ------------------------------------------------------------- | ------------------ | ---------------------- |\n| Inter+SimCLR          | Best neural score                                             | 0.0                | 0.0                    |\n| Zhuang+GPT+Supervised | Best task score                                               | 0.33               | **0.5**                |\n| Inter+GPT+Supervised  | Best neural score out of supervised models                    | 0.0                | **0.5**                |\n| Resnet+Mamba+SimCLR   | Best task score out of self-supervised models                 | 0.0                | **0.67**               |\n| Mouse Neural Data     | Score of stimuli decoded per mouse, then averaged across mice | 0.44               | 0.0                    |\n\nFinally, a figure of the RDMs for the best neural fitting supervised and self-supervised model is already included in the appendix (Figure A3.)\n\n### Task performance measurement other than “top-5 accuracy”\nFor self-supervised learning, all the models are first pre-trained with self-supervised losses. After we have the best pre-trained checkpoint, we freeze the model weights and add a trainable classification head, which is a common approach for evaluating downstream task performance for self-supervised models [1]. This assembled model is then finetuned with labels in a supervised fashion and we save the best model based on its accuracy on the validation set and report the test accuracy (see line 187-189); hence, reporting the task performance under “top-5 accuracy” is consistent for both supervised and self-supervised learning.\nIn our preliminary experiments, we tried self-supervised learning with the instance recognition loss [1], however, the training is highly unstable and the results are not satisfactory, which suggests that in instance recognition as an objective/measurement is less promising for tactile processing. Thank you for bringing this up, we will mention this in our revised paper.\n\n[1] Wu Z, Xiong Y, Yu SX, Lin D. Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018. p. 3733–3742.\n\n\n### Brain regions for rodent neural data & potential more fine-grained analyses\nIt is all from S1 (barrel cortex specifically). Our current neural dataset does not parametrize convexity and concavity for us to run a tuning curve analysis. However, our focus is on building the models which will be useful for such future datasets and subsequent downstream analyses to further separate models, since our model parameters are optimized on a task (rather than our specific neural dataset) and can be readily evaluated in new settings.\n\n### Identifying the best-fitting layer\nThe best neural fitting layer is the layer with the highest neural score, so a separate split of data was not used.\n\n### Explanation of the IntersectionRNN\nThe following equations define the recurrent computations at layer $\\ell$ and time step $t$ for IntersectionRNN, more details about the computation of different RNNs used in our paper can be found in [2], and we plan to include these details in our revised paper.\n\n$$\nm_t^\\ell = \\tanh(W_m^\\ell * x_t^\\ell + U_m^\\ell * s_{t-1}^\\ell + b_m^\\ell)\n$$\n\n$$\nn_t^\\ell = \\textbf{relu}(W_n^\\ell * x_t^\\ell + U_n^\\ell * s_{t-1}^\\ell + b_n^\\ell)\n$$\n\n$$\np_t^\\ell = \\sigma(W_p^\\ell * x_t^\\ell + U_p^\\ell * s_{t-1}^\\ell + b_p^\\ell + 1)\n$$\n\n$$\ny_t^\\ell = \\sigma(W_y^\\ell * x_t^\\ell + U_y^\\ell * s_{t-1}^\\ell + b_y^\\ell + 1)\n$$\n\n$$\ns_t^\\ell = p_t^\\ell \\circ s_{t-1}^\\ell + (1 - p_t^\\ell) \\circ m_t^\\ell\n$$\n\n$$\nh_t^\\ell = y_t^\\ell \\circ x_t^\\ell + (1 - y_t^\\ell) \\circ n_t^\\ell\n$$\n\nNotation\n- $x_t^\\ell$ is the input at time $t$ and layer $\\ell$\n- $s_t^\\ell$ is the state at time $t$ and layer $\\ell$\n- $W^\\ell$ and $U^\\ell$ represent learnable weight matrices\n- $b^\\ell$ are bias terms\n- $\\circ$ denotes element-wise multiplication\n- $\\sigma$ is the sigmoid function\n- $\\tanh$ is the hyperbolic tangent function\n- $\\text{relu}$ is the rectified linear unit function\n- $*$ denotes a linear transformation or convolution depending on context\n\n[2] Aran Nayebi, Javier Sagastuy-Brena, Daniel M. Bear, Kohitij Kar, Jonas Kubilius, Surya Ganguli, David Sussillo, James J. DiCarlo, Daniel L. K. Yamins\nbioRxiv 2021.02.17.431717; doi: https://doi.org/10.1101/2021.02.17.431717\n\n\n### Ethological validity of the SimCLR loss\nThe augmentation for SimCLR is an approximation for head motion of the rodents (see in Fig. 2 b, middle, for example, the rotation augmentation mimics head rotation of the rodents seeing the same object). From a training perspective, it enforces invariance of the same object for the model; hence, the “sample-twice” mechanism of SimCLR together with the proposed tactile augmentation ethologically approximating how rodents rotate their heads and provide different views of the same object to improve the learning results.\n\n### Inductive biases and prior work in rodent tactile processing\nAs mentioned in line 98-101, prior work in rodent tactile processing is only limited to supervised learning, a small number of model architectures, and crucially does not compare the models to brain data. With our Encoder-Attender-Decoder framework, we can systematically explore and search for the inductive biases necessary to rodent tactile processing, which is further strengthened by our results in neural alignment, as discussed in line 320-324.\nTo sum up, our work is fundamentally different from previous works and the findings are well-supported by both task performance *and* neural alignment of the models under comprehensive search with a broad range of different neural network architectures (e.g., SSMs, RNNs, transformers).\n\n### Integration of insights into robotic tactile systems\nThere is currently no established or widely-used method to process tactile data in robotics, due to so much variety in hardware implementation. Yes, our intention is to convey that recurrent EADs would be a good candidate for such tactile hardware as they become more capable. At the moment, animals and simulation environments are more capable than current robotic hardware; therefore, our motivation for this work is to better understand a biological tactile sensing system so that, down the line, we can try to replicate biological tactile capabilities in robots using the inductive biases discovered in this paper. As such, the primary result of this work is in a neuroscientific scope and currently not deployed on robots. Still, our work (1) provides new tactile augmentations which may also be used to train self-supervised policies on robot tactile data, and (2) serves as a validation of low-dimensional force sensors as sufficient for object discrimination, as opposed to requiring high resolution images as tactile data (visuotactile sensors like GelSight). Both provide strong evidence for developing future robotic tactile systems based on force/torque sensors, other than existing sensors purely rely on vision, and can be potentially adapted for building such models."}, {"review_id": "MB0zPVq6PK", "reviewer": "Reviewer_qLD8", "summary": "The paper tackles rodent whisker-based touch from both an ML and a neuroscience angle.\nThe authors (i) port the WHISKiT Physics simulator to a 30-whisker mouse array and generate two large ShapeNet-derived datasets of realistic force-and-torque sequences; (ii) introduce a PyTorch \"Encoder-Attender-Decoder\" search space that spans feed-forward, state-space, and convolutional recurrent (ConvRNN) encoders plus optional attention modules; (iii) train 62 model variants under supervised categorization and contrastive self-supervised objectives that use force/torque-specific augmentations; and (iv) compare each layer's representations to barrel-cortex population recordings via noise-corrected RSA.\n\nThey find that (i) ConvRNN encoders (i.e. IntersectionRNN) outperform ResNet and SSM baselines on tactile categorization, with (ii) the best ConvRNN-based models “saturating” explainable neural variance, matching or exceeding inter-animal consistency without any fitted read-outs. Task accuracy and neural alignment are linearly related, and contrastive SimCLR models (with tactile augmentations) equal supervised models in neural predictivity, offering a label-free proxy. GPT-style attenders give modest but consistent gains over no-attention or Mamba variants.", "questions": "Following up on my comments above, the paper would benefit from a targeted robustness ablation, stronger significance tests, and a leaner, more transparent methods presentation. Specifically:\n* Models are trained on passive sweeps of 9k ShapeNet meshes but evaluated neurally on active whisking to six cup-like shapes. Ceiling-level RSA might therefore hinge on an accidental projection onto a tiny stimulus sub-space. This needs at least one ablation to show robustness. For instance: retrain (or at least fine-tune) the best ConvRNN-EAD on a dataset that includes the six concave/convex objects (held out only for validation) and report RSA, to verify it remains near ceiling after ablation.\n* The lack of statistical testing undermines confidence in architecture ranking. We cannot tell if IntersectionRNN > ResNet is reliable across mice. To address this, the authors should provide hierarchical statistics (e.g. linear mixed-effects with random intercept per mouse) or paired permutation tests on RSA scores.\n* Six stimuli yield a noisy ceiling estimate. Please report split-half reliability distribution; give bootstrapped 95 % CIs for the inter-animal RSA. Add SEM bars or violin plots for the model-to-mouse RSA distribution (n = 11). Clearer ceilings would raise confidence. Revealing wide uncertainty without acknowledging it would lower significance.\n* Several critical details are buried or missing. Please move to main text or clearly state in appendix: full definition of IntersectionRNN cell and unroll scheme, exact tactile augmentation parameters (rotation range, flip prob., temporal reversal prob.), and optimizer/learning-rate schedule/batch size/etc.", "limitations": "Mostly yes. Related to my comments above, the training-test domain gap should be better acknowledged/addressed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This paper delivers an end-to-end demonstration from realistic whisker physics through large-scale training to barrel-cortex validation. It is allegedly the first work to reach the noise ceiling in tactile cortex and thereby may illuminate key somatosensory inductive biases. The open release of code, simulator, and data further boosts value for both neuroscience and robotics. However, the current draft may over-claim: training on passive ShapeNet sweeps but testing on a six stimulus active-whisking set risks spurious ceiling-level fits, and architecture rankings lack formal statistics.\n\nQUALITY\n\nStrengths:\n* From a biomechanically faithful simulator to large-scale model training and direct neural comparison, the experimental flow is coherent and correctly executed.\n* 60+ Encoder-Attender-Decoder variants cover feed-forward, state-space, recurrent and attention architectures under both supervised and self-supervised regimes.\n* Split-half reliability filtering, noise-ceiling correction and layer-wise RSA are the accepted best practices for small-stimulus neural datasets.\n\nWeaknesses:\n* Models learn on passive sweeps of ShapeNet objects but are judged on active whisking to six simple shapes. This mismatch may inflate RSA by chance; an ablation including these test shapes in the training mix (held out for validation) is missing.\n* Bar plots lack formal significance tests; we do not know if IntersectionRNN > ResNet is reliable across mice\n* Six objects restrict the dimensionality of representational geometry and make it easier to “hit the ceiling.” Confidence intervals on the noise ceiling itself are not shown.\n* Search budget (GPU-days, hyper-parameter tuning strategy) is only sketched, making it hard to gauge efficiency or carbon cost.\n\nCLARITY\n\nStrengths:\n* Figures are information-dense yet readable; architecture, dataset and result diagrams are clear.\n* The summary of tactile augmentations vs. image-style augmentations is easy to follow and motivates the SSL results\n\nWeaknesses:\n* Introduction and related-work sections are bloated with robotics-hardware history that obscures the main contributions.\n* Key implementation details (augmentation hyper-parameters, exact RSA equation, IntersectionRNN definition) are relegated to the appendix. Some should appear in the main text.\n\nSIGNIFICANCE\n\nStrengths:\n* First convincing “goal-driven” account of somatosensory cortex\n* demonstrates that recurrent encoders plus force/torque-aware contrastive learning outperform common feed-forward baselines\n\nWeaknesses:\n* Impact on mainstream ML may be limited. The architectural lesson (“use ConvRNNs for smooth temporal signals”) is incremental relative to earlier vision work.\n* The neural dataset’s narrow scope means conclusions about “saturating variance” could be overturned once richer recordings emerge.\n\nORIGINALITY\n\nStrengths:\n* Combines three elements that have not previously appeared together: (i) a 3-D physically correct whisker array, (ii) a systematic temporal-network search that spans ConvRNN, SSM and attention, and (iii) noise-corrected RSA on barrel-cortex data.\n* Shows that SimCLR with bespoke tactile augmentations rivals supervised learning. This is a novel result in somatosensation and a direct answer to an open question from Zhuang et al. 2017\n\nWeaknesses:\n* The Encoder-Attender-Decoder framing is more a tidy taxonomy than a fundamentally new modelling concept; each module comes from prior work.\n* Attention module analysis is modest. Gains are small and mechanistic interpretation remains speculative", "was_revised": true, "final_justification": "The authors have addressed my main concerns (passive sweep vs active whisking, noise ceiling, stats) and pointed me to other details that I either overlooked or that were clarified. I have raised my score accordingly", "rebuttal": "Thank you for taking the time to review and provide your valuable feedback! We address the weaknesses and questions as follows.\n\n### Passive sweeps vs. active whisking and including neural fit shapes during training\nThis is a great question. We are already generalizing because the shapes that are used in the neural evaluation are not present, even in distribution, in the ShapeNet training data. The reason why we chose to do passive sweeps for the large sweeping dataset on ShapeNet is that, especially with the temporal flipping used in the tactile augmentations, active whisking is isomorphic to doing multiple passive sweeps. Therefore, passive whisking is a more scalable implementation of active whisking, which is important for generating pretraining dataset diversity, which is in turn is important for matching brain data by the “Contravariance Principle” [1, 2, 3].\n\n[1] R. Cao and D. Yamins. “Explanatory Models in Neuroscience, Part 1: Taking Mechanistic Abstraction Seriously.” Cognitive Systems Research, vol. 87, Sep. 2024, p. 101244. ScienceDirect, https://doi.org/10.1016/j.cogsys.2024.101244.\n\n[2] D. Yamins and J. DiCarlo. “Using Goal-Driven Deep Learning Models to Understand Sensory Cortex.” Nature Neuroscience, vol. 19, no. 3, Mar. 2016, pp. 356–65. PubMed, https://doi.org/10.1038/nn.4244.\n\n[3] A. Nayebi*, N. C. Kong*, C. Zhuang, J. L. Gardner, A. M. Norcia, and D. L. Yamins. “Mouse Visual Cortex as a Limited Resource System That Self-Learns an Ecologically-General Representation.” PLoS Computational Biology, vol. 19, no. 10, Oct. 2023, p. e1011506. PubMed, https://doi.org/10.1371/journal.pcbi.1011506.\n\n\n### Low number of stimuli and confidence intervals for the noise ceiling\nThe mouse neural dataset we are evaluating against only has 2 different shapes and 3 distances which we can use as unique stimuli. This was the most recent viable candidate for somatosensory cortex data that was available. We agree this is an unfortunate limitation that reduces the generalizability of our findings; though we note that low numbers of stimuli are a common occurrence in neuroscience and our task-optimized modeling approach, along with our inter-animal consistency analysis, highlights the need for future neural datasets to move forward in this concrete direction of collecting more stimuli. This was noted in the “Limitations and Future Directions” paragraph in the Discussion section: *“Most importantly, current tactile neural datasets remain limited in stimulus diversity and the number of object conditions tested, restricting the captured neural variability and could be the reason why inter-animal consistency values are lower for the statistical average between animals than the current pointwise empirical maximum of 1.3.”* (Note: Statistically, Pearson's correlation value (r) is within the range [-1, 1] but the split-half noise correction can result in values >1 when self-consistency is low.) Finally, since we train the models on the large-scale tactile whisking task rather than fit to the neural data directly, we note that our models can be evaluated *without change* on future neural datasets that collect more stimuli. Therefore, our models and results here are not the final say, but a platform for future exploration and evaluation.\n\n### Statistical significance of model comparisons across mice\nThe error bars in Figure 4 show the SEM of the median neural fit score across animals.\nAdditionally, we find that the Wilcoxon significance on the neural score across 11 mice for Inter+SimCLR (best neural score model) & Resnet+Mamba+SimCLR (best task score out of self-supervised models) gives statistic=2.0, p=0.0425 which is significant (p<0.05). We will include this additional result in the paper!\n\n### Search budget and computational efficiency\nWe already report the GPU hours in line 191 in the main text and the training details (e.g., hyper-parameter tuning strategy) in appendix A2 Model Training. We will provide a more detailed description on the search and training cost in our revised paper.\n\n### Missing details in main text & relevance of robotics in introduction\nWith the additional page provided for the camera-ready version, we are happy to include the augmentation hyper-parameters, exact RSA equation, IntersectionRNN definition in the main text. We will also revise the robotics-hardware history in the introduction section to clarify the primary message and result of our work. \n\n### Impact on mainstream ML and contribution to architectural insights for temporal tactile modeling\nOne of our main contributions is the Encoder-Attender-Decoder (EAD) framework, which enables comprehensive and systematic model search with diverse neural network architectures. Despite the similar architectural conclusion, our tactile processing is different from prior vision works as the input of the model is forces/torques other than pixels that are easier to understand. We also found that we had to create our own data augmentations for self-supervised tactile learning, as typical image augmentations did not scale for this dataset [Figure 5c]. Furthermore, to the best of our knowledge, we are the first to compare tactile models to brain data for neural alignment test; hence, our findings are well-supported by both the EAD framework and the neural alignment test for temporal tactile processing.\n\n### The EAD architecture as a unified search framework\nAs mentioned in abstract line 3-4 and line 166-168, we agree that the EAD framework offers a comprehensive solution that allows searching through different model architectures other than a new modeling concept for tactile processing.\n\n### Attention module analysis and performance gains\nCompared to the choice of encoders and augmentations, the performance gain of different attention modules are indeed relatively small, as discussed in paragraph “GPT-based Attenders Provide Modest Improvements in Task Performance and Neural Alignment” in line 274-285, hence, more analyses are focused on the encoders and augmentations."}]

Author rebuttal:

Meta-review: This paper systematically explores neural network architectures for tactile perception inspired by the rodent whisker system. The authors propose an Encoder-Attender-Decoder framework and train various models using realistic simulated tactile data from virtual whisker interactions with 3D objects. They demonstrate that convolutional recurrent neural networks outperform other architectures on tactile object categorization tasks. and exhibit remarkable alignment with neural recordings from rodent somatosensory cortex.
It reports the first application of model-brain-alignment methods to tactile processing across a broad set of model architectures and both supervised and self-supervised training objectives. It provides a powerful and direct comparison between a large set of models and rodent somatosensory cortical responses, surpassing average inter-animal consistency. It quantitatively identifies nonlinear recurrence as a key inductive bias in rodent tactile processing, guiding future development in embodied tactile AI systems.
Because of the many strengths, the paper is deemed of great interest for the NeurIPS community.
Decision: Accept (oral)