DOI: 10.5061/dryad.rxwdbrvh3
title: Spike-timing based coding in neuromimetic tactile system enables dynamic object classification

description: Coding dynamic tactile information in spike timing is essential to human
 haptic exploration and dexterous object manipulation. Conventional
 electronic skins generate frames of tactile signals upon interaction with
 objects and are unfortunately ill-suited for efficient coding of temporal
 information and rapid feature extraction. Here, we report a neuromorphic
 tactile system that uses spike timing, especially the first-spike timing,
 to code dynamic tactile information about touch and grasp. This strategy
 enables the system to seamlessly code highly dynamic information with
 millisecond temporal resolution on par with the biological nervous system,
 yielding dynamic extraction of tactile features. Upon interaction with
 objects, the system rapidly classifies them in the initial phase of touch
 and grasp, thus paving the way to fast tactile feedback desired for
 neuro-robotics and neuro-prosthetics.
description: The method is provided in SM. In brief, the raw data of spike
 trains in voltage were generated and collected by using the e-skin that
 touched surfaces and grasped objects. The raw data was converted to
 digital spike trains with 1 as a spike and 0 as non-spike. The processed
 spike trains were analyzed to study the encoding performance of the system
 and were used to train and test the spiking neural networks for
 classification tasks. The tabular data provided are corresponding to the
 main text figures (i.e., Fig 2, Fig 3 and Fig 4). 
description: # Spike-Timing Based Coding in Neuromimetic Tactile System Enables Dynamic
 Object Classification ## Explanation of Figures * Fig 2C: Typical spike
 trains in the ensembles of afferents (with indices 29 through 32 and 36
 through 39) on the fingertip in response to touching the hard hemisphere
 (left) and soft hemisphere (right). The spike trains have been
 digitalized. The center of spike trains is intentionally aligned to t=2s
 just for illustration. * Fig 2D: Activity over population, which is the
 time-dependent spike rate by counting spikes in all afferents within a bin
 size of 0.01 s, in response to touching the hard hemisphere (left) and
 soft hemisphere (right). * Fig 2F: 1st-spike patterns (radar maps)
 illustrating the 1st-spike sequences and latencies of the ensemble of
 afferents, in response to touching the hard hemisphere (left) and soft
 hemisphere (right). The distance from the pole defines the relative
 1st-spike latency to t0, while the angle denotes the artificial afferent
 that fired the 1st spike. Here, all unrecruited afferents are set at t=10s
 just for illustration. The moment that the first activated afferent fires
 its first spike is intentionally set at t0=0s. * Fig 3A: Visualization of
 the outputs of the trained network of spiking neurons for testing samples
 using t-SNE in response to touching four surfaces varied in curvature and
 stiffness. Each point represents the outputs of a testing sample. Symbol
 denotes the actual label of the sample. There are 10 testing samples in
 each surface category. * Fig 3B: (Top) time-dependent classification
 accuracy of all testing samples that were classified according to
 curvature, stiffness, and both curvature and stiffness, respectively.
 (Bottom) time-dependent first-spike count and total spike count, both of
 which were accumulated over time. The spike count counts spikes from all
 afferents. t=0 denotes the moment that the first activated afferent fires
 the first spike. * Fig 3C: Time-dependent classification accuracy of the
 testing samples in each surface category, that were classified according
 to curvature and stiffness, respectively. * Fig 3D: Statistics of the time
 needed for half of the testing samples in each category being classified
 correctly as curvature or stiffness. * Fig 3F: Stimulating-spike-train
 output frequencies for surfaces in each category with different stiffness.
 * Fig 3G: Correlation between hindlimb twitching angle and surfaces’
 stiffness, which is calculated based on the stimulation frequency used. *
 Fig 4A: Typical time-dependent activity over population during grasping an
 apple by subject 1. t=0 denotes the moment that the first activated
 afferent fires its 1st spike. * Fig 4B: Typical 1st-spike pattern of
 grasping the apple. The tactile afferents are re-sequenced to show the
 cooperativity between different parts of the hand during grasping. All
 unrecruited afferents stayed at t=0. The moment that the first activated
 afferent fires its first spike is intentionally set at t0=0.1s just for
 illustration. So, the timing of all spikes is increased by 0.1s
 accordingly. * Fig 4C: Visualization of the outputs of the trained network
 of spiking neurons using t-SNE in response to the grasp of 22 objects.
 Each point represents the output of one grasp event, of which the object’s
 actual label is denoted by the corresponding symbol. * Fig 4D:
 Time-dependent classification accuracy and 1st-spike count during
 grasping. * Fig 4E: Maximum first-spike latency, averaged over testing
 samples in each object category, plotted against the classification time
 of a = 52%. Maximum first-spike latency is the latency between first and
 last appeared first spikes in all afferents. Linear fitting result and
 correlation analyzing results are also presented. * Fig 4F: Normalized
 first-spike count (a = 52%), plotted against the corresponding
 classification time. Normalized first-spike count is the first-spike count
 at the moment normalized by the total recruited neuron number when the
 object is fully grasped. Linear fitting result and correlation analyzing
 results are also presented. ## Description of the data and file structure
 Data in this upload are organized based on the figure they appear in the
 publication: Spike-timing based coding in neuromimetic tactile system
 enables dynamic object classification ## Sharing/Access information Links
 to other publicly accessible locations of the data: N/a Other sources: N/a