dataesr/scientific-paragraphs-categorization huggingface dataset:
https://github. com/gatechatl/ JUMPg/ PGMiner (2016) [63] Command √ - - √ √ This workflow allows acquisition of mass Improves genome line spectrometric data, peptide identification against annotation preprocessed sequence databases, assignment of statistical confidence to identified peptides, and mapping confident peptides to gene models. https://github. com/olalonde/pgtools PROTEO-FORMER Command √ - √ √ √ RIBO-seq NGS data are processed to delineates Identification of (2015) [64] line, Virtual proteoforms. RIBO-seq-derived sequences are novel translation machine, then translated and mapped to a public data- products Galaxy tools base, creating a custom search database for pep- tides to MS/MS matching. PGTools (2015) [65] Command √ - - √ √ The software is divided into 2 phases: Phase 1 Improves genome line contains 8 modules to analyse MS/MS data using annotation known proteins databases.
Phase 2 contains 5 modules and 7 customized databases that allow MS/MS data to be analysed against the genome. That software includes applications, libraries, customized databases and visualization tools. NextSearch (2015) [66] Command - - - √ √ Nucleotide EXon-graph Transcriptome Search Improves genome line identifies peptides by directly searching the nu- annotation cleotide exon graph against tandem mass spec- tra. NextSearch outputs which are the proteome- genome/transcriptome mapping that can be vi- sualized using public tools. ProteoAnnotator (2014) Command √ - √ √ √ MS spectrum are queried by one or several Improves genome [52] line, Stand proteomics databases search engines (MASCOT, annotation alone OMSSA, X!Tandem or MSGF+) and results are application converted into GFF adding genome coordinates and statistical confidence values.
It exports mzIdentML files. http://www. proteoannotator. org Peppy (2013) [67] Command √ - N/A √ - This workflow generates a peptide database Improves genome line, Stand from a genome, tracks peptide loci, matches annotation alone peptides to MS/MS spectra and assigns FDR application confidence values to those matches. http://geneffects. com/peppy Protk (released in 2012) Command √ - √ - √ It is a suite of tools for proteomics providing the Improves genome line, Galaxy following analysis tasks: (i) MS/MS data search annotation tools with X!Tandem, Mascot, OMSSA and MS-GF+; (ii) peptide and protein inference with Peptide.
We demonstrate the efficacy of ReStoCNet and the presented HB-STDP based unsupervised training methodology on the MNIST and CIFAR-10 datasets. We show that residual connections enable the deeper convolutional layers to self-learn useful high-level input features and mitigate the accuracy loss observed in deep SNNs devoid of residual connections. The proposed ReStoCNet offers >20× kernel memory compression compared to full-precision (32-bit) SNN while yielding high enough classification accuracy on the chosen pattern recognition tasks.
We note that  Suri et al. (2013)  proposed an STDP-based learning rule employing constant switching probabilities, where the potentiation and depression windows extend over the entire STDP timing window. On the contrary, HB-STDP contains dead zone in the STDP timing window, where the switching probability is zero. We visually demonstrate the significance of dead zone for efficient feature learning using binary fully-connected SNN. We present HB-STDP based layer-wise unsupervised training methodology for ReStoCNet, where we train the binary kernels interconnecting successive convolutional layers using HB-STDP. Once a given layer is trained, we forward propagate the spikes from the input through the trained layers and update the binary kernels of the following convolutional layer. After all the convolutional layers are trained, we feed the input dataset, estimate the spiking activations of the spatially pooled convolutional spike maps by accumulating the spikes at every time instant and decaying the resultant sum between successive spike timing instants, and pass them on to the fully-connected layer, trained using error backpropagation  (Rumelhart et al.
, 1986) , for inference. We validate the efficacy of ReStoCNet and the HB-STDP based unsupervised training methodology on the  MNIST (LeCun et al. , 1998)  and CIFAR-10 datasets  (Krizhevsky, 2009). We show that residual connections enable the deeper convolutional layers to extract useful high-level input features and effectively mitigate the accuracy degradation observed in deep SNNs devoid of residual connections  (Lee et al. , 2018b). We note that  Masquelier and Thorpe (2007) ,  Panda and Roy (2016) ,  Lee et al. (2016) ,  Stromatias et al. (2017) ,  Srinivasan et al. (2018) ,  Tavanaei et al. (2018) ,  Kheradpisheh et al. (2018) ,  Ferré et al. (2018) ,  Thiele et al. (2018) ,  Lee et al. (2018b,a) , and  Mozafari et al. (2018)  have demonstrated convolutional SNNs composed of full-precision kernels.
We propose ReStoCNet, a deep residual convolutional SNN composed of binary kernels, for memoryefficient neuromorphic computing. 2. We present HB-STDP, an STDP-based probabilistic learning rule incorporating Hebbian and anti-Hebbian learning mechanisms, for training the binary kernels constituting ReStoCNet in a layer-wise unsupervised manner for hierarchical input feature extraction. 3. We validate the efficacy of ReStoCNet on the MNIST and CIFAR-10 datasets, and show that residual connections enable the deeper convolutional layers to learn useful high-level input features and mitigate the accuracy loss incurred by STDP-trained deep SNNs without residual connections. all the convolutional layers are trained, the respective spike maps are spatially pooled using average pooling with 2×2 unit-weight kernels followed by Integrate-and-Fire (IF) spiking non-linearity to produce the pooled spike maps.
3 and enhances the scalability of deep SNNs. After all the convolutional layers are trained, we feed the input dataset and spatially pool the spike maps of the convolutional layers. Spatial pooling is the mechanism used to suitably combine the neighboring pixels of a convolutional feature map to reduce the map size (height and width) while retaining the salient features. Spatial pooling also renders the network invariant to slight translations in the input features  (Jaderberg et al. , 2015). We perform a class of spatial pooling operation known as average pooling with 2×2 kernels composed of unit weights and stride length of 2 as detailed below. The spikes in every 2×2 non-overlapping region of the convolutional maps are summed up and normalized by the kernel size (4 for a 2×2 kernel) to produce the pooled output maps, which are then fed to a layer of Integrate-and-Fire (IF) spiking neurons to generate the pooled spike maps.
On the contrary, input images when pre-processed by normalizing the raw pixel intensities to zero mean and unit variance result in normalized images with positive and negative pixel intensities. The normalized pixels with negative intensities are mapped to inhibitory pre-neurons firing unit negative spikes. The normalized input maps containing excitatory and inhibitory pre-neurons offer richer spike-encoding of the image patterns, resulting in efficient STDP-based feature learning. We find that input normalization is critical for natural images like those from the CIFAR-10 dataset  (Krizhevsky, 2009)  that do not have clear separation between the region of interest and the background unlike digit patterns from the MNIST dataset  (LeCun et al. , 1998). Binary synapses require a probabilistic learning rule to prevent rapid switching of the weights between the allowed levels, which could otherwise render the synapses memoryless.
RESULTS We first validate the efficacy of HB-STDP, by visually demonstrating the significance of having distinct potentiation and depression windows separated by a dead zone for efficient feature learning, using two-layer binary fully-connected SNN trained on the MNIST dataset. We then comprehensively evaluate ReStoCNet and the presented HB-STDP based unsupervised mini-batch training methodology on the MNIST and CIFAR-10 datasets. We show that the residual connections are critical to achieving efficient unsupervised learning in deeper convolutional layers and minimizing the accuracy degradation incurred by STDP-trained deep SNNs without residual connections. We use the classification accuracy on the test set and the synaptic memory compression obtained by using binary kernels as the evaluation metrics for ReStoCNet compared to full-precision (32-bit) SNN under iso-accuracy conditions.
4(a). 2. eHB-STDP2 -This is a variant of the eHB-STDP learning rule where the dead zone is replaced with a wider Hebbian potentiation window as depicted in Fig. 4(b ). 3. eHB-STDP3 -This is an alternative variant of the eHB-STDP rule where the dead zone is replaced with a wider anti-Hebbian depression window as illustrated in Fig. 4(c ). Note that the excitatory↔inhibitory synaptic weights are fixed a priori and are not subjected to STDP-based learning. We simulated the fully-connected SNN using BRIAN  (Goodman and Brette, 2008) , which is an open-source SNN simulation framework, on the MNIST dataset. The input image pixels are converted to Poisson spike trains firing at a rate constrained between 0 and 63. 75Hz depending on the respective pixel intensities for a simulation period of 350ms.
The binary fully-connected SNN of 400 neurons trained using eHB-STDP yielded 79. 94% accuracy on the MNIST test set, which is higher by >8% compared to that achieved using the remaining eHB-STDP variants. The accuracy can be further improved by increasing the number of excitatory neurons as shown in Fig. 5(b). We now estimate the synaptic memory compression offered by the binary SNN compared to full-precision (32-bit) SNN, which is specified by synaptic memory compression = #input neurons × #excitatory neurons f ull-precisionSN N × 32 #input neurons × #excitatory neurons binarySN N × 1 (6) where #input neurons is 784 for the MNIST dataset. Fig. 5 (b) indicates that binary SNN of 6400 neurons offers comparable accuracy (∼92%) to that provided by full-precision (32-bit) SNN of 1600 neurons  (Diehl and Cook, 2015) , leading to 8× synaptic memory compression under iso-accuracy conditions.
ReStoCNet for MNIST Digit Recognition The MNIST dataset contains 60,000 training patterns and 10,000 test patterns of handwritten digits that are stored as 28×28 Greyscale images. In this work, we developed a custom simulation framework using Pytorch  (Paszke et al. , 2017)  to evaluate ReStoCNet and the presented HB-STDP based unsupervised training methodology. The simulation parameters for the Leaky-Integrate-and-Fire (LIF) neuron in the convolutional layers and the Integrate-and-Fire (IF) neuron in the spatial pooling layers are shown in Table  3. The binary kernels in every convolutional layer are initialized to logic high state (w high ) with a probability, p high , which is specified by p high = α weight init f an in + f an out (7) where α weight init is the proportionality constant controlling p high , and f an in and f an out are the total number of input and output synaptic weights, respectively, for a given convolutional layer.
The remaining kernel weights in the convolutional layer are initialized to logic low state (w low ). The firing threshold of the LIF neurons in every convolutional layer are initialized to zero. We first simulated a 16C3-2P-10FC ReStoCNet, composed of single convolutional layer with 16 maps and 3×3 binary kernels followed by pooling layer whose spiking activations are directly fed to the final softmax layer. The input image pixels are mapped to excitatory pre-neurons firing at a rate constrained   3. We trained the convolutional layer in ReStoCNet using 2000 MNIST digit patterns with a mini-batch size of 200. We thereafter fed the entire training dataset to ReStoCNet, spatially pooled the spike maps of the convolutional layer, and low-pass filtered the pooled spike trains over a simulation period of 100ms to estimate their spiking activations.
ReStoCNet for CIFAR-10 Image Recognition The CIFAR-10 dataset contains 50,000 training images and 10,000 test images, 32×32×3 in dimension, spanning 10 output classes. We pre-processed the CIFAR-10 images using global contrast normalization followed by ZCA whitening  (Krizhevsky, 2009). Global contrast normalization is performed by subtracting and scaling the pixel intensities of each input channel by the corresponding mean and standard deviation computed over the training set. The normalized image is then transformed by multiplying with whitening filters as explained in  Krizhevsky (2009) , which enables a network to learn higher-order pixel correlations. Fig. 7  illustrates a few original and pre-processed images from the CIFAR-10 dataset. The simulation parameters used for training the convolutional layers are provided in Table  3  while those used for training the fully-connected layer are listed in Table  4.
The binary kernels in ReStoCNet-1 are trained using 5000 images, with mini-batch size of 200, for simulation period of 25ms per mini-batch training iteration. Note that we used a simulation time-step of 1ms. Fig. 8 (a) illustrates the low-level input features self-learnt by the binary kernels, enabled by the e/iHB-STDP based unsupervised training methodology. The shallow ReStoCNet-1, wherein the fully-connected layer is trained on the entire dataset, yielded 64. 31% test accuracy that is higher than an accuracy of 59. 42% obtained using randomly initialized binary kernels and zero firing thresholds in the convolutional layer. In order to determine if accuracy loss is incurred as a result of using binary kernels, we trained ReStoCNet-1 composed of full-precision (32-bit) kernels using standard exponential STDP rule  (Song et al.
Fig. 9(d)  shows that ReStoCNet-2 provides only modest improvement in accuracy as the number of output maps is increased in the second convolutional layer. The accuracy limitation is caused by the inability of the unsupervised training methodology to effectively optimize an over-parameterized network. Finally, we evaluated a deeper 36C3-36C3-36C3-2P-1024FC-10FC ReStoCNet, referred to as ReStoCNet-3, composed of three convolutional layers as depicted in Fig. 1. We inverted the residual inputs to the third convolutional layer to ensure diversity in the residual maps received by the second and third layers from the input layer. We trained the third convolutional layer with the same hyperparameters (shown in Table  3 ) as those used for training the second convolutional layer, albeit on a different subset of 5000 images from the CIFAR-10 dataset.
Second, it reveals that the maximum achievable accuracy is limited by the STDP-based unsupervised training methodology as further corroborated by Fig. 11 , which illustrates the unsupervised clustering capability of ReStoCNet-3 for different training images from the CIFAR-10 dataset. In order to visualize the efficiency of unsupervised clustering offered by ReStoCNet-3, we reduce the dimension of the pooled spiking activations of the convolutional layers using Principal Component Analysis (PCA) followed by t-Distributed Stochastic Neighbor Embedding (t-SNE)  (Maaten and Hinton, 2008) , and plot the first two t-SNE components for the training images. The t-SNE dimensionality reduction technique computes pair-wise similarities between the data points (images) in the high-dimensional space and projects them to a low-dimensional space that preserves the measured similarities.
We refer the readers to  Maaten and Hinton (2008)  for a review of the t-SNE algorithm for visualizing high-dimensional input data. Fig. 11(a)  shows the t-SNE scatter plot for 15,000 training images spanning three different classes from the CIFAR-10 dataset, namely, airplane, bird, and frog. The primary objective of any machine learning model is to cluster the images per class together while ensuring sufficient separation among different classes. The t-SNE scatter plot of the pooled spiking activations of ) indicates that, although distinct clusters are formed for the images in each class, there exists considerable overlap among different image clusters.
On the contrary, very few works have benchmarked convolutional SNNs, trained using unsupervised learning algorithms, on the CIFAR-10 dataset. Panda and Roy (2016)  proposed spike-based convolutional Auto-Encoders, where the kernels in every convolutional layer are trained in an unsupervised manner using error backpropagation to regenerate the input spike patterns. Ferré et al. (2018)  presented convolutional SNN (without residual connections), where the kernels are trained using a simple Hebbian STDP learning rule. Table  6  shows that ReStoCNet provides 4-5% lower accuracy than that reported in both the related works. In particular, a 256C3-2P-1024FC-10FC ReStoCNet yields 4. 97% lower accuracy than that provided by the 64C7-8P-512FC-512FC-10FC convolutional SNN  (Ferré et al. , 2018)  while offering 21.
On the CIFAR-10 dataset, ReStoCNet provided 66. 23% accuracy and 21. 7× kernel memory compression, albeit with 5% accuracy degradation compared to full-precision convolutional SNN. We believe that ReStoCNet, with event-driven computing capability and memory-efficient probabilistic learning with binary kernels, is ideally suited for neuromorphic hardware implementations based on CMOS and stochastic emerging device technologies like Resistive Random Access Memory, Phase-Change Memory, and Magnetic Tunnel Junctions that can potentially lead to much improved energy efficiency in battery-powered IoT edge devices. , Bichler, O. , Gamrat, C. , et al. (2015). Spin-transfer torque magnetic memory as a stochastic memristive synapse for neuromorphic systems. IEEE transactions on biomedical circuits and systems 9, 166-174 Wijesinghe, P.
ACKNOWLEDGMENTS This work was supported in part by the  Center for Brain Inspired Computing (C-BRIC) , one of the six centers in JUMP, a  Semiconductor Research Corporation  (SRC) program sponsored by  DARPA , by the  Semiconductor Research Corporation , the  National Science Foundation ,  Intel Corporation , the  DoD Vannevar Bush Fellowship , and by the  U. S. Army Research Laboratory  and the  U. K. Ministry of Defense  under Agreement Number  W911NF-16-3-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the  U. S. Army Research Laboratory , the  U. S. Government , the  U. K. Ministry of Defence  or the  U. K. Government.
Recording data with BrainVision recorder software (versions A and B) Data is recorded using custom workspaces in the BrainVision Recorder software. Workspaces contain user-defined recording settings, such as amplifier parameters, cap configuration, electrode  positions and channel names. The three different workspaces needed are provided in the Supplementary Materials 1, 2 and 3. First, impedance measurements should be checked by two individual 32-channels workspaces corresponding to each participant/amplifier. For organization purposes, we named one participant as Subject A (e. g. left participant while facing the screen) and the other as Subject B. Finally, data from both participants are recorded in a 64-channel workspace, where the first 32 channels belong to the amplifier connected to the USB2 Adapter's port labelled Fiberoptic 1 and the second half belong to the amplifier connected to the port labelled Fiberoptic 2.
This is facilitated by a 25-pin serial port y-cable: female/2 male (see Fig. 6 ) that connects one stimulation computer (with E-prime, Psychtoolbox, or other stimulation software) to both recording computers via amplifiers. The two recording computers thus simultaneously receive the same trigger/marker, which can be used to synchronize the datasets. The impedance measurements and data collection require two individual 64-channels workspaces that correspond to each participant, amplifier and computer.
Necessary hardware (64-channel measurement per participant) ActiveTwo AD-Box with 7 Â 8 channel Amplifiers with (retrofit) optical fiber input ('master' AD-box) ActiveTwo AD-Box with 7 Â 8 channel Amplifiers ('slave' AD-box) Battery box (attached to AD-Box). 2 units Battery Charger. 2 units USB2 Receiver + cables. 1 unit Data acquisition computer with BioSemi ActiView recording software. 1 unit Stimulation computer. 1 unit Fiber optic cable. 2 units Headcaps. 2 units Pin-Type Active-electrodes. 4 sets (BioSemi electrode sets have 32 electrodes on a common connector) Flat-Type Active-electrodes for mastoid reference and/or EOG/EMG/ECG. Head Box. 2 unitsthis is optional and depends on the number of Flat-Type Active-electrodes that need to be connected and the (practical) functionality of the input sockets in the AD-Box.
Fig. 8  shows the BioSemi ActiveTwo hyperscanning setup. Two caps with 64 electrodes (installed after cap placement) are each connected to a separate AD-Box with amplifiers. One of the two AD-Boxes is the 'master' system, which is connected to a USB2 Receiver, and is (Daisy chain) linked to the 'slave' system via a fiber optic cable (the AD-boxes need a different 'speed' setting: see Figs. 8  and 9 ). This allows considerable spatial separation between participants without losing temporal alignment between the two recordings. The USB2 Receiver receives triggers/markers from the stimulus computer, and this information is combined with the data from both AD-boxes and sent to the software acquisition computer via USB2. In this implementation, the optical fiber input socket in the 'master' AD-box is necessary, but it is not part of the standard AD-box configuration.
New systems can be ordered with this feature, and existing systems can be upgraded by BioSemi (provided that they are compatible with the retrofit). Each participant has individual REF and GND (active CMS/passive DRL loop) channels, which are connected to the individual AD-boxes. In the acquisition software (ActiView), the EEG data from both participants (and the triggers/markers) can be viewed separately, but the data is stored in one data-file with distinctive labels for each participant. 8. Overall schema of the EEG hyperscanning setup for the BioSemi ActiveTwo system. The data from the 'slave' AD-box is fed to the 'master' AD-box via a fiber optic cable (in blue), from which it is sent to the USB2 receiver that sends all data and trigger/marker information to the acquisition computer. Up to three 'slave' AD-boxes can be linked to the 'master' via a Daisy chain connection.
Acknowledgements This work was supported by the  Social Sciences and Humanities Research Council (SSHRC)  of  Canada  ( IDG 430-15-00647 ) and the  Natural Sciences and Engineering Council (NSERC)  of  Canada  ( RGPIN-2017-06053 ) to  Philip J. Monahan. PB was supported by the  Associative Research Program of CONICYT  under Grant Basal Funds for Centers of Excellence  FB0003  and the  FONDEQUIP  project  EQM 150003.
The main objective of ViSEAGO package is to carry out a data mining of biological functions and establish links between genes involved in the study. We developed ViSEAGO in R to facilitate functional Gene Ontology (GO) analysis of complex experimental design with multiple comparisons of interest. It allows to study largescale datasets together and visualize GO profiles to capture biological knowledge. The acronym stands for three major concepts of the analysis: Visualization, Semantic similarity and Enrichment Analysis of Gene Ontology. It provides access to the last current GO annotations, which are retrieved from one of NCBI EntrezGene, Ensembl or Uniprot databases for several species. Using available R packages and novel developments, ViSEAGO extends classical functional GO analysis to focus on functional coherence by aggregating closely related biological themes while studying multiple datasets at once.