dataesr/scientific-paragraphs-categorization huggingface dataset:
B. Spintronic-based Dropout Model Description This section describes the Spintronic-based Dropout (called in the following -SpinDrop) neural network model. As mentioned previously, normal Dropout sample a random Bernoulli mask with a fixed dropout probability p = [0, 1]. As the dropout is usually implemented at software level or in a digital fashion, the dropout probability itself is deterministic. On the other hand, implementation of the SpinDrop module with stochastic and analog components will result in a dropout probability p which can vary and can be considered as stochastic. The stochasticity here could also be coming from manufacturing variations and/or thermal noise. With SpinDrop, the feed-forward operation of a layer l becomes: pl = p l + ϵ, ϵ ∼ N (µ, σ 2 ) Ml ∼ Bernoulli(1 -pl ) Z b l-1 = sign(Z l-1 ) Ẑb l-1 = Z b l-1 * Ml W b l = sign(W l ) Z l = S l (W b l Ẑb l-1 α + b l ) (10) Here, the mean µ and standard deviation σ comes from device technology, M denotes dropout mask with process variation, * denotes elementwise product, p is the dropout probability with variations and the subscript b denotes binary matrix or vector.
In addition, a wordline decoder with the capability to enable multiple consecutive addresses is used. One SpinDrop module, described in the previous section, is connected to each wordline pair to implement the Dropout concept described earlier. A bitline conditioning circuit is used to set the bitline according to the inputs. One ADC per sensed bitline and a sourceline conditioning circuit are connected to the output, to sense and convert the state of the enabled cells. Further to that, they are provided to a CMOS-based Accumulator-Adder module to sum up the partial Matrix-Vector product results according to Equation  2. Finally, a comparator and a running average CMOS circuit complete the schematic, ensuring that the computation achieves the predictive mean needed for the Bayesian inference presented in Section II-B4.
The sourceline periphery (ADC, accumulatoradder, and so on) can be shared by multiple sourcelines using MUXes to save chip area by reusing temporary un-useful components. The output of the CMOS periphery of the crossbar can then be provided to input to the next crossbar that also respects the same architectural design. Moreover, as Bayesian inference with Dropout approximation requires spatial and temporal independence, the probability that a neuron is dropped is independent of one another and also from one input to another. To achieve temporal independence, the proposed SpinDrop module randomly activates and inactivates each word-line with probability p and 1 -p, respectively, during each forward path. In addition, to achieve spatial independence, a separate SpinDrop module is used for each word-line of the crossbar.
As only a limited number of cells per bitline can be reliably sensed at once  [34] , our architecture supports the computation of group-wise CiM operations. Based on parallel measurable cells Z CIM , the array with Z l-1 inputs (stored in 2 × Z l-1 rows) is split into groups of Z groups = Z l-1 /Z CIM. Each group is selected one by one via the wordline decoder, and the SpinDrop modules are used to drop out an input by disabling the wordline of the respective input neuron. The implementation of the SpinDrop module is performed by adding a pass-gate that allows either the access to the classical decoder or to the stochastic wordline (WL). Here one SpinDrop module is used per wordline, but this can be reduced down to four (depending on the maximal CiM operation group size) and thus multiplexed among the different group operations.
A. Simulation Setup For the architectural simulation, we first obtained the circuit characteristics of the peripheral blocks described in Section III-C. The Acumulator-Adder, Comparator and Averaging circuits were synthesized with the Synopsys Design Compiler using the TSMC 40 nm low power PDK based standard IP libraries. Decoding and sensing for the CiM operation were evaluated at the circuit-level array using NVsim  [35]. For doing that, we adjusted the NVsim simulation with the data presented in Section IV-B to account for four active cells, thus modeling the CiM operation. We have also replaced the single bit sense amplifiers with multi-bit ADCs. The results for each individual component are shown in Table  I. To evaluate both predictive performance and uncertainty estimations, we trained an MLP with four layers (256 neurons per layer for three hidden layers the last one being dependent on the dataset) on the MNIST dataset and a CNN with a LeNet-5 and one with a VGG topology (each with nine layers) on the MNIST and CIFAR-10 datasets.
For both, we used the Adam optimization algorithm with default settings in PyTorch to minimize the cross-entropy loss function with λ = 1×10 -5. Model precision τ can be derived from λ value. We trained the MNIST dataset longer (400 epochs) than the CIFAR-10 dataset (100 epochs) due to the network sizes. We have applied RandomHorizontalFlip and RandomResizedCrop type data augmentation to CIFAR-10 datasets to improve accuracy. In total, 10000 input for each dataset were used and each of the data has the same shape (28 28) as it is needed for MNIST dataset. The random noise in each dataset changes for each Monte-Carlo sample for Bayesian NN. To evaluate robustness to thermal fluctuation (epistemic uncertainty) of the proposed SpinDrop BayBNN, we inject random Gaussian noise N (0, I) to the weighted sum of each layer.
For the RESET signal an amplitude of 300µA for a duration of 4ns has been used. This current has to be high enough to ensure that the MTJ is reliably switched. This evaluation on the SpinDrop module aims at appraising the target accuracy of the probability generated by the module. This is done due to the fact that our objective is to quantify the effects of the variability and the stochasticity on the target accuracy. For doing this, we performed several Monte Carlo analyses (i. e. 100 switchings of the MTJ within 20 Monte Carlo runs, which is equivalent to 2000 Monte Carlo simulations on a MTJ). Note that the MC simulations performed here are different from the MC sampling required for Bayesian inference. The probability of dropping the neuron from the crossbar output will be equal to : P Dropout = 1 -P Switching ( 11 ) As we can see in Figure  8 , the generated probability is not fixed but follows a Gaussian distribution.
C. Circuit-level simulation A 20 × 10 crossbar array is implemented according to the architecture presented in Figure  4. With this implementation, we intend to illustrate the Dropout concept impact on a larger crossbar with accurate Spice-level simulation. For this validation, the crossbar outputs are connected to a winnertakes-all (WTA) circuit  [36]. WTA circuits inhibit all its inputs except the one corresponding to the highest current at the output of the crossbar. Therefore, the prediction is directly implemented in the crossbar. To implement the proposed SpinDrop module, another peripheral circuit is added that selectively allows the activation of the previous layer to the stochastic wordline of the current layer. In Figure  8 , the impact of the Dropout rate on the overall accuracy is depicted.
D. Architecture Simulation Data from Table  I  were used to approximate the delay and the inference energy of a single image forward pass for MNIST dataset. Figure  9  shows the delay and energy consumption of both MLP and CNN topologies, implemented with Binary and Bayesian Binary styles. In each graph, we compare the BNN values with the resources needed for the proposed SpinDrop BayBNN with 10 MC-samples taken for the predictive performance and a CiM group size of Z CIM = 4. Taking more MC-samples does not improve predictive performance, as shown in the Figure  12. Note that BayNN implementation tends to be more computationally expensive in contrast with conventional BNN implementation since one must compute the expectation (average) of many inference results. To perform a single inference with the BayBNN, each layer is evaluated using multiple CiM operations to calculate the results of the matrix-vector multiplication between the input to the layer and its weights.
Each of these CiM operations gives a partial sum of the final neuron output, which is generated by adding up the partial sums in the accumulator-adder. After all partial sums are added up, the comparator is used to implement the threshold activation (sign (x)) function. This is done for all layers to get a prediction for a single When performing the BayNN inference, this procedure has to be done multiple times, depending on the number of inference samples, in which the SpinDrop module is used to randomly select the neurons during the CiM operation. To get the final result, the average circuit is computing the running average of the MC runs. However, the main contribution to energy and delay are the individual forward passes occurring during the predictive mean computation. Therefore, the total delay and energy consumption scale linearly with the number of inference samples for the Bayesian inference.
In Table  II , the proposed approach is compared with State-Of-The-Art (SOTA) implementation in terms of energy consumption, all based on MNIST dataset. When compared to an NVM implementation  [21] , the proposed approach with STT-MRAM technology is ×4. 65 less energy consuming. It should be noted that the RRAM implementation is done only with 2 hidden layers, whereas our implementation is realized on a Lenet-5 topology. When considering the FPGA implementation, the energy consumption is ×10 times lower when compared to an implementation with two hidden layers  [37] ,  [38]. With a Lenet-5 topology  [39] , the evaluated energy is ×20 better. One can conclude on the energy efficiency of the proposed approach with the SpinDrop module and the MTJ crossbar array.
E. Uncertainty Estimation Here, we have used Dropout with dropout probability of 20% on all the layers and crossbar array for uncertainty estimation. a) Epistemic Uncertainty: In critical applications, any NN results must be correct and trustworthy, otherwise an error flag should be raised. Such kind of NNs should only predict an answer when the input distribution matches the trained one D train. Point Estimate NNs can't infer "Undecided" if it receives out-of distribution data. We found that when a BNN gets out-of distribution data, it predicts random MNIST labels. For example, it mostly predicts MNIST labels 0 (with a frequency of 28. 28%) and 8 (with a frequency of 62. 17%) when dataset D 2 (random Gaussian noise) is applied as  an input. In fact, our evaluated point-estimate NN predicts 66.
e. , low variance. As a result, the proposed SpinDrop BayBNN can detect up 100% of out-of distribution data from the dataset D 1 • • • D 6 , as shown in Figure  10. To further emphasize the significance of uncertainty in detecting out-of distribution data, we have also assessed the situation in which the model predicts when uncertainty is moderate (or moderately certain) and high (that means not certain). Nonetheless, the prediction confidence is still considered high. A quantile score of 50% (median of the MC samples) and 90% is considered for moderate and high uncertainty analysis. If the model predicts despite moderate uncertainty, its ability to detect out-of distribution data could decrease by as much as 61. 68%. However, the model can not detect any out-of distribution data for most of the datasets when it predicts despite high uncertainty, as shown in Figure  10.
This demonstrates the significance of uncertainty estimate in spotting out-of distribution data. Additionally, our proposed method is also robust against random data poisoning. When 20% of the MNIST validation data is poisoned with random Gaussian noise (dataset D 2 ) with random labels, our proposed Dropout based Bayesian BNN can detect 84. 45% of poisoned data and achieves an accuracy of 96. 28% on predicted inputs. On the contrary, the accuracy of the point-estimate NN decreases from 98. 83% to 81. 19%. Consequently, our proposed SpinDrop BayBNN improves inference accuracy by 15%, in addition to obtaining fail-safe property. Data poisoning in our context is considered to occur when 1) an adversary is aware of inference data of the model, and has the power to alter a small fraction of the inference data in order to degrade the trained model's overall accuracy, or 2) the data generation process is noisy.
F. Analysis of the Predictive Performance 1) Comparison With State-of-the Art Algorithms: Predictive performance of our BayBNN, assuming there are no variations in the SpinDrop module, is comparable to the full precision BayNN  [6]  as shown in Table  V  for the MNIST dataset on LeNet-5 CNN topology. For a fair comparison, we have used the same network size used in  [6]. They use MaxPool as the activation for the convolution layers, however, our proposed SpinDrop BayBNN uses Sign(x) activation function for all the layers where activations are applied. When hyperbolic tangent (Htanh) is used as activation function for linear layers, our proposed SpinDrop BayBNN achieves slight improvement in accuracy for up to 1000 MC samples. However, the predictive performance of  [6]  with ReLU activation is slightly better than the others.
This analysis shows that binarizing BayNN can still achieve comparable predictive performance to full precision BayNNs. However, the hardware implementation for our solution may lead to a smaller area and better powerperformance product thanks to simpler activation functions and binary weights. Here, a smaller dropout probability of 20% is used in our analysis. Additionally, we have explored many different kinds of implementations with different location for Dropout layer. There are summarized in later in Section IV-F3. Furthermore, our proposed BayBNN achieves comparable inference accuracy to the state-of-the-art (SOTA) point estimate BNN algorithm for VGG, and ResNet-18 CNN topology with CIFAR-10. However, on CIFAR-100 datasets, our method outperforms SOTA point estimate BNN algorithm by 3.
6%, as summarized in Table  III  and IV. In general, we have found that taking T Monte-Carlo samples and averaging them increases inference accuracy of NNs model trained with and without Dropout. 2) Performance of BayBNN with SpinDrop: Although, the predictive performance of BayBNN (algorithmic, i. e. , no variations in the SpinDrop module) is comparable to the full precision implementation, it should also be robust to the variations of the Spintronic-based implementation, SpinDrop. The evaluations show that the predictive performance of the proposed BayNN with SpinDrop considering variations is also comparable to algorithmic Dropout, as shown in Table  VI  for MNIST and CIFAR-10 datasets on LeNet-5 and VGG topoligy, respectively. We have performed two experiments to evaluate the robustness of the proposed approach to variations in the SpinDrop module.
In one case, we have trained a NN with normal Dropout but during Bayesian inference, we have evaluated the model against our proposed SpinDrop Dropout with up to 3× the standard deviation σ of the manufacturing variations. In this case, the predictive performance of both MNIST and CIFAR-10 datasets improves slightly (+0. 66%) compared to original algorithmic Dropout BayBNN. Here, the σ of the SpinDrop module is as high as 3. 3% (3×). That means, the Dropout probability of each neuron can fluctuate by ±10% from the trained value, without any impact on the performance. In our experiments, SpinDrop with a dropout probability of 20% is used for all the layers and all crossbars. In the other case, when the NN is trained with the Spin-Drop techniques instead of the original Dropout, it would expect Dropout probability p = N (0, σ).
In this case, the predictive performance also improvement is slightly better in comparison, e. g. , the accuracy improvement of 1% achieved. Variation in the dropout probability leads to more sparsity during Bayesian inference, as a result, accuracy improves slightly. 3) Analysis of Dropout Rate and Location: We have used a smaller Dropout rate 20% in our analysis for both MNIST and CIFAR-10 datasets. The results for other smaller dropout rate, e. g. , from 10% to 20% are summarized in Table  VII. Here, for each Dropout probability p, we have taken up to 50 MC-samples for the Bayesian inference. Our results show that the predictive performance improved for all the Dropout rates by up to 1. 7% compared to BNNs. Usually, the default Dropout probability of 50% is used during training. Although, this improves the predictive performance of full precision NNs, it does not improve the  performance of the binary NNs, as shown in Figure  13.
Execution Task A task e executes instructions. It is represented as a tuple S e = (I e , θ, τ, m, st, r) where I e is the next expression being evaluated, θ maps free variables to their values (as bound by lets or pattern matching), while τ does the same for variables that are bound to recursive functions by letrec. In this model, modules are named sets of functions used during the initialization of τ. We will not focus on the rules around letrec, which can be found in appendix. We also store information dedicated to generate signals: m is the name of the current module, st stores the execution stack and r stores the reason of the end of the process if known. Notice that the only role of st is to store the information needed to build error messages.
Discussion The semantics we proposed here contrasts with previous small steps semantics in two ways. First, it describes in a more precise way the Core Erlang monitoring and signal system. Second, it is based on the official documentation, rather than the current implementation. We tried to develop a semantics that is close enough to previous formalizations so as to make it easier to establish links between both, which we hope will come handy for the next part of our works. We are currently working on an implementation of this semantics in Maude  [8] , which will be available, with an updated and further documented semantics, at  [7]. Our next step will be to leverage abstract interpretation in order to generate causal explanations of an observed faulty behavior. Although not based on exactly the same language as our work,  [9]  and  [13]  seem to be a good starting point.
2) Library-modulated Cumulated Responsibility Vector (Λ) -weighing the library responsibility vector, by giving more importance to nodes known to harbor an external reference library 3) Property-modulated Cumulated Responsibility Vector (Ω) -weighing the library responsibility vector by the (physicochemical, target binding, etc. ) property values mapped on every node CLS is a regular vectorial space that can be visualized by dimensionality reduction methods -such as GTM, as advocated here. Since the CLS descriptors are themselves based on the GTM projections of compound chemical space, the CLS maps will be further referred to as meta-GTMs (µGTMs)  9. In this work, we aimed to analyze the CLS populated by 2497 virtual DELs and ChEMBL using µGTM. ChEMBL was selected as a reference database due to its richness in molecules with high functional diversity (displaying a broad range of biological activities), thus being a good reference for the selection of diverse DEL that may be further used for primary screening.
DATA 1. ChEMBL ChEMBL28 database containing molecules biologically tested against more than 15 000 targets was selected as a reference library  10. Its standardization was performed in one of our previous works  11  in conformity with the protocol implemented on the Virtual Screening Web Server of the Laboratory of Chemoinformatics at the University of Strasbourg, using the ChemAxon Standardizer  12. It was then filtered according to the rules of DEL-likeness derived in our previous work  5. Hence, until otherwise is stated, in the text, we always refer to the filtered ChEMBL28 containing 1 605 370 molecules. After standardization and filtering, ISIDA fragment descriptors of type IA-FF-FC-AP-2-3 (ISIDA fragment sequences of 2 and 3 atoms labeled by their CVFF force field types and formal charge using all paths) were calculated  13.
Library Descriptor Vectors A single compound on the GTM is described by a responsibility vector, outlining the fuzzy levels of association of the compound to each map node. To encode a whole library by a single vector, the sum of responsibility values over all molecules of the library in each node of the GTM can be calculated to give a Cumulated Responsibility Vector CRV (see Figure  2 ). Since GTM nodes are predominantly associated with specific "chemotypes" (common structural traits -scaffold, pharmacophore, size, etc. ), the CRV allows to quantify the occurrences of such chemotypes in a library  5. Figure  2. Scheme describing how a responsibility vector is obtained for a single molecule using GTM (top) and how a CRV is obtained for the whole library of compounds (bottom). To obtain a library-size independent library descriptor vector Φ, CRV values can be normalized  5  by the library size N.
Instead, one map per vector type was created. Φ and Λ-based µ-maps were used to analyze the CLS from the library structural similarity point of view. On Ω-based µ-maps, the CLS was analyzed by considering both the property and structural similarity of libraries. Five properties were selected for analysis: MW, logP, number of H-bond donors, number of H-bond acceptors, and Quantitative Estimate of Drug-likeness (QED). Like with any GTM, a µGTM landscape can be colored based on projected properties, starting with basic cumulated responsibility (some CLS zones being more "crowded" in terms of resident libraries than others), library "status" (being a member of top100 similar libraries relative to some reference database, for example) or other characteristics (underlying reaction type, number of building blocks engaged in the library compounds, etc.
CONCLUSIONS In this study we propose Chemical Library Space (CLS) visualization, using µGTM. In the context of a large pool of 2497 virtual DELs and ChEMBL database (used as reference), CLS The proposed method of CLS analysis and visualization using µGTM represents an efficient and useful tool for: 1) providing a senseful bird's eye view of the whole CLS and simplifying the analysis of inter-library relationships; 2) analysis of the CLS from different perspectives, positioning libraries by either chemical space similarity or/and property distribution similarity; 3) the selection of a compound library covering the desired chemical and property space out of thousands of possible ones, given the appropriate reference database. In this regard, it can be beneficial to add this method to the toolkit of medicinal chemists that deal with the selection of a screening collection to verify beforehand if the designed library covers the desired chemical and property space or identify closely lying analogous libraries with similar properties on µGTM.
Data and Software Availability The data used in this work are available in the public domain resources: biologically relevant compounds from ChEMBL 10 (version 28) -https://www. ebi. ac. uk/chembl/, eMolecules 19    building blocks that were used for DEL generation using eDesigner  15  are partially available on the website https://www. emolecules. com/products/building-blocks and Enamine 20 building blocks are available on the website https://enamine. net/building-blocks. GTM software used for CS and CLS visualization herein is available under paid license.