dataesr/scientific-paragraphs-categorization huggingface dataset:
The solid lines represent the best fitting functions, i. e. , a decreasing exponential function y = 1. 64 -0. 75 exp(-3x/10 4 ) for rats and an increasing exponential y = 0. 82 -0. 10 exp(-4x/10 4 ) for humans. These functions can be used to choose the best fuzzy exponent of these datasets according to the series length. Figure  8  shows the average time (over three human HRV series) spent calculating the different multiscale fuzzy entropy-based variants at increasing segment sizes (up to 12,000 samples). As expected, the computational time required to run any method increases with the segment size. However, MMFE has the highest computational cost, while MFE showed the lowest. CMFE and RCMFE show virtually the same computational time.
The replacement of the rigid similarity of SampEn (Heaviside function) by the smooth fuzzy function in FuzzyEn seems to be the most relevant improvement for a good estimation of entropy in short time series. However, one must be aware that optimal fuzzy exponents' choice is crucial to obtain high accuracy. In a previous study with the same dataset, we showed that MMFE provides better estimates of the original MSE than MMSE when proper choices of the fuzzy exponent n are made  [19]. Here, we showed that CMFE, RCMFE, and MFE also have a dependence on n and that the optimal exponents found for MMFE are virtually the same for CMFE, RCMFE, and MFE, as can be seen in Figure  6. Although both the rats and humans datasets represent health conditions, the best exponents for rats increase with series length, while it decreases for humans.
This is likely to be a consequence of the different species, but it still has to be investigated together with datasets with pathological HRV series. Nevertheless, the fitting equations provided in Figure  6  can be used for the choice of the optimal fuzzy exponents in the dataset evaluated here. The measure stationarity is a possible issue concerning long-term time-series as previously pointed out  [27]. However, all time-series and analyzed segments are supposed to be taken at baseline physiological state. We introduce an illustrative analysis of two time-series data, i. e. , one human and one rat, in the appendix to this paper presented in Figure  A1. To check the influence of the fuzzy function on the accuracy of fuzzy-based MSE variants, we calculated the mean squared error of MFE using an alternative fuzzy function, i.
e. , exp(-0. 6931 × (d/r) n )  [22]. Interestingly, this alternative function also presented a dependence of the minimum error on the choice of the exponent n. However, the range of optimal n values is markedly different from the ones found for the original fuzzy function, and curiously, the exponent always increases with the series length for both datasets (rats and humans). The extensive evaluation of different fuzzy functions is out of the present study's scope, and one must be aware that changing the fuzzy function requires the search for the optimal fuzzy exponents. For an extensive review on the possible fuzzy functions and their differences, please refer to  [28]. For entropy estimators based on a similarity function between patterns (such as SampEn and FuzzyEn), the tolerance factor, r, is commonly defined as a percentage of the signal's SD, making the results comparable within signals with different magnitude.
Conclusions In this study, several SampEn-and FuzzyEn-based MSE variants were calculated in short HRV time series, and their accuracy to estimate the actual MSE was investigated. All fuzzy entropy-based algorithms provided better accuracy (given the fuzzy exponent's proper choice) compared to the variants based on SampEn. Moreover, all FuzzyEn-based algorithms evaluated showed similar accuracy. Therefore, since MFE is the most simple and cost-effective algorithm among them, we recommend the use of MFE for the analysis of short-term time series. The results also indicate that different fuzzy functions may provide good accuracies. However, the dependence of the fuzzy exponent (n) to the series length may vary from one function to another and on different datasets. Further studies are necessary to determine the optimal fuzzy exponents in datasets with pathological signals.
e. , 15,892 beats. Figure6shows the optimal fuzzy exponents of fuzzy-based MSE variants for each segment size. These exponents were utilized to calculate the minimum error curves for multiscale fuzzy entropy-based variants (see Figure5). The curves were fitted to exponential functions, which can be employed to find the best fuzzy exponent of those datasets according to the time series length. Note that for the HRV series from rats, the optimal fuzzy exponent increases with the series length, decreasing the HRV series of humans. For the sake of comparison, Figure7shows the mean square error of MFE using the alternative fuzzy function exp(-0. 6931 × (d/r) n ). The errors are illustrated for the fuzzy exponent ranging from n = 1. 3 to n = 3. 0 (rats) and from n = 2. 0 to n = 5. 5 (humans). Results show that MFE with this alternative fuzzy function also presents dependence on the segment size, similar to the original fuzzy function (Figure1).
Note that the errors are calculated regarding the MSE of full-length time-series, i. e. , 15,892 beats. Figure6shows the optimal fuzzy exponents of fuzzy-based MSE variants for each segment size. These exponents were utilized to calculate the minimum error curves for multiscale fuzzy entropy-based variants (see Figure5). The curves were fitted to exponential functions, which can be employed to find the best fuzzy exponent of those datasets according to the time series length. Note that for the HRV series from rats, the optimal fuzzy exponent increases with the series length, decreasing the HRV series of humans. For the sake of comparison, Figure7shows the mean square error of MFE using the alternative fuzzy function exp(-0. 6931 × (d/r) n ). The errors are illustrated for the fuzzy exponent ranging from n = 1.
Related work. Related work in privacy-preserving machine learning encompasses SMPC and fully homomorphic encryption (FHE) techniques. FHE only needs a single round of interaction but does not support efficient non-linearities. For example, nGraph-HE  [9]  and its extensions  [8]  build on the SEAL library  [55]  and provide a framework for secure evaluation that greatly improves on the CryptoNet seminal work  [27] , but it resorts to polynomials (like the square) for activation functions. 1 The code is available at github. com/LaRiffle/AriaNN. SMPC frameworks usually provide faster implementations using lightweight cryptography. MiniONN  [44] , DeepSecure  [52]  and XONN  [50]  use optimized garbled circuits  [63]  that allow very few communication rounds, but they do not support training and alter the neural network structure to speed up execution.
Last, works like Gazelle  [33]  combine FHE and SMPC to make the most of both, but conversion can also be costly. Works on trusted execution environments are left out of the scope of this article as they require access to dedicated and expensive hardware  [32]. A concurrent work from Boyle et al. [13]  was made public shortly after ours. Their approach also provides improvement over previous algorithms for private comparison using function secret sharing, and their implementation results in the same number of rounds than ours and similar key size (approximately n(λ+n), where n is the number of bits to encode the value, it accounts for correctness and is typically set to 32, and λ is the security parameter and usually equals 128). However,  [13]  is not intended for machine learning: they only provide an implementation of ReLU, but not of MaxPool, BatchNorm, Argmax or other classic machine learning components.
On the other side, the information obtained by the data owner about the model (i. e. the prediction on a given input) is the same as in MLaaS. Model inversion techniques  [64]  can leverage multiple calls to the model to try to build a new model with similar performance. There are not many defenses against this, except limiting access to the model, which is usually the case in MLaaS where data owners are given a quota of requests. Also, attacks like membership inference  [57]  or reverseengineering  [26, 31]  methods could be used to unveil information about the dataset on which the model was originally trained. Using differential privacy  [1, 25]  during the initial training of the model can provide some guarantees  [48]  against these threats, but it has a tradeoff between privacy and utility as differentially private models usually have poorer performance.
Training Phase Using Autograd These operations are sufficient to evaluate real world models in a fully private way. To also support private training of these models, we need to perform a private backward pass. As we overload operations such as convolutions or activation functions, we cannot use the built-in autograd functionality of PyTorch. Therefore, we have used the custom autograd functionality of the PySyft library  [54] , where it should be specified how to compute the derivatives of the operations that we have overloaded. Backpropagation also uses the same basic blocks than those used in the forward pass, including our private comparison protocol. Therefore, the training procedure Train described in Algorithm 8 closely follows the steps of plaintext training, except that the interactions between the secret shared data and model parameters use the protocols we have described in Section 4.
Experiments In order to simplify comparison with existing work, we follow a setup very close to the work of  [60]. The reason why we compare our work to  [60]  is that it provides the most extensive experiments of private training and evaluation we are aware of. We are aware that  [60]  also provides honest-majority malicious security, but we only report their results in the honest but curious setting (where they obtain the best runtimes). We assess private inference of several networks on the datasets MNIST  [43] , CIFAR-10 [39], 64×64 Tiny Imagenet  [53, 62]  and 224×224 Hymenoptera which is a subset of Imagenet, and we also benchmark private training on MNIST. More details about the datasets used can be found in Appendix E. 1. More precisely, we assess 6 networks: a 3 layers fully-connected network (Network-1), a small convolutional network with maxpool (Network-2), LeNet  [42] , AlexNet  [40] , VGG16  [58]  and ResNet18  [30]  which to the best of our knowledge has never been studied before in private deep learning.
The description of these networks is available in Appendix E. 2. Our implementation provides a Python interface and is tightly coupled with PyTorch to provide both the ease of use and the expressiveness of this library. To use our protocols that only work in finite groups like Z 2 32 , we convert our input values and model parameters to fixed precision. To do so, we rely on the PySyft library  [54]  which extends common deep learning frameworks including PyTorch with a communication layer for federated learning and supports fixed precision. The experiments are run on Amazon EC2 using m5d. 4xlarge machines for CPU benchmarks and g4dn. 4xlarge for GPU, both with 16 cores and 64GB of CPU RAM, and we report our results both in the LAN and in the WAN Input: Model on the server S, initialized with parameters θ Output: Model trained using the data from the clients (C i ) i=1.
Inference Time and Communication Comparison of experimental runtimes should be taken with caution, as different implementations and hardware may result in significant differences even for the same protocol. We report our online inference runtimes in Table  3  and show that they compare favourably with existing work including  [44-46, 59, 60]. For example, our CPU implementation of Network-1 outperforms all other studied frameworks by at least a factor 2× in the LAN setting and even more in the WAN setting. For larger networks such as AlexNet and VGG16, we have an execution time which is slightly higher than  [60]. One reason for this can be that we use a Python interface to serialize messages and communicate between parties, while  [60]  uses exclusively C code. However, we are more communication-efficient than  [60]  for models starting from LeNet, with a typical gain of 7% to 30% on CIFAR-10.
Test Accuracy Thanks to the flexibility of our framework, we can train each of these networks in plaintext and need only one line of code to turn them into private networks where all parameters are secret shared, or to fixed precision networks where all parameters are converted to fixed precision but computation is still in plaintext. Comparing the performance of private models with their fixed precision version helps us to understand if fixed precision by itself reduces the accuracy of the model, and gives an estimate of the loss that is related to using secret shared computation. We compare the accuracy of several pre-trained networks in these 3 modes in Table  4  by running a private evaluation with FSS, a fixed precision using only PySyft and a public evaluation where the model is not modified.
Computation and Communication Analysis We have provided in Table  6  a small analysis of how the compute time can be decomposed. We use AlexNet on the Tiny Imagenet dataset as it is the biggest network on which we could use a batch size higher than 64 both on CPU and GPU and hence amortize the serialization and communication cost. Thanks to the efficiency of our Rust implementation, function secret sharing only accounts for 16% for the online runtime when we use CPUs, and most of the time is spent doing matrix multiplications and convolutions. This last part uses the underlying PyTorch functions on integers which are significantly slower than when they run on floats. This motivates us to use GPUs for which such operations are far more efficient. In the GPU setting, the distribution of time is indeed much more balanced, and having function secret sharing directly running on GPUs avoids going back and forth between the CPU and the GPU.
C. 5 BatchNorm Approximation The BatchNorm layer is the only one in our implementation which requires a polynomial approximation during training. We have therefore experimented how this approximation can alter the behaviour of a deep network such as ResNet18 on a simple dataset like the hymenoptera dataset 3. We follow the PyTorch transfer learning tutorial 4 and use a pretrained version of ResNet18. We retrain all the layers for 25 epochs, but we replace the BatchNorm layers with our approximated version which behaves as such: as a general rule the inverse for the variance is computed using 3 iterations of the Newton methods and we use the result of the previous batch as an initial approximation. For the first batch of the epoch where no approximation is available, we use instead 50 iterations.
E. 3 Models Architecture Unless otherwise specified, our models follow their standard architecture as provided by the torchvision library (version 0. 5), except for smaller models such as Network-1, Network-2 and LeNet which are respectively detailed in  [46] ,  [44]  and  [42]. For the CIFAR-10 version of AlexNet, we follow the architecture of  [60]  which includes BatchNorm layers and is available on the FALCON GitHub. For the 64×64 Tiny Imagenet version of AlexNet however, we used the standard architecture from PyTorch since it allows us to have a pretrained network and the version of FALCON seemed non-standard. We have adapted the classifier parts of AlexNet, VGG16 and ResNet18 to the different datasets we use. -For AlexNet on Tiny Imagenet, we use 3 fully connected layers with respectively 1024, 1024 and 200 neurons, and ReLU activations between them.
-For VGG16 we use 3 fully connected layers with respectively 4096, 4096 and 10 or 200 neurons for the last layer depending on the dataset used. We also use ReLU activations between them. -For ResNet18, we use a single fully connected layer to map the 512 output logits to the appropriate number of classes. Note also that we permute ReLU and Maxpool where applicable like in  [60] , as this is strictly equivalent in terms of output for the network and reduces the number of comparisons. More generally, we don't proceed to any alteration of the network behaviour except with the approximation on BatchNorm. This improves the usability of our framework as it allows us to use pre-trained neural networks from standard deep learning libraries like PyTorch and to encrypt them with a single line of code.
Acknowledgments We would like to thank  Geoffroy Couteau ,  Chloé Hébant  and  Loïc Estève  for helpful discussions throughout this project. We are also grateful for the long-standing support of the OpenMined community and in particular its dedicated cryptography team, including  George Muraru ,  Rasswanth S ,  Hrishikesh Kamath ,  Arturo Marquez ,  Yugandhar Tripathi ,  S P Sharan ,  Muhammed Abogazia ,  Alan Aboudib ,  Ayoub Benaissa ,  Sukhad Joshi  and many others. This work was supported in part by the  French project FUI ANBLIC. The computing power was graciously provided by the  French company ARKHN.
5 - - - - - - - - - - CrypTFlow MNIST - 0. 008 - - - - 0. 034 - - - - 0. 058 - - - AlexNet VGG16 ResNet18 LAN LAN LAN WAN LAN LAN LAN WAN LAN LAN LAN WAN Framework Dataset Prep. CPU GPU CPU Comm. Prep. CPU GPU CPU Comm. Prep. CPU GPU CPU Comm. AriaNN CIFAR-10 0. 091 0. 15 0. 078 0. 34 0. 95 0. 94 1. 75 1. 55 1. 99 12. 59 - - - - - FALCON CIFAR-10 -0. 043 - 0. 13 1. 35 -0. 79 -1. 27 13. 51 - - - - - AriaNN 64×64 ImageNet 0. 27 0. 33 0. 20 0. 48 1. 75 3. 42 7. 51 6. 83 8. 00 53. 11 - - - - - FALCON 64×64 ImageNet - 1. 81 - 2. 43 19. 21 -3. 15 -4. 67 52. 56 - - - - - AriaNN 224×224 Hymenoptera - - - - - - - - - - 10. 02 19. 88 13. 90 24. 07 148 LAN Accuracy Model Dataset time (h) Private Fix prec. Public Network-1 MNIST 0. 01 98. 2 98. 2 98. 2 Network-2 MNIST 0. 18 99. 0 99. 0 99. 0 LeNet MNIST 0.
36 128 128 9. 98 128 128 14. 7 AlexNet VGG16 ResNet18 Batch Size Preprocessing Batch Size Preprocessing Batch Size Preprocessing Framework Dataset CPU GPU Comm. (MB) CPU GPU Comm. (MB) CPU GPU Comm. (MB) AriaNN CIFAR-10 128 128 24. 6 64 14 277 - - - AriaNN 64×64 ImageNet 128 64 88. 4 16 3 1124 - - - AriaNN 224×224 Hymenoptera - - - - - - 8 1 3254.
We consider a human protein as a "shared target" if it was reported to bind both to a SARS-CoV-2 protein and another virus' protein, according to the PPI data retrieved from http://viruses. string-db. org (Supplementary Table  1 ). Out of the 332 human proteins directly targeted by SARS-CoV-2, only 18 of them were found to be also targeted by other viruses, among which Herpes viruses, HPV type 16, Reovirus or Encephalomyocarditis virus (Supplementary Table  1 ). Figure  5a  shows that SARS-CoV-2 does not indeed cluster with any other virus on the basis of shared protein interactors (Supplementary Fig. 2 ). We then extended our clustering analysis to biological pathways and processes in which these targeted proteins are involved. The R package clusterProfiler  50  allows to perform enrichment analysis of gene clusters and was used to identify statistically enriched Reactome pathways  51  and Gene Ontology terms  52  potentially targeted by the viruses although through multiple different proteins.
Considering enriched Reactome pathways, SARS-CoV-2 was shown to have the highest similarity with Bunyavirus and Reovirus (Fig. 5b ). The same clustering analysis on Biological Processes as defined by the Gene Ontology database showed that SARS-CoV-2 clusters with Rotavirus C, another virus of the Reoviridae family (Supplementary Fig. 5 ). These two methods to assess virus similarities (based either on their targeted proteins, or on their relative enriched pathways among these proteins) are complementary. with a cutoff of p value <0. 005. Viruses for which we could not find any enriched Reactome pathway do not appear on the enrichment heatmaps. Complete results of these enrichment analyses are available as Supplementary material (Supplementary Data 1 and 2). To investigate whether SARS-CoV-2 would cluster with other viruses at a higher distance, we extended the clustering analysis to the human proteins located one node further of the proteins directly targeted by viruses (referring to them as second-order interactors, Supplementary Data 6).
Noticeably, even when all the considered approaches are combined to reach an integrated view of the virus clusters, we observe the same similarity between SARS-CoV-2 and viruses such as Herpes. It is worth mentioning that the SARS-CoV-2 outbreak is very recent and its PPI is not yet available on the STRING repository. Therefore, for this particular virus, we relied on a study published in Nature 17 , in April. We acknowledge the possibility that our results might be affected by the limitations of the currently available data sets. Overall, our framework opens the doors for further analyses of viral agents from the perspective of combining statistical physics and computational biology, highlighting the sensitivity of macroscopic functions, such as spectral entropy, to small variations across interaction networks and, more specifically, virus-host interactomes.
Even though other analyses, such as the perturbation propagation patterns, lack the same sensitivity, according to our results it provides microscopic details about the interactions between viral and human proteins that complement the macroscopic view, together enhancing our understanding of the novel SARS-CoV-2 from a new perspective, which can provide a mathematical ground for the exploration of further clinical treatments and biological understanding. The most likely application in this direction is drug repurposing, i. e. , the identification of new roles of an existing drug to discover previously unknown therapies for untreated diseases. Usually, drugs are combined together to trigger their most direct effects, i. e. , at the first-order neighborhood of their targets: however, this approach does not account for potential interference at a systemic level, and databases of empirically discovered side effects have to be taken maintained to be interrogated  61, 62.
Methods Overview of the dataset. The human interactome used in this study combines PPI from two of the largest repository publicly available to date, namely STRING v10. 5  12  -publicly available at https://string-db. org/cgi/download. pl-and BIO-GRID v3. 5. 182  75, 76  -publicly available at https://downloads. thebiogrid. org/ BioGRID/Release-Archive/BIOGRID-3. 5. 182/). For a consistent analysis, all protein names and aliases have been standardized to follow the common nomenclature of official symbols of NCBI gene database (ftp://ftp. ncbi. nlm. nih. gov/gene/DATA/ GENE_INFO/Mammalia/ (accessed: 28/03/2020)  77  ). In the following we will refer to this comprehensive network, in standardized format, as BIOSTR. The virus-host interactions for 93 viruses are collected from the STRING database-publicly available at http://viruses.