dataesr/scientific-paragraphs-categorization huggingface dataset:
SV-based nuclear tree The input SV vcf file (homo_and_hetero_noDoublonsInCoordinates. vcf. gz) was generated using the non-redundant SV dataset. Based on the presence/absence information of these identified SVs in each assembly entry, a phylip-formatted 0/1 matrix was generated accordingly and used for tree building. IQtree (version 1. 6. 12; options: -s $prefix. phylip -st MORPH -m MK+ASC -bb 1000 -alrt 1000 -nt $threadspre $prefix. iqtree -safe) was used to generate the phylogenetic tree. A thousand rounds of ultrafast bootstrap (UB) and approximate likelihood-ratio test (aLRT) were used to assess the branch supports.
When using SQK-LSK108 kit for FLO-MIN106 MinION flow cells, twice 140 ul ABB washes were performed. When using SQK-LSK109 kit for FLO-PRO002 flow cells, twice 250 ul L Fragment Buffer (LFB) washes were performed. The final library was eluted in 15 ul Elution Buffer and loaded into the MinION or PromethION flow cells according to the ONT manuals. Raw fast5 files were basecalled using Guppy (v3. 4. 5) followed by Porechop (github. com/rrwick/Porechop) removal of adapters and barcodes. The entire project generated close to 204 Gbp of Nanopore sequencing data. Sequencing statistics are detailed in Supp. Table16. For fast5 file storage/sharing, singlecom/scottgigante/picopore), ensuring all files contain only data necessary for re-basecalling. Next single-fast5 files were converted into multi-fast5 files using the ont-fast5-api (github.
Acknowledgments We thank  B. Dujon  and  B. Llorente  for their valuable feedback on the manuscript. This work was supported by the  Agence Nationale de la Recherche   ANR-16-CE 12-0019  and  ANR-18-CE12-0004. This work was also partially supported by the  ANR-15-IDEX-01  (to G. L. ),  Fondation pour la Recherche Médicale  ( EQU202003010413  to G. L. ), the  European Research Council  ( ERC Consolidator Grant   772505  to J. S. ), the  Guangdong Basic and Applied Basic Research Foundation  ( 2019A1515110762  to J. -X. Y. ),  Guangdong Pearl River Talents Program  ( 2019QN01Y183  to J. -X. Y. ), and  National Natural Science Foundation of China  ( 32070592  to J. -X. Y. ). J. S. is a Fellow of the  University of Strasbourg Institute for Advanced Study (USIAS)  and a member of the  Institut Universitaire de France.
Introduction Disentanglement, a process aimed at obtaining neural representations with identified meaning, is a crucial component of research on interpretability  (Rudin et al. , 2022). A form of disentanglement that received a lot of interest from the NLP community is the separation between syntax and semantics in neural representations  (Chen et al. , 2019; Bao et al. , 2019; Zhang et al. , 2019; Chen et al. , 2020; Huang and Chang, 2021; Huang et al. , 2021). Previous works perform disentanglement using paraphrase pairs as information for semantics, and/or constituency parses as information for syntax. The 1 github. com/ghazi-f/QKVAE dependence of models on labeled data is known to entail high cost (see  Seddah et al. , 2020 on syntactic annotation) , and to often require new labels to handle problems such as concept drift  (Lu et al.
, 2021). Using Cross-Attention, our model encodes sentences into two latent variables: z sem to infer values for V , and z syn to assign keys in K for values in V. These keys and values are then used in the Attention mechanism of a Transformer Decoder to generate sentences. We show that z syn tends to contain syntactic information, while z sem tends to represent semantic information. Additionally, comparisons with a supervised model show that it needs a considerable amount of data to outperform our model on syntactic and semantic transfer metrics. Our contributions can be summarized as follows: • We describe QKVAE, a model designed to disentangle syntactic information from semantic information by using separate latent variables for keys and values in Transformers Attention. • We run experiments on a dataset for English which empirically show that the two types of latent variables have strong preferences respectively for syntax and semantic.
Similar to Dec, we define QKVDec to be the auto-regressive version of QKVDec. The QKVAE decoder yields probabilities for the generated tokens by using this operator on values given by z sem concatenated with embeddings d, and keys given by a linear transformation on z syn : v = Cat(d;z sem ), k = M s (z syn ) ∀ i s. t. 1 ≤ i ≤ |w| : wi = QKVDec(w 0 ,. , w i-1 ; k; v) w i ∼ Categorical(Softmax(M w ( wi ))) where M s is a linear layer. 5 While ADVAE already uses Cross-Attention to encode and decode latent variables, our model uses separate variables to obtain keys and values for Multi-Head Attention in its decoder.
Setup Data To compare our model to its supervised counterparts, we train it with data from the English machine-generated paraphrase pairs dataset ParaNMT  (Wieting and Gimpel, 2018). More specifically, we use the 493K samples used by Chen et al. 2020 ) also provide a curated set of triplets formed by a target sentence (target), a semantic source (sem_src),and a syntactic source (syn_src). The semantic source is a paraphrase of the target sentence, while the syntactic source is selected by finding a sentence that is syntactically close to the target (i. e. edit distance between the sequence of PoS Tags of both sentences is low 9  ) and semantically different from the paraphrase (has low BLEU score with it). Contrary to paraphrases in the training set of ParaNMT, paraphrases from this set were manually curated.
Therefore, we use Meteor (M) in conjunction with ParaBART  (Huang et al. , 2021)  a model where BART  (Lewis et al. , 2020)  is finetuned using syntactic information to produce neural representations that represent maximally semantics and minimally syntax. We measure the cosine similarity between sentences according to ParaBART embeddings (PB). ii) Syntax: We use the script of  (Chen et al. , 2020)  to produce a syntactic tree edit distance (STED) between the constituency trees of sentences, as was done to assess VGVAE. Additionally, following the evaluation procedure designed by  Huang and Chang (2021)  for SynPG, we measure the Template Matching Accuracy between sentences, where the template is the constituency tree truncated at the second level (TMA2). TMA2 is the percentage of sentence pairs where such templates match exactly.
Finally, we show that VGVAE, a supervised model, needs more than 50K samples to outperform QKVAE on both syntactic and semantic transfer. We plan to extend this work in three directions: i) Finding ways to bias representations of each z sem l towards understandable concepts; ii) Applying QK-VAE to non-textual data since it is data agnostic (e. g. to rearrange elements of a visual landscape. ); iii) Investigating the behavior of QKVAE on other languages.
Acknowledgments This work is supported by the  PARSITI  project  grant  ( ANR-16-CE33-0021 ) given by the  French National Research Agency (ANR) ,  the Laboratoire d'excellence "Empirical Foundations of Linguistics " ( ANR-10-LABX-0083 ), as well as the  ON-TORULE  project. It was also granted access to the  HPC resources of IDRIS  under the allocation 20XX-AD011012112 made by  GENCI.
To date, most solutions for anomaly detection have not fulfilled all of these requirements. In addition, the interpretability of why an instance is predicted to be abnormal is hardly concerned. In this paper, we propose the so-called FedeX architecture to address those challenges. The experiments show that FedeX outperforms 14 other existing anomaly detection solutions on all detection metrics with the liquid storage data set. And with Recall of 1 and F1-score of 0. 9857, it also outperforms those solutions on the SWAT data set. FedeX is also proven to be fast in terms of training time of about 7. 5 minutes and lightweight in terms of hardware requirement with memory consumption of 14%, allowing us to deploy anomaly detection tasks on top of edge computing infrastructure and in real-time. Besides, FedeX is considered as one of the frameworks at the forefront of interpreting the predicted anomalies by using XAI, which enables experts to make quick decisions and trust the model more.
I. INTRODUCTION An Industrial Control System (ICS) is an automation system that controls and monitors functionality in industrial processes. Wireless and control devices of ICSs are widely deployed in industrial sectors and critical infrastructures such as power grids, water treatment facilities. As illustrated in Fig. 1 , a typical ICS comprises multiple control loops connected with human-machine interfaces (HMIs), and remote diagnostics and maintenance functions based on network protocols. Nowadays, AI and Bigdata present excellent potential in migrating the manufacturing paradigm to smart The associate editor coordinating the review of this manuscript and approving it for publication was Wentao Fan. manufacturing (SM), as it enables Industrial Internet of Things (IIoT) based systems to operate in real-time and be more precise and efficient  [1].
From another perspective, recently, IIoT has been designed to make use of edge computing technology to perform computational tasks right at the edge of a network. This avoids offloading intensively computational tasks to a cloud centre as traditional IIoT would deploy. Edge computing can solve a serious drawback in a smart factory such as latency required for transmitting, receiving and processing big data collected from IoT devices. Within that context, Federated Learning (FL) -a distributed machine learning mechanism  [4]  was found to be a promising scheme for edge computing in a distributed environment. In addition, as IIoT evolves, the amount of data collected from ICS systems will increase exponentially. As a result, network latency and bandwidth become a barrier when these data are sent to the cloud from distributed edge nodes  [5].
Thanks to the XAI function, experts could define which real features contribute mainly to the anomaly. Besides, FedeX enables faster system response capability upon attacks since detection is deployed near anomaly sources. With FL aggregating distributed models into a united global model, FedeX can yield even higher detection performance than the centralized learning manner with Accuracy, Precision of up to 0. 99; Recall, F1-score, and AUC of up to 1 at maximum on the SCADA liquid storage infrastructure dataset, as illustrated in Table  2. Additionally, as evaluated in Table  3 , FedeX is proven to outperform 14 other anomaly detection reference methods on all detection metrics with the main case study of SCADA liquid storage. For the cross validation case on the SWaT dataset, with Recall of 1 and F1-score of 0.
II. RELATED WORK A short summary of related work is briefly described in  Centralized or Federated learning manners, XAI inclusion. Some important performance metrics such as running time and hardware consumption are also considered. In more detail, for the non purely time-series data type, we can find a range of anomaly detection methods using different algorithms. Recently, in  [14] , the author proposed a Logical Analysis of Data (LAD-ADS) solution using a rule-based method to detect anomalous behaviours in ICS systems over the Secure Water Treatment (SWaT) dataset  [22] , which is, in our opinion, not a purely time-series data scenario. LAD-ADS performs detection by extracting rules from a huge of data in the past. However, rule-based systems are often complex and challenging to manage and determine the cause of detected anomalies.
In the same scenario of ICSs,  [15]  proposed a state-aware anomaly detection method that uses the CUSUM (Cumulative Sum) control chart to the state-dependent detection threshold. The training process is done centrally on a large amount of data. In fact, the data in ICSs are often distributed; so using a centralized solution can cause disadvantages such as latency for sending all raw data to the central cloud and huge computer resource consumption for training. Moreover, in CUSUM  [15] , no detection performance metric such as Accuracy, Precision, Recall, or F1-score was revealed except the false alarm rate. In  [12] , the authors presented a methodology called MADICS for Anomaly Detection in ICSs using a semi-supervised anomaly detection paradigm with five main steps. The performance of MADICS in terms of Recall is slightly low over its testing dataset.
In addition, this mechanism requires a large amount of data for semi-supervised learning, which faces data privacy issues when transmitting a large amount of raw data for training, resource capacity, and computational resources of the system. In terms of detection performance, FedeX is also proved to outperform the previously proposed solutions MADICS  [12] , LAD-ADS  [14]  in the same factory contexts. In  [13] , a statistical window-based anomaly detection method was adopted by using various deep-neural network architectures, showing effectiveness in detecting the attacks in a SWaT infrastructure. However, the authors also indicated that their work needs to be improved with the interpretability of the outcomes and the behaviour detection of fault ICS components. For the time-series data type, we have observed various proposed AD solutions.
With a similar lack of performance testing on the edge hardware, work  [18]  proposed an FL-based anomaly detection approach for IoT networks based on the combination between Gated Recurrent Units (GRUs) and Long short term memory toward detecting anomalies with decentralized on-device data. However, the performance of the proposed method is not good enough in the distributed scenario; accuracy in each FL client is just around 90% on average. In several other studies  [7] ,  [19] , the authors developed and investigated attack detection solutions in ICS cyberspaces. In  [19] , the authors proposed an attack detection model that uses a Deep Neural Network and a Decision Tree classifier to identify cyber-attacks in the ICS context with an F1-Score of 93. 83% with the ICS gas pipeline dataset, which is higher than other algorithms such as SVM, LSTM, Naïve Bayes, Decision Tree (DT), DNN, Random Forest (RF).
Study  [7]  used the semi-supervised techniques by leveraging K-means and Convolutional Autoencoder to protect the ICS system from cyberattack. Like in  [19] , the experiments of the proposed methods were performed with the gas pipeline dataset and the water storage tank dataset. However, the anomaly detection performance of the proposed method still needs to be improved. In contrast to our study, these studies only focused on evaluating the performance of detection algorithms and did not consider other important metrics when being implemented in the edge environments of an ICS such as detection time and power/memory/bandwidth consumption. In addition, we have found a variety of classic machine learning algorithms for anomaly detection purposes based on the SWaT data. Accordingly, Logistic Regression (LR), Linear Discriminant Analysis (LDA), k-nearest neighbours (KNN), Naïve Bayes (NB), Support Vector Machine (SVM), and Classificationand Regression Tree (CART) are reported by  [8] ; one-class Support Vector Machines (SVM) and Deep Neural Networks (DNN) are reported by  [9].
The evaluation is conducted with a subset of NSL-KDD Dataset -an old network intrusion detection dataset released in 1999. Even though the combination of solutions to solve the black-box problem of DNN helps domain experts intuitively access the insight of the DNN algorithms, classification accuracy improvement is required when producing predictions in the test set. Very recently, the authors in  [21]  have proposed to use XAI to interpret anomaly detection outcomes of the multiple Bi-LSTM learning models in an ICS ecosystem. The scope of the ICS is the smart factory of steam-turbine power generation and pumped-storage hydropower generation. This paper can be considered as the forefront of interpreting anomaly detection in the ICS ecosystem.
Furthermore, this solution reduces workloads offloading up on the central cloud server as traditional centralized computing architectures do. As illustrated in Fig. 2 , the FedeX operation consists of 6 main steps, as follows: Step 1 : The edge device uses the sensing data collected from nodes within a zone    More specifically, in the detection process, there are two modules that work together at each zone: the local anomaly detection model -FedVAE-SVDD and the explanation module, as shown in Fig. 3. Any instance x will be fed to the FedVAE module and reconstructed as output x. Then the loss value vectors u = (xx) will be passed to a trained SVDD model to calculate the distance between it with the centre I of the hypersphere defined in SVDDd(u, I ). Accordingly, based on the comparison between d(u, I ) and radius R, the instance x is predicted as an anomaly or normal.
The VAE structure is illustrated in Fig. 5. Given an input dataset X = {x 1 , x 2 ,. , x n } characterized by an unknown probability distribution q(X ). The aim is to approximate this true distribution q(X ) by using a parametrized distribution Q θ (X ) with the parameter θ. Let Z be a random vector jointly distributed with X , which represents a latent encoding of X. Since the computation cost of Q θ (X ) is high, it is needed to introduce function P φ (Z |X ), with φ defined as the set of real values that parametrize P, to speed up the calculus. This function is to approximate the posterior distribution Q θ (Z |X ). For VAEs, the objective is to jointly find a pair of optimal model weights θ * and φ * through a backpropagation process intended to minimize a differentiable loss function L (θ,φ).
(θ * , φ * ) = argmin θ,φ L (θ,φ) (X ) (1) L (θ,φ) is the evidence lower bound (ELBO) loss function is defined as follows: L (θ,φ) = -log(Q θ (X )) + D KL (P φ (Z |X ) || Q θ (Z |X )) (2) where: Q θ (X ): The probability distribution to characterize the input dataset X P φ (Z |X ): Function to approximate the posterior distribution Q θ (Z |X ) D KL (P φ (Z |X ) || Q θ (Z |X )): Distance between two distributions Q θ (X ) and P φ (Z |X ), calculated by using the Kullback-Leibler divergence function as follows: D KL (P φ (Z |X ) || Q θ (Z |X )) = P φ (Z |X )log P φ (Z |X ) Q θ (Z |X ) dZ (3) Because of the stochastic sampling in the latent space, we cannot directly do the backpropagation process through the neural network. Therefore, the parameterization trick should be used to overcome this problem.
• After the first model's weights are published to the aggregated model topics, the Cloud Server awaits requests from the VAE model configuration from each zone. • Local models are trained at each edge based on their own dataset. • In each communication round, the weights of the trained models ω (c) r are sent to the Cloud Server for FL. VOLUME 10, 2022 • The Cloud then uses the formula (5) to calculate the weight of the federated global model: ω r+1 = 1 C C c=1 ω (c) r (5) where: C is the number of zones. ω (c) r is the weight of the local model of zone c at round r. ω r+1 is the federated global model's weight at round r + 1. • Finally, the weight from the federated global model is sent downward to update the local model of each zone. In fact, as described in Algorithm 1, an edge will need to send the weight matrix to the cloud in many iterations during the back propagation process until the VAE model converges.
D. AUTOMATIC THRESHOLD DETERMINATION -FEDVAE-SVDD In this subsection, we will describe the automatic mechanism to determine a threshold for the Anomaly Detection model (FedVAE) to work efficiently. In order to do that, Support Vector Data Description (SVDD) is deployed to go over this request accurately while keeping the real-time assurance. Usually, experts in the industry establish the threshold after attempting a range of values, then select the one that best balances the requirements (performance, true positive, or false negative,. etc). SVDD also works well as an outlier detection algorithm, especially with high-dimensional datasets, but just like all SVMs, it does not scale to large datasets. Therefore, we suggest a combination of FedVAE and SVDD, as a moderate addition: FedVAE serves as the main anomaly detection model for the distributed system, while SVDD, trained with a small set of error vectors from the output of the FedVAE model, can correspond to finding a small region that encompasses all instances.
As the average marginal contribution of a player across all possible coalitions, the Shapley value tells us the payout to which the player is allocated fairly in a game. For a decision made by a black-box model, each feature of a data point can be considered as a player. Mathematically, the Shapley value of a feature in a learning model can be generally defined as: ξ n (f , a) = P⊆a |P|!(m -|P| -1)! m! [f a (P) -f a (P \ n)] (16) where, ξ n is the Shapley value of feature n f is a ''black-box'' model that needs explaining a is an input datapoint a is the simplified data input, which maps to a via a particular function mapping h a that satisfies a = h a (a ). P is one of all possible subsets of feature set, considered as a coalition m is the number of features in the dataset Due to the fixed input size, commonly, the features of a model omitted in the Eq.
(  16 ) are substituted with random input values from the background dataset. It can be seen that the total possible subsets of an m-feature set used for interpretation is 2 m , which leads to the massive complexity of computing Shapley values if m increases. Therefore, to deal with this issue, without calculating all combinations, Kernel SHAP  [26]  can be employed by sampling feature subsets and  VOLUME 10, 2022  then fitting them into a linear regression model:  (17)  In this linear regression model, the variables a i (i = 1, 2,. , m) are features encoded according to their presence (a i = 1) or absence (a i = 0), and the output value or label is the prediction value of the model f. After the training, the coefficients γ i can be interpreted as approximated Shapley values. Y = γ 0 + γ 1 a 1 + γ 2 a 2 + γ 3 a 3 +.
γ m a m In this work, our FedeX architecture is applied with SHAP to explain the impact level of features on the anomalies that are predicted by the FedVAE-SVDD model, via their SHAP values. As depicted in Fig. 3 , the Kernel SHAP takes the FedVAE model and the test data as inputs to construct a local linear regression explanation model. Subsequently, the explanatory model computes the SHAP values of classified anomalies and displays them visually. As feature values are measured by sensors, by using this explanation, operators or domain engineers can easily determine the sensors likely causing the abnormality and make a faster detection response. Through such actual maintenance, experts can validate and trust the proposed anomaly detection model more. These will be demonstrated in the case study of a SCADA liquid storage infrastructure in Section IV-C2.
IV. PERFORMANCE EVALUATION In this section, we evaluate the FedeX architecture in various aspects. From the detection performance perspective, the FedVAE-SVDD learning model is evaluated in comparison with different cutting-edge solutions for Anomaly detection in ICSs. The resource requirement of FedVAE-SVDD over the embedded edge device is also taken into account. Moreover, the results of using the XAI-SHAP technique to interpret the predicted results of FedVAE-SVDD are also described with the main case study in a SCADA liquid storage infrastructure dataset  [27].