dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2009 , Koch et al. , 2013 , Kuksa, 2015 , Nielsen et al. , 2003]. Of these approaches, NetMHC/NetMHCpan have been shown to achieve state-of-the-art for MHCpeptide binding prediction  [Nielsen et al. , 2016 , Trolle et al. , 2015]. In this article, we apply machine learning techniques from the natural language processing (NLP) domain to tackle the task of MHC-peptide binding prediction. Specifically, we introduce a new distributed representation of amino acids, named HLA-Vec, that maps amino acids to a 15dimensional vector space. We combine this vector space representation with a deep convolutional neural network (CNN) architecture, named HLA-CNN, for the task of HLA class I-peptide binding prediction. Finally, we provide evidence that shows HLA-CNN achieves state-of-the-art results for the majority of different allele subtypes from the IEDB weekly automated benchmark datasets.
Dataset To control for data pre-processing variabilities, we decided to use an existing post-processed training dataset so prediction algorithms could be more directly compared. The dataset used was filtered, processed, and prepared by  Luo et al. [Luo et al. , 2016]. This dataset contained HLA class I binding data curated from four widely used, publicly available MHC datasets: IEDB  [Vita et al. , 2015] , An-tiJen  [Toseland et al. , 2005] , MHCBN  [Lata et al. , The input is the peptide. The embedding layer substitues the individual amino acids with their 15-dimensional vector space representation. This is followed by two 1-dimensional convolutional layers preserving input length using 32 filters of size 7. The output of the 2nd convolutional layer is reshape into a 1-dimensional vector and is fully connected to the next layer of the same size.
This fully connected layer is then fully connected to a logistic output unit. The architecture is generalizable to allele subtypes of any length. 2009], and SYFPEITHI  [Rammensee et al. , 1999]. Target indicator indicating binding or nonbinding was readily given as one of the column in the processed dataset. Peptides that contained unknown or indiscernible amino acids, denoted as "X" or "B", were removed from the dataset prior to training. Dataset was split into 70% training set and 30% validation set. The test datasets were obtained from IEDB automatic server benchmark page (http://tools. iedb. org/auto bench/mhci/weekly/). Allele subtypes with less than 500 training examples were excluded from testing. The lack of training data is a well-known weakness of deep neural networks as the model may not converge to a solution or worst yet, may overfit to the small training set.
Distributed Representation Distributed representation has been successfully used in NLP to train word embeddings, the mapping of words to real-value vector space representations. More generally, distributed represen- tation is a means to represent an item by its relationship to other items. In word embeddings, this means semantically similar words are mapped near each other in the distributed representation vector space. The resulting distributed representation can then be used much like how BLO-SUM is used for sequence alignment of proteins  [Henikoff et al. , 1992]  or peptide binding prediction by NetMHCpan  [Andreatta et al. , 2015]. That is, we encode amino acids with their vector space distributed representation to be useable by downstream machine learning algorithms. Other amino acid encoding includes Atchley factors  [Atchley et al.
, 2015]. In this paper, the skip-gram model is used. The interested reader is encourage to consult the relevant references for further details of the CBOW or GloVe models. A short overview of the skip-gram model is given here for completeness. As originally formulated by Mikolov  [Mikolov et al. , 2013] , in the skip-gram model, given a sequence of words w 1 , w 2 ,. , w T , the objective is to maximize the average log probability: 1 T T t=1 -c≤j≤c,j =0 log p(w t+j |w j ) ( 1 ) where T is the total number of words (i. e. total number of amino acids) in the dataset, c is the context window size (i. e. number of words to the right or left of the target word, and p(w t+j |w t ) is defined as: p(w O |w I ) = exp (v w O v w I ) W w=1 exp(v w O v w I ) (2) Here, v w and v w are two vector space representations of the word w.
In this paper, the skip-gram model is used with a context window of size 5, 5 negative samples, and 15-dimensional vector space embedding. Various other dimensional size were explored, however, 15-dimensions gave the best results on 10-fold cross-validation of HLA-A*02:01 subtype. The entire post-processed dataset by Luo et al. [Luo et al. , 2016]  was used to learn this new distributed representation. The 15-dimensional vector space distributed representation, HLA-Vec, is summarized in Table  1. Experimental results indicate using our proposed HLA-Vec encoding showed performance gains over Asgari's representation, Atchley factors, or Kidera factors. Description of the experiements results can be found in the Supplementary Material.
Convolutional neural network Convolutional neural networks (CNN) have been studied since the late 1980s and have made a comeback in recent years along with renewed interested in artificial neural networks, and in particular of the deep architecture varieties. Much of the recent fervor has been spurned in part by both accessibility to large training datasets consisting of over millions of training examples and advances in cheap computing power needed to train these deep network architectures in a reasonable amount of time. Although originally proposed for the task of image classification  [LeCun et al. , 1989 , Krizhevsky, 2012 , Simonyan et al. , 2014] , CNN have been found to work well for general sequence data such as natural language sentences  [Kalchbrenner et al. , 2014 , Kim, 2014]. It is with this insight that we propose a convolutional neural network for the task of MHC-peptide binding prediction.
The number of epoch was less important as we arbitrarily set max epoch to 100 but enforce early stoppage if the loss function stops im-proving for 2 consecutive epochs. Solutions were found to have converged under 40 epochs for all test sets. The dataset was most abundant in 9-mer HLA-A*02:01 allele (10547 samples) therefore this specific 9-mer subtype was used for network architectural design and hyperparameter tuning. Dataset split of 70% training and 30% validation was used to determine the optimal architecture and hyperparamters. While the network architecture was designed using a single allele subtype of length 9, HLA-CNN framework is robust enough to accept and make prediction for allele subtypes of any length. Each test datasets of different allele subtypes and peptide lengths are treated as completely separate tests.
For a specific test dataset, the training dataset is filtered on the allele subtype and peptide length. The resulting smaller training subset is then used to train the HLA-CNN model. Due to the random nature of initialization in the deep learning software framework used, five prediction scores are made for each test sets. The final prediction used for evaluation purposes is taken as the average predicted score of the five predictions. Two commonly used evaluation metric for peptide binding prediction task are the Spearman's rank correlation coefficient (SRCC) and area under the receiver operating characteristic curve (AUC). The state-of-the-art NetMHCpan  [Andreatta et al. , 2015 , Trolle et al. , 2015] , a shallow feed forward neural network, and a more recently developed bipartite network-based algorithm, sNebula  [Luo et al.
Results We have introduced the HLA class I dataset. We formulated the HLA class I peptide data as an equivalence of text data used in NLP machine learning tasks. We have proposed a model to learn a vector space distributed representation of amino acids from this HLA class I dataset. We have also described our deep learning method and how it takes advantage of this new distributed representation of amino acids to solve the problem of HLA class I-peptide binding prediction. Next, we show the result of the learned distributed representation followed by the performance of our model against the state-of-the-art prediction model and another recently developed model.
g. points closer to each other in the original, high-dimensional space are grouped closer together in the low 2-dimensional space. We color this low dimensional representation with various physicochemical properties to see if any pattern can be discerned using this unsupervised machine learning technique. In Fig  2 , we see the 2-D visualization of HLA-Vec colored by various physicochemical properties, including hydrophobicity, normalized van der waals volume, polarity, and net charge  [Asgari et al. , 2015]  from the Amino acid Physicochemical properties Database (APDbase)  [Mathura et al. , 2005]. As can be seen, there are some structure in the graphs for hydrophobicity, polarity, and net charge; factors important for covalent chemical bonding. The clusters of magenta-colored amino acids are almost separable from clusters of green-colored amino acids with the exception of a few outliers.
HLA-peptide binding prediction The results of our HLA-CNN prediction model against NetMHCpan and sNebula on the two latest IEDB benchmarks are shown in Table  2. As AUC is a better measure of the goodness of binary predictors compared to SRCC, for evaluation purposes between models, we say one algorithm is superior to another if it scores higher on the AUC metric. On these latest IEDB benchmark datasets, our algorithm achieved state-of-the-art results in 10 of 15 (66. 7%) test datasets. This is in contrast to NetMHCpan, which acheived state-of-the-art results in only 4 out of 15 (26. 7%) and sNebula in 4 out of 15 (26. 7%). In the 10 allele subtypes where our model achieved state-of-the-art results, our model averaged a 9. 3% improvement over the previous state-of-the-art. As the binary cross-entropy loss function for this binding prediction problem operates on binary-transformed indicator values, any sort of ranking information encoded in ic50 binding measurements are loss in the objective and is a secondary task.
Run-time Analysis Though we did not do so, due to the relatively small peptide binding dataset compared to ones typically seen in NLP, both HLA-Vec and HLA-CNN can be parallelized on a GPU for faster computation. Using a single thread of a quad 3. 5GHz Intel Core i7 machine, HLA-Vec learned on the entire dataset took 33 seconds. HLA-CNN trained on the largest allele subtype, A*02:01 9-mers, took less than 10 minutes to run.
Conclusion In this work, we have described how machine learning techniques from the NLP domain could be applied to bioinformatics setting, specifically HLA class I-peptide binding prediction. We presented a method to extract a vector space distributed representation of amino acids from available HLA classI-peptide data that preserved property critical for covalent bonding. Using this vector space representation, we proposed a deep CNN architecture for the purpose of HLA class Ipeptide binding prediction. This framework is capable of making prediction for any length peptides or any allele subtype, provided sufficient training data is available. Experimental results on the IEDB benchmark datasets demonstrate our algorithm achieved state-of-the-art binding prediction performance on the majority of test sets over existing models.
426 0. 804 2015-08-07 1028790 A*02:03 9 55 ic50 0. 539 0. 696 0. 477 0. 629 0. 373 0. 746 2015-08-07 1028790 A*02:03 10 35 ic50 0. 208 0. 750 0. 419 0. 697 0. 307 0. 837 2015-08-07 1028790 A*02:06 9 55 ic50 0. 630 0. 770 0. 510 0. 848 0. 578 0. 819 2015-08-07 1028790 A*02:06 10 35 ic50 0. 572 0. 768 0. 525 0. 680 0. 638 0. 920 2015-08-07 1028790 A*68:02 9 55 ic50 0. 534 0. 806 0. 482 0. 713 0. 581 0. 909 2015-08-07 1028790 A*68:02 10 35 ic50 0. 272 0. 620 0. 591 0. 813 0. 722 0. 991 Average:. 511. 778. 436. 735. 521. 836 These benchmark datasets came from IEDB and encompasses the two most recent datsets. Allele sub- types with fewer than 500 training examples were excluded from these test datasets. SRCC stands for Spearman's rank correlation coefficient and AUC stands for area under the receiver operating characteristic curve.
Introduction Due to the rapidly advancing technologies of network communication, the Internet is going to connect everything from everywhere. New concept of Internet of Things (IoT) appears and is associated with the future Internet of 5G. IoT connects a large number of heterogeneous devices, such as "instance cameras," "wireless sensor network" (WSN), "smart meters," and "vehicles," while providing open access to a variety of data generated by such devices to provide new services to citizens and companies  [1]. However, as the resources of IoT's front devices are constrained, many security mechanisms are hard to be implemented to protect the IoT networks. Some lightweight encryption methods are considered as the core technology to build the security mechanism of IoT  [2] , but considering the increments of the hacker's computation ability (the usage of Cloud Computing, Distributed Computing, Quantum computation, etc.
), those lightweight cryptography methods are going to be crushed in the foreseeable future. Other kinds of security enforcement methods, such as intrusion detection system should be considered to protect the IoT networks  [3]. Intrusion detection system (IDS) is an efficient technique to detect attackers when cryptography is broken  [4]. It can detect malicious activities or policy violations by monitoring the network traffics or system actives  [5]. IDS is normally a stand-by device or third-part software which will not inquire many changes to the current system. It is suitable for the resource constrained or inherited systems to protect their network security. Many recent works have noticed the security problem of IoT system, and a number of intrusion detection methods are proposed and developed, such as  [4, [6] [7] [8] [9] [10].
However, most of the proposed methods are still limited to data mining and can only give an intrusion view of WSN, MANET, Zigbee, or other subnets of IoT, and a uniform intrusion detection method for the whole IoT networks is rarely discussed. Meanwhile, as the network packets digging and statistic feature training usually require many computation resources, such methods are hard to be implemented in some cases of IoT environments. In this article, we present an automata based intrusion detection method for the networks of Internet of Things. Our method uses an extension of Labelled Transition Systems to propose a uniform description of IoT systems and can detect the intrusions of IoT networks. The used automata model can describe the combination of heterogeneous networks with terms and graphs, and the proposed IDS structure and algorithm can detect the intrusions by comparing the abstracted actions flows, which can solve the aforementioned problems.
Paper Contribution. By using automata theory, many complicated problems can be described and solved. In this article, we use an extension of Input Output Labelled Transition System to solve the uniform description problem of the heterogeneous IoT networks and propose a corresponding intrusion detection mechanism for IoT network. To achieve this purpose, a set of procedures including collected data grouping, packet data translation, anomaly data detection, and intrusion classification are designed and proposed. Comparing with the existing methods, the benefits of our work can be listed as below: (1) To our knowledge, this is the first time of using automata theory to model and detect the intrusions of IoT networks. By using the proposed automata methods, we can map the IoT system to an abstract space, where a uniform security evaluation structure can be built.
Intrusion Detection System. The concept of intrusion detection was first proposed by Anderson in the year of 1980 The main advantage of misuse IDS is that it can accurately and efficiently detect instances of known attacks. The principal disadvantage is that it lacks the ability to detect the truly innovative attacks. On the other hand, anomaly IDS  [18]  does not require prior knowledge of intrusion and can thus detect new intrusions. But it may not be able to describe what the attack is and may have a high false positive rate. An IDS normally contained four major components: Event Monitor, Event Database, Event Analyzer, and Response Unit  [19]. The Event Monitor is responsible for detecting the system or environment actives and converts them as some specific formats and store them in the Event Database.
The Event Analyzer retrieves the modeled actives from the Event Database and analyzes them in order to detect the intrusions. Once the unusual actives are detected, the Response Unit produces reports to a management station to warn a risk. IDS focuses on detecting and preventing the intrusive activities, which were not detected by conventional system security mechanisms. For some inherited systems, because of some historical or economic reasons, some powerful security mechanisms are hard to be deployed. However, the IDS can be used to solve this problem, because it needs nothing to change the target system.
Intrusion Detection Approaches of IoT Networks. Although the proposed automata model can be used to describe the communications of an IoT system and can make the comparison of different subnets of IoT become possible, to adopt this model into an intrusion detection system, a set of cooperated devices and some existing approaches are also needed. Just like the general IDS system, the proposed automata based IDS of IoT networks also consist of four major components: Event Monitor, Event Database, Event Analyzer, and Response Unit. A general view of the proposed IDS can be presented in Figure  5. In this article, although the four components are developed in our system, our description will mainly focus on the Event Analyzer and Response Unit.
Event Database. In our method, the network event is described as the abstract action flows, and such network actions are described with transitions of the proposed Glued-IOLTS model. Three databases should be implemented in our IDS: Standard Protocol Library, Abnormal Action Library, and Normal Action Libraries are required. The Standard Protocol Libraries store the description of the standard protocols through Glued-IOLTS. The Normal Action Libraries store the possible action flows which are created from the Standard Protocol Libraries. The Abnormal Action Libraries store the recognized anomaly actions flows for the system. These three databases should be stored on the cloud and can be visited directly by the Event Analyzer.
Such transition sequences are then taken as the input to the intrusion verification part. In our method, we have two phases of intrusion verification. Intrusion Detection Phase 1. The results of Action Flows Abstraction Model are used to be checked with an Abnormal Action Library, which is stored in the Event Databases. This library is a predefined database that is stored on the cloud next to the IoT system (Fog Computing  [11] ). If the transition sequence matches with the one stored in the Abnormal Action Library, we remark such message as an intrusion and output it as the result of the intrusion detection system. If the input sequence does not match any stored sequences in the Abnormal Action Library, the action flows go to the second phase of the intrusion detection. Intrusion Detection Phase 2.
Mobile Information Systems The router is connected with a server, and on the server, we use MySQL to build three database tables: Standard_Protocol, Abnormal_table, and Normal_table, which are corresponding to the three databases in our IDS methods. We use port mirroring on the router (a plug-in is needed to be installed on the OpenWrT router) and mirror the packets of WAN to the connected server. We install Wireshark  [28]  on the server side to collect and analyze the forwarded transmitting packets from IoT gateway. In our experiment, the RADIUS applications are taken as the services executed on the tested IoT networks  [29]. The RADIUS protocol is an application layer protocol, which transmits data through UDP traffics. It uses the port number 1812 or 1645 to communicate. So when the monitor (Wireshark) obtains the IP traffics, by checking the port number of the UDP messages, the RADIUS messages can be distinguished.
This sequence contains a replay-attack. And the third sequence is not a complete sequence. If the IDS only verifies the signature of the message, it will not find the problem of the second transition sequence. In our IDS approach, we only need to search this transition trace in the corresponding reachable graph, which is a nonanomalous profile of the target system. The proposed Java tools will visit the Standard_Protocol table (the Standard Protocol Library) on MySQL database, and the nonanomalous profile of RADIUS protocol can be presented as the Glued-IOLTS of Figure  11. In this selected experiment, the verified traffics contain two RADIUS sessions and after the "message concatenation and classification," two different message sequences are obtained (they are listed in the bottom-left of Figure  11 ).
Conclusion Internet of Things is an important part of the future 5G, and the security of IoT will relate to many important scenarios of the future 5G and has become the core requirement of the network development. However, as the resources of IoT devices are constrained, many security mechanisms are hard to be implemented to protect the security of IoT networks. In this article, based on the automata theory, we proposed a uniform intrusion detection method for the vast heterogeneous IoT networks. Our method uses an extension of Labelled Transition Systems to propose a uniform description of IoT systems and can detect the intrusions by comparing the abstracted actions flows. We designed the intrusion detection approach, built the Event Databases, and implemented the Event Analyzer to achieve the IDS approaches.
Acknowledgments This work is sponsored by the  National Key R&D Program of China  (Grant  2016YFB0800700 ), the  NSFC  (Grants  61602359  and  61402354 ), the  China Postdoctoral Science Foundation Funded Project  (no. 2015M582618 ), the  111  project (Grant  B16037 ), and the  Fundamental Research Funds for the Central Universities  ( JB150115  and  JB161508 ).
The recently proposed principal component analysis network (PCANet) has been proved high performance for visual content classification. In this letter, we develop a tensorial extension of PCANet, namely, multilinear principal analysis component network (MPCANet), for tensor object classification. Compared to PCANet, the proposed MPCANet uses the spatial structure and the relationship between each dimension of tensor objects much more efficiently. Experiments were conducted on different visual content datasets including UCF sports action video sequences database and UCF11 database. The experimental results have revealed that the proposed MPCANet achieves higher classification accuracy than PCANet for tensor object classification.
A. Performance in UCF sports action database The experiments are carried out on UCF sports action video dataset. It contains approximately 200 sport action videos at resolution of 720 × 480, included in 9 classes, which are typically featured on broadcast television channels such as the BBC and ESPN. Before feeding the UCF samples to MPCANet, the tensorial input needs to be normalized to the same dimension in each mode. Every video is converted from color to gray. The primary individual, who doing sport, in each frame, is cropped with dimension 400 × 250, which is then resized to 80 × 50 pixels in order to reduce the computational complexity. The normalized time-mode dimension is chosen to be 20, which keeps a complete action as much as possible. Thus, each video sample has a canonical representation of I1×I2×I3 = 80×50×20.