dataesr/scientific-paragraphs-categorization huggingface dataset:
Recently, a Cambrian explosion of a novel, non-volatile memory (NVM) devices known as memristive devices have inspired effort in building hardware neural networks that learn like the brain. Early experimental prototypes built simple perceptrons from nanosynapses, and recently, fully-connected multilayer perceptron (MLP) learning systems have been realized. However, while backpropagating learning systems pair well with high-precision computer memories and achieve state-of-the-art performances, this typically comes with a massive energy budget. For future Internet of Things/peripheral use cases, system energy footprint will be a major constraint, and emerging NVM devices may fill the gap by sacrificing high bit precision for lower energy. In this paper, we contrast the well-known MLP approach with the extreme learning machine (ELM) or NoProp approach, which uses a large layer of random weights to improve the separability of high-dimensional tasks, and is usually considered inferior in a software context.
Previous work has been done on porting ELM-inspired systems to nanotechnological substrates in order to take advantage of high density and energy-efficiency, such as the use of domain-wall nanowire logic systems  [20] , or resistive memory crossbar systems  [21] ,  [22] ; however, all previous proposals critically relied upon the energy-intensive offsite computing requirement of computing a large matrix inverse. As first proposed in  [23] , this inverse can be locally and sequentially approximated on-chip; however, until now this approach has never been rigorously benchmarked compared to ELM software performance, or competing nanotechnology-ready algorithms i. e. approximate backpropagation. Concretely, we contrast the efficiency of these categories of neural networks built from memristive nanosynapses: • Online MLP systems trained by standard backpropagation, that learn according to a modern cost function, and with contrasting differential neuron designs • Online NoProp systems, where the first layer realizes random weights given intrinsic device-to-device variation, a simple neuron design optimized to reduce energy consumption provides projections, and where the output layer is trained sequentially via a sign-based implementation of the classic Widrow-Hoff learning rule.
B. NANOELECTRONIC LEARNING IMPLEMENTATION To explore computing possibilities of networks built from nanodevices the MNIST database of handwritten digits  [38] , consisting of 60, 000 training digits and 10, 000 test digits, was selected. The neural network architecture consists of: • A number of input neurons, L. For the MNIST task, L = 784 is set. • A number of hidden layer neurons, M performing projection and/or gradient accumulation, where M can vary as a parameter. • A number of output neurons N. Again for MNIST, N = 10 is set according to the ten digit classes. Like in standard computer vision tasks, images are vectorized before system presentation. A simple analog encoding scheme maps images on a pixel-by-pixel basis into voltage values. Given a numeric input channel or pixel of index i, X i a voltage within the range -V read < V i < +V read is assigned: V i = 2V read (X i /L max ) -V read , (5) where L max is the maximum pixel intensity, and V read is a voltage that does not alter nanodevice conductances.
To smooth the effect of random initial conditions (device conductances) upon learning outcomes, Monte Carlo simulations were run with different seeds. The mean of five system's learning performances is shown in the following section unless otherwise noted. For off-chip comparisons, scripts were written in Python, integrating functions available in the TensorFlow packages to build software ANNs with floating point synapses  [51]. Software gradient learning system weights were always obtained with the same activation function and hidden layer dimensions, so as to make a fair comparison with the companion nanosystem, and an appropriate, constant learning rate. For off-chip imported or software comparisons, NoProp second-layer weights were always obtained analytically via regularized ridge regression.
A. EFFECTIVENESS SIGN-BASED ON-CHIP TRAINING Previous work  [52] ,  [53]  has argued that sign-based gradient learning can serve as a satisfactory on-chip approximation for fully featured gradient descent in multilayer perceptron systems. We confirm this result, as our MLP results with a constant learning rate and using tanh and ReLU hidden neurons can achieve within 0. 2% and 0. 8% of the results obtained with software synapses in Tensorflow, as visible in Table  1. For the first time, we analogously confirm the validity of our sign-WH appropriate on-chip learning with NoProp/ELM systems. We have directly contrasted the ability of our proposed system, which learns locally with only presynaptic activation and post-synaptic error values obtained from labels as in Fig. 2 (a), to a comparison system which collects hidden layer activations, computes output weights off-chip analytically and then imports them before inference.
B. BEST PERFORMANCE ON TASK Fig. 4 (a) demonstrates the convergence of in-situ NoProp systems of varying synapse quality alongside a fully software neural network. As visible, this ex-situ learning approach is faster to converge; at M = 3000, maximum 94. 3% performance is analytically obtained given around 50, 000 samples. In contrast, a full three epochs or 180, 000 training samples are required to achieve on-chip convergence, which slightly outperforms the software system at 94. 5%. Using the mini-batch stochastic gradient descent learning policy (mini-batches of b = 100 images), cross-entropy and softmax error as the cost function and following sign-based gradient descent, we examined the success of several MLP systems using varying hidden layer activation functions. The strongest performing system used tanh; this system achieved superior performance on the test set of the MNIST database as compared to the NoProp system as well as others, at mean 96.
3% recognition-but only when the system is built from perfect (linear) devices. As visible in Fig  4(a) , this system approaches the performance of a software batch learning system after 6-7 epochs (360k-420k training samples). When using the non-linear model, this performance advantage disappears. Moreover, as visible in Table  1 , the generic (ReLU ) and best rectifier system struggle to completely match software results. Moreover, these systems slightly under-perform the NoProp systems when the perfect model, and substantially under-perform them when using the non-linear model.
A. RELATIONSHIP TO STATE-OF-THE-ART NVM GRADIENT LEARNING A key result is that, unlike NoProp, memristive gradient systems learning with a sign approximate of the gradient (s-WH in both layers as sign(δ k x j,k )), require g > 256 or eight bit weight resolution to do so effectively ( losing 1% or less compared to software comparison systems). This observation is consistent with the machine learning literature, which has suggested that less than 8 bit weight resolution can significantly impede network generalization and accuracy when gradient updates are binarized  [71] ,  [72]. In  [37]  a parameter analysis shows that g = 64 or 6 bits are sufficient for NVM backprop online learning. This is not necessarily an inconsistent result as the updates used to manifest backprop in that case are precisely calibrated, which requires additional overhead on the core to calculate magnitudes of δ k x j,k for every row and/or column.
B. ADVANTAGES AND CONSTRAINTS OF NVM NoProp LEARNING The online learning NoProp learning systems we analyzed were only able to converge to software quality results with sufficient writable depth, and perfect (linear) evolution. Yet, unlike gradient-learning alternatives, the required writable depth requirement is gentle, with 6 bits g = 64 being ideal but 4-5 bits (g = 16 -32) performing well too. In  [77] , 4 bits were shown sufficient to implement weights executing simple STDP (temporal correlation) learning, which is a similar operation to s-WH. In addition, a memristive two-layer system using the locally competitive algorithm in the first layer (unsupervised) and supervised learning in the second layer also required between 4-6 bits depending on device considerations  [78]. These consistent results could inspire further  VOLUME 7, 2019  research on the potential advantages of random/unsupervised pre-processing layers cooperating with later supervised ones.
V. CONCLUSION In this work, we have designed an energy-efficient in-situ learning rule and system inspired by software-based NoProp/ELM systems, and found that it can successfully compete with imported weight (inference-only) memristive as well as software alternatives. Using the rule, we obtained state-of-the-art results for single hidden-layer memristive ELM: 94. 5%, and multiple hidden-layer memristive ELM (M-HL ELM): 96. 83% on the MNIST task, the latter of which was competitive with results obtained with a memristive MLP system with similar constraints (96. 3%. ) We also found that NoProp/ELM systems learning with this rule obtain special properties. Foremost, they manage to learn with synapses with less than 8 bit quality, while this serves as a necessary minimal requirement for MLP online learning systems learning with the sign-based learning rule.
Additionally, we discovered that device non-linearity has a differential impact between NoProp/ELM v. MLP online learning styles. While only perfect/linear devices can fully emulate software results in all cases, NoProp/ELM systems are notably more resilient to systems learning with non-linear synapses. Lastly, we found that device-to-device variability, asymmetry and cycle-to-cycle noise again have a differential impact between the two learning approaches. NoProp/ELM systems seem to be more resilient to several of the effects; however, no system is immune to them. An energy estimation of online learning in memristive MLP and NoProp systems-taking into the array's core operation costs as well as key peripheral CMOS systems for a few candidate nanosynapses classes-showed that NoProp systems save energy head-to-head versus competitor MLP systems.
ACKNOWLEDGMENT This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the  U. S. Department of Energy  or the  United States Government. Sandia National Laboratories  is a multimission laboratory managed and operated by  National Technology & Engineering Solutions of Sandia, LLC , a wholly owned subsidiary of  Honeywell International Inc. , for the  U. S. Department of Energy's National Nuclear Security Administration  under contract  DE-NA0003525.
The work of C. H. Bennett was supported in part by the  Paris-Saclay Nanodesign Lidex , in part by the Chaire nanofiabilite, and in part by the  Sandia's Laboratory-Directed Research and Development Program. The work of  V. Parmar  and  M. Suri  is supported by  DST-SERB-Core Research Grant  (Number  CRG/2018/001901 ),  Government of India ,  CNRS-PICS Grant  and  CYRAN AI Solutions. The work of  M. Marinella  was supported by the  Sandia's Laboratory-Directed Research and Development Program. The work of  D. Querlioz  was supported in part by the  European Research Council  Grant  NANOINFER  under Grant  715872  and in part by the  CNRS PICS  Grant.
This paper presents the main results of the experiments conducted using the MMT-IoT security analysis solution run on a IoT Fed4Fire+ platform (Virtual Wall -w. iLab proposed by IMEC, Belgium). MMT is a monitoring framework developed by Montimage, and MMT-IoT is the tool that allows monitoring and analysing the security and performance of IoT networks. The results obtained concern two principal advancements. First, the adaptations made to deploy MMT-IoT on the IoT platform in order to run the tool on the platform's IoT devices. Second, the deployment of the software allowed us to run preliminary tests on the selected platform for performing initial validation and scalability tests on this real environment. To this end, Montimage defined and implemented three test scenarios related to security and scalability with 1 or more clients.
The challenge of security monitoring on IoT network arises when trying to detect these attacks on devices that have strict resource limitations. Furthermore, existing centralised monitoring techniques (Intrusion Detection and Prevention Systems) cannot handle the large amounts of data that needs to be analysed, and have been designed to work on the edge of the networks and cannot cope with IoT networks that lack clear boundaries. In this paper, we present MMT-IoT, a security tool intended for addressing the requirements for security monitoring on IoT networks, and its application in industrial settings. MMT-IoT allows capturing IoT network traffic near the IoT devices and analyses them to detect potential attacks. In this work we take advantage of the industrial Fed4Fire testbed to deploy MMT-IoT on a near-to-real-life scenario, validate its security detection properties and perform initial scalability tests.
Montimage Monitoring Tool (MMT) for IoT Networks The Montimage Monitoring Tool (MMT)  [8]  is a modular monitoring framework that allows detecting behavior, security and performance incidents based on a set of formal properties (written in XML) and embedded functions (written in C or any script or interpreted language). MMT allows real-time data capture, metadata extraction, correlation of data from different sources (i. e. network, applications traces and logs, operating systems), complex event processing, and distributed analysis. It uses temporal logic to detect given security properties (expected or anomalous), and statistical and machine learning-based analysis for detecting more sophisticated activities and behaviour. It is relatively easy to extend by adding new: i) properties and embedded functions; ii) plugins for parsing any structured message; iii) new dashboards for visualising data, statistics and alarms; and, iv) instructions to trigger reactions (e.
g. mitigation or blocking of attacks). In order to correctly adapt this approach -designed initially for traditional Ethernet networks -to IoT networks, it was required to split the network extractor (sniffer) in two parts: the MMT-IoT Sniffer (a Contiki-based IoT device), and the MMT-IoT Bridge (a Linux-based tool). The former is the IoT endpoint that is in charge of sniffing the packets and forwarding them -using a USB line -to a more powerful machine. The latter recovers the transferred packets from the USB line and injects them (encapsulated using the ZEP protocol) in the loopback interface of the machine, making the packets ready for analysis by the MMT-Probe and MMT-Security. Figure  1  summarises the general architecture of the solution. Concerning the MMT-IoT Sniffer, the implementation of this architecture was achieved by introducing modifications in the network drivers to make the sniffing feature work.
The structure of the inserted callbacks is depicted in Figure  2. Finally, the MMT-IoT Bridge is responsible for capturing the packets sent through the USB line and making them available for the security analysis performed by the MMT-Probe and MMT-Security; both part of the MMT software. This security analysis is performed by a set of security rules -previously assessed by a network security engineer -which codify the set of network events that need to be correlated for detecting security issues. It is important to notice that computaiton complexity of detecting an attack is given by the rule itself; complex attacks require more complex rules which correlate a higher number of network events. Considering this, the computation complexity will be taken by MMT-Probe, and not MMT-IoT, which only redirects the traffic to MMT-Probe.
Virtual Wall -w. iLab The w. iLab platform [5] is an IoT and 5G emulation testbed that allows running experiments on nodes on real IoT deployments. This platform was designed by the IMEC, Belgium. It provides "bare metal" access to its nodes, i. e. , it gives root access to physical machines that will be used to run the experiment. This allows the experimenter to have full control of the nodes on the testbed. The deployment of the MMT-IoT and MMT-Probe software and the execution of the tests are performed remotely without requiring major interventions from the operators. For this, we created credentials on the iMinds platform and performed a reservation of the Intel NUC nodes from the "Datacenter" floor of the platform. The jFED-Experimenter tool was required to design an experiment to access these nodes.
-IoT Router: A gateway running a routing protocol to allow communications within the IoT network. -MMT-IoT: A node running the Montimage software under test. We used the deployment described above to perform initial validation and scalability tests in scenarios that contain respectively 1, 2, 3 malicious clients. We used these configurations to pursue both objectives previously mentioned: (1) the security analysis validation, by means of determining the number of detected attacks; and (2) the scalability of the MMT-IoT solution, by means of analyzing the number of extracted packets on each scenario. This latter aims to be a first test of the scalability of the MMT-IoT software, aiming to determine the amount of information an IoT sniffer is capable to handle. To deploy the testing scenarios we used the nodes provided by the w.
iLab platform, each one composed of a Linux machine with two Zolertia Re-Mote IoT nodes. On each node we used the Zolertia Re-mote nodes to install the corresponding device type (in form of an IoT firmware) and generate the test traffic. Additionally, we installed the MMT-IoT Bridge, MMT-Probe and MMT-Security software on the MMT-IoT Linux machine. This was done in order to read the packets extracted by the IoT sniffer and perform the security analysis on the same node. The Ping Client IoT sensors were configured to trigger the attack every 10 seconds. At each triggering, the client sent a burst of 10 ICMP ping packets equally spaced within a second. Additionally, an RPL router image was deployed in the "IoT-Router" machine in order to allow packets to flow through the network. All the MMT software was deployed in the MMT-IoT machine, including the MMT-IoT sniffer (in the Zolertia remote connected to that node), the MMT-IoT Bridge (running on the same NUC machine) and the MMT-Probe (also running on the NUC machine).
The packets extracted with MMT-IoT Sniffer (using the tcpdump tool) and the MMT-Probe logs are used to check the number of detected attacks in the scenario. In this figure one can observe peaks each 10 seconds. These peaks correspond to the automatic triggering of the attacks, i. e. they show the moment when the clients started to send the ICMP ping packets. In these particular instances we observe a raise in the extracted traffic since there was more data available to be processed. In the 3-clients scenario we see that after 3 minutes of execution the peaks appear more often. We conjecture that this behaviour is due to some type of "desynchronization" between the three clients, and the different attacks appear more frequently. An interesting observation is the limit of the extracted packets per second.
Introduction Today pesticides, antimicrobials and other pest control products used in conventional agriculture are questioned and alternative solutions are searched out, including active plant-based products. The Knomana (KNOwledge MANAgement on pesticides plants in Africa) project aims to identify plants used as biopesticides, currently from the scientific literature, and to implement methods to support the exploration of knowledge by the potential users (farmers, advisers, researchers, retailers, etc. ). About 30000 descriptions of plant uses have been collected and recorded according to a data model designed through meetings with domain experts. Each plant use is described using 36 attributes such as the plant taxonomy, the protected system (i. e. crop, animal and human being), or the preparation method.
Each conceptual structure classifies the objects of one category according to the (initial) attributes and the relational attributes, thus according to the relations that the objects of this category have with objects and object groups (concepts) of another (or the same) category. Considering a real-word context such as the Knomana project dataset, although a data model has been set for data collection purpose, there are still many ways to encode the data model into a relational context family. In this paper, we study the impact of the definition of a relational context family on the practicability of RCA on the Knomana project dataset. The first question is raised by the fact that relational contexts represent binary relations between objects, but the data model we deal with contains a ternary relation.
Section 2 presents Relational Concept Analysis and the RCAExplore tool which is used in our evaluations. In Section 3, we propose two encodings of the initial data model into relational context families. Then, in Section 4, we first show the size and computation time of several conceptual structures for the two encodings on an excerpt of the Knomana dataset restricted to a few key plants designated by the domain experts. Then we study a simple, yet real, exploration scenario using both encodings, to show their relevancy with respect to the studied question. Section 5 exposes related work. We conclude and give perspectives of this work in Section 6.
Other quantifiers can be found in  [11]. RCA process consists in applying FCA first on each object-attribute context of an RCF, and then iteratively on each object-attribute context extended by the relational attributes created using the concepts from the previous step. The RCA process stops when the families of lattices of two consecutive steps are isomorphic and the extended object-attribute contexts are unchanged. In the following, we consider a small example from the Knomana dataset. Object-attribute contexts Plants and Pests (Table  1 , (a) and (b)) respectively represent a set of plants and a set of pests. Pests are described by their family (e. g. Callosobruchus maculatus -CallosobruchusM in Table  1 -is a member of the leaf beetle family, Chrysomelidae), while plants are described by the parts (e.
Family and Conceptual Structures The Knomana database gathers descriptions of plant uses, each one characterized using 36 data types, including the protecting plant, the protected organism, the controlled aggressor (also called pest and disease), the method adopted to prepare the product to be applied, or the reference to the document describing the use. Currently, the Knomana database comprises 28700 plant use descriptions manually entered from 250 documents (mainly scientific publications). The descriptions include 966 plant species, originated from 60 territories, used to protect 39 species of organism (animal, vegetal, and human) against 253 species of aggressors (Bacteria, Chromista, Eukaryota, Fungi, Insecta, and Virus). In the data model, data types are grouped as data classes to represent the three main entity categories of the system: biopesticide, protected system and targeted organism.
To represent the biological system, these three main entity categories (or data classes) are linked through a ternary relationship. As the relational contexts of RCA are binary relationships, two different data models have been designed. The first one, called M1 (see left-hand side of Figure  3 ), consists in reifying the ternary relationship as a specific data class, the latter supporting binary relationships with each of the three main data classes of the biological system. The second one, called M2 (see right-hand side of Figure  3 ), consists in establishing binary relationships between the data classes of the biological system, corresponding to projections of the ternary relationship. The relation directions have been determined in order to obtain classifications and propagation of relations according to the first question (Q1) asked by the experts (introduced below).