dataesr/scientific-paragraphs-categorization huggingface dataset:
Scalable Curation With a growing number of datasets, it becomes increasingly costlier for consumers to find knowledge graphs suitable for their purposes, and upon finding graphs, for them to gauge their quality. This highlights the need for scalable curation mechanisms, which we will try to address in the following chapter. Free participation and censorship-resistance in our architecture has two sides. On the one hand, these characteristics ease the publication of useful and high-quality data for everyone. On the other hand, they make the infrastructure prone to being flooded with low-quality and less relevant data. Any entity can easily create as many KGs as desired. The gas costs act as a deterrent for spamming the network. Even so, the number of legitimate graphs could potentially increase to be high enough, as to make the cost of finding suitable graphs among them non-trivial, assuming there exists a channel from which consumers can find the address of all graphs.
SPARQL Queries Although querying triple patterns and compositions thereof would suffice for some applications, it falls short for others. In order to allow SPARQL queries, we build on the Linked Data Fragments framework  [29]  by implementing the Triple Patterns Fragments interface. By doing so, the TPF client decomposes a SPARQL query into triple patterns, retrieves the corresponding responses, and computes the final SPARQL response therefrom. In implementing the TPF interface, some specific points had to be taken into account. TPF has been design with REST APIs in mind. In our architecture, the client and the implementation of the TPF interface reside on the same node, and they communicate via faster intra-process means. In addition, the original TPF interface implementation runs on a server and fetches data from a local database, whereas with our solution, triples are stored across peers, and in case the required triples are not replicated locally, each triple pattern query is automatically translated into requests to fetch triples from other peers.
Finally, pagination and control functions such as nextPage, are a requirement of the TPF interface; currently pagination is done after fetching all of the triples matching a pattern. Federated SPARQL queries -i. e. queries requesting several graphs at onceare performed in a similar manner, by utilizing the TPF interface. In a nutshell, the query is first split according to the various implied graphs to obtain distinct Linked Data Fragments; then we process as described above; before joining together the results. However, whereas TPF originally requests the result of triple patterns from servers via HTTP, in our architecture, all triple pattern queries are done simply over different graphs which exist on the same P2P filesystem following the concerned hash addresses.
Complete Evaluation Process More generally, as already presented in Figure  2 , the solution has two layers: one on top based in Ethereum which offers managing features and the bottom one on IPFS where data is actually stored. As a consequence, during the query phase, the system takes into account this specificity. We present in Figure  3  the details of the query process around the information retrieval. Indeed, the first step requires to look for graph addresses in the Registry, then to obtain the correct root hashes from each Graph (taking e. g. into account the version number). Once these information are obtained, we are done with the Ethereum side and can decompose the query in order to get the triple pattern from the IPFS. Once they are retrieved, the answer can be constructed and the complete process to evaluate a SPARQL query is done.
Experimental Validation In this section, the results of a benchmark performed on the prototype implementation is presented. To verify the correct functionality of the architecture and its implementation, the source code also includes a test suite. Moreover, we generated a WatDiv  [2]  dataset with scale 1 which contains 107 665 triples, stored the dataset on our system and performed the 20 queries provided in the WatDiv (v0. 6) packaging, comprising of linear queries (L), star queries (S), snowflake-shaped queries (F) and complex queries (C). The benchmark was executed on a computer with Intel(R) Core(TM) i7-2640M CPU @ 2. 80GHz, SSDSA2BW16 disk and 8 GB of memory, running a Linux kernel (v4. 19. 1). We set up virtual nodes for IPFS to test our architecture. The goal was more to validate the feasability of the approach rather than benchmarking it.
By default the SDK maintains a cache of results fetched from IPFS. To measure query execution times, each query has been executed 5 times with a warm cache, and 5 times with a cold cache. Results of the queries have been compared for correctness against the ARQ engine 5. Loading Phase -Storing the dataset comprises of two main phases, constructing the index tree locally and storing the tree on IPFS from leaves to the root. Constructing the tree locally took 1 503 ms, and storing it on IPFS took 18. 52 minutes and translated into 318 972 IPFS PUT requests. Query Execution -Figure  4  displays execution times measured for the aforementioned queries using a logarithmic scale. The difference between default and w/o cache traces is in caching the results of IPFS get requests in the engine. Table  1  outlines measurements of metrics during each query, providing additional insight into factors potentially influencing the execution times.
Triple patterns denotes the number of triple patterns each SPARQL query is decomposed into by Triple Pattern Fragments client, IPFS gets is the number of GET requests to IPFS, Repeated paths is the number of paths that had been requested from IPFS during the same query and returned triples denotes the total number of triples that have been returned from IPFS to construct the final SPARQL result. Our implementation can be seen as a global linked open data repository which facilitates storing KGs and retrieving triples from either single graphs or a multitude of them. In particular, the metrics shown in Table  1  point to the number of decomposed triple patterns and IPFS requests as a potential factor that correlates with execution time. The repeated paths metric reemphasizes the benefits of a cache for intermediate results retrieved from IPFS.
Related Work There has been extensive research on centralized RDF data storage and retrieval. A survey of such storage and query processing schemes has been done by Faye et al. [11] , in which triplestores are categorized based on multiple factors. These factors include native vs non-native and in-memory vs disk-based storage solutions. Non-native solutions for example are triplestores that use an existing data store, such as relational databases. Hexastore  [31]  stores six indices, enabling efficient lookup of triple patterns for each parts of the triple, including subject, predicate and object. This gain in performance comes at a cost in storage. When it comes to querying data from remote servers, Verborgh et al. argue that there's a spectrum between data dumps and SPARQL endpoints, and that there's a tradeoff along the spectrum between factors including performance, cost, cache reuse, bandwidth, etc.
g. GridVine  [1]. Two general strategies have been observed by Filiali et al. , either retrieving all relevant triples from other peers and evaluating the result of the query locally, or propagating the query and partial results through the network, as in QC and SBV  [20]. Unlike the aforementioned protocols, in this study we don't design a custom P2P network specifically built for RDF data storage, but use the live global IPFS filesystem  [4] , which is simultaneously being used for other purposes. Triples are indexed as a Hexastore. To process queries, all relevant triples are fetched, and result is evaluated using the Triple Pattern Fragments  [29]  framework. Sicilia et al. [26]  explore publishing datasets on IPFS, either by storing the whole graph as a single object or by storing each dereferenceable entity as an object.
Furthermore they propose using IPNS to refer to the most recent dataset version. In this study, versioning is handled by a smart contract on Ethereum. English et al. [9]  explore both utilizing public blockchains for the semantic web, improving on the current URI schemes, storing values on the Bitcoin network, and creating ontologies for representing blockchain concepts. We share the idea that blockchains and web of data are complementary, and use the Ethereum blockchain to store metadata, and perform curation for KGs.
Conclusion & Future Work In this article, we proposed a novel architecture for a fully decentralized linked data infrastructure, which has a very low barrier to entry, is censorship-resistant and benefits from fault-tolerance properties of its underlying open technologies. Due to immutability of each version of a dataset, consumers can cache the data objects they interact with, and perform queries even while offline. By replicating these data objects, they are at the same time contributing to the availability of those datasets. In addition, we explored two mechanisms which allow a community to come to consensus over a collection of datasets which they find relevant or highquality in a distributed manner, by utilizing smart contracts to align the incentives of participants. IPFS replicates a document on every node that interacts with it.
The methodology followed in order to carry out this survey then started with a literature review on the previously cited domains. The main criteria applied to this literature review were database, year of publication, and type of publication, prioritizing survey papers published in journals with Impact Factor. The main keywords to select the bibliography were cyber-physical systems, intelligent buildings, building energy management systems, communications, smart metering systems, and cyber-security, always mixed with the term smart grid. Regarding each one of the different sections of the paper, the final validation of the selected references was conducted using a top-down approach, starting from the most general concepts to the most accurate ones: SG and IBs viewed as two systems of systems (SoS) interacting together; presentation of the inherent characteristics of an IB to finally focus on the technical aspects of energy management and communication technologies; and lastly, the cyber-security concerns of communications employed in IBs that contribute to the energy management are detailed according to the OSI model, to better delimitate the proposed solutions.
Also using Matlab/Simulink, a FLC for a residential building was designed in  [81]. A recent work using FLA proposed a solution for processing the environmental data to advise building users with the aim of achieving minimum energy consumption  [82]. Considering the distributed nature of an MAS, this technique is prevalent in the management of complex systems in general, and in the energy management of SGs in particular  [83, 84]. The work conducted in  [85]  studied several MAS dedicated to power engineering applications, as did that presented in  [86] , whereas  [87]  proposed two solutions for the supervision and analysis of a large quantity of data from a multisource electric network. An MAS for controlling production units, storage equipment and charges, based on Matlab/Simulink, was developed in  [88].
Attackers can clone the SM or compromise it. Moreover, cryptographic secrets can be extracted through side-channel attacks such as electromagnetic waves or laser injections. Few papers consider physical attacks due to the necessary of physical access  [124]. However, the SG includes IBs, which are open to external people, so a first possible solution is to protect physical access. The use of crypto processors and secure elements or firmware can prevent the lack of tamper protection. Energies 2021, 14, 2733 14 of 37 From a cyber-side point of view, attacks usually target databases and applications. Attackers exploit software flaws and misconfigurations in order to disrupt data management applications. Illegal access to employees' computers by means of social engineering attacks can lead to the introduction of malware programs that can help intruders access the database.
In this context, attacks against databases can be launched in order to tamper with or steal data with the aim of manipulating the energy market. Stock ciphering data can prevent such leakage and tampering  [125] , whereas random access to the database can solve privacy problems. Finally, in SG metering networks, data are generally exchanged in a single or multihop path. Some structures and devices of an SG that are usually targeted by cyber-attacks include SCADA network devices, phasor measurement units (PMUs), and AMI. Smart meters are basic components of the AMI, and represent a significant back door for eventual attacks to IBs and SHs, in which they are most frequently installed  [126]. Attacks on SMs, which are caused by the lack of security and privacy in SM wireless communications, can disrupt energy provision, destroy hardware, and cause loss of customer data.
Finally, homomorphic solutions enable complex mathematical operations to be performed on ciphered data without decryption, which improve confidentiality, especially in database applications  [194]. Symmetric encryption is faster than asymmetric encryption, which reduces energy consumption. However, lightweight cryptography mechanisms, such as tiny AES, have been designed to fit with constrained networks  [195]. To ensure the authentication and integrity of messages, a message integrity code (MIC) is generated thanks to a hash function and a secret key. In this context, HMAC-SHA-1 and HMAC-MD5 are examples of hash protocols  [196]. Still, MIC and ciphering use keys whose creation, distribution and maintenance represent a challenge in constrained environments, including in SGs  [197]. First, a common key shared by all the participants of network communications, called a global key, is the easiest solution and saves memory.
Detection Systems The first step in deploying the best countermeasure is to detect and identify the type of attack using an intrusion detection system (IDS). IDS can detect signatures, anomalies, or can be based on specifications  [210]. In signature-based IDS, attack detection is performed on the basis of known patterns, which is the method employed by antivirus software. A database of misuse is created by the network administrator to be compared with the run time network activity in order to detect any abnormal behavior. In SGs, IDS using the deep packet inspection method can be deployed to secure SCADA  [211] , while an IDS based on 50 signatures has been used in ModBus communications  [212]. However, this kind of IDS, very popular in classic networks, is not suitable for AMI, where the storage of signatures overflows the memories of constrained SMs, and the comparison algorithms cannot be run using a limited CPU.
IDS can be deployed in HAN to detect attacks on IEEE 802. 15. 4  [216]. A petri colored network can be implemented in IDS to model communications between SM  [217]. However, specification-based IDS requires the creation of a database by the administrator, as well as additional computational power. Anomaly-based IDS is therefore the more widely adopted solution for SG, but it again represents a challenge with respect to its deployment and realization  [218]. Indeed, classic IDS are too heavy and energy consuming to be deployed in constrained devices as SM. Some solutions are starting to be published, but it is important to identify the right place to implement them  [219]. First, centralized IDS employed in powerful trusted third parties such as GW can detect several types of attack. However, centralized IDS requires a global vision of communications, which is complicated in large networks and presents a single point of failure.
In decentralized solutions, IDS is deployed in constrained devices, which can exhaust their memory and energy, limiting the number of signatures in the database  [220, 221]. Therefore, IDS uses a rule of uniqueness, which is effective, while the behavior of the attacker (modification of number of packets, no optimum path) fits this rule. Consequently, in order to detect all the attacks described in the scientific literature, a great number of IDS is necessary. The last solution is a hybrid IDS, where detection is performed by an SM while decisions are taken by more powerful devices, at the cost of additional communications for IDS applications. In conclusion, in SGs, IDS will be able to help detect attacks when some remaining issues have been resolved, such as the space available for system implementation.
Lightweight ciphering at the MAC layer must be deployed to prevent privacy leaks. However, ciphers at the MAC layer lead to interoperability problems. There is also a lack of solutions for hiding MAC header metadata in order to prevent traffic analysis in AMI. Authentication, access control and trust are important features for allowing SG deployment. Authentication can prevent integrity attacks, but the needs are different depending on the level of the SG. For example, energy providers need reliable authentication schemes to link SM with its consumers in order to provide energy cost estimations. In the database, authentication is required for access to the data. New SM or devices can be deployed in run time in the SG. These devices are managed by different companies. Establishing trust between two devices managed by the same company is easy, but it is more complicated when there are two entities.
This study is part of an extensive project whose overarching goal is to freely and publicly offer datasets of illuminance at any time and any location in an operational way similar to the McClear model  [13, 14]  with inputs on atmosphere variables originating from CAMS. To achieve this project, a step-by-step approach was adopted in which the performance of the clear-sky procedure is separately assessed to be better able to later understand the performance of the modelling of the cloud effects. The accuracy of the presented method is evaluated at two sites providing high-quality 1-min measurements of illuminance under cloudless conditions. Overall, this study contributes to answering the growing need of construction industries and investors on the spatial and temporal availability of illuminance at ground level for optimal building design strategies.
The station field is flat and is covered by natural grasses without trees. Only measurements of the global illuminance on a horizontal surface were available and measured by a Global Photometric sensor model LI-210. These measurements have been downloaded from the website midcdmz. nrel. gov/apps/sitehome. pl?site=BMS, accessed on 1 December 2020. The relative uncertainty of the measurements is approximately 8% (  [22] ). In addition, measurements of broadband diffuse and global irradiances received on a horizontal surface and of direct irradiance at normal incidence were collected at both stations with 1 min of temporal resolution. With the assumption that cloudless instants identified by scrutinizing broadband irradiances are also cloudless instants in the illuminance measurements, the three datasets of broadband irradiances have served to select the cloudless instants instead of a single dataset of global illuminance at Golden or two datasets at Vaulx-en-Velin.
The SoDa service (http://www. soda-pro. com/, accessed on 1 December 2020) provides an access to the CAMS aerosol optical properties together with TOC and TWV. θ s is calculated with the SG2 algorithm  [21]. libRadtran offers several solar spectra; that of Gueymard  [23]  was selected for computing the solar spectral irradiance Io Nλ received at the top of atmosphere at normal incidence at any time, λ being the wavelength. Air Force Geophysics Laboratory (AFGL) vertical profiles are used and selected at any location using the map of Gschwind et al. [14]. The Shuttle Radar Topography Mission dataset is used to extract the ground altitude. The albedo of the ground in the daylight spectral band denotes the portion of the incident sunlight reflected by the ground in this band and is unknown in most cases.
For aerosol type, "urban" and "continental average" types from OPAC (Optical Properties of Aerosols and Clouds: The Software Package OPAC,  [28] ) were used at Vaux-en-Velin and Golden respectively. This could affect the estimates and particularly the RMSE, as the local atmosphere may experiment other aerosol types. An additional numerical validation was carried out by comparing the outputs of the presented method against those from detailed spectral calculations by libRadtran serving as reference to assess the accuracy of the method itself on both global illuminance at horizontal surface and direct illuminance at normal incidence. An independent sample of 10,000 atmospheric states has been randomly built following the marginal distribution variables described earlier in Section 3. 2. Overall, squared correlation coefficient of 1.
Acknowledgments: The authors thank the  NREL Solar Radiation Research Laboratory in Colorado, USA  (midcdmz. nrel. gov/apps/sitehome. pl?site=BMS, accessed on 1 December 2020)) and  Habitat Sciences Laboratory at the National School of Public Works in Lyon, France  (idmp. entpe. fr/stafr. htm,accessed on 1 December 2020) for making their data freely accessible.
Data Availability Statement: All used measurements at NREL, Golden, CO and Vaulx-en-Velin were freely available and downloaded through the following websites idmp. entpe. fr/mesfr. htm and midcdmz. nrel. gov/apps/sitehome. pl?site=BMS respectively, accessed on: 1 Decmeber 2020. The McClear atmospheric products are accessible after registration and were downloaded from: http: //www. soda-pro. com/, accessed on 1 December 2020. Ground surface images can be downloaded from the following website: https://worldview. earthdata. nasa. gov/, accessed on 24 April 2021).
We managed to extract these travel indicators from AIS messages. We use a dataset composed of AIS messages provided by the Danish Maritime Authority. This extraction follows several steps, adapted to the volume of data thanks to ARLAS PROC/ML, an open source and scalable processing platform. The results are visible with ARLAS Exploration, an open-source solution  (ARLAS Development Team, 2021a)  we have developed to visualize and explore interactively such amounts of geographical data. We will see how this valuable information can be extracted from such amounts of AIS messages thanks to these tools.
Data compression The creation of these 'course' objects allows an important compression of the information. Indeed, all the static information about the vessel is much less redundant and the aggregated dynamic information is much smaller. Moreover, we applied a geometry simplification algorithm to lighten the travel geometry without losing information. We also compute the geohashes covered by the travel geometry, to facilitate the geo exploration of these courses (only these geohashes could be enough to observe the flux, but we chose to keep also the detailed travel). From more than 31 million GPS locations representing 7 days of data, we eventually get around 14,000 course objects. In our context of Big Data analysis, this compression is really valuable.
This makes the computation scalable by increasing (or decreasing) the number of executors that are able to compute partitions. For instance, to get 14,000 courses from a dataset of 31 million GPS positions, the cluster was configured with 4 nodes of 8 cores and 52 Go memory, split into 6 executors (computation workers) of 4 cores and 30 Go memory. Since executors need sufficient memory on some steps of the process that require loading the entire vessel's time series, we had to limit parallelization with few executors to assign more resources for each one of them. That being said, the whole dataset was processed in about one day with this setup. If we need to ingest a larger dataset or to compute this one faster, we can add nodes to the computation cluster and/or give them more cores and memory.