dataesr/scientific-paragraphs-categorization huggingface dataset:
The architecture development and prototyping of "scientific data lakes" is being carried out as a part of the DOMA  [6]  project (WLCG  [7] , CERN). The work on the project is closely related to: • the development of data management systems and data streams for processing and analyzing information in the exabyte range; • the development and testing of scenarios for connecting computer centers to the "data lake" infrastructure depending on the level of their resources and the quality of communication channels (including supercomputer centers); • the development of methods for determining the popularity (demand) of particular scientific data sets and data management methods. In this paper, we will mainly discuss the development of the Russian Scientific Data Lake prototype. This work is carried out by Russian research centers (NRC KI -PNPI, ISP RAS) and universities (SPbSU, MEPhI, RUE) in cooperation with international scientific centers (CERN, JINR, LAPP, UNAB).
Problem description and challenges Taking into account the new requirements for data storage and data processing systems for the HL-LHC project, the R&D activities are currently underway on technologies for building such systems using the "data lake" concept which would work as a part of distributed heterogeneous high-throughput global computing system. To obtain relevant research results, each of the possible technologies must be tested using the uniform methodology. It is also necessary to demonstrate and describe a methodology for connecting relatively small computing systems without associated storage elements to already existing distributed computing infrastructure. This methodology should significantly decrease the "entry threshold" for the small sites (analysis facilities), which for some reason cannot provide deployment and support of dedicated data storage systems.
Provided methodology needs to reduce the complexity of deployment and maintenance requirements of components being installed, as well as bring down the overhead costs, which would allow even smaller sites to work effectively with the data lake. To reduce the complexity of installing and deploying of software components, the modern container technologies can be used, which allow for almost instantaneous deployment of many identical instances of a prepared software stack that requires just a few parameters to configure. Concerning the reduction of overhead costs, the authors see an increase in the efficiency of using the CPU resources of computational nodes by minimizing the time the CPU is idle due to data I/O operations. In the standard workflow for distributed data processing of scientific experiments a task allocated to a computational node must first receive input data from the storage system, and then write output data into it.
Both operations can take a significant amount of time compared to the whole task processing time, although they practically do not occupy the CPU. Minimizing the time for data reading and writing at the beginning and at the end of a computational task can significantly increase the efficiency of using the CPU resources. Combining these two directions of increasing the data processing efficiency discussed above, the development of a data lake prototype with one central storage, which has high reliability, capacity and availability, and several satellite computing sites with small storage resources used as a short-term data storage (cache or buffer) will be shown in this paper. At the same time the data caching for reading and writing will be considered separately, since the expediency of its use, as well as the technologies used for this, depend on the types of computational tasks.
Data caching Three main configurations of the caching system were evaluated and studied (Fig. 1 ): 1. a dedicated caching server -central control of caching, possibility to quickly change the storage hardware without touching the computational nodes; 2. a local caching server on each computational node with no data exchange between the nodes -good for fast deployment of multiple independent instances; 3. a local caching server on each computational node with data exchange between the nodes forming a distributed cache (one of the computational nodes additionally acts as a control node) -a somewhat more complicated configuration than 2 but with better LAN utilisation and higher cache hit rates. Configuration 2 has been used for primary evaluation tests only. This is due to the fact that the number of repeated requests to the same input data file in LHC workflows is low, and is less than the average number of computational nodes even on relatively small sites, which leads to the nearly zero hit ratio for an isolated node-local cache, rendering it practically useless.
It is more interesting to compare configurations 1 and 3. In configuration 1 all the burden of storing, searching and copying files from the main storage lies on a dedicated server, almost completely freeing up worker nodes. In the case of configuration 3 with a distributed cache there are savings due to the lack of a dedicated server, but there is an additional load on the worker nodes. Also, configuration 3 allows to create a scalable hybrid CE + SE system (compute plus storage) based on one unified configuration containing all the necessary components. Such a system can be scaled horizontally by adding more instances, which is extremely convenient when deploying to cloud resources. In addition to this, there appears, although not very high, the probability of reading the input file from the cache located on the same worker node as the requesting task, which further optimizes the data transfer.
Data buffering In order to further optimize the usage of CPU resources for write-oriented workloads the development of a buffering system for output data was started in 2020. In the case when the analysis task generates a significant amount of output data, the time required to transfer this data from a worker node to a remote storage can be accounted for as a wasted CPU time. If the generated data is temporarily stored on a fast local server (a buffer) then the CPU time will be saved and the efficiency of the site will increase. This can be expressed, among other things, in the average processing time of a single task. The proposed scheme is shown in Figure  2. The testbed for the data caching system was deployed at the resource centers at JINR (Dubna), PNPI (Gatchina), MEPhI (Moscow), and RUE (Moscow).
The monitoring system To estimate the efficiency of computer resources usage and to monitor the software and hardware infrastructure in the process of testing the Russian Scientific Data Lake prototype, a unified monitoring system was developed using modern web and database technologies. At the current stage of the prototype development the following distributed data sources are used to monitor the operation of the deployed software infrastructure: • XRootD log files, which contain complete information about the XCache service operations; • Billing relational database (dCache) based on PostgreSQL DBMS. The database contains information about operations (billinginfo) and requests for an operation (doorinfo) • A special API provided by the monitoring system for distributed production and analysis of ATLAS experiment data -BigPanDA  [13] , to obtain information and necessary metrics describing test jobs.
In particular, the job type, the queue of the computing resource and the node where the job was completed, the final job status, error messages, the list of input files, the amount of consumed RAM, time metrics: total job execution time; time of job initialization; loading time of input files; time of uploading output files; time of transformation of input files into output ones, etc. • Accounting database based on MySQL DBMS. The database contains information about the local jobs that describe the current state of computing elements. ElasticSearch was chosen as a unified data storage, which is part of the ELK stack. It provides efficient access to aggregated information from distributed sources due to the flexibility of storage schemes. The general architecture diagram of the unified monitoring system is shown in Figure  3.
Using the PNPI computing infrastructure, all the necessary components included in the monitoring system were deployed, in particular, the ElasticSearch storage, the Logstash input data flow collector, the Kibana visualization system and a specially developed web application for monitoring and analysis of the test jobs execution results, which implements specific data visualizations which are not natively available in the Kibana. Thus, data streams from all sources pass through the Logstash collector, which converts heterogeneous data into keyvalue format and stores it in specially dedicated indexes of ElasticSearch document-oriented non-relational storage. In the expanded Kibana visualization system, dashboards were created for the software components of the Russian Scientific Data Lake prototype, which consist of graphs, tables and other available types of visualizations.
The following monitoring dashboards were implemented in the Kibana system using the collected data: • XRootD monitoring. Dashboard is designed to monitor the XRootD component and is based on processed data from log files; • Billing monitoring. This monitoring section allows to get information about the work of dCache, which is extracted from two separate tables of the Billing database combined and exported to the ElasticSearch index; • Jobs monitoring. Based on information about all test jobs executing on the testbed; • Accounting monitoring. The "data lake" includes sites with a computing element based on CREAM CE, an outdated system at the moment, and computing elements based on ARC CE, which is currently recommended for use in the WLCG. Due to the limited number of available visualisation types in the Kibana system and the lack of customizing functionality, we developed a special web application for monitoring and analyzing the results of test jobs.
, 2019; Jabbar et al. , 2019). Therefore, contemporary businesses need to understand how different industry sectors and contexts give rise to different relationships between tactical activities, such as service quality management, and overall performance outcomes through the application of modern technologies, such as cloud computing and complex data processing techniques  (Bolton et al. , 2004; Jabbar et al. , 2019; Stephany et al. , 2020). Few studies have been conducted on the differences in the technological adoption of real-time information sharing (RTIS) across service industry sub-sectors. This applies particularly to emerging markets (see Appendix -A). In Malaysia, RTIS has only recently been adopted in the services sub-sectors to share up to date information with customers in relation to daily routine operations and performances-i.
Cronin and Taylor (1992)  found considerable differences in the relationship between satisfaction and repurchase intentions among the four service industry sectors of dry cleaning, pest control, banking, and fast food. Bretschneider and Wittmer (1993)  explained the differences found in the adoption of microcomputer technology in different sectors. The influence of satisfaction and loyalty intentions on shareholder value varies between industry sectors; for example, between the retail trade, accommodation, and food services in North America  (Russo and Fouts, 1997). Desai et al. (1998)  investigated the implementation of database management in both the retail and travel industries. They found that these industries are more likely than financial services to gain a competitive advantage from investing in information technology.
Methodology A quantitative approach was adopted whereby the variables were measured through a survey and the data were collected in numerical form. Scholars have recommended the quantitative research method as one of the most suitable ways to address research problems-i. e. , to test hypotheses, test theories, and examine the correlation between different variables and measures-linked to the perceptions of a large group of individuals in regard to a phenomenon. Creswell and Clark (2017)  and  Hair et al. (2016)  believed that quantitative research also enables researchers to gather data from a large sample to address specific questions. Research designs and related aspects need to be selected carefully as part of an overall methodology.
9%) to the overall service category, which plays a significant role and to the overall service industry (33. 1%) (Ministry of Finance Malaysia, 2017). The data were collected from mid-level managers of SMEs from three provinces of Malaysia-i. e. , Selangor, Perak, and Kedah-who were directly involved in customer operations. They had been identified as the best suited respondents as they had relevant knowledge both in theory and in practice. As they are involved in customer care operations, and work with the customer service personnel of their organizations, mid-level managers are normally in close contact with their firms' customers. Additionally, they also play a part in achieving the strategic goals of their organizations. A survey questionnaire was sent out to potential respondents. A total of 313 responses were received.
Acknowledgements. The authors have received support from the  Center for ADvanced MOdeling Science (CADMOS). The third author also acknowledges funding from the  European Union 's  Horizon 2020 research and innovation programme  under grant agreement  N. 800898 , project  ExaQUte -EXAscale Quantification of Uncertainties for Technology and Science Simulation , as well as from the  Swiss National Science Foundation  under the Project n. 172678 "Uncertainty Quantifi-cation techniques for PDE constrained optimization and random evolution equations". The authors thank the anonymous referees for their valuable comments.
, 2019b] , but reworked, in order to best meet the requirements of our use case on public housing. With this metadata system, we also propose a framework for data lakes that we consider to be versatile and well suited for Data Intelligence. By using this framework, users can integrate all types of data and run various kinds of analyses without giving the data lake a predefined shape. Moreover, we introduce HOUDAL (public HOUsing DAta Lake), a data lake implementation based on our framework and adapted to our use case. This implementation allows users to create and manage metadata in the data lake and to interact with data stored in the lake. This paper is organized as follows. Section 2 presents the state of the art on data lakes. Section 3 explains our use case about public housing. Section 4 presents our data lake framework.
Metadata system The metadata system is the keystone of a data lake, and guarantees its proper functioning. In the absence of an effective metadata system, the data lake becomes unusable and is then referred to as a data swamp. Several proposals have been made for the metadata system of a data lake. As discussed in Section 2, the metadata system greatly influences the framework of the data lake. Thus, the choice of the metadata system is crucial, and the use case we are dealing with will also influence this decision. As mentioned in Section 3, our use case mostly concerns structured data, which are then reworked through various data wrangling operations. However, reworked data often end up being stored in not very common formats, such as. RData or. pkl files. These unconventional formats are considered as unstructured data by most tools, because the files can only be exploited by the right tools (in this case, R for.
, 2019b]. This study showed that apart from our metadata system, namely MEDAL, no other system offers the 6 key functionalities, the most complete systems implementing only 5 of the functionalities. Moreover, MEDAL is a metadata system based on data semantics, which best suits our constraints. Thus, we consider that MEDAL is the most relevant metadata system to use for our data lake. MEDAL is based on the concept of object, which designates a set of homogeneous information, as well as a metadata typology in three categories: intraobject, inter-object and global metadata. Intra-object metadata refer to metadata associated with a given object, while inter-object metadata are about relationships between objects, and global metadata add more context to all the data lake's metadata  [Sawadogo et al.
, 2019b]. For our data lake, however, we have reworked MEDAL. The main change lies in the introduction of three main concepts, that generalizes some of MEDAL's. The first concept is data entity, that generalizes versions and representations of an object in the sense of MEDAL. A data entity can represent a spreadsheet file, an image, a database table, and so on. The second concept is grouping, which is a set of groups, and data entities can be gathered into groups. For example, we can reproduce the object notion of MEDAL, by creating a grouping "semantic object", each group of this grouping being an object in the sense of MEDAL. Note that it is possible to have several groupings, thus enabling, among other things, to gather data entities according to their structure or zone if such groupings are needed.
Thanks to this, it becomes possible to reproduce classic frameworks as presented in Section 2. Lastly, the third concept is process, which generalizes the MEDAL's notions of transformation, update and parenthood relationship. It refers to any transformation applied to a set of data entities and that generates a new set of data entities. With processes, data lineage can be tracked, allowing the data processing chain to be monitored. A process can represent a script or a manual operation made by an user. Let's illustrate with an example. A CSV file is stored in the data lake, thus creating a data entity in the metadata system. This file is then processed by a data cleansing script: a second file with cleaned data is generated, and also stored in the data lake. Thus, a second data entity is also created, as well as a process representing the script.
Data stored in the lake are not only meant to be stored, but also analyzed. All types of data analysis can be considered, but since our final goal is to combine BI and BA within Data Intelligence, we focus on the presence of BI and BA analysis. Among these analyses, we are particularly interested in the creation of "advanced indicators" through BA methods (for example with machine learning), which then enrich BI analyses, whether at the data warehouse level or directly in dashboards. To illustrate the notion of advanced indicator, our use case is a perfect example: once predictions are made, they are recorded in the BI dashboard, and cross-referenced with data in the star model. This gives us an example of BA analyses that enrich BI analyses. It is important to note that since the data lake can contain reworked data, it can also contain a data warehouse and/or dashboards.
Our proposal of a data lake's framework combines data ingestion, storage, analysis, and the metadata system. Firstly, data are extracted from sources, and go through the ingestion layer. There, metadata are created and stored in the metadata system, while data are saved in the storage area. Then, users can browse metadata to find useful data, and run either BI or BA analyses on them thanks to the data analysis layer. In the end, if an analysis generates new data, it is possible to store it back into the data lake: in this case, data go through the ingestion layer, in order to create metadata, and so on. This is referred to as "back-feeding" the data lake. The schema-on-read property of a data lake implies that the data schema is specified at the query. That is why data lakes operate in Extract -Load -Transform (ELT) mode, which is opposed to the Extract -Transform -Load (ETL) mode typical of data warehouses, where data are transformed before being loaded.
Furthermore, with the (Re)Load step, we insist on the fact that the data lake must be the central piece of the whole data analysis chain. Of course, data are first extracted from sources outside the lake; however, when the user reworks data from the lake, whether through BI or BA analyses, the results may have to be stored in the lake. Thus, for these reworked data, the framework "restarts" at the Describe step. Figure  1  summarizes this approach in a schematic way. To illustrate the framework, we refer to the use case. First, data sent by the landlord are ingested into the data lake. Data scientists then process data, and backfeed the data lake with these reworked data, as well as the transformation scripts. In the end, the two predictions are made, and then exploited in a BI dashboard. With all this, we have a perfect example of the creation and exploitation of advanced indicators, as presented before.
Placing the data lake as the central piece of the knowledge extraction from data chain gives us several advantages. Firstly, it is one of the fundamental properties of the lake, i. e. the ability to store of all types of data in their native format, that benefits us. Indeed, we store in the lake both raw data sent by the landlord, reworked data, predictive models, dashboards, but also project tracking documents. This centralized storage of all data also means data are not scattered and environments are not multiplied. This was a major issue for our use case without the data lake, and it is quite a common problem, whether in BI or BA projects: the data as well as the processes can be stored locally on a computer, or on the company server, or on a cloud, or to be retrieved directly on a client's virtual computer, and so on.
An additional advantage is that metadata must be entered directly when data are inserted into the data lake. Indeed, whether for BI or BA projects, there are several ways to generate post-project documentation. For example, in BI, DataGalaxy 1  provides an ergonomic data catalog to manage data governance. On the other hand, having to maintain this documentation after the project has been set up is an expensive, error-prone task, in addition to being tedious, and therefore rarely done properly. With the data lake, data governance is done upstream : we can define this as "on-write data governance", and there is little need to maintain this information afterwards.
Selection of technologies Although the concept of a data lake is still relatively recent, and there is no consensus yet on the architecture that a data lake should have, there are already several tools and technologies available to implement this type of solution. We therefore conducted a study to identify which tools could be used to meet the requirements of our use case and remain the most faithful to our vision of the data lake and its framework. We have identified four possible ways to implement the data lake. All-in-one tool. The first is to use all-in-one tools, such as Apache Kylo 2. This is defined as "a selfservice data lake management software platform for data acquisition and preparation with integrated metadata management". Both data storage and analysis are handled with Kylo, as well as the metadata system.
However, this option was abandoned for three main reasons. First of all, the metadata system proposed by Kylo is not exhaustive enough to let us implement our goldMEDAL-based metadata system. Also, Kylo is mainly oriented to manage structured data. Although we mainly have source data in. csv format in our use case, this becomes problematic when dealing with files of various formats. The last reason concerns data analysis: the data preparation part of the tool is certainly interesting, especially in our case, but as far as more advanced analyses (BI and BA) are concerned, we would have had to manage them outside 2 https://kylo. io/ Kylo. This would have had several unattractive consequences, such as managing metadata outside Kylo. Proprietary clouds. The second option considered is to use proprietary clouds, such as Microsoft's Azure 3 or Amazon's AWS 4.