dataesr/scientific-paragraphs-categorization huggingface dataset:
Execution environments 325 The Hardware architecture for each solution is described below, all running on UNIX Virtual machines. THREDDS implementation is using two TDS instances. One accesses the single file benchmark datasets from local disk. The second accesses the single-file benchmark datasets from an on-prem object store appliance. The two TDS instances provide us with an interesting comparison for the benchmark testing. The TDS service manages data stored on a local file system is a dual Xeon Gold 6138 machine with 512GB memory and a 330 120TB RAID running CentOS. The TDS that serves data stored on an object store, is a VM on a cluster co-located with the object store, it has 4 CPUs with 12 GB of RAM and 80 GB disk and is also running CentOS. The CNES HPC cluster was used to deploy Pangeo on a Conda environment.
PBS Pro drives this cluster, and GPFS operates the distributed file system. The main libraries used with our Pangeo environment are Dask, Dask-jobqueue, Zarr, and Pyinterp. Pyinterp was used to perform the interpolations. For each test, PBS was used to limit the resources allocated on the cluster. 335 We repeated the tests several times to obtain the most reliable metrics. Nodes in the CNES HPC have the following characteristics: • CPU: 24 cores / 2. 2 GHz (Intel Broadwell) • Memory: 128 Go / 2400 MHz • Disk storage : 2 x SAS 15Klaps/mn / 300 Go / RAID0 / 500 Go 340 • Network : Gigabit Ethernet • Infiniband : FDR 56Gb The Parquet Cube solution is deployed on the CLS big data infrastructure (Cloudera version 5. 13. 1). 345.
Ingestion of NetCDF data 355 The first THREDDS deployment is using the current NetCDF format, so no conversion is required. The second one is using the S3 storage of a unique object per dataset. The conversion uses the NCO `ncrcat` command to generate a single object store. Concerning Pangeo-Zarr solution, a python script was developed using the xarray module to convert NetCDF into Zarr format. XARRAY module was not used to convert the big volume of the datasets and a specific script was developed using NetCDF4 Python module for input read and using Dask/Zarr for output write. 360 The ingestion service for the Spark-Parquet conversion is implemented as a Java/Scala job encapsulated in a docker (see git repository for a complete description). For big datasets, we can see the impact of the compression method and of the replication of data.
There is no replication for 365 the Pangeo-Zarr or THREDDS implementations. It is to consider that HDFS storage is replicating 3 times the data chunks to monitor the disaster recovery data integrity (so the effective disk storage is 3 times the presented figures). For the Lake Water Quality, the ingestion size in the Parquet format could be optimize by the generation of a second dataset with variables first_observation_day, last_observation_day, number_of_observations which are metadata and not observations. This can be discussed according to the usage of the lake water quality physical measurements. 370  The Zarr conversion on the HPC was not measured precisely, but using all the expected nodes in the CNES HPC, the ingestion of all the datasets took approximately 1 day. In comparison, the Parquet ingestion for all the datasets was 7.
Incidence of geographical and time coverage The sea_surface_height_above_geoid variable of the global-analysis-forecast-phy-001-024 dataset was extracted (for Pangeo-Zarr and Spark-Parquet using 10 cores of 4 Gb). 395 We can see that the execution time is quite linear between 1 month and 6 months for THREDDS-NC and THREDDS-S3 with a coefficient equal to 1 for a given area, but due to the striding strategy, it is not possible to conclude on a linear factor for the 400 geographical dimension.
Incidence of number of cores One of the key factors in the execution time is the number of cores to process data. The diagram below illustrates the results for the extraction of sea_water_potential_temperature, eastward_sea_water_velocity,northward_sea_water_velocity, 440 sea_surface_height_above_geoid variables for the global-analysis-forecast-phy-001-024 dataset over a 3 months time period.
Incidence of the number of time steps The test is to assess if the number of time steps in a dataset is a key factor in terms of performances and to try to find out what 450 the limits are in the different implementations. The request is the same: 4 variables, the sea_water_potential_temperature, eastward_sea_water_velocity, northward_sea_water_velocity, sea_surface_height_above_geoid, extracted first from the global-analysis-forecast-phy-001-024 daily dataset and then from the global-analysis-forecast-phy-001-024-hourly-t-u-v-ssh the hourly dataset having 24 more time steps than the daily. The size of the first one is approximately twice as the second one since it includes more variables. The THREDDS-S3 implementation provided a single value for the hourly dataset and is not 455 considered in the analysis.
Incidence of not significant data The idea was to compare the incidence of the data type: significant continuous data over wide areas (seas) versus lot of small data spots (lakes). The extraction for the Africa region is performed with 10 cores and with 50 cores for both Africa and Europe. The execution time of the extraction for the non-continuous data is greater than the global extraction for the oceanographic data with 50 cores (See Figure  12 ). In the Parquet case, with 100 cores, the result of the European and African coverage extraction is 5 times greater than the oceanographic data worldwide extraction performed with 50 cores. Even if the tested data is not a big amount of volume, it is necessary to read a lot of chunks to extract it. These formats are not optimized for gridded data with a poor 475 geographical coverage distribution, or the chunk size should be optimized for this kind of dataset.
Parallel processing results The Jupyter Notebook in the CNES HPC is used to run the process of the sea level yearly mean, and its conversion in a Spark/Scala job is run in Zeppelin notebook on the CLS big data architecture. 530 In future versions, UNIDATA intends to make the partitioning strategy customizable by dataset (in order to avoid having too many small files when we can group data by month for example). While the initial S3 access supports a number of data access 540 patterns, the work of this study has shown that there are issues with handling very large datasets and requests that result in very large response sizes. To better support large collections of data and reduce the need for rewriting existing datasets, the THREDDS team is working to implement support for virtual aggregation of datasets stored in S3 object stores.
The current study has suggested a number of possible items for future work: 545 • Requesting NetCDF-4 files with NCSS is RAM intensive and requires the resulting NetCDF-4 file to be written on the server before the server can start sending the HTTP response. The TDS CdmRemote Service can return a ncstream response which can start as the data is being retrieved. This would, to some extent, give a better indication of data access times by separating it from the time required to write the NetCDF-4 file. • Further investigation is needed on caching of S3 reads in the TDS to see if changes might improve performance and 550 memory usage. • Once virtual aggregations of S3 datasets are supported, look for possible performance improvements. • Longer term, the THREDDS team will look at how to parallelize reads from S3 object stores.
Their storage should be in the same chunk or closed chunks to avoid additional read access. Then, a tuning could be performed to adjust the chunk size to the dataset hierarchy and size. in this study, the daily partitioning was suitable for the tested data but a specific test could have been performed to estimate the impact of the partition size. A parameter could be customized to adjust the dataset structure with the storage partition size. Additional tests will lead to recommendations according to datasets characteristics, but a classification of the dataset types should be studied first to help users of the Parquet Cube solution to tune it. 580 Reducing the ingestion time by scaling the ingestion process is an easy step to be performed. This scalability is implemented in the operational environment at CLS premises to ingest files in parallel and speed up the ingestion of datasets with a long time coverage.
Conclusion 585 The Parquet format is a convenient implementation to open the data to a large community of users. Through a scalable process in a big data infrastructure, the ingestion of NetCDF data facilitates the processing of gridded data to extract easily the expected piece of information and make high added value processing chains or artificial intelligence modelling. The presented results demonstrate that the Parquet format is compliant with the typical users'use cases in processing big volume of gridded data: extraction of a subset of data to study or modelize, enrichment of positions with environmental 590 variables, parallel processing to tune models is becoming accessible to scientists and compliant with operational processing requirements. Alternatives could be runs of the benchmarks using Parquet/Dask/Python, TileDB or Cloud Optimize Geotiff formats for instance.
Data availability Gridded Data is around 2 Tb and is available on a NAS storage at CLS premises. The access can be provided on request to download data through HTTP. Please send a mail to bigdata4science@groupcls. com that will provide a slot time, an URL and password for downloads. You can also perform the tests with the rolling year on line on the Copernicus services, but the availability of these datasets is not guarantee in the long term.
https://doi. org/10. 5194/gmd-2021-138 Preprint. Discussion started: 2 July 2021 c Author(s) 2021. CC BY 4. 0 License. • THREDDS-S3 for a new possibility using THREDDS services managing data stored in S3 storage (Amazon Simple Storage Service), • Pangeo-Zarr for the Pangeo/Python/Zarr/Dask implementation, • Spark-Parquet for the Cloudera/Hadoop/Spark/Parquet solution 2 Methods 70 To compare the existing THREDDS-NC, THREDDS-S3, Pangeo-Zarr and Spark-Parquet solutions, a selection of various datasets and use cases is realized.
o of variables • Areas, defined as rectangles (top left, right down corners in lon, lat): 125 o North Sea: small size to check the incidence of filled value due to shore areas: [lon(-10,10),lat(50,60)]=100°2 o Atlantic Ocean: small size (4 times the surface defined for the North Sea): [lon(-30,10),lat(50,60)]= 400°2 o Pacific Ocean: medium size with check of 180° management (that can required 2 extracts to merge): [lon(140,200), lat(0,35)]=2100°2 130 Global coverage: [lon(-180,180),lat(-90,90)]=32400°2 • Time depth: from 1 month to 12 months For the land dataset, the criteria of subsetting are: • 1 variable: Turbidity_blended_mean • Time coverage: 135 o 1 day: 2020-02-29 o 1 month: 2019-09-01 to 2019-09-30, o 3 months: 2019-09-01 to 2019-11-30, https://doi. org/10. 5194/gmd-2021-138 Preprint.
A second AIS dataset is set up to estimate the time period incidence on a worldwide distribution:• World1 (all positions of all vessels for 1 day) : 2 578 627 positions • World1D (one position by vessel for 1 day) : 176 735 positions 175 • World5D (one position by vessel by day for 5 days): 876 941 positions • World10D: 1 736 849 positions • World20D: 3 588 894 positions • World50D: 8 897 349 positions https://doi. org/10. 5194/gmd-2021-138 Preprint. Discussion started: 2 July 2021 c Author(s) 2021. CC BY 4. 0 License. 180 Then, the incidence on the number of extracted variables and of the number of cores to enrich the location are also studied. 2. 3 Parallel processing scenario 3 details Run: Calculate the annual average elevation of the oceans on the globe over 25 years -Fill value to ignore -Output : curve plot (png) 185 -Conf : 4 go per executor, 5 cores per executor (optimum), up to 50 cores Metrics:.
For the tested datasets, the Parquet storage is designed to get a daily parquet file per folder to obtain a timely indexation, with 300 a storage size compliant with good I/O performances. The compression format is Snappy since it is splittable (see Cloudera documentation), performant in read and write access for any dataset. Additional metadata columns are added in the footer of the Parquet format to quickly filter data between physical values ranges for instance (see Faster Performance for Selective Queries).
The along track dataset is less resource demanding and is available on Zenodo with the Digital Object Identifier 10. 5281/zenodo. 4894095, https://zenodo. org/deposit/4894095 605 The requests with all the parameters for subsettingand along track enrichment are available on Zenodo with the Digital Object identifier 10. 5281/zenodo. 4905665, https://zenodo. org/deposit/4905665 The source code ingestion is under L-GPL license and identified with the 10. 5281/zenodo. 4905679 DOI. It is available as an archive in Zenodo, https://zenodo. org/deposit/4905679 or in the Github repository https://github. com/clstoulouse/parquetCubeIngestion 610 https://doi. org/10. 5194/gmd-2021-138 Preprint. Discussion started: 2 July 2021 c Author(s) 2021. CC BY 4. 0 License.
of requests is performed to evaluate the incidence of the geographical coverage and of the number of positions. A first AIS dataset includes tracks on the area of North Sea and Rotterdam: (the filename contains the down sampling time • Zone: North Sea, Rotterdam, globe to estimate the geo coverage factor 160 • Time depth: from 1 day to 1 year • Number of cores • Number of variables to extract 165 A first set period) • NorthSea. csv : 3 207 817 • Rotterdam1H. csv : de 05/04/2019;00:00:00 à 05/04/2019;23:59:59 : 152 484 positions • Rotterdam15m. csv : de 06/04/2019;00:00:01 à 06/04/2019;23:59:59 : 362 557 positions • Rotterdam12H. csv : de 02/04/2019;00:00:00 à 02/04/2019;23:59:58 : 76 764 positions 170 • Rotterdam1m. csv : de 03/04/2019;00:00:00 à 03/04/2019;23:59:59 : 1 025 641 positions RotterdamAll.
5G and beyond private network use cases: from requirements to proof of concept This section investigates the three use cases selected for implementation at two trial sites in Europe and Taiwan in the 5G CONNI context of private 5G networks : Process diagnostics by Computer Numerical Control (CNC) and sensing data collection (UC-1), Process diagnostics using virtual or augmented reality (UC-2) and Robot platform with edge intelligence and control (UC-3). For each use case, some implementation aspects and challenges are described for the demonstration implementations and studies. The analysis of non-functional and functional requirements is summarized in Tables  1  and 2 , respectively. The two production facilities are each equipped with their own RAN and MEC. The user data plane is terminated locally with appropriate network functions being present in the local edge cloud.
e. , CNC lathe, a robot and a conveyor) will be studied. This equipment will be connected to an industrial PC running the online monitor system. The collected data will be uploaded to the analysis system in an edge data center using the 5G CPE and gNB. A detailed analysis of the requirements will be performed. Lastly, the setup will be extended to 11 CNC cells distributed throughout the shop floor, and a network emulator will introduce different impairments (bandwidth limitations, latency and packet loss) to assess the impact on the applications.
UC-2: process diagnostics using augmented/virtual reality Conventional machining process planning relies on time-consuming Trial-and-Error (T&E) activities to determine force-related parameters (e. g. , feed rate, spindle speed, depth and width of cut). In order to reduce the T&E cost, additional diagnosis models are used to detect vibration, collision, acoustic emission, temperature, or energy consumption-input that goes beyond that of traditional computer-aided manufacturing software to generate tool paths along a workpiece geometry. Virtual or Augmented Reality (VR/AR) can help process engineers reveal full productivity of machines by superimposing graphical objects (e. g. , 3D models, charts, vector fields and text messages) on top of a video streaming on head-mounted displays. Here, high data rate along with low latency properties of 5G can be utilized, while high fidelity 3D scenes with millions of polygons are rendered remotely and transferred wirelessly lightweight VR/AR device in terms of video streams to facilitate high mobility on the shop floor.
The data collection is achieved by a secured open platform communications unified architecture client server pair. The control agent is a software module located in the edge data center. The agent, implemented by the Unity3D software, will act as a bridge between user and 3D scene. In this setup, the user device will be a tablet PC or head-mounted display with a camera module. Using the camera module, the relative position and orientation between the end user and the equipment can be identified and transformed into corresponding viewport and navigation commands to the 3D model. Sensing data can be associated with Fig. 3  Implementation of AR/VR for process diagnostics corresponding 3D components to show process status information. For example, vibration levels can be shown on the 3D model of the spindle as color-coded contour.
A 5G link can interconnect the robot with the backend to fulfill the speed and reliability requirements imposed by the exchange of motion control messages (e. g. , target values of joint positions or speeds) and feedback (e. g. , joint angles and torques). In our planned setup (cf. Fig. 4 ), we use a 7-degree-of-freedom Franka Emika's collaborative robotic arm called Panda to demonstrate the performance of the underlying 5G network between the edge controller and the robot. The Panda robot is normally connected to a workstation PC via Ethernet and requires communication to the controller at 1 kHz. Franka uses a real-time robot control code using UDP messages via the Franka Emika interface. This interface includes a motion generator and a controller, through which a programmer can send commands to the robot.
g. , in terms of network slices. In such a case, a unified 5G management or even 5G core would be preferred, so that the production assets work the same way at the new location as they did at the old one, while minimizing manual reconfiguration of the wireless system including UEs. Favorable solutions in this scenario would require the 5G Core to be located in the enterprise domain for central management or in a central cloud with unified management across multiple companies. Alternatively, multiple interconnected private 5G Cores that share a common database of user profile could be used.
Another ongoing Rel-17 study item is Study on Enhanced Support of NPN  [28]. This study focuses on the credential and subscription handling in NPN. The onboarding and provisioning procedures will be defined, and the entities handling the subscription will be specified. Furthermore, the study aims to investigate enhancements to the service requirements for audio-visual content. These topics are very relevant to 5G CONNI and contributions to 3GPP are planned. Several other ongoing study and work items regarding unlicensed access, dynamic spectrum sharing, edge applications and flexible local area data networks are related to NPN, highlighting the relevance of these topics that is reflected in the standardization.
There is often more than one stakeholder involved in building and operating the private 5G network. Possible locations of elements include enterprise data center, enterprise headquarters data center, enterprise site, service/cloud provider central cloud, MNO central cloud, MNO edge cloud, and MNO site. The locations of elements specify the distribution of 5G elements, i. e. , the 5G deployment model or architecture, which has some implications with respect to the operator model, operation and management. Tasks during the lifecycle points out that the private 5G network must clearly know the detail of its lifecycle subtask. This dimension can be used to know which tasks are involved in building the private 5G network.
Core Similar to the RAN, the Core Network (CN) is not necessarily owned by the stakeholder that defines which subscribers and devices are allow to use the 5G communication service. In fact, that organization can use the mobile network offered by a traditional MNO or an equipment/service provider as a service, so even subscriber management is delegated to a third party, technologies such as dedicated core network and network slicing enable this scenario. Nevertheless, such organization is still responsible for the administration and provisioning of the subscribers database, including their service profile, and for the distribution and configuration of physical SIM cards or eSIM to users. The Authentication Server Function (AUSF)/Unified Data Management (UDM) are then responsible for authenticating the device by exploiting the information contained in its SIM card, such as the international mobile subscriber identity, the subscription permanent identified and relative keys.
In data routing process, the telecom operator is responsible for the network routing of the edge computing platform away from the applications, sending data to the target applications, and processing the data completed by the applications to the target users. In addition to the typical scenarios mentioned above, non-network-related enterprises may also obtain the edge computing platform through buyouts or leases and then install the applications on the platform. In this case, the platform that the telecoms help build is inline with the requirements of enterprises, so it is a cooperative relationship. The enterprise is responsible for the cost of establishing the edge computing platform and the requirements to configure the edge computing platform. The telecom operator is in charge of the establishment of network connectivity, network transmission security, application onboarding functions, and transmission performance according to the requirements of enterprises.
Applications There are a plethora of different applications that can be offloaded onto a MEC platform. In the industrial domain, these applications range from simple data collection and database systems to control logic functions of controllers to more complex systems, such as manufacturing execution systems or even enterprise resource planning software. Depending on the type of the application, the MEC platform is either deeply integrated into the 5G system and located close to a machine or production line, or it provides computing capabilities for a large number of machines, sensors, etc. , that may even span multiple factories. Application ownership and governance is more flexible than that of the MEC platform, as applications can be provided by many suppliers, such as telecom operators, enterprises, or application providers, and each can manage their respective applications individually.
The edge node resides inside the enterprise firewall and keeps traffic and user data local to meet low latency, data security and edge computing requirements. The edge node can have a very small footprint to be deployed in each commercial building, factory, warehouse or enterprise, or it can be deployed at an aggregation point, to be used by multiple private networks. The hybrid model can be viewed as part of a long-term transition strategy: the enterprise can start outsourcing with a simple MVNO model (see Sect. 5. 2), in case an adequate enterprise IT management team is not put in place initially. Once a favorable status is reached, the enterprise can begin a transition to a fully private network (see Sect. 5. 1), in which the entire network is owned by the enterprise. If SIM cards are allowed to roam between the private and public RAN, the manner in which they are ordered and deployed must be agreed upon between the enterprise and the MNO.