dataesr/scientific-paragraphs-categorization huggingface dataset:
In practice it means storing data in object storage and loading it on the fly at massive speeds in VMs, started on demand and disposed of when the analysis is done. We leverage the Google Cloud Platform to do just that. We store the historical dataset in Google Cloud Storage, we start servers on the fly when a user needs them and shut them down when they are done. To mount the data in memory and deliver analytics in the hands of business users, we use the ActiveViam platform. In the following section we explain in practice how we did it for a 50TB dataset (500 historical days, 100GiB per day). In short, we managed to start a 1600-cores, 64TB RAM cluster from scratch, load the 50TB in-memory and be ready to serve real-time queries, all in less than 20 minutes.
Using Scripts to Set up a Whole Infrastructure: In-memory Computing without the Hassle It can be difficult to take full advantage of in-memory computing. Managing tens of server nodes dynamically, optimizing data loading from cloud storage, efficient memory allocation of billions of records in terabytes of RAM or parallel computing on hundreds of core and NUMA architectures are difficult tasks. Fortunately, in a public cloud model the cloud and the software providers take care of all issues related to configuration and optimization, and client organization only need to start the servers to access almost immediately a fully-functioning analytics platform. In the case of Google Cloud and ActiveViam, you can start and stop 100 servers if you want with a short script calling cloud APIs or by describing resource templates in tools such as the Google Deployment Manager.
All servers start with a given image, in our case a Linux operating system configured with memory intensive settings. Example of an allocation loop from the script used in the benchmark: for ((i=0; i < 100; i++)) do NODE_NAME="$INST_NAME-$NODE_TYPE-$i" changeLine "TC_NODE_NUM" "$i" "$STARTUP_SCRIPT_FILE" echo "=== Creating new $NODE_TYPE node [$NODE_NAME]. " # Create node gcloud compute instances create $NODE_NAME \ --custom-cpu 32 --custom-memory 624 \ --custom-extensions \ --min-cpu-platform skylake \ --zone $ZONE \ --metadata-from-file startup- script=$STARTUP_SCRIPT_FILE \ --tags http-server,https-server \ --image-project ubuntu-os-cloud \ --image-family ubuntu-1604-lts $MACHINE_SPECS \ --async -quiet done Note that in this scenario, we take advantage of "custom" instances in the Google Cloud Platform.
Data Loading The next challenge is data loading. The dataset must The End of the Nightly Batch: How In-memory Computing and the Cloud Are Transforming Business Processes be transferred from cloud storage to the cluster within minutes to satisfy the on-demand constraint, which means we must move 50TB in minutes. This is where you can really leverage the world class infrastructure of a large public cloud provider. We have 500 days of historical data, so we load 5 days per node (500GiB). The objective is that each one of the 100 nodes downloads its data at maximum speed, and that they all do it at the same time. At the time of this test the maximum bandwidth of the network interface in a GCP instance was 16 Gbit/s. So, an instance had the theoretical capacity to transfer data at 2GiB per second.
In this case the data is actually downloaded from Google Cloud Storage and the throughput of a data transfer is far less than that, being HTTP based. ActiveViam developed a special connector that opens tens of HTTP connections to blob storage and performs the transfer in parallel, transparently reconstructing blocks. With this trick, a node running the ActiveViam connector can saturate the 16 Gbit/s cap and actually download 500GiB in about 5 minutes. Of course, when 100 instances download their 500GiB concurrently and at full speed, Cloud storage becomes the next bottleneck. This is where cloud infrastructures are really put to the test, and during our benchmark we observed impressive scalability from Google Cloud Storage. The 50TB dataset was entirely transferred from storage to the 100 nodes cluster in less than 10mn, as shown on Figure  1.
That's a monumental 90GiB/s effective throughput between storage and compute, reached with possibly the simplest and least expensive storage solution in the cloud platform. So, it takes a few minutes to start the GCP instances, 10 minutes to transfer the data, and another few minutes for the ActiveViam platform to get its inmemory data structures ready. All together the entire cluster is ready in about 20 minutes. It is fast enough to be started up at the beginning of the work day, or at any time at the request of business, without having to schedule a batch.
Benchmarking Queries for Interactive Analysis Finally, we looked at the performance of interactive queries on this on-demand cluster. We performed two different queries and check if they follow Gustafson's Law  (Gustafson, 1988) : queries on twice the dataset should take the same time if you also double the CPU power and RAM. Query 1 performs aggregations on all the records of the dataset, Query 2 performs aggregations on a filtered selection of records (about 10%). We scaled up the cluster step by step, each time doubling its size, from 1 VM to 100 VMs. Below you can see that the scale-up is good: the query times are close to constant. In the end, we only observe a 10% difference between a query running on a single VM and a query running on a 100-VMs cluster. Query times, as shown on Figure  3 , remain well under halfa-second, which is fast enough to provide end users with a comfortable, responsive experience for analysis.
With the cloud, you use exactly what you need now. If the business grows, you can switch to a larger resource almost seamlessly -as long as the software is designed to cope. We have shown that it is possible to load data from the Cloud storage and launch ActivePivot instances very quickly to answer to a business query that, in previous architectures, would have to wait for the next nightly batch. However, this design assumes that the business applications producing the data consumed by ActivePivot have actually pushed their data in real-time to the Cloud storage. In the next section we will discuss the CloudDBAppliance project, in which we aim to build a platform that will be able to contain both the operational database and the analytical one, both in-memory, allowing for the shortest delays between a change in the operational data and its availability for analytical queries.
CloudDBAppliance: Operational Database and Analytics, All in One The CloudDBAppliance project is funded by European Union's Horizon 2020 research and innovation programme. It aims at producing a Cloud Database Appliance for providing a Database as a Service able to match the predictable performance, robustness and trustworthiness of on-premise architectures. This project will deliver a cloud database appliance featuring both the next-generation hardware for in-memory computing and the necessary software for processing high update workloads on an operational database, performing fast analytics and applying data mining / machine learning techniques on a data stream or on a data lake, the robustness being protected by redundancy and failover mechanisms. We have tested the ActivePivot analytical database on the hardware built during this ongoing project.
The platform we tested is a Bull Sequana S800 with 8 processors Intel(R) Xeon(R) Platinum 8158 CPU @ 3. 00GHz, (which represents 96 cores), each constituting a NUMA node with 512 GB of RAM (4 TB in total). For our experiment, we have used the well-known TPC-H benchmark's dbgen tool  (TPC, 2019)  to generate data at two different scales: we used a factor of 100, then 500, to generate respectively 107 GB then 542 GB of data. Those data quantities are roughly representative of the two smaller use case sizes cited in Table  1 , for which the platform would have enough memory to co-host the operational database and the analytical database. The data (as generated in CSV format) was stored on SSD disk on the machine. We evaluated the total startup time, from launching the command line that starts the analytical database until the analytical database is able to answer queries on the full dataset.
We are currently implementing a connector between the LeanXcale database, the operational database that was selected for the CloudDBAppliance project (CloudDBAppliance, 2018), and the ActivePivot analytical database. We expect throughputs to be much higher, since the data will be pushed directly from the in-memory database, so:  The data will be read from RAM, not SSD disk;  the data will already be in a binary format, no need for parsing the CSV into actual data types;  the data will already be broken into individual records, removing the constraint of having to read one single CSV file sequentially. We can envision several scenarii for the best usage of the platform's capacities. In one scenario, the operational database is always on, but the rest of the machine resources are shared between different kinds of applications: the ActivePivot analytical database can be started on-demand to compute complex analytical queries, and be shut down after usage to leave resources to other applications such as machine learning Spark jobs performed on the datalake.
In that scenario, the main difference from using a datawarehouse built from nightly batches is that the data will be fresher, since the application can be loaded on-demand from the latest version available in the operational database. In another scenario, the ActivePivot is always on, receiving data updates continuously from the operational database. This way analytical queries can be performed at any time without any loading phase and reflect real-time changes in the operational data. In both scenarii, the whole machine is always up because the operational database has to be always available. The main advantage of being in the Cloud rather than using an on-premise machine resides in the ability to easily switch from one configuration to another when the needs evolve, instead of having to buy a new server.
CONCLUSION While it's not feasible to try out every use case, we believe we demonstrated through our tests that one of the most demanding business cases can be addressed fully without the need for a recurring nightly batch. If financial risk analytics can be performed without scheduled batches through a combination of cloud infrastructure and in-memory computing, we are convinced that there aren't many real-world business scenarios that cannot be addressed in the same way. Furthermore, while it serves its purpose for decades, nightly batch processing now appears not only a second-rate option at best, but simply wasteful compared to the alternative. It is wasteful in time, in capital and in power. Fortunately, the transition to a better model is now easier than ever for organizations. We are currently participating in a research project that aims at building a cloud platform on which an operational database is integrated with analytical capabilities, allowing for real-time data processing without hassle.
Introduction A motivation to do gopaw came from the introduction of the HDF5 [1] file format in the Geant4  [2]  analysis toolkit to store histograms and ntuples. See the related poster at the same CHEP Sofia conference about the introduction of HDF5 in Geant4, and we refer to  [3] ,  [4] ,  [5] ,  [6]  for the Geant4 analysis toolkit. It is fine to produce such files, but having interactive analysis tools that understand them would be highly welcome. We already improved our GUI oriented ioda  [7]  program so that it knows these files, but being aware that a lot of HEP people prefer command typing, we look also for a keyboard oriented application. In HEP history, CERN-PAW  [8]  has been very popular and the minimal typing to open a file (H/FILE) and plot an histogram (H/PLOT) or do an ntuple projection (N/PRO), is rather appealing to do a refactoring of this program by using our softinex  [11]  technology.
It can also reproduce the style of CERN-ROOT by using the root_def. kumac script. See figure  3. (For ROOT, see  [12] ). Final images are done by traversing a scene graph with a "renderer". For screens, we have a GL-ES renderer that permits to do graphics on MacOS, Linux, Windows, iOS and Android by using the OpenGL libraries available on all these platforms. For MacOS and iOS, the announcement by Apple, at their WWDC of June 2018, to deprecate their OpenGL library [13], in favour of their Metal proprietary rendering engine, compromises our GL-ES strategy, but we foresee to have, in a short future, a dedicated Metal renderer for the Apple devices. We have also offscreen renderers to produce Postscript, png or jpeg files. These are available in gopaw through the PICTURE/PRINT command.
Python to replace COMIS As for CERN-PAW, we can define functions to be plotted or to fill histograms, etc. In CERN-PAW it was done by using the COMIS FORTRAN interpreter. In gopaw, we use a Python [14] kernel to do that. For example, in pawex10. kumac, CERN-PAW draws a fractal with the command: fun2 10 mandel. f [1] -2. 4. 8 [1] -1. 2 1. 2 ' ' whilst in gopaw it is done with: fun2 10 mandel. py [1] -2. 4. 8 [1] -1. 2 1. 2 ' ' We show in figure  4  and figure  5 , the two versions of the mandel script. Note that we added the command: /SESSION/PYTHON <file>. py in order to execute other python scripts. In gopaw/1. x, the Python kernel which is used is the one of a Python/2. 7 release.
Being positioned on a directory, the famous command: h/plot <storage id> permits to read and plot and histogram. Ntuple commands, as N/PLOT, work on ntuples found in ROOT files, HDF5 files, but also on table TBL objects found in FITS files. For FITS, we have also the IMAGE_HDU command extension to show (astronomical) images stored in a file. See figure  1. Note that gopaw is not able to read HBOOK files since we have no C library to do that, and that we do not want to tie to any FORTRAN code and the old CERNLIB anymore.
Platforms Portability is a high priority for us. gopaw runs on MacOS, Linux, Windows-10 and we have also a docker container on docker hub  [15]. The core engine (including KUIP and Python) can be built also on Android and iOS and we have embedded a "kumac reader" in our ioda application. On Android and iOS, we have not yet finished the work to have an effective way to input commands from a virtual keyboard, and then have not yet a gopaw application on the Apple AppStore and GooglePlay stores. Having a gopaw application on an iPad makes definitely sense for us and it remains a nice target to work on.
Availability Public releases are deposited on GitHub at [16] and the web pages are at  [17]. Why doing gopaw? Beside the motivation to read Geant4 HDF5 files, it is also to try to answer a more fundamental question: what is the best user API to do (HEP) physics? At the arrival of ROOT in the 1990's, a user told us: with CERN-PAW I had the feeling to do physics, with ROOT I have now the feeling to type C++. There is some truth here. Especially for someone, as the author, involved in interactive and graphics software engineering for physics. If we look at the today software situation in HEP, it is particularly shocking to see what a new comer has to learn of technicalities before being operational in manipulating data to get a plot ready for publication. We even do not want to mention the pain to build an event display exploiting local graphics capabilities of devices that people have in hands now (laptops, tablets, smartphones).
(The vocabulary in Spark is "map", 'reduce' and, if used on tabular data, is similar to the "ntuple" and "projection" used in HEP). But then, why not doing an extra quantum leap by going toward a specific dedicated language? Some language "HEP data oriented" where the keywords would be de facto "run", "event", "cut", "histogram", "ntuple", "fit" and "plot" beside, or in place of, generic keywords as "for", "if", "class". And then, why not a voice API ? "Hi gopaw, apply a pt cut of 10 GeV on the last run and show me the plot". Science fiction? Quite not, if there is an adequate language at hand; the technology is around the corner to be able to do that.
Conclusions We presented the first release of gopaw to help exploiting HDF5 files produced by using Geant4 analysis tools. This CERN-PAW refactoring is done in C++. It keeps the C KUIP found in CERNLIB along with the command file descriptions of CERN-PAW, which ensure a similar behaviour as the original FORTRAN program. It comes with a new graphics layer based on a scene graph manager co working with two renderers; a GL-ES one for displaying on screen, and an offscreen one to produce Postscript, png or jpeg files. It uses a Python 2. 7 kernel to script functions, the "f2c' of MINUIT (extracted from ROOT) to do fitting, a new C++ implementation of VECTOR commands and the single instruction interpreter exlib::yacc::cexpr_eval to replace SIGMA. gopaw has also extensions to read histograms and ntuples in a ROOT file, and exploit tables and images found in FITS files used in astronomy.
This is a central difference to the domain of IT security, where the processing organisation, its business secrets and company assets are the paramount subjects of protection. Therefore, in the realm of personal data protection with its fundamental rights underpinning, the following questions present themselves whenever a personal data processing operation is intended: -Which data shall be collected and processed, and to which extent? -In which way shall the data be processed, using which means? -For which purposes shall the data be processed, and by whom? -Is a transfer to and/or storage at other parties/foreign countries foreseen? A concise knowledge about the points above is a necessary precondition enabling data subjects to exercise their rights granted by the GDPR. Such rights include the right to be informed, to access, rectify, or erase one's own personal data.
In 1997, when there was still a lot of enthusiasm about PICS, the idea came up to also use such a transparency scheme for privacy and data protection. In fact, people found out that a lot of http protocol chatter was stored in log files and used for profile building. People did not realise that such profiling was happening as the browser was totally opaque about it. The US Senate threatened the advertisement industry with legislative action. Industry feared that such legislation would damage their revenue model. It was better to improve the user experience than to continue to provide a picture perfect example on why legislation was needed. This created a discussion within W3C about technical remedies to the opaqueness of the HTTP protocol. Major vendors like Microsoft joined a W3C Working Group to make a tool that could replace legal provisions.
The project decided deliberately against taking into account legacy enterprise databases and went for the pure research by requiring all systems to be RDF in order to work. The main argument was that integrating those legacy systems with their legacy interfaces would allow a malicious actor to circumvent the enforcement engine of the PRIME system. Given those practical constraints, the PRIME project produced good scientific results, but its practical relevance remained marginal. It was not really usable for real world systems unless someone would create a brand new system from scratch. PRIME had improved the understanding of the difficulties with transport anonymity and also advanced considerably the state of the art concerning the sticky policy paradigm. Drawing conclusions from this initial experience with a privacy-enabled data management system, most of the partners of the PRIME project created the PrimeLife project 10  to now concentrate on the challenges for such a privacyenabled system in relation to real world systems.
When the PrimeLife proposal was created, the predominant data format was XML  [3] [10] and the predominant software architecture was Web Services 11. PrimeLife concentrated on those technologies and reduced the dependency on RDF. The attacking model was changed. An assumption was made that a company wanted to do the right thing. In fact, rights management systems are very hard to enforce once the entire computing resources are in the hand of the attacker. Consequently, PrimeLife had a security boundary between the client side and the server side, and a focus on data usage control on the service side. This allowed efforts to push boundaries on both sides. PrimeLife continued integrating ways to achieve results and checks without consuming personally identifiable information while further developing a system to create data value chains across company borders.
One special merit of PrimeLife was the invention of the term 'downstream data controller'. This terminology allowed researchers to better repartition and assess the respective responsibilities between data controllers and responsibilities further down in the data value chain. It allowed clearing the fog in discussions where people partly did not realise that they were talking about the same thing. Or, that they believed they were talking about the same person, while meaning another one. The term 'downstream data controller' greatly reduced the confusion. For PRIME, the sticky policy paradigm was rather easy to implement as RDF with its URIs  [7] [16] 12  on every triple had a natural way of addressing a specific data packet or data record. In fact, this is like a sentence in English language.
To make a policy sticky, it has to be attached to the data record it applies to. Within RDF data, every object has a URI. So it is sufficient to create a new triple with a policy that points to the data record it wants to apply to. The URI used is a world unique identifier. This means wherever the data travels, the relation to the policy remains intact. This allows data value chains across company borders to honour the policies expressed and even impose the respective rules to subsequent downstream data controllers. In PrimeLife, because web services were used, the stickiness of the policy made a rather complex system necessary. All was using standard data formats, namely XACML  [30]  and SAML[2] to create, manage, transport and execute policy statements along a given data value chain. While this was very pragmatic at the time, technology has moved on and different things have to be used today.
PrimeLife had a rather holistic view, taking into account the data life-cycle and providing new ways to manage access control in a way that was much closer to the needs for business, e. g. depending of the role of the person wanting to access certain information. PrimeLife had an entire work package concentrating on user interface issues. Data self-determination, if taken seriously, requires the data subject to understand what is happening. This is not only an issue for transparency and the provision of information from the server/service side. It is also an issue of cognitive limitations by humans confronted with the huge wealth of information transitioning through today's communication systems. PrimeLife experimented with icons and logos and user interfaces and did usability testing in a lab environment  [19].
One of the lasting outcomes was a logo: Users were tasked to find out about a privacy tool PrimeLife had developed. It was collecting data about who is collecting what to identify trackers and show them to the user  13. Users had the task to find out who collects what about them. The challenge was, whether the users would find the tool and click on the right logo. All kinds of logos were tested. But only with an icon with footsteps on it, users found the tool easily. PrimeLife was also very successful in the scientific field testing out role based access control and a new type of group based social networking. It created a new policy language that was able to express all the metadata in standard languages and extended XACML to carry more metadata. After PrimeLife, there have been further research and development efforts focused on providing better transparency for users of digital services.