# Storage and Processing of Big Data for Geomagnetic Support of Directional Drilling
## 3 Materials and Methods  
When switching to the new BD technology, it was necessary to migrate all data and duplicate the functionality of the original MAGNUS system. Today, the latter includes:  
* Original geomagnetic data from 22 ground-based observatories;
* Processed data and calculated indices;
* Geomagnetic data from low-orbit satellites;
* Derived parameters obtained after processing satellite and ground-based data;  
Figure 2: Map of geomagnetic observatories in Russia and neighboring countries. * The web system for visualizing and uploading accumulated data, available on the official website of the MAGNUS hardware-software system, [[http://mag.gcras.ru](http://mag.gcras.ru)]([http://mag.gcras.ru](http://mag.gcras.ru)) (accessed on 3 September 2024);
* The set of application programming interfaces (API) in different programming languages. At the start of the migration, the DB contained more than 400 GB of original and processed data. Due to limitations in the performance of the MAGNUS DB, some of the available information was not loaded, so about 10 TB of satellite and related geophysical data were accumulated in the file storage. Working with these data within the original architecture of the MAGNUS system was impossible. Prior to the migration, an analysis of the market for technological solutions for the BD DBMS was conducted. An important parameter was the use of open-source software. In this case, we could be sure that the system would function and develop regardless of external sanction changes. The main feature of systems built on Big Data technology is the implementation of parallel computing [36]. This allows the query to be distributed between several servers (segments) using the query optimizer (master) [37]. The concept of the system is shown in Figure 3.  
The most widely used open-source data management system in the IT industry at the moment is the Greenplum DBMS. An analysis of its functionality confirmed this system as the optimal choice for deploying a geomagnetic database. In addition, the decision to choose Greenplum was based on the availability of a pool of specialists working with the system, as well as a version of its Russian assembly, the Arenadata database, which has become widespread in the commercial sector, especially after the introduction of sanctions and the departure of many technology giants from the market. Among the major users of this system are VTB, the Federal Tax Service of Russia, Moscow Exchange, and many others. The Arenadata database is a distributed DBMS that uses the concept of massively parallel processing (MPP) and is based on the open source DBMS-Greenplum. Analytical massively parallel DBMSs are designed to store and process large volumes of data--from single units to hundreds of terabytes of data. The pilot version of the MAGNUS BD system was deployed on the basis of three physical servers, the technical characteristics of which are given in Table 1.  
Figure 3: Arenadata database operating principle. Arenadata DB software version 6.26.2.55 was installed using the Arenadata Cluster Manager (ADCM) interface. The ADCM provides the ability to create, configure, and update clusters via a graphical interface or using an API. All operating system services are configured automatically, under the supervision of a specialist who promptly resolves any technical difficulties that arise.