# Storage and Processing of Big Data for Geomagnetic Support of Directional Drilling
## 4 Results  
The current MAGNUS system, created in 2016 and based on a classic DB, stores data in a form aggregated by hours. This approach reduces the query time but complicates data retrieval by value. In this regard, the assessment of data availability intervals could only be performed by a complete enumeration of the values for each component. If, in the case of minute measurements, such an approach was justified, then, when switching to second and especially to satellite data, the assessment of availability became a cumbersome and computationally expensive process. The new MAGNUS BD system, already in its pilot version, achieves high performance by partitioning data by criterion (time) and dividing between physical segments. As a result, the assessment of data gaps is implemented separately, at each observatory. Thus, in the new version of the system, data on the time interval for which there is information, as well as the volume of available data, for any observatory, can be obtained by executing a simple query in SQL. Table 2, as an example, presents the result of executing such a query for 11 observatories by MAGNUS BD. At the same time, we also have the opportunity to quickly obtain information on the availability of data for one observatory for the desired period of time. Figures 5 and 6 show histograms with information on the volume of data for the Mikhnevo (MHV) and Klimovskaya (KLI) observatories per day for the time period from 1 May 2023, to 30 June 2023. The gap in the data on June 2 and 6 is due to technical problems with the scalar magnetometer after power failures. The synchronous absence of data from the scalar (F) and vector (X, Y, Z) magnetometers is due to power failures that affected all devices at once. Figure 4: Structural diagram of the MAGNUS BD system. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  
Figure 5: Volume of available data (in percent) from the Mikhnevo observatory in the period from 19 May 2023, to 30 June 2023. The MAGNUS BD system showed strong results in comparison with the original MAGNUS system. As a result, when executing data requests for the Klimovskaya observatory for different periods of time, the new system showed an increase in speed up to six times (Table 3). At the same time, as it should be in the case of a BD system, the larger the request, the higher the increase in the speed indicators of its execution. This was demonstrated when executing a data request for the entire observation period for all 22 observatories, shown in Figure 7.  
The basis of effective geomagnetic navigation in directional drilling is the processing of geomagnetic data in real time to obtain the direction to magnetic north with an accuracy of 0.1\({}^{\circ}\) for declination \(D\) and 0.05\({}^{\circ}\) for inclination \(I\), as well as, additionally, the modulus of  
Figure 6: Volume of available data (in percent) from the Klimovskaya observatory in the period from 1 May 2023, to 30 June 2023. Figure 7: Speed of data query execution for the entire observation period for 22 observatories, sec. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Data Request Period**} & \multicolumn{2}{c}{**Processing Speed, s**} \\ \cline{2-3}  & **MAGNUS BD** & **MAGNUS** \\ \hline Day & 1.3 & 0.1 \\ Month & 1.6 & 1.6 \\ Year & 5.1 & 22.6 \\ All available data interval & 41.1 & 246.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Speed of query execution for the Klimovskaya observatory. the full vector F with an accuracy of up to 50 nT according to the ISCWSA error model [22]. Space weather events such as magnetic storms and substorms near the production site lead to strong variations in the angular components of the vector \(B\) (Figure 8a). Taking into account such rapid variations, an error occurs depending on the processing speed of the incoming data. Figure 8a shows the declination and inclination record during the magnetic storm of 1-3 December, 2023, obtained at the Klimovskaya magnetic observatory (IAGA code KLI).