# Seamless Weather Data Integration in Trajectory-Based Operations Utilizing Geospatial Information
## 2 Materials and Methods
### Seamless Weather Integration for TBO
#### 2.2.2 Technological Framework  
Distributed processing system construction using Apache Spark. The 4 DT-Wx prototype system utilizes the Apache Spark framework (version 3.5.1) [38] to efficiently process large-scale weather data and aircraft trajectory information. Apache Spark is a Scala-based open-source computing platform that enables the parallel processing of large datasets, capable of analyzing static or streaming data with high performance and speed through in-memory operations [39]. In this study, Apache Spark's local mode was configured to maximize the efficient use of resources in a single high-performance workstation (Figure 3). In the SparkSession configuration, about 80% of the system's total memory of 256 GB, i.e., 200 GB, was allocated as the driver memory. This configuration was chosen based on our system's workload characteristics for processing large, complex weather datasets. The 80-20 split between driver and executor memory was determined through experimental optimization, balancing in-memory data processing needs with executor operations and system overhead. Additionally, the default parallel processing level and shuffle partition count were set to 256, four times the number of logical processors. This setup effectively distributes and parallelizes data processing tasks on a system with 64 logical processors, ensuringthe efficient handling of data-intensive tasks for real-time weather data integration in TBO environments. For performance optimization, we used the Kryo serializer and enabled the compression option. The Kryo serializer is more efficient than Java's native serialization, improving performance during data serialization and deserialization [40]. Enabling the compression option improves the overall system performance by reducing network transfers and disk 1/O. This Spark configuration effectively leverages the fault-tolerant dataset concept [41]. The Resilient Distributed Dataset (RDD) is the foundation of the Spark framework, enabling in-memory computation on large-scale clusters [42; 43; 44; 45; 46]. The 4 DT-Wx system utilizes RDD to efficiently process weather data and aircraft trajectories. _Python-based data processing through PySpark._  
Utilizing Spark's functions in a Python environment using PySpark, we performed efficient weather data processing, trajectory information analysis, and complex numerical calculations. Apache Spark supports various programming languages, including Java, Scala, Python, and R, enabling the development of real-time stream processing applications, allowing users to manage real-time data in a scalable, high-throughput, and fault-tolerant manner [47; 48; 46]. In this study, we used the following libraries for main data processing and analysis: Numpy for efficiently processing large multidimensional arrays and matrices, Pandas for high-performance data structures and tools for data manipulation and analysis, Pyroj for coordinate-system transformation and geospatial data processing, and cecodes for decoding and encoding weather data in GRIB and BUFR formats [49]. _Dynamic partition allocation and performance optimization._  
In this study, we implemented a dynamic partition allocation method to optimize the system performance. The optimal number of partitions was dynamically determined based on the numbers of files and CPU cores, which improved the processing speed by efficiently utilizing system resources. Therefore, each file processing time was precisely measured through performance analysis and monitoring. The performance was compared and analyzed for various partition quantities (4, 8, 16, 32, 64, 128, and 256). By monitoring CPU and memory usage in real time, we could identify and optimize system bottlenecks. To verify the scalability of the system, we performed tests on datasets of different sizes (50, 100, 500, 1000, 1500, and 2000 files) and different numbers of partitions. This allowed us to verify the linear scalability of the system and determine the optimal parallel processing configuration.