# Nodule Detection with Convolutional Neural Network Using Apache Spark and GPU Frameworks
## 3 Our Proposed Methodology
### Pre-Processing
Time taken to process the input images are taken in the y-axis, whereas the various platforms used for comparison are taken on the x-axis. The in-memory computation and Resilient Distributed Dataset (RDD). This provides a functional interface to partition the data across the cluster. Figure 7: Example of a Spark shuffle. We converted the Dicom scan image of the patients into RDDs for loading and passing through the nodes. We adopted HDFS as a file storage system for our lung nodule dataset and Spark for training the model. Even though a single node spark did not show promising results than a single GPU, clustered nodes can process the images in a few seconds. Overall, for our model and the training dataset, a GPU can process and train the images in 1678 s. Figure 9b shows the variation in batch size with all the model configurations. The throughput metrics are all in MB per second. For this investigation, we only considered the best results from each category.The main focus was to present the throughput difference between Spark and GPU and CPU TensorFlow. We found that model workload remains almost stable for Spark and GPU, yet Spark gives better throughput marginally in Figure 9c,d. When the dataset is tremendous and could not fit into single GPU memory, we can use distributed spark clusters to achieve convergence faster. TensorFlow on Spark provides fast processing in a distributed environment. The batch size of the data and the learning rate should be fixed according to the training accuracy to increase the throughput. In such a case, before a model gives an impressive performance, it needs to be trained repeatedly to achieve the proper hyper-parameters. Hence, Spark distributed training leads to better results when the deep neural network is complex and the dataset size is large. If we have shallow architecture and can fit the training data into a single machine, parallelism leads to computational overhead. We can adapt this setting to serverless architecture, as well. AWS, Microsoft Azure, and Google Cloud Platform are three gigantic heads in Cloud platforms. All three of them use Apache Spark on their cloud run for distributed parallel computing. Apache Spark performs well than GPU and has fault tolerance for a dataset. Spark has a replication system that allows it to reconstruct the data frames by itself in failure or crash. Spark also comes with built-in tools for machine learning algorithms and deep learning models, which lets us apply the ML model directly on batch data or real-time streaming data without any latency. It can be scaled horizontally and vertically without major hassle in coding and our experimentation results. Nevertheless, in the case of GPU, scaling is tedious, both concerning cost and coding. Figure 8: General architecture of Distributed Apache Spark and deep model. ## 5 Discussions & Future Work

This research intends to identify lung nodules using a deep model trained using the Apache Spark environment. Our paper enhanced the accuracy for detecting the lung nodules in patients scans by our proposed deep learning model NG-CNN. By eliminating the visual Gaussian noise in the CT scans, we can classify the lung scans' nodules at an early stage, which is notably crucial in predicting the disease to save a life. In addition to our proposed model, we also performed a performance evaluation to find the most reliable platform to run the deep learning model. Apache Spark shows promising results when integrated with distributed TensorFlow. As mentioned above, Spark has in-memory processing as it keeps intermediate results in RAM and built-in libraries for data analysis, machine learning, streaming live data, and graph analysis. Moreover, it also offers lazy evaluation and maintains a series of transformations with fault-tolerance. Hence, unless we perform the tasks, Spark can memorize the overall transformations.