# Nodule Detection with Convolutional Neural Network Using Apache Spark and GPU Frameworks
## 3 Our Proposed Methodology
### Pre-Processing
The size of the dataset is extensively large, which makes it challenging to work. The lung's raw image size is 512 \(\times\) 512, roughly 40 million pixels, which take up 1 GB of system memory to load the data. A significant hindrance is to load the patient's data for every training step and iteration. Loading the data takes up even more time than training the same. Detailed experimentation was done by comparing it with Apache Spark, GPU, and CPU. The accuracy of the deep learning model is directly proportional to the size of the training data. When astronomical data is fed, the network architecture shows impressive performance. Training a massive amount of data on a single machine is computationally challenging. If the dataset cannot fit into a single machine's memory, training the neural network in distributed clusters will save the training time and memory. Hence, to analyze the training time and efficiency, comparative experiments on data parallelism is conducted in this section. Spark operations that sort, group, or join data by value need to transfer data between partitions when constructing a new DataFrame from an existing one between stages, in a process called a shuffle. With a physical plan for CPUs, the DataFrame data is transformed into RDD row format and usually processed one row at a time. Spark supports columnar batch, but in Spark 2.x, only the Vectorized Parquet and ORC readers use it. Figure 7 shows that data is grouped by value and exchanged between partitions

Figure 6: Comparison of the proposed method (NGN-ACDNN) Non-Gaussian Noise aware Autoencoder Convolutional Deep neural network with other deep models (CAE-DLF) Convolutional Autoencoder Deep Learning Framework, (CNN) Convolutional neural Network, (DBN) Deep Belief Network: **(a)** describes the Training loss and Validation loss for our proposed CNN model; **(b)** describes the Training accuracy and Validation accuracy for our proposed CNN model; **(c)** describes the Comparison of test accuracy of our proposed model with other deep neural network models; **(d)** describes the ROC curve comparison of our model with others. (white rectangles) when creating a new DataFrame (blue rectangles) from an existing one between stages. Spark deep learning usually parallelize the dataset instead of the model itself. Figure 8 shows the general architecture of the distributed spark cluster. The data is distributed among the Spark workers. The gradients are updated via the parameter server in each iteration. Since the lung dataset we used is comparatively medium-sized, and one server was enough to process it, we did not parallelize the workers' deep learning model. Lung image data is parallelized among the workers by parameter averaging. Training the model in a cluster consists of the following steps. * Parameters, such as weight, and biases are randomly initialized based on NG-CNN architecture. * First copy of the parameters is distributed to all the Spark worker nodes. * The lung image dataset is divided upon the worker nodes, and they train their subset of data. * Update the calculated parameters after a certain number of iterations from the server node. * Repeat from step 2 till the training converges. The equation for parameter averaging in case of 2 worker nodes is given below. \[W_{i+1}=\frac{1}{2}\sum_{j=1}^{2}W_{i+1,j}, \tag{19}\]

where \(W\) represents the parameters, and i indicates the current values and \(j\) the updated values. Similarly, in case of 4 worker nodes and one server, the parameter averaging would take place as follows:

\[W_{i+1}=\frac{1}{4}\sum_{j=1}^{4}W_{i+1,j}. \tag{20}\]

The worker nodes hold a copy of our neural network model, and the lung images are distributed among the nodes. The hyper-parameters such as the learning rate, number of epochs and iterations, number of hidden layers and neurons are changed according to each training results. In Figure 9a, Spark with a four-node cluster shows equal performance to GeForce GPU.