## 2 Proposed Method
### Machine Learning Pipeline
Taking those threshold numbers into account, the shuffled lot data directories within each label subdirectory were individually analyzed to split them into two groups, namely training and test sets. If a particular lot's data directory was considered as complete (i.e., it had a JSON file, the two multispectral images, and at least one RGB image) and supposing that using its files for testing did not exceed the corresponding threshold, then it was copied under the same structure to another location in order to feed the test set. Otherwise, the lot data directory was also copied, but to grow the training set. Thereby, the training set (\(\sim\)75% of all data) was used to train and tweak the model, while the test set (\(\sim\)25% of all data), with no overlapping with training set, was only incorporated at the time of the diagnosis evaluation. The data distribution after the above mentioned process was importantly imbalanced, as seen in Table 2. It can be noted that Stage 1 was not included in the table. This was due to the fact that only one sample was identified in that stage. Consequently, it could not be used for the model construction, and therefore, it could be considered as not relevant. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**\# of Samples** & **Stage 0** & **Stage 2** & **Stage 3** & **Stage 4** & **Total** \\ \hline Training & 711 & 55 & 90 & 112 & 968 \\ Test & 149 & 12 & 18 & 23 & 202 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Data distribution between the training and test sets by each CLR development stage. Figure 10: Machine learning summarized pipeline model. After the two sets were correctly obtained, one submodel was generated for each file type, i.e., sensor data (JSON), RGB, RGN, and RE. For the JSON files, Multi-Layer Perceptron (MLP) was used, whereas Convolutional Neural Networks (CNNs) were implemented to classify the RGB, RGN, and RE images. For that purpose, the data in the training set were first divided into four subdirectories according to the file type, while preserving the same structure. Then, each of them was preprocessed so that the noise was removed from the images, and the irrelevant keys in the documents were also identified and eliminated. Figure 11 illustrates an example of preprocessed image files. After that, the corresponding data were loaded within each submodel's generation, divided into feature data (the files themselves) and label data (the names of the label subdirectories that contained the files), and permuted. Thereby, the data were randomly mixed while it was still possible to know each feature's respective label unequivocally. Then, if applicable, the data were normalized and structured to scale the input and format, as was recommended when using deep ANNs. The normalization used for this experiment was the z-score (subtracting the mean of the feature and dividing by its standard deviation), which scaled the data to have the properties of a standard normal distribution [53]. Upon having the data prepared, different architectures and hyperparameter values were tried to train the submodel to tune it to reach higher performance values on the predictions. The technique used for tuning the submodel is called grid search with cross-validation. It consisted of executing an exhaustive search over specified hyperparameter values for an estimator to find out which combination achieved the best performance, which was by default the higher accuracy, but different metrics could be chosen. One candidate estimator for each combination of hyperparameters was built and evaluated, so that the best estimator, its attributes, and its average performance could be extracted once the search was complete [54]. Furthermore, the procedure for measuring the average performance of each candidate estimator during the generation of the submodel is called \(k\)-Fold Cross-Validation, where \(k\) separate learning experiments are run on the the same estimator to calculate \(k\) performance values and average them.