# Vineyard Gap Detection by Convolutional Neural Networks Fed by Multi-Spectral Images
## 3 Materials and Methods
### The Images and Datasets  
Before training and testing, a dataset was created for each image format mentioned (see Figure 1). The generated maps were divided by file types, and one class object was identified in every image--the vine gap. The manual identification was made by one person. The software used to manually draw the bounding boxes is mentioned in the next section. Images were prepared before training the CNN; this included cropping each map into same-sized pieces, that could or could not be overlapped with one another. Cropping the map into smaller images resulted in a closer look into a certain space of the field. By using zoomed images of fixed spaces, the neural networks can more easily detect the gaps in an image. This allows YOLO [20], and its successors, to work better, since YOLO networks give worse results when objects are farther away. In Figure 2, the bounding boxes and labeled regions include parts of the top and bottom vegetation, which trained the networks to search for gaps encapsulated by plants. Although this is a better way to identify objects and regions, as it will be discussed, networks struggled with patches of dirt that mimicked these characteristics. As mentioned before, later in the research process, more generated maps were received, which were integrated into the datasets and increased the amount of data available to retrain Mask-RCNN. In Table 1, we describe the number of images we had, after cropping each map, for each image type, and how much data we used in training and testing. We implemented a cropping algorithm (see Algorithm 1) that in conjunction with Algorithm 2 automatically divided each set into two subsets (training and testing) by giving a percentage of approximately how many images we wanted for testing. As shown in Table 1, we used 10% first, and later, when we received more images, we increased the testing data to approximately 20% due to having more images. As seen in Tables 1 and 2, the percentage amounts were not fixed; instead, they averaged out to the percentages discussed previously of 90-10% and 80-20%; this is because, as it was studied, there were no advantages in achieving perfect ratios in this case, and its effects would have been small, since it would have changed the datasets by a couple of images. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Multi-Spectral Index** & **Total** & **Train** & **Perc. (\%) Train** & **Test** & **Perc. (\%) Test** \\ \hline RGB & 1696 & 1590 & 94\% & 106 & 6\% \\ CIR & 1575 & 1414 & 90\% & 161 & 10\% \\ NDWI & 1575 & 1418 & 90\% & 157 & 10\% \\ NDRE & 1352 & 1226 & 91\% & 126 & 9\% \\ NDVI & 327 & 281 & 86\% & 41 & 14\% \\ GS & 650 & 583 & 90\% & 67 & 10\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of images per multi-spectral index. First dataset used for training. Total: total number of images; Train: total number of images used in training; Test: total number of images used for testing. Approximately 10% of Total was used for testing. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Multi-Spectral Index** & **Total** & **Train** & **Perc. (\%) Train** & **Test** & **Perc.