# Region-Based Convolutional Neural Network for Wind Turbine Wake Characterization in Complex Terrain
## 2 Materials and Methods
### Model Accuracy Metrics and Criteria  
The testing dataset comprised 59 scan images chosen at random that contained approximately 300 instances in total of wakes and wake fragments. Model success is dependent on both wake characterization and wake identification success when compared to subjective annotation. Areas of intersection (in pixels) of masks and subjective annotations are denoted as the intersection over union (IoU). Thus, each wake mask output by the CNN was associated with a unique IoU value when compared to subjective annotation. For the proof of concept, the IoU needed for a successful wake mask was 0.6 or greater (60% of the CNN mask's area in pixels intersects the subjectively annotated wake's area in pixels, Table 1).  
Wakes with obscured shapes because of CNR filtering were evaluated separately with the same criteria for failure and success. Successes and failures were calculated per unique wake, and thus each scan can produce multiple instances of successes and failures,  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Success** & **Failure Modes** \\ \hline  & (1) Wake/wake fragment shape is displaced relative to ground truth \\ Wake/wake fragment is correctly identified and shape of wake/wake & (-0.6 IoU). \\ fragment is adequately masked (\(\geq\)0.6 intersection over union [IoU]). & (2) Wake/wake fragment is undetected (false negative). \\  & (3) Wake/wake fragment is identified as present when it is not present \\  & (false positive). \\ \hline \hline \end{tabular}
\end{table}
Table 1: Criteria for success and failure when evaluating wake mask results from the convolutional neural network (CNN).  
Figure 7: Example model diagnostic showing the validation and training loss curves as a function of model iterations. The training loss curve shows the rate of model learning as a function of the number of learning cycles (iterations). High loss values indicate higher classification and identification errors. The validation–loss curve highlights the generalizability of the model (indicating extent of overfit or underfit to the dataset when training).  
depending on the number of wakes and masks observed. All success rates were calculated through the following expression:  
\[\textbf{{Success}}\ \textbf{{Rate}}=\ \left(\textbf{1}-\frac{\textbf{{failure}}}{ \textbf{{success}}}\right)\]  
Occasionally, wake masks returned by the model identify nested wakes, which occur when one wake mask is within another. Seven percent of wake masks returned for the full resolution testing set contained nested masks. Nested model predictions for bounding boxes and masks are commonly seen in image processing applications and can be mitigated through adjusting parameters during training (such as adjusting non-maximum suppression parameters [47]). For this use case, it is possible that a small number of nested predicted wakes were unavoidable because of the nature of velocity contours and the coarse spatial resolution of the scans. Further, when analyzing the nested wakes subjectively, their aggregated mask was often sufficient when identifying the wake and wake fragment geometries. Consequently, nested wakes are treated as single entities herein and counted in only one IoU estimation.