## 3 Materials and Methods
### Object Detection Using Deep Learning Techniques  
In addition to the need for accurate segmentation of the objects within the Petri dishes, our workflow must also provide each object's position and class (monolayer, forming spheroids, and spheroids). A perfect match for this task is the state-of-the-art instance segmentation method called Mask R-CNN [6]. This segmentation method is based on the original Faster R-CNN [12], which is capable of detecting individual object instances within an image with a class prediction. Additionally, Mask R-CNN provides a binary segmentation mask for each segmented object; this binary mask, in combination with the corresponding classification results, is crucial for our analysis of the individual cell states. In addition to the fast R-CNN and Mask R-CNN predictors of pytorch, for the final Mask R-CNN model, a pretrained resnet50 fpn backbone was used. Based on previous work in this field [13], we included a Unet-based [14] segmentation approach called Unet++ [15] using a pre-trained RegNetY120 backbone out of the segmentation model Pytorch framework [16]. In the previous publication, this semantic segmentation algorithm provided not only a binary segmentation mask for each needed class (cell states), but also an approximation for an instance segmentation based on the connected components labeling result. Due to the property of the early spheroid stages not being able to overlay each other, we wanted to detect these stages using an xor approach, which was realized using a multiclass Unet++ with a softmax activation function applied within the final network layer, or the Mask R-CNN. In contrast to the early stages, the final spheroids are able to overlay other objects. These spheroids are detected by an additional binary classification, Unet++, realized using a sigmoid activation function within the final layer. The combined segmentation of these two neural networks used in this publication (one for the monolayer and forming spheroid class and one for the spheroids themselves) outperformed all of our approaches using a single multiclass Unet++ for the classification of all spheroid stages. Both model architectures were trained using the Pytorch lightning framework [17] under Ubuntu 22.04, including data augmentation techniques provided by the albumentations framework [18] for an artificial extension of the available image data. For additional processing steps scikit-learn was used [19]. These augmentation techniques included horizontal/vertical flip, rotation, Gaussian blurring, random shifts of the individual color channels, the brightness and contrast of the whole image, and color saturation. In addition to these data augmentation methods applied to the individual images, we also included image cropping for an extension of our dataset by cropping smaller parts out of the initial image. For final evaluations, subimages with the size of 768 \(\times\) 768 pixels were used. Using the combination of data augmentation and image cropping, we artificially extended our training dataset by a factor of 25. All our models were trained for a maximum of 2000 epochs, selecting the best validation model out of these epochs as the final model used for our analysis, where the data splitting is shown in Figure 2. Both architectures (Mask R-CNN, Unet++) were also trained using two different types of input. On the one hand, we trained our models using the provided raw RGB images sourced from our imaging setup. In addition to extensive evaluation and the potential for improved results, we also used the LAB colorspace, converting these images to their corresponding luminance representation using the LAB color space. This has already shown improved results compared with RGB colorspace in other biomedical applications [20]. Using the luminance information as a standalone image, it is possible to reduce the impact of color shifts between the images due to slight changes in the imaging setup.