## 3 Materials and Methods
### Case Study
#### 3.2.4 Training Deep Learning Classifiers  
1. Using CNN architecture for image classification  
For each object defined by the organization, public images were collected through a Google search. The goal was to create training data for each object. We classified the images of each object into two categories (exists/does not exist). Figure 4 presents the number of training images collected for each object. Figure 3: The number of images collected each year using the API. According to the data, the number of images collected annually has increased significantly since 2017. Most of the posts were collected between 2019 and 2022. It is important to note that in this study, publicly available images were used for training purposes without obtaining the explicit consent of the image owners. The classification models were trained and evaluated using fivefold cross-validation for each object. The CNN architectures evaluated were ResNet50, ResNet152, VGG16, VGG19, InceptionV3, EfficientNetV2B2, EfficientNetV2L, EfficientNetV2S, EfficientNetV2M, and EfficientNetB0. We ran the CNN architectures with the default settings for each model (dropout = 0.2, number of neurons in middle layer = 1024, activation function = relu, activation function in output layer = sigmoid, optimizer= adam, learning rate = 0.001). A single Intel Core i7-10510U (16 GB RAM) was used in the experiments. As part of the preprocessing, each image was resized to \(224\times 224\times 3\). It was necessary to do this to construct the transfer learning neural network, as it ensured compatibility with the input requirements of the pre-trained model. To fine-tune the pre-trained model, the convolutional layer was frozen. A fully connected layer was created, with an output layer consisting of two neurons (one for each class). Training the model involved 100 iterations. However, to prevent overfitting, we stopped the process when performance on a validation dataset declined. Our model was evaluated using validation loss as a stopping criterion. Losses were calculated using binary cross-entropy. For each of the six objects, we selected the model with the highest F-score across the five folds, resulting in six optimal models. A test set was then used to test each on new data. 2. Using YOLOv5 for object detection  
We tagged each object separately for the YOLOv5 models. Tagging was performed using the Roboflow site ([[https://roboflow.com/](https://roboflow.com/)]([https://roboflow.com/](https://roboflow.com/)) (accessed on 1 February 2024)), used to build computer vision models. A feature for tagging images is provided by Roboflow. The resolution of each image was changed to 640 \(\times\) 640 \(\times\) 3 during preprocessing. Google Colab was used to train the model with the following parameters: epochs = 100, batch = 16, weights = yolov5s (initial weights). A YOLOv5 model was created for each object, and the training data were divided into 80% for training and 20% for validation. Similarly to the training data for the CNN models, the training data for the YOLOv5 were created from public images retrieved from Google. Figure 5 shows the number of images and total number of tags for each object used to train the YOLOv5 model. Considering that an image can contain multiple instances of the same object, there are more tags than images. In order to test the YOLOv5 models, we used the training data from the CNN models presented in Figure 4.  
Figure 4: Data used for training the CNN architecture for each object type (member, screen, property, map, hat, and pin). The pink color represents the number of images that are not defined as containing the specific object, while the blue color represents the number of images that were defined as containing it. Using the BLIP model for image captioning  
The BLIP model is an image captioning model pre-trained on the Common Objects in Context (COCO) dataset based on Vision Transformers (ViT) architecture. The HuggingFace repository includes the BLIP model under the name \"Salesforce/blip-image-captioning-base\".