## 4 Artwork Recognition in a Real Environment
### CNN Artwork Recognition Models  
Three different CNN architectures were used to train models for artwork recognition using the TensorFlow framework, including the VGG19 [31], InceptionV3 [32] and MobileNetV2 [33] architectures.  
The first subset was used to train the 3 CNNs architectures. The subset was randomly divided into 3 sets, a 70% training set (5950 images), a 20% validation set (1700 images, used to fine-tune the model and improve its performance after each epoch) and a 10% test set (850 frames, used to test and compare the accuracy of the created models). The frames were subsampled to a size of \(224\times 224\) pixels and normalized to the range of \([-1,1]\) for the InceptionV3 and MobileNetV2 models, and to the range of \([0,1]\) for the VGG19 model. Random transformations such as random crops, variations in brightness and horizontal flips were applied to the images of the training subset, prior to the subsampling and normalization step. Random transformations resulted in a model with rotation-invariant convolutional filters without explicitly enforcing it on the model architecture. The rotation invariance was required in the proposed framework since a smartphone is a constantly moving device and the user might not take the capture completely leveled.  
Figure 6: Example of the ambient light intensity variations in the hall.  
Figure 7: Example of figures of virtual visitors superimposed on images showing paintings, in order to simulate occlusions (the human silhouettes used originated from a freely available dataset [30]).  
The networks were trained using the layer configurations shown in Table 2 and were initialized with the pretrained weights of ImageNet. Each CNN was trained for 20 epochs and a batch size of 128, using a Google Colab environment configured with a GPU runtime and 13 GB of RAM.  
Since the purpose of the trained networks was to be incorporated into a smartphone app, the models were converted to the app-compatible TensorFlow Lite format (.tflite). Additionally, by taking the advantage of the post-training quantization technique ([[https://www.tensorflow.org/lite/performance/post_training_quantization](https://www.tensorflow.org/lite/performance/post_training_quantization)]([https://www.tensorflow.org/lite/performance/post_training_quantization](https://www.tensorflow.org/lite/performance/post_training_quantization))), accessed on 11 July 2022, which reduces model size and improves CPU performance with a slight reduction in model accuracy, quantized TensorFlow Lite versions of the models were developed. The characteristics of the resulting CNNs are listed in Table 3.  
While the accuracy for all architectures when tested with the 10% test subset was quite similar, the models differed in their file size and the inference time for each frame. The ideal CNN architecture for the app would lead to a model file with the smallest size, the lowest inference time, and the highest accuracy. Although the quantized MobileNetV2 model led to the smallest file size and lowest inference time, it failed to achieve a high accuracy, making the MobileNetV2 model the preferable choice for the development of the app.