# Egyptian Hieroglyphs Segmentation with Convolutional Neural Networks
## 3 Materials and Methods
### The Dataset  
Training the Mask R-CNN needs the availability of a suitable dataset containing a number of different Egyptian hieroglyphs (as large as possible), along with a segmentation mask associated with each of them (the _ground truth_). An amount of pictures are available online, for example in the repositories of museums, such as the collection of \"Egyptian Museum in Turin\" (Italy) [34] or \"The Met\" in New York (US) [35], but, unfortunately, all these images are not segmented. Furthermore, most of the images are not yet ready to be used for training, having different dimensions, resolutions, point of views, etc. Thus, likewise in many machine learning problems, the construction of a dataset for training purposes needs an accurate selection and preparation of the images. More specifically, better results are expected if the images contain symbols as uniform as possible (in terms of dimension). Often, images contain both hieroglyphs and drawings, so that, even though Detectron2 works with input images of arbitrary size, cropping patches from larger images was performed to extract \"text\" areas.  
In this work, we used different resources to create our dataset. Part of the images comes from the dataset used by the authors of \"GlyphReader\" [36; 37]. Images from this dataset represent the stelae contained in the Egyptian Pyramid of Unas, and each image contains many hieroglyphs. The number of images is limited (ten) and, more importantly for their use in a training process, they are quite similar to each other, i.e., they come from the same stelae and have the same support, so that they lack the \"hetereogeneity\" necessary to make the network learn the generic features important for segmentation. Therefore, the dataset was extended by adding images representing different writing styles and supports, available thanks to the online resources cited above. Our final dataset is then composed by 101 images, each containing multiple glyphs. The dataset was split in training (59 images), validation (14 images) and test set (28 images). Some examples are shown in Figure 2.  
Figure 2: Examples of images (on different supports) belonging to our dataset: papyrus on the top and bas-relief on the bottom.  
After image selection, all the instances in the training and validation sets were manually segmented and labeled using the _VGG Image Annotator_ tool (VIA) [38] in order to create the \"true\" segmentation mask of all the hieroglyphs contained in the images. VIA allows images to be annotated with bounding-boxes, circles or, as in our case, polygonal masks. The annotations were saved in a _json_ file, which can be read by the Detectron2 framework. At the end of the manual segmentation process, the gathered instances of hieroglyphs and associated masks for the training set were 2198 and those for the validation set were 564. An example of the outcome of this manual segmentation process is shown in Figure 3.  
Since in this work we were only interested in the segmentation task, we labeled each hieroglyph in an image as belonging to the generic \"_glyph_\" class, so that, at a pixel level, we are dealing with a binary classification task between _background_ and _glyph_.  
It is worth noting that hieroglyphs are represented by more than 700 different ideograms. The segmentation process benefits from using a training dataset populated with the majority of hieroglyph classes. In our case, it can be noticed that the classes of glyphs are not exhaustively represented and, in some cases, just a few instances per class are present. This fact can be an evident limitation to the generalization capability of our model. Despite that, in the Experimental Results section, we will show that the network is able to also segment glyphs that were not present into the training dataset.