dataesr/scientific-paragraphs-categorization huggingface dataset:
Materials and Methods 3. 1. Materials 3. 1. 1. Samples from the Archaeological Site of Scifì. Two types of mortars of the visible building, respectively, belonging to "phase I" and "phase II," have been investigated by XRF spectroscopy. At first inspection, they appear similar, and the analysis was aimed at establishing if the compositional homogeneity is maintained also at elemental level. Furthermore, four pottery fragments of different typology, labelled SC1, SC2, SC3, and SC4, belonging to a set of shards excavated from this archaeological site were analysed by XRF, Raman, and IRT techniques. All samples are dated back to the late imperial period, and precisely to the 4th-5th century AD, with the exception of the SC1 fragment, which is of Greek-Hellenistic age, about 3rd century BC. No archaeometric data exist on these samples, so their characterization was a prerequisite for their cataloguing.
In addition, a brick from the "tower," for which a local production is reasonable, has been also analysed by XRF spectroscopy. All the information relative to the investigated samples are reported in Table  1. Regarding the pottery fragments, the colour of their ceramic body and coating is described according to Munsell soil colour chart, whereas their photos are shown in Figure  3. We performed an XRF investigation of bricks and stones from the facade of the abbey. In particular, data have been collected on a red brick and on black, white, and yellow stones (Table  2 ). The photo of the analysed samples is reported in Figure  4. The goal of the investigation was to make some hypothesis concerning the local production of the used materials and to support archaeological and/or geological classification.
In our study, XRF measurements were performed by using a portable XRF "alpha 4000" (Innovex-X system) analyzer that allowed the detection of various chemical elements having an atomic number (Z) between phosphorus and lead. The apparatus was equipped with a Ta anode X-ray tube excitation source and a high-resolution Si PIN diode detector with an active area of 170 mm 2. The instrument operated in "soil" mode and two sequential tests were performed on each sample. The operating conditions were 40 kV and 7 μA for the first run and 15 kV and 5 μA for the second one, for a total spectrum collection time of 120 s. The two sequential measurements allowed us to detect elements from P to Pb. A Hewlett-Packard iPAQ Pocket PC was used to control the instrument and data storage. The calibration was performed by soil LEAP II and was verified using alloy certified reference materials produced by Analytical Reference Materials International.
The fluorescence signal was collected for about 60 s per run in order to increase the statistics. For all the investigated samples, the lines detected ~8. 15 keV and ~9. 34 keV are attributed to the L α and L β transitions of Ta anode. When Cu is present, the L α line of Ta is superimposed to the K α line of Cu. In the case of mortars, bricks, and stones, because of their heterogeneity both in compositional and dimensional terms, we decided to proceed in the following way: we selected only samples that, at a visual inspection, appeared similar (i. e. , in colour, granulometry, and absence of macroscopic inhomogeneities). For each sample, several measurements (~50) on many different points were performed. Almost the 80% of the collected data turned out to be reproducible (the reported spectra are, for each specimen, representative of this class of data).
XRF, Raman, and IRT Results on the Pottery Fragments. In the case of the pottery fragments, XRF data were collected on the painted surface of each sample. The elemental composition, as obtained by the spectral analysis, is reported in Table  3. Nevertheless, since, as well known, the X-ray source has a penetration depth of the order of few micrometers, the XRF signal will generally reflect not only the contribution coming from the pigmented layer, but also the one due to the preparation substrate. This latter is principally constituted by calcite and contributes to the high fluorescence lines of Ca and Sr. In order to overcome this difficulty and unambiguously identify the key elements responsible for the pigmenting agents, we also performed measurements on the back of the fragments that have been used for comparison.
The observed increasing of the Fe K α line allows to hypothesize, for both the analysed fragments, a Fe-based red pigment, such as an earth (inorganic pigment of differently hydrated iron oxide). In addition, in the case of SC2 fragment, the observation of Mn can support the hypothesis of the use of umber (combination of iron oxide, manganese oxide, and clay). XRF spectral features of SC3 sample appear very similar to those of SC4 fragment, indicating a comparable elemental composition starting from which, taking also into account their common archaeological classification, similar receipts for the pigmenting agents of the finishing layer can be supposed. Finally, as far as SC1 fragment is concerned, the observation in the XRF spectrum (data not shown) of Fe as key element indicated a Fe-based pigment for the black surface, probably magnetite (Fe 3 O 4 ), whose use in Greek-Hellenistic potteries is already reported in literature  [33].
0 1 0 2 0 3 0 4 0 0 20 40 60 K 훼 Zr K 훽 Sr K 훼 Sr K 훼 Rb K 훽 Fe K 훼 Ca K 훼 Fe K 훽 Mn Counts (a. K 훽 Zr K 훼 Zr K 훽 Sr K 훼 Sr K 훼 Rb K 훽 Fe K 훼 Ti K 훼 Fe Counts (a. u. ) (b) The molecular composition of the pigments was attempted by Raman spectroscopy. Unfortunately, in the case of SC1 and SC2 fragments, no reliable data were obtained, mainly because of fluorescence, but also as a consequence of the heterogeneity of the surface and the deposited contamination. On the other side, the Raman spectra collected for samples SC3 and SC4 are reported in Figures  9(a  In both cases, haematite (α-Fe 2 O 3 ) is clearly identified, by the observation of its characteristic peaks centered, on average, at ~229 cm -1 (stretching Fe-O), ~300 cm -1 (bending O-Fe-O), ~414 cm -1 (bending O-Fe-O), and ~612 cm -1 (stretching Fe-O)  [34].
Cultural assets are generally partially annotated with textual labels that may be used as additional information for their classification. To our knowledge, leveraging this additional information has not yet been used for the classification of cultural data. In this paper, we propose a novel approach for the classification of cultural content. Our approach is multimodal, as, in addition to asset visual features, we leverage the available asset metadata. The experimental results for a painting dataset show that leveraging additional data in the classification improves the classification performance. In the following sections, we present the impact of digital technologies on cultural heritage, and also give a literature review of the work related to cultural data classification and annotation using computer vision and machine learning.
Feature Extraction Approaches In the past, visual recognition and image classification were performed using feature extraction approaches. These approaches are designed to examine the low-level features of an image by analyzing its pixel structure. The approaches used to extract these features include SURF, SIFT, HoG, GIST, Color Features, etc. Once extracted, the low-level features are used as inputs for supervised or unsupervised machine learning models such as SVM, KMeans, Random Forests, etc. The resulting models are then used to classify or cluster the images  [16, [18] [19] [20]. These methods perform very well in some applications and scenarios, but their performance is limited in general classification tasks and big data environments where deep-learning-based approaches outperform feature extraction approaches  [21].
Using feature extraction techniques, many methods have been proposed to tackle the challenge of cultural heritage classification. The authors of  [11]  published a dataset collected from the Rijksmuseum of Amsterdam consisting of more than 100 thousand artifacts. They also provided baseline results of a four-task classification challenge using SIFT features and Fisher vector encoding. The authors of  [10]  combined HoG, SIFT, GIST and Color Features to classify painting genres, while the authors of  [22]  proposed a ranking-based method to classify paintings using Random Forests. The authors of  [23]  presented a histogram-based combination of local and global features for the classification of paintings. Several other researchers have attempted to combine feature extraction and feature learning.
Indeed, the results of this contribution surpassed the baseline results of the Rijksmuseum challenge set in  [11]. The authors of  [29]  studied the use of multiple image patches to improve the accuracy of CNNs for the classification of paintings using transfer learning. In  [30] , the authors tried to use both painting images and brush stroke information for the identification of paintings using two parallel deep residual networks. The approaches in  [31, 32]  explore Siamese convolutional networks for painting matching and retrieval in large databases. Both solutions can be used to capture visual links between assets and thus cluster assets sharing the same visual features. A large cultural dataset, called OmniArt, consisting of more than half a million assets in addition to a multitask classification approach, is published in  [33].
The remainder of this paper is organized as follows. In Section 2, we present the different techniques and tools used to design and implement our approach. These include the various datasets, the machine learning tools such as CNNs and concepts such as transfer learning. In this section, we also describe the architecture of our framework. In Section 3, we depict some of the experimental results using a designed experimental setup with data relating to paintings and compare our multimodal input architecture with single input traditional CNNs. In Section 4, we discuss the results and interpret the pros and cons of our approach. Finally, in Section 5, we draw our conclusion and give some perspectives for future work.
• The WikiArt Dataset This dataset is a collection of more than 140,000 paintings which can be accessed from the Wikiart. org website. However, the website does not provide an easy means, such as a public API, to perform data collection. We designed a custom script based on the Python library beautifulsoup  [34]  to capture the most relevant data fields directly from the site's webpages. The data were then inserted into a MySQL database for further filtering while preserving the integrity of the original data scheme. The data fields selected were the artist, the media, the genre, the year and the style. Some samples from the dataset are shown in Figure  1. perform data collection. We designed a custom script based on the Python library beautifulsoup  [34]  to capture the most relevant data fields directly from the site's webpages.
The data were then inserted into a MySQL database for further filtering while preserving the integrity of the original data scheme. The data fields selected were the artist, the media, the genre, the year and the style. Some samples from the dataset are shown in Figure  1. • The Metropolitan Museum of New York (MET) Dataset The Metropolitan Museum of New York (MET) recently published a collection of more than 200,000 cultural artifacts under the Creative Commons open access license. The data represent images of assets along with their metadata, however the metadata are not fully available for the majority of the assets. The collection is accessible from a CSV file, nevertheless, the downloading of the images required the writing of a custom script to harvest them from the asset webpages on the museum website.
• The Rijksmuseum Dataset More than 100,000 assets have been published by the Rijksmuseum of Amsterdam (which is often referred to as the Rembrandt museum, as it hosts a large collection of the artist's works). The collection is accessible via an API and has been the subject of multiple research contributions  [11, 33]. As a result, many researchers have hosted the collection on archival platforms. However, we noted that the museum had updated its collection with multiple new assets, and thus we relied on the museum's API to collect and organize the harvested assets. In this work, we only focused on paintings. Some samples from the dataset are shown in Figure  3. The data pre-processing step is very critical for our approach. For this phase, we discarded the non-relevant samples in the aforementioned datasets.
• The Metropolitan Museum of New York (MET) Dataset The Metropolitan Museum of New York (MET) recently published a collection of more than 200,000 cultural artifacts under the Creative Commons open access license. The data represent images of assets along with their metadata, however the metadata are not fully available for the majority of the assets. The collection is accessible from a CSV file, nevertheless, the downloading of the images required the writing of a custom script to harvest them from the asset webpages on the museum website. The script was also based on the Python library beautifulsoup  [34]. Some samples from the dataset are shown in Figure  2. perform data collection. We designed a custom script based on the Python library beautifulsoup  [34]  to capture the most relevant data fields directly from the site's webpages.
• The Metropolitan Museum of New York (MET) Dataset The Metropolitan Museum of New York (MET) recently published a collection of more than 200,000 cultural artifacts under the Creative Commons open access license. The data represent images of assets along with their metadata, however the metadata are not fully available for the majority of the assets. The collection is accessible from a CSV file, nevertheless, the downloading of the images required the writing of a custom script to harvest them from the asset webpages on the museum website. The script was also based on the Python library beautifulsoup  [34]. Some samples from the dataset are shown in Figure  2.
• The Rijksmuseum Dataset More than 100,000 assets have been published by the Rijksmuseum of Amsterdam (which is often referred to as the Rembrandt museum, as it hosts a large collection of the artist's works). The collection is accessible via an API and has been the subject of multiple research contributions  [11, 33]. As a result, many researchers have hosted the collection on archival platforms. However, we noted that the museum had updated its collection with multiple new assets, and thus we relied on the museum's API to collect and organize the harvested assets. In this work, we only focused on paintings. Some samples from the dataset are shown in Figure  3. The data pre-processing step is very critical for our approach. For this phase, we discarded the.
• The Rijksmuseum Dataset More than 100,000 assets have been published by the Rijksmuseum of Amsterdam (which is often referred to as the Rembrandt museum, as it hosts a large collection of the artist's works). The collection is accessible via an API and has been the subject of multiple research contributions  [11, 33]. As a result, many researchers have hosted the collection on archival platforms. However, we noted that the museum had updated its collection with multiple new assets, and thus we relied on the museum's API to collect and organize the harvested assets. In this work, we only focused on paintings. Some samples from the dataset are shown in Figure  3. often referred to as the Rembrandt museum, as it hosts a large collection of the artist's works). The collection is accessible via an API and has been the subject of multiple research contributions  [11, 33].
Data Pre-Processing The data pre-processing step is very critical for our approach. For this phase, we discarded the non-relevant samples in the aforementioned datasets. Unfortunately, the required labels for the data we collected were incomplete. We found that paintings were the most complete and fully annotated category, especially the samples collected from Wikiart. org. The most relevant labels we selected for the paintings were the artist name, the year of creation, the genre, the style and the medium. We selected a total of 43,594 paintings annotated for all of these labels. To fully evaluate the performance of our approach, we created four datasets regarding the number of paintings per artist, as we found that the artist attribution is the most relevant task for painting categorization (see Table  1 ).
Each of the new sub-datasets retains only paintings of the artists that have a higher artwork count than the dataset threshold. For example, the >50 dataset will only host paintings by artists that painted more than 50 paintings. This was intended to evaluate the impact of the number of paintings per artist on the results, as often artists do not change style or genre, and their paintings are only created within a specific time frame. This can help the model in learning correlations between labels. The five textual labels were encoded using One Hot Encoding and the year label was divided into bins of 10 years. Each entry in the datasets consists of a painting identified by its content ID, which points to the image file and its five annotations. At the training stage, the media, the style and the genre are fed to the model along with the asset image.
The artist and the year are the output of the model. Table  2  represents a dataset sample of five entries. At the training stage, for each item in the dataset, the data are split into two sets: The input set X, which consists of the visual capture, the genre, the style and the medium; and the output set Y, which consists of the artist name and the creation year. For visual data, we chose to resize the images to a minimum of 300 pixels either in length or width (preserving the aspect ratio). This was mainly done to increase the performance of the online data augmentation while training the models. The main purpose of data augmentation is to increase the validation and generalization performance of the trained CNN by avoiding the phenomenon of overfitting, where the network becomes unable to accurately classify samples not used in the training phase.
The data augmentation generally extends the training dataset with an augmented dataset composed of the same images with distortions and adjustments such as flips, zoom, rotations, whitening, contrast and lighting adjustments, etc. This new augmented set will help the network avoid overfitting by training the network with slightly modified samples at each iteration. In our experiments, we initially tested data augmentation with the single input network to evaluate its impact (comparing the training of normal images with augmented images). We found that, other than the techniques we used (horizontal flips and random crops), any data augmentation will decrease and deteriorate the generalization capability of the network, as generally the visual features found in paintings are the same (see Figure  4 ).
For example, paintings created by an artist who lived between 1835 and 1875 can never be attributed to 1885, as the artist in question was not alive at that time. For a human, such correlations are straightforward after identifying the artist. However, for a machine learning system, these correlations are not so obvious. Our network learns such correlations implicitly by jointly training it to predict the two output labels and thus making predictions more accurate. Our selected dataset is smaller than large datasets such as ImageNet used to train large-scale CNNs  [41]. As a result, we relied on transfer learning from models trained for the ImageNet challenge in order to benefit from their visual feature extraction. A deeper analysis of these networks revealed that they are very good at extracting and learning low-level image features even for data that were not used to train them.
In fact, the majority of domain-specific image recognition tasks rely on transfer learning  [42]. For the visual features part, we used pretrained CNNs such as ResNet50 with the ImageNet weights as initialization. We designed a set of deep-learning-based classifiers using transfer learning (for the visual feature learning part). We conducted several tests and compared the classification performance for different base CNNs, including the ResNet50  [39] , VGG16, VGG19  [37]  and InceptionV3  [43]  CNNs (these networks were the winners of the ImageNet challenge). We designed four networks based on the following CNNs: ResNet50, VGG16, VGG19 and InceptionV3. The InceptionV3 network has an input image resolution of 299 × 299 pixels, and we therefore increased the patch size in the online data augmentation process while training the network.
The tests were performed on a single-input network; an image of a painting was used as input, and the two outputs were the artist and the year of creation. The accuracies shown in Table  4  are the average results after five-fold cross-validation for the dataset of 300 paintings or more per artist. We found that the difference in performance between the tested networks was not very large. This was also the case in the ImageNet challenge, where we could only observe single digit improvements between these networks from year to year. Nevertheless, the ResNet50 network was found to have the best validation accuracy for our dataset, and fewer training parameters, compared to the other networks. After the selection of the ResNet50 network as the base CNN for the image path, and for comparison purposes, we trained the model on the four datasets mentioned in the data pre-processing section.
We also used the single-input network with the ResNet50 network as the base CNN to compare a single-input CNN with the multimodal network (multiple inputs). Both multimodal and single-input networks were trained using the SGD optimizer with the hyperparameters outlined in Table  3. Our first tests with the Adam optimizer  [46]  were mostly comparable to SGD performance, although the SGD was more consistent and yielded slightly better results. Figure  10  outlines the training and validation accuracy for the multimodal classifier on the >300 dataset. processing section. We also used the single-input network with the ResNet50 network as the base CNN to compare a single-input CNN with the multimodal network (multiple inputs). Both multimodal and single-input networks were trained using the SGD optimizer with the hyperparameters outlined in Table  3.
Our first tests with the Adam optimizer  [46]  were mostly comparable to SGD performance, although the SGD was more consistent and yielded slightly better results. Figure  10  outlines the training and validation accuracy for the multimodal classifier on the >300 dataset. Since we used cross-validation, the training and validation operations were performed five times. Each time, we perform the training for 100 epochs, which takes 5-7 h. The validation is mostly instantaneous as it is a feed-forward computation.
Approach Evaluation To evaluate our approach, we trained our model on the datasets discussed in Section 2. 1. We mainly evaluated the model prediction accuracy in scenarios where we leveraged the available information concerning the asset along with its image (visual features + textual features). We also compared the performance of the multimodal classification with other scenarios where only the 2D capture is fed to the model as input while tasked with predicting the same labels. Although we used two outputs in our implemented models (artist and creation year), for our experiments, we mainly focused on predicting the artist class, as it is the most relevant and widely used classification metric for painting categorization. The tests were performed on the four datasets mentioned in Section 2.