dataesr/scientific-paragraphs-categorization huggingface dataset:
We develop a multimodal classifier for the cultural heritage domain using a late fusion approach and introduce a novel dataset. The three modalities are Image, Text, and Tabular data. We based the image classifier on a ResNet convolutional neural network architecture and the text classifier on a multilingual transformer architecture (XML-Roberta). Both are trained as multitask classifiers and use the focal loss to handle class imbalance. Tabular data and late fusion are handled by Gradient Tree Boosting. We also show how we leveraged specific data models and taxonomy in a Knowledge Graph to create the dataset and to store classification results. All individual classifiers accurately predict missing properties in the digitized silk artifacts, with the multimodal approach providing the best results.
However, the material vulnerability of these objects and the institutional fragility of many local heritage organizations have rendered them relatively hidden from the public. As regards the information about that heritage, many descriptions, and images of objects exist within in-house databases that are only available as local files. In other cases, those records are uploaded by many museums across the globe, in siloed repositories and heterogeneous, often incompatible formats. A few of them provide public access to the images and metadata of their silk objects through APIs, many more through their websites, but harmonization and integration efforts have been very scarce. Therefore, it is very hard for general audiences, historical experts, and industry (e. g. , fashion designers) to access this knowledge.
This article presents methods that enable further annotation of these museum objects through a multimodal classification approach that trains models to predict missing metadata from images, text descriptions, and other (available) metadata. The outcome is then further used to enrich an underlying knowledge graph that feeds the ADASilk exploratory search engine. Domain experts can easily assess the quality of the automatically generated annotations through rich visualization and connections between the items. Figure  1  shows examples of the kinds of images we are discussing. While Table  1  shows some examples of text descriptions from various museums. Both show respectively their associated metadata.
Contributions The main scientific contributions of this paper are related to our research hypotheses. We introduce a multimodal machine learning approach, adapted to the cultural heritage domain, for predicting the properties of digitized artifacts. We perform an in-depth analysis of the performance of our classification models, i. e. , models based on individual modalities and the multimodal classifier. Additionally, we introduce a novel dataset 2  to the cultural heritage and multimodal analysis domains that includes data for four different tasks and three different modalities. It consists of harmonized text and image data from heterogeneous, multilingual sources that went through different stages of preprocessing, cleaning, and enrichment like domain expert-guided entity linking and grouping.
Data and Labels The data used in this work belongs to the cultural heritage domain. More specifically, it is related to silk textiles produced in Europe, primarily in the period between the 15th and the 20th centuries. In the domain of cultural heritage, we cannot expect all class labels to be equally common or equally correlated. For example, in some locations, more silk fabric objects were produced than in others. Similarly, we know that the production of silk fabric objects in a given location likely intensified after a certain point in time and possibly subsided after a certain date. We also know that catalogs are curated by humans and often have strong thematic biases. For example, certain ultural Heritage Artifacts museums focus almost exclusively on objects created within one location.
The data we use in this work was aggregated from 12 different museum or collection websites. Each museum may have different standards for how it collected the underlying objects and how it digitized the information related to these objects. Importantly, this gives each museum its own standards for how to write text descriptions, how to create images, and how to annotate properties. Regarding these properties of digitized artifacts, accurately representing them requires adequate data modeling capabilities and considerable domain expert collaboration. This collaboration is also important in creating a dataset for machine learning. Labels need to be mapped from annotations made in different languages and grouped into domain-relevant classes. Due to the partially automated nature required to create the dataset, challenges arise that are common in such processes: label text requires normalization such as correcting typos, unifying the styles of dates, and matching different locations to specific countries.
Image Classification. In the context of this paper, the classification of images aims to predict the abstract properties of the silk fabrics depicted in the images. Whereas it may be relatively straightforward to learn to classify the material of a depicted piece of fabric, the prediction of semantic information such as the production place of the fabric, the period of time in which the fabric was produced, or the technique used to manufacture the fabric is assumed to be much more challenging. Furthermore, it is assumed that there are interdependencies between these properties of silk fabrics, e. g. , a certain production technique may have been more common during a certain period of time. This is why multitask learning is investigated for image classification. However, standard multitask classification frameworks require one reference label for every task to be learned during training for every training sample; The challenge we have to face is that in real-world data, as they were collected for the dataset presented in this paper, there may be many training samples for which annotations are unavailable for some target variables to be predicted.
Another challenge that is uncommon is the reduced variable overlap between images and text descriptions. Not only is our work subject to a comparatively small dataset, restricted by historical reality and difficulties of data collection, but we must also deal with the fact that for most archives of culturally relevant objects, many objects that have been photographed have no corresponding textual description. In fact, we'll see that less than half of all objects have both these modalities. Another challenge, uncommon outside of retrieval scenarios, is that we can have multiple images with different angles and focus per each individual object while it makes no sense to talk about multiple text descriptions per object. Yet another challenge we need to deal with, common to many real-world applications but not to research datasets, is that we do not have all properties for all objects.
For example, for a given object, we might know what material and techniques were used but not when or where it was made. Finally, our dataset, although drawn from several museums, contains under 30k objects and approximately 11k text descriptions. This effectively makes it small compared to general datasets, but not uncommonly so for a dataset in the cultural heritage domain. 2 Related Work   [2]. CIDOC-CRM can, however, only be considered a starting point for ontologies that deal with museum data, such as in our case. The fact that it is not difficult to extend it makes it, however, easy to add more domain-specific classes and properties as necessary in projects such as ours. There are more and more efforts of different CH organizations to adopt Semantic Web technologies and build knowledge graphs: Culture-Sampo is the result of integrating heterogeneous cultural content  [3].
The challenges consisted amongst others of converting legacy data into linked data and making it heterogeneous. Getty ULAN was used as structured vocabulary to find connections between two referenced persons, for example. One similar example is ArchOnto  [4] , which specifically addresses the challenges of CH data from and for national archives. Both can be an inspiration for our work, but given how fine-grained the vocabularies in Cultural Heritage domains can be, it is still necessary to deal with language and domain-specific vocabulary differently in each case. The training data used for our experiments is fully extracted from the SILKNOW Knowledge Graph that relies on classes and properties defined by CIDOC-CRM and its direct extensions CRMsci (Scientific Observation Model) and CRMdig (Model for provenance metadata).
Knowledge Graphs and Culture AI Knowledge graphs allow the representation of multi-source information about many entities and their relationships to each other. The data stored in a Knowledge graph can then be used for many tasks, especially when structured knowledge of a specific domain is relevant, e. g. , the development of product designs  [5]. Other common domainspecific fields are Medicine, Cybersecurity, and Finance. Knowledge graphs are also frequently used to aid product development and research for language-based tasks such as question answering systems, recommender systems and information retrieval  [6, 7]. A knowledge graph can also help with textual metadata-aided visual pattern extraction and recognition  [8]. Lastly, as we deal not only with images, but also textual metadata, the SemArt project can be considered related: it is a multi-modal dataset for semantic art understanding.
Thus, a trained CNN can be used to predict a class label for an object with unknown properties from an image depicting that object. CNN-based classifiers are also applied in many works addressing attribute prediction for depicted objects in the context of cultural heritage, where the focus is on making predictions for images showing paintings  [8, 16]. In  [17] , the artist, the genre, as well as the style of a painting are learned by means of a variant of AlexNet  [15] , achieving on average 68. 3% correctly classified images for the three variables in the WikiArt dataset (http: //www. wikiart. org/). A ResNet18  [18]  was used to predict a painting's artist in  [19]  which reported an 82. 5% overall accuracy on the Rijksmuseum dataset  [20]. In both cases, there is one CNN per classification task, and network weights pretrained on a variant of the ImageNet dataset  [21]  are used to improve the classification performance.
In contrast to  [31]  as well as  [24] [25] [26] , we adapt the training of the network weights such that hard training examples get a higher impact on the weight updates. In this way, we want to deal with the problem of class imbalance in the training data, aiming to improve the classification performance for underrepresented classes. For that purpose, we combine a variant of the focal loss  [33]  with the multitask softmax cross entropy loss used in  [31] , leading to a training strategy that focuses on hard training examples in a multitask scenario while allowing for missing class labels at training time for some of the tasks to be learned. Furthermore, we investigate the prediction of four variables instead of three like in  [31]  and evaluate our methodology on a much larger dataset consisting of images from several museum collections instead of only one.
In fact, even correctly linking information known to represent a certain label (e. g. , material) from tables can be challenging in the presence of spelling issues. Second, we work with multilingual data from multiple sources, which presents additional challenges. Finally, our dataset contains many more samples and uses automatic labelling based on information present in catalogs rather than externally annotated. We are also proposing to improve the handling of class imbalance through the use of the focal loss in text classification. The focal loss was compared to (binary) cross-entropy in the context of multilabel text classification  [43]. It was shown to outperform cross-entropy in both the datasets used. The datasets used are significantly different from ours in that they have many more labels and, in particular, many small classes with a very small number of very large classes, forming a "long-tailed" distribution.
Multimodal Classification in Cultural Heritage A joint image-text neural network architecture for classifying images of paintings by artist and year was presented in  [47]. Their text input consisted of a limited set of labels (style, media, and genre) rather than text descriptions. Conceptually, this is similar to CLIP-Art  [48] , an application of CLIP  [49]  to the retrieval of artwork images. CLIP learns to associate a small text vocabulary akin to labels with images through joint contrastive pretraining. This was applied to The iMet Collection dataset  [50] , possibly the most similar dataset to our own, it includes images of artworks associated with labels (also called "tags") that describe what is visually depicted in the object (e. g. , "Dragons"), its visible properties (dimensions, medium) as well as other culturally relevant properties (e.
g. , country of origin). Another very relevant dataset is in this context is Artpedia  [51] , a dataset of images of paintings associated with textual descriptions tagged as either "visual sentences" that describe the scene depicted in the painting or as "contextual sentences" that describe other aspects of the painting such as its historical context. The tasks for which this dataset was created consist of separating visual from contextual sentences and the retrieval of the correct image for a given text. The differences between the related work and our work are clear. We propose handling images, multilingual text, and tabular data as equal modalities. We also propose to handle data from multiple collections. 3 Data.
Knowledge Graph The SILKNOW Knowledge Graph lies at the center of all efforts to create a unified representation of the metadata of European silk textiles, particularly from the 15th to the 19th century. The data used in our experiments was collected from 16 sources, most of which are public online museum records, for which we built crawling and harvesting software. In addition to that, we have data from the SILKNOW 3  project partners Garin and the University of Palermo (Sicily Cultural Heritage). The dataset used in the experiments was created from a full export of all objects in the knowledge graph, which consists of the metadata of 38,873 unique silk objects before any preprocessing steps. This export includes in total 74,527 unique image files. In order to model this heterogeneous data from so many sources, we chose and relied strongly on the CIDOC Conceptual Reference Model (CRM).
We also developed our own SILKNOW ontology  4 to extend CIDOC-CRM with further classes and properties. This was done for cases where it did not cover some specifics of the silk textile domain and for storing the confidence score of metadata predictions, once we started integrating the results of those predictions into the KG. In order to develop a converter that could unify all the original data with all these classes and properties into one knowledge base, mappings have been created by domain experts. And on a technical level, all museum records had to be harvested and were first converted into a common JSON file format through our crawler software but each array inside this format still had the original field labels from the museums before the final conversion. For example, the majority of museums have a field for describing the production time of a silk object, but in most cases, museums use different names for their field.
Moreover, the museums are from all over the world and use different languages for both the field names and their values. This is why we created a mapping for, e. g, a field named "Date" (Metropolitan Museum of Arts) and the class E12 Production with the property P4 has time-span and another class E52 Time-Span. Likewise, a mapping rule was written for the field named "date text" (API of the Victoria and Albert Museum) and for the (Spanish) field named "Datación" (Red Digital de Colecciones de Museos de España). Another very central part of our knowledge representation is the SILKNOW Thesaurus, a controlled vocabulary that contains many explicit and multilingual concept definitions for materials, techniques, and motif depictions relevant to these silk textiles. Thanks to this thesaurus, a lot of information and entities from explicit categorical fields of the original museum records could be linked, without any advanced machine learning techniques.
The string literal could just be matched with the (multilingual) labels of the thesaurus and then replaced with a unique concept link. This explicit representation of knowledge forms the core of the dataset used to predict missing metadata. This includes cases where a categorical value is either not given at all or "hidden" in longer textual descriptions and not explicitly semantically annotated. Once all the modelling, download, conversion, and enrichment steps were taken, the final knowledge graph was uploaded onto a SPARQL endpoint from where all the data across languages and museums can be queried the same way. To make access easier, we also developed a RESTful API, so it is not necessary for web developers to write SPARQL queries, and an aforementioned exploratory search engine on top of this API, called ADASilk.
Extracting and Normalizing Labels The development of the SILKNOW Knowledge Graph is a combined effort of data processing that relies on a data modelling and annotation process created in collaboration with domain experts. This is especially true for the SILKNOW Thesaurus. The group labels used in the experiments in this work are based on the hierarchy and relations of concepts of the silk textile domain described in this controlled vocabulary. As described in Section 3. 1, many categorical property values can be easily extracted, linked, and automatically normalized thanks to the SILKNOW Thesaurus. This means that many concepts are accessible even though there were originally different strings, including typos in some cases, synonyms, or written in different languages. An example would be a weaving technique like "Damask", which would be "Damas" in French and "Damasco" in Spanish and Italian: for all these, we replace the string literal with one link to the same concept.
In addition to the SILKNOW Thesaurus, we also use linked open data like such as GeoNames 5  to normalize and link place names. Matching strings with our thesaurus and other controlled vocabularies was not without challenges. As will also be discussed in Section 6, misspellings or unconventional punctuation usage could still cause the matching process to fail. To give an example: If the string value of a record was "silk; gold thread" the latter would not have been linked, due to a bug that did not properly consider a semicolon as a separator. Other such cases existed as well, as the development of the SIL-KNOW Knowledge Graph is an ongoing process and concurrent with this work. See Figure  2  for an illustration of a museum record in the knowledge graph. The aforementioned hierarchy defined in the SILKNOW Thesaurus can be used to select specific types or subtypes of properties.
To refer back to the previous example, we could select only objects with the weaving technique "Damask", but also only objects made with "Two-coloured damask" which is even more specific. Based on the Thesaurus, we can also make sure that we only choose objects based on equivalent levels of this hierarchy. Based on this enrichment and the linking process, we created a pipeline to extract the dataset based on pre-specified criteria. We first developed a comprehensive SPARQL query that outputs all museum objects described in the Knowledge Graph (KG) and includes, if available, the most relevant properties: the identifier of the object in the knowledge graph, the museum where the description comes from, the text description, and Fig. 2:  A record from the MET museum with a missing property represented in the knowledge graph using our ontology and controlled vocabularies URL links to the images that illustrate the object.
Dataset Preparation and Properties The goal of the dataset preparation is the conversion of the knowledge graph data with normalized and grouped labels described, respectively, in Sections 3. 1, 3. 2, and 3. 3, into a dataset for the experiments in Section 6 using the classification methods described in Section 4. The first step was to select the records in the knowledge graph that were relevant to the domain. The second step as to select only records that contained a value for one of the variables to be predicted, i. e. , labeled samples. Uncommon labels, with a total frequency below 150, were discarded. The final step was to randomly split the records into disjoint sets: • a training set consisting of 60% of the data for supervised learning; • a validation (or development) set, consisting of 20% of the data for hyperparameter tuning and multimodal supervised learning; • a test set, consisting of 20% of the data, for the final evaluation of the proposed methods.
Given that the objective is to train and evaluate a multimodal multitask approach on records, that regularities exist within each collection (i. e. , museum) that comprises the data, that the text modality is also multilingual, and that both modalities and task-specific labels may be missing from a record, we believe the most reasonable way to split the dataset is a random split of records. The distribution of the data, in terms of records per variable, set, and class label, sorted by count, can be seen in Table  3. The table includes separate columns with the count of all records, columns for records with images, and columns for records with text. The first can be used by the tabular classifier, the second by the image classifier, and the third by the text classifier. Because records can have multiple images, while they may or may not have text, the exact class proportions can change depending on the modality.
Overall, class imbalance exists for all variables. An additional overview of the modalities with their overlap can be found in Table  5. We can see how 27. 120 or 96,60% of the 28,077 records about annotated fabric objects contain at least ultural Heritage Artifacts Finally, the distribution of samples over the museums can be found in Table  4. Each record in our dataset corresponds to an object in a museum. Each record can have multiple images associated with it, but at most, one "text". Text data in our dataset consists of descriptions of fabrics or objects made mostly of fabrics. These descriptions range in length from short sentences to multi-sentence paragraphs to multi-paragraph texts with thousands of words. Some descriptions focus primarily on a single aspect, such as a scene depicted or the history of the object, while   4 Methods.
Image Classification The goal of the image classification is to predict one class label per classification task, i. e. , the prediction of a class label for each of the target variables technique, timespan, material, and place, for an image that illustrates an object. For that purpose, an image classifier is trained using all images of all records contributing to the dataset described in Section 3. 4. We propose to use a convolutional neural networks (CNN) for that purpose, motivated by the success of CNN in image classification. As there are many records with annotations for more than one of these variables, we propose to train the classifier to predict all classes simultaneously in a multitask framework, exploiting the inherent relations between the variables to learn a joint representation that is used by task-specific classification heads.