dataesr/scientific-paragraphs-categorization huggingface dataset:
Methods for dealing with class imbalance As already discussed, our dataset has an imbalanced class distribution. We have already proposed the use of the focal loss for the image and text classifiers. To serve as a comparison, we implemented two other common strategies for dealing with class imbalance. These are Weight Rescaling (WR) and Random Uniform Sampling (RUS). We will evaluate them on the validation set. In the case of the image and text classifiers, they will be subject to separate hyperparameter tuning and shown in separate tables to provide a comparison to the proposed use of the focal loss. In the case of the tabular and multimodal classifiers, weight rescaling will be part of hyperparameter tuning, as we're not proposing an alternative. Weight rescaling works by assigning a weight to each class in the cross entropy loss function.
Image Classification For all experiments in the frame of image classification, we use the split of the dataset described in Section 3. 4 in order to train the CNN for image classification presented in Section 4. 1. 1 by means of the training strategy described in Section 4. 1. 2. We use all images that are assigned to a record for training and classification, assigning the class labels of the corresponding records to all images associated with it. As pointed out in Section 4. 1. 1, for records associated with multiple images, all images are classified by the CNN at test time, and the image-based prediction having the highest class score is chosen to be the final result.
Experimental setup. The workflow of our experiments is as follows: The training dataset is used to update the weights w T Rt , w T F C T of the CNN with early stopping. The model parametrization and hyperparameters leading to the lowest loss are calculated on the validation set. In this context, we compared different training strategies dealing with class imbalance discussed in Section 4. 5: the focal loss, weight rescaling, and random uniform sampling. Furthermore, we tuned the hyperparameters listed in Table  9  and described in Section 4. 1, choosing the values achieving the highest average F1 scores on the validation set. Finally, all test set records for which at least one image is available are used for an independent evaluation, with the hyperparameters values tuned on the validation set.
7%, followed by technique having four classes with 76. 8% correct predictions and timespan with five classes with a OA of 64. 0%, whereas place with nine classes has the lowest OA of 62. 2%. An analysis of the task-specific F1 scores in connection with the class distributions of the respective task indicates a dependency of the F1 score on the degree of class imbalance. Taking the ratio of the number of image examples for the majority class, i. e. , the class with the most labeled examples in the dataset, in relation to the number of image examples for the minority class, i. e. , the class with the fewest examples, a negative correlation between this ratio and the achieved task-specific F1 score can be observed for the focal loss experiment, where a similar behavior can be observed for the softmax experiment.
Text classification The results for the text modality, shown in Table  14 , are better, in terms of F1 and OA, than the results for the image modality (Table  11 ) or the tabular modality (Table  16 ). This is not surprising, since the properties we are predicting are important to domain experts. They, or their taxonomy subclasses, are often included in the  Even when they are not, we can intuitively expect some degree of similarity between text descriptions of objects with similar underlying values for these properties, either globally or at least within the same museum. The biggest disadvantage of the text classifier is that text descriptions are present in the dataset far less often than images, as shown in Table  5. Once adjusted for missing modality, given that more than 60% of the records are missing this modality, the text classifier actually performs the worst of any modality, as shown in Table  20.
ultural Heritage Artifacts We analyzed about 20 misclassified English language test set examples for each task. In around half of the cases, there was no direct information that could've allowed an accurate classification. E. g. , no location is mentioned when attempting to classify place or no date is mentioned when attempting to classify timespan. This forces the classifier to rely on other statistical regularities present in the text to provide a classification. The material task is particular. Its most common class, "animal fibre", is a de facto background class. All records in the dataset should be of silk fabrics, which means the material they are made of is an "animal fibre". Some have other materials too. These other materials can correspond to a vegetable fibre (e. g. , cotton) or a thread with some metal (e.
Cramers' V is a symmetric measure that gives a value between 0 and 1 for the association between two nominal variables. We see that the values are, by themselves, relatively low, with only material and museum having a value above 0. 5. The association between museum and the other properties was the highest (first column or row) which again reinforces the belief that the curated nature of museum collections and its impact on the other properties is learnable from this dataset. The strength of the association does not directly translate into feature importance as measured by information gain (Table  17 ). Rather, it seems that a particular combination of these associations is being learned by the tree boosting algorithm in such a way that results in a relatively effective classifier. Fig. 9 : Association between features of the tabular classifier (Cramer's V).
Integrating and visualizing the predictions To model the predictions as part of the SIL-KNOW Knowledge Graph ontology we use classes and properties of the Provenance Data Model  with rdf:object, the type of the predicted property through rdf:predicate and its fitting CIDOC-CRM property type. The property rdf:subject connects the statement to the production class (E12) of the object in the Knowledge Graph. Every prediction is inserted in the appropriate part of the existing KG. For example, if a material value gets predicted, it gets inserted with the CIDOC-CRM property P126 employed at the production class of the object. See Figure  12  for an illustration of the data model. The prediction models were only trained on group labels (Section 3. 3), and can thus only predict those. It is sometimes necessary to map them back to a more concrete concept of the SILKNOW Thesaurus.
If, for example, "Damask" is predicted in form of its facet link http://data. silknow. org/ vocabulary/facet/damask it will be automatically converted to http://data. silknow. org/vocabulary/ 168, as facet links are too general for concrete category values. All predictions are converted one after another using the described data model and saved in the Terse RDF Triple Language (TTL) file format which is uploaded and stored as its own graph identified by http://data. silknow. org/predictions. This makes it possible to always identify and eventually separate predictions from original values obtained from the museums. In total, 98,379 predictions were made for 19,248 distinct objects. These were uploaded into the SILKNOW Knowledge Graph. In our exploratory search engine ADASilk, the predictions are displayed differently from values that come originally from the museums: They are shown in blue together with their confidence score as a percentage next to them.
Conclusions We presented results for three individual modality classifiers, as well as multimodal results. In terms of our original hypothesis, presented in Section 1. 2, we showed that we were in fact able to ultural Heritage Artifacts l Heritage Artifacts 29 accurately predict missing properties in the digitized silk fabric artifacts that made up our dataset. While the quality of predictions varied between individual modalities, we showed that the multimodal approach provided by far the best results (74. 2% vs 55. 6%) and even that combining any two modalities outperforms any single modality. To recapitulate, our contributions included the already mentioned multimodal approach tailored specifically to the challenges we faced in the multimodal scenario, including the incomplete overlap of data across modalities.
The individual approaches of each modality-specific classifier also provide a useful contribution to the automated classification of cultural heritage objects. The image and text classifier offer the possibility of being applied to data outside a Knowledge Graph (KG) or database, possibly even directly submitted by a system user. The tabular classifier, on the other hand, offers the possibility of classifying data in a KG or database when no text descriptions or images are present by relying on other properties. The text classifier provides the best results of any single modality when text is present. Including text descriptions when classifying cultural heritage objects is thus highly advantageous and should be considered in any practical work. It is also important to remember that in most practical situations, including inside a KG or other knowledge bases, images are more common than text descriptions of objects in the cultural heritage domain, and thus image classification may be the only viable approach to automatically assigning metadata to a large portion of any cultural heritage dataset.
When adjusted for records with missing modalities, because images are more common in records than text descriptions, the image classifier actually outperforms the text classifier. Although useful in fewer circumstances, once also adjusted for missing modalities, the tabular classifier is the best-performing single modality classifier. This modality is probably the best starting point for predicting missing metadata inside a KG or a database. For both text and image classification, significant improvement of results will probably have to lean more heavily into self-supervised learning, as supervised data scarcity is likely to remain a challenge in the near future. This applies to individual modalities and to the fusion approach. Similarly, class imbalance and label noise are challenges for all modalities, and different ways to handle them could be found not just at the level of the individual modalities, but perhaps also with a change of fusion approach.
Data The data we used in our work originally comes from many museum sources and is from a very specific Cultural Heritage domain: historical, European silk fabrics. We applied common methods to process such data and developed an ontology and a Knowledge Graph out of the original museum texts and images. Such an effort comes typically with challenges, which in our case consisted mostly of a small amount of (training) data, domain specificity, different styles of writing texts and capturing images of objects, different languages (in the case of texts), and finally simply annotation errors, typos, and other errors that happened during the original digitization. Not all of these challenges could be completely overcome. In fact, the metadata gaps constitute part of the motivation to conduct this research work.
The exclusion of some data from our dataset was necessary to ensure that the classes present had a viable number of samples. This was, however, very much alleviated through the grouping process, which was possible through our domain expert-designed thesaurus about silk fabric concepts. In the end, we were able to publish a cultural heritage dataset that can be used for automated classification for multiple modalities and enable the study of multimodal approaches. In this work, we also provide the data modelling of how metadata predictions for data such as ours can be represented within knowledge graphs or other knowledge bases.
Text classification When text descriptions are present, the text classifier provides the best results of any single modality. It seems, thus, that the text classifier was able to overcome the primary challenges it faced: small dataset, domain specificity, cross-linguality, and museum-specific text styles. This was primarily achieved by the choice of XLM-R as the basis of the text classifier. Our results indicate that the use of the focal loss, originally introduced in computer vision, can help improve results in text classification and mitigate class imbalance within the cultural heritage domain. Future work can include the evaluation of more recent advances in loss functions for class imbalance. While more focus on class imbalance is desirable, future work must also deal with noisy labels, as this is an issue in the dataset and very likely to be an issue in any practical application within this domain.
Tabular classification Tabular classification was not emphasized in this work due to its more limited uses. Despite this, it proved very useful and provided surprisingly good results when considering that, by definition, this modality present in database records. In the future, class balance could be tackled by generating artificial examples using nearest neighbor data generation techniques common in tabular classification. The inputs to the algorithm could also be switched to finer-grained labels (e. g. , "cotton" instead of "vegetable fibre") as the requirements for the number of examples are more relaxed in inputs than in target labels. Other methods could also be explored. Neural Networks have become competitive algorithms, in certain cases, for tabular classification. It would be interesting, in this case, to replace the one-hot input labels with word embeddings which could make clear the similarities between them (e.
A comprehensive analysis of this approach was beyond the scope of this work. Perhaps different approaches to integrating predictions from multiple modalities beyond fusion could also prove fruitful. For example, using the predictions of image and text modal to replace missing non-target values in the input to the tabular classifier. Data Availability. The dataset used in this article is publicly available at https://zenodo. org/record/6590957. The entire Knowledge Graph from which the dataset was created is also available at https://zenodo. org/record/5743090. Finally, the thesaurus used in this work is available at https://skosmos. silknow. org/thesaurus/.
Code Availability. The code used to perform the experiments reported in this work, namely all the classifiers, is available online at https://github. com/silknow/ multimodal cultural heritage. The tools used to create the dataset are also publicly available. The crawler is available at https: //github. com/silknow/crawler/ and the converter at https://github. com/silknow/converter/. Other code relevant or used is this work may be found in the project collection of code repositories at https://github. com/silknow/.
Second, a systematic research design needs to be followed, which is guided by the research hypotheses. Third, the researchers implement evaluation criteria to assess potential biases and ensure the methodological rigor of the case studies  (Herriott & Firestone, 1983; Yin, 2014). We have already established a testable hypothesis derived from theoretical considerations in the previous chapter. Hence, we will now focus on the data collection and analysis as well as the case selection and description. We use diverse methods of data collection to build rich and comprehensive data and to offer a holistic perspective  (Canhoto, 2021; Creswell, 2003). First, we conducted semi-structured interviews with three key informants who hold a distinct position in the case firm that ensures access to relevant information: the CEO, who is also the founder and owner of the firm, the director of new business and consulting, and the senior strategist.
The aim of these interviews was to reveal how specific dimensions of digital technologies influenced the firm's competitive strategies and how they contributed to scaling the firm's operations. After a general part, in which we covered their use of digital technologies and their understanding of scaling, we specifically addressed the three dimensions -automation, individualization, and interdisciplinarity -identified in our theoretical framework. We asked the interviewees to provide concrete examples to illustrate the information they gave us wherever possible. Second, we collected secondary data such as newspaper articles, reports, and media broadcasts (see Table  1 ) to avoid being trapped in the case  (Johnston et al. , 1999). By using multiple types of data, creating a database of materials, and following our case study protocol we were able to ensure triangulation, reliability, and quality control.
To scale its operations, Netural not only offers services based on digital technologies, but it also founds spin-offs. These spin-offs include Roomle, a VR software for planning and setting up rooms, VIVELLIO, a blockchain-based application for managing health data, and Storyblok, a headless content management system (CMS). The VR based planner Roomle allows the virtual furnishing of rooms and forms the background technology of numerous furnishing and furniture stores in Austria, Germany, and Sweden. With over six million installations, it is one of the most widely used products in this sector (P15). Roomle had 20 employees in 2021. Between 2016 and 2020 the firm scaled considerably with its total assets more than doubling from 0. 38 million USD in 2016 to 0. 80 million USD in 2020. The firm has received several awards and achieved one of the most successful exits in Austrian history (P16).
Today, nocode platforms allow for developing individual solutions. " (I3). However, Netural's market offerings also need to be based on an interdisciplinary approach to ensure a competitive advantage. Typically, this implies using digital technologies in addition to incumbent technologies; "How can we enhance a business to deliver value? That is the main question. The answer is increasingly digitalization. Around 1900, the answer was industrialization, using machines etc. " (I3). By simultaneously combining automation and individualization with profound interdisciplinary knowledge, Netural's business model can be replicated in new markets "We replicate our business model across markets by providing country-specific language and buying options based on the same product" (I1). This replication strategy allows the firm to decrease its costs; "there is a bigger market for the same product with lower marginal costs" (I1).
Moreover, it also leads to increased customer benefits due to individualization; "we can excel in delivering highly individualized solutions for unique and complex business cases in a very reasonable timeframe (e. g. , 6-12 months compared to 2-3 years with a conventional software development approach" (I2). The possibility to simultaneously combine automation and individualization ultimately allows the firm to gain new customer segments; "we begin to get more and more customers who set the technology and systems they want us to use to implement a solution" (I3). In our explanatory case study, we find the pattern that Netural's approach to simultaneously (1) rely on automated processes, (2) provide individualized market offerings, and (3) use an interdisciplinary knowledge base within the digital era leads to better scaling performance.
We discuss the advantages and limitations of these methods and the metrics used to assess their performance, proposing that accuracy and contiguity are the most meaningful metrics. Finally, we propose the field of alignment-based polyploid phasing would greatly benefit from the use of a well-designed benchmarking dataset with appropriate evaluation metrics. We consider that there are still significant improvements which can be achieved to obtain more accurate and contiguous polyploid phasing results which reflect the complexity of polyploid genome architectures.
Briefly, physical separation methods attempt to only sequence one chromosome at a time, sidestepping the polyploid phasing problem by sequencing individual chromosomes  [10]. De novo haplotype assembly methods ambitiously attempt to simultaneously recreate the different haplotypes and resolve the structure of the genome, typically relying on long-range sequencing methods such as Hi-C  [11]. Alignment-based phasing methods map the sequencing reads to a reference sequence and identify variable positions, which are then used as input to a phasing algorithm that outputs predicted haplotypes. In this review, we discuss the different paradigms in the field of alignment-based polyploid phasing methods and how the performance of these methods is evaluated. We also propose that it would greatly benefit the field to standardize the performance metrics used to evaluate proposed methods, including the generation of gold standard datasets to systematically benchmark against.
Trends in polyploid phasing solutions All alignment-based phasing methods share the same pre-processing steps. First, a reference sequence must be chosen or assembled de novo, to serve as a guide. Then, the sequenced reads are mapped to this reference sequence and variable positions are identified. Finally, the dataset of reads, reduced to their variable, phase-informative positions, is used as input for the phasing method (Fig. 1 ). The methods proposed as solutions to the polyploid phasing problem are highly varied in their approaches and mathematical and conceptual underpinnings. To provide a coherent framework for this review, we delineate the development and usage of different strategies, identifying four major trends: Population inference methods, which leverage mapped reads of known genotypes in related individuals or in a population to infer the haplotypes in a sample.
Objective function optimization The objective function optimization strategy seeks to solve the phasing problem for single individuals. This method defines an objective function, which it then seeks to minimize algorithmically (Fig. 3 ). The objective function is typically a measurement of how well the predicted haplotypes correspond to the reads in the dataset. For example, for MEC (Minimum Error Correction) optimization, the objective function counts Alignment-based phasing methods invariably require the following steps: DNA sequencing of the sample, which fragments the DNA into sequenced reads. The reads are then mapped to a reference sequence and heterozygous sites are identified by variant calling. The dataset of reads associated with their variable positions is then input to a phasing method and predicted haplotypes are output.
These predicted haplotypes therefore conform to the structure of the reference sequence that was aligned to initially. how many mismatches there are between the predicted haplotypes and the set of mapped reads. The intuition is that a low MEC score implies a highly accurate phasing. Another variant of this method is the MFR (Minimum Fragment Removal) method, in which the objective function is minimized when the predicted haplotypes and the set of mapped reads are in perfect agreement after the removal of as few reads as possible. Typically, but not always, objective function optimization methods cast the dataset of reads as a matrix, and implement known or novel algorithms and heuristics intended to minimize the chosen objective function in the matrix. Objective function optimization methods showcase a variety of heuristics and statistical methods.