dataesr/scientific-paragraphs-categorization huggingface dataset:
The microscopic differences between the four datasets become evident in the detail views in Figure  1c , f, i, l. Conducted electrolytical Struers A2 etching (source) does not emphasize the sub-grain boundaries and culminates in comparatively slender carbide film appearance overall. In contrast, target 1 (T1) and to a lesser extent target 2 (T2), highlight the hierarchical structure in the lath-bainite regions (see arrow annotations in Figure  1f , i). However, these two target domains are also accompanied by etching artifacts (see ellipse annotations). In T2 and T3, as highlighted by the rectangle annotations, some grain boundaries appear faded due to their grain boundary inclination and the progressed etching state. Along with the pronounced contrast and wider carbide films, this renders it evident that T2 was over-etched.
Pre-training datasets The first pre-training dataset used in this study is ImageNet  25. Further, the apparent domain gap between ImageNet and our target datasets motivated us to test an additional pre-training dataset with a smaller domain gap. Therefore, we selected a SEM dataset of nanoscientific objects  26  , which comprises approximately 22k images, non-uniformly distributed in 10 classes: biological, fibers, films and coated surfaces, microelectromechanical systems (MEMS) and electrodes, nanowires, particles, porous sponges, pattern surfaces, powders, and tips. In the following, we refer to this pre-training dataset with NanoSEM. Before using the NanoSEM dataset, a pre-processing cleaning step was performed. Images with many burned-in measurement annotations were discarded and SEM annotation bars were cropped to avoid spurious correlations between annotation bars and class predictions, known to occur otherwise.
Pre-training & fine-tuning procedure for domain generalization experiments For the ImageNet pre-training, we used pre-trained weights provided by the python package Segmentation models pytorch  30. Concerning our self-performed NanoSEM pre-training, we passed the U-Net encoder output to an auxiliary classification head  30. This head consists of a global average pooling layer, followed by 50% dropout and a linear layer with a sigmoid activation. The auxiliary classification head facilitates encoder training on classification datasets. For NanoSEM pre-training, ImageNet weights were used as an initialization for the trainings, making this process a two-step pre-training (from ImageNet to NanoSEM to the final task). The 18,750 NanoSEM images were split into 15k images for training and 3,750 images for testing purposes.
Pre-training used an Adam optimizer with a constant and encoder-layer independent learning rate and was run for 100 epochs. The obtained pre-trained models were transferred to the segmentation task by just copying the weights of the model performing best on the pre-training task. The full model was then fine-tuned on the source dataset (without frozen layers) with a reduced learning rate for the pre-trained encoder (10× lower than the decoder learning rate). The two-stage pre-training process along with fine-tuning is summarized in Figure  2. Please note that the individual learning rates applied at the pre-training and fine-tuning stage are of major importance. An optimization of the learning rate used for pre-training on NanoSEM has been carried out and is given in the Supplemental. In case of the VGG16 U-Net model, aside from random initialization, either the two-stage pre-training or ImageNet pre-trained weights were used as initial conditions before pursuing fine-tuning to the source domain.
Unsupervised domain adaptation -An adversarial framework To additionally take advantage of unlabeled target domain data, which often can be created in abundance effortlessly, the UDA method of Tsai et al. 7  has been implemented. This adversarial framework can be used to train a semantic segmentation unsupervised domain adaptation task. It is based on the original idea proposed by Ganin et al. 20  The code of  7  was adapted to make it compatible with our data. Figure  3  depicts the training process in a simplified fashion. As proposed in the original framework  7  , we utilize a DeepLabv2 with a ResNet-101 as the segmentation architecture, which facilitates comparability with the corresponding domain generalization experiments described at the end of Section 2. 4. 2. In the UDA framework, annotated source and unannotated target domain data are fed into this segmentation model (shared weights).
Data augmentation Data augmentation is a common practice in ML for increasing the labeled data amount without additional annotation cost. It consists in applying transformations to the data before passing it to the model. The main objective is to render the network invariant to specific transformations. As part of this work, a simple flip and 90°rotation pipeline (with probability 0. 5) was first tested (marked as basic). Moreover, an extended pipeline making use of, amongst others, elastic transformation and optimized for the source domain in our last publication  5  was implemented (marked as extended). Both pipelines have been built with the Albumentations package  31. Full details about the pipelines are given in the Supplemental. Data augmentation was applied to train all our models except for the NanoSEM pre-training.
Supervised-learning on the source domain Models were trained on the source dataset in a supervised fashion. Five-fold cross-validation was implemented to investigate the influence of different training data configurations. The VGG16 U-Net architecture has been used to test aforementioned pre-training and data augmentation settings in order to maximize the phase segmentation task performance. A batch size of 12 with a constant learning rate of λ = 5E -3 was utilized. The models were trained for 200 epochs, except for experiment S. 3, which required more iterations due to the random initialization coupled with the extended augmentation pipeline (increased data variance). This experiment was thus extended to 400 epochs. Additionally, a ResNet-101 DeepLabv2 model was trained in a supervised manner.
This segmentation architecture corresponds to the one used in the UDA framework, rendering the results comparable. For this architecture, only the pre-training on ImageNet was implemented. The models were trained for 550 epochs with an initial learning rate of λ = 1E -3 and a polynomial decay with a decay factor of 0. 9. Note that a larger epoch number was required for this architecture because of ResNet-101's higher number of model parameters. The mentioned training epoch numbers were chosen to ensure proper convergence of the validation mIoU. The results are given in Table  2. The results in Table  2  demonstrate that pre-training helps the model in this low-data regime. Indeed, we observe the systematic trend that the two-stage NanoSEM pre-training outperforms the ImageNet one, which in turn surpasses random initialization, regardless of the used augmentation pipeline.
Similarly, the extended augmentation pipeline consistently grants better results than the basic one, which in turn compares favorably to the unaugmented result. Thus, the best performance is observed with the NanoSEM pre-training and the extended data augmentation pipeline, reaching 80. 2% mIoU as an average over the five folds (S. 9). Experiment S. 10 consists in the supervised training of the DeepLabv2 architecture on the source domain. Concerning the ImageNet pre-training case, DeepLabv2 slightly outperforms the VGG16 architecture on this task (cf. S. 10 to S. 6). In the following sections, all mIoU values in Table  2  are utilized as the reference value to compute equivalent DG experiments' RDT metrics. Analogously, model S. 10 serves as the reference for all the UDA-based models.
Model Pre Lastly, the T3 (LOM) dataset has been investigated. Along with the UDA performance, the domain generalization DeepLabv2 model is reported as the sole baseline in Table  5. The VGG16 U-Net models are omitted, considering their poor performance on this target domain. In this case, the large scatter between the five folds' results made any conclusion about the relative domain generalizability of individual models on T3 impossible to draw. While domain generalization seems compromised on this target dataset, a tremendous improvement of 23. 6% mIoU is experienced when using the UDA method. Moreover, the scatter over the five folds is reduced substantially. Considering the few prediction examples in Figure  6 , it is apparent that UDA turns a completely unusable model into a convincing one without requiring any labeled data in the target domain.
Supervised-learning on the source dataset Achieving the results above with this source data set (see Table  2 ) contrasts the common preconception that DL techniques require a large training data quantity. Specifically, satisfactory results on this complex microstructure segmentation task were attained despite using barely more than 100 tiles (27 native SEM images) for training. This can be explained by two factors. First, the data has been acquired in a very repeatable manner. Images of each dataset (i. e. , domain) were drawn from an individual etched specimen, and reproducible imaging conditions were applied among images of the same domain. It has to be underlined that this results in comparatively low intra-domain variance, which might not be representative for large-scale datasets acquired by multiple operators or even different institutions.
Second, the native micrographs exhibiting a high resolution and rich feature density, 27 such images still represent an appropriate learning foundation for our binary segmentation task. The results in Table  2  emphasize that pre-training improves the performance of the trained models, giving up to 1. 5% mIoU improvement between the best random-initialized model (S3) and the best NanoSEM pre-trained model (S9). Moreover, pre-trained weight initialization led to faster model convergence. Even with further training, it was observed that random initialized models did not catch up to the pre-trained ones. Therefore, the dataset is situated in the low-data regime mentioned by He et al. 11  , where pre-training elevates the performance irrespective of training iterations. In addition, the two-step NanoSEM pre-training shows better performance compared to ImageNet in all cases.
In contrast, pre-training solely on NanoSEM resulted in poor model performance (not reported here). These observations are in line with Gonthier et al. 16  , who performed a two-step pre-training process as well to gradually bridge the domain gap between real-word image datasets and artwork datasets. Please note that this gradual pre-training procedure, compared to conventional pre-training, introduces a further learning rate hyperparameter, which is known to affect the final task performance sensitively. Therefore, relatively more learning rate optimization is required for the pre-training and fine-tuning steps. More details on the learning rate variation are given in the Supplemental. This two-stage pre-training on ImageNet and NanoSEM might be called into question considering the slight performance increase over sole ImageNet pre-training (1% improvement from experiment S6 to S9).
However, it has to be emphasized that NanoSEM is far from being the optimal pre-training dataset for our target task. Indeed, it entails the following limitations: • Structures in certain classes such as MEMS, patterned surfaces, and tips contain shape-related features but barely any apparent microstructural ones, making the learned weights possibly sub-optimal for the final task. • The image formation is complicated in SEM and depends on a multitude of settings. Most images in the NanoSEM dataset, depending on the class, were either acquired with an Everhart-Thornley (SE2) detector or in-lens detector. Therefore, concerning the detector class, only the latter portion of the pre-training data matches the acquisition of both source and target datasets. Generally, the SE2 detector exhibits a more pronounced topography sensitivity due to its location and orientation, while the in-lens detector combines surface topography and, to a lesser extent, material contrast.
• NanoSEM represents a classification task. Hence, only the encoder of our segmentation model could be pre-trained. While it can be assumed to be domain gap dependent, there is no quantitative understanding to which extent and how many layers of a segmentation model would benefit from such a decoder pre-training. Conventional pre-training was reported to primarily help the models' first layers to learn general features  10. • For pre-training standards, NanoSEM is comparatively small. Despite these inadequacies, the underlying rationale of utilizing this NanoSEM dataset for pre-training was that high-level characteristics such as noise levels typical image textures can be learned. However, we assume that a more extensive dataset involving a micrograph segmentation task of arbitrary alloy would prove beneficial over NanoSEM.
For instance, The ultra-high carbon steel micrographs collection subset introduced in  9  would have been appropriate if not for its low quantity. A more promising candidate could be the recently published Aachen-Heerlen annotated steel microstructure dataset  32  containing annotated martensite-austenite islands. While this datasets' annotations exhibit a systematic offset at instance boundaries potentially causing adverse effects during learning, such tendencies presumably can be unlearned during fine-tuning. With respect to data augmentation, a systematic increase in performance is observed when applying the two pipelines, which is not surprising considering the low amount of data used for training the models. Lastly, it appears that the DeepLabv2 architecture achieves better results compared to the U-Net one (compare experiments S.
Transferability to other domains Pre-training not only helps improving performances on the source domain but also brings generalizability to the trained models. Table  6  provides the obtained mIoU on the T2 domain when applying the best source domain trained models after 100 and 200 epochs of fine-tuning. The 200 epoch results were already contained in Table  4. Generally, pre-trained models perform better, especially with NanoSEM. Interestingly, when pre-training with ImageNet, models' generalization power decrease between 100 and 200 epochs. It can be assumed that, during prolonged training, weights are tweaked such that very dataset-specific features are progressively replacing general ones. While after 100 epochs, using NanoSEM pre-training achieves the best results, the difference between ImageNet and the two-stage NanoSEM after 200 epochs is marginal.
Moreover, considering the small test sets, it can not be excluded that the T1 test images potentially being easier to predict on average contributes to the better T1 performance. Overall the UDA framework gave a 2. 5% mIoU boost on this target domain (T1. 11 compared to T1. 10). Furthermore, training the UDA framework with T1 as the target domain gave models that perform better on the source domain with 79. 7% mIoU, granting a 0. 3% boost compared to experiment S. 10. Similarly, such small domain gaps led to the same observation in the context of urban images  35. On the other hand, T2 and T3 have broader gaps with respect to the source domain due to stronger etching and different imaging modalities, respectively. Despite the large domain gap of the T3 dataset with the source one, UDA performed substantially better on this dataset than on T2, culminating in a 23.
6% mIoU improvement over the DG experiments, which is mirrored in Figure  6. As a reference, fully supervised training on T3 presented in our last paper 5 achieved 79% mIoU, showing that employing UDA results in a 6% drop only, despite not relying on target labels. This result with respect to bridging modalities is promising and in line with literature where domain adaptation in the medical field was successfully applied to transition between computer tomography and magnetic resonance imaging  21. Note that UDA models trained with T2 and T3 datasets scored 78,8% and 75,8% mIoU on the source domain, respectively, falling short compared to the fully-supervised training (Exp. # S. 10: mIoU = 79. 4%). Indeed, UDA reaches a compromise between source and target, which is detrimental to the source domain when large domain gaps with the target are involved.
This observation confirms that T1, T2, and T3 are gradually increasing the domain shift with respect to the source domain. The difference in UDA performance on the target domain between T2 and T3 could be attributed to the 5× larger data amount available for the latter set (see Table  1 ), where 48 unannotated training tiles for T2 might be insufficient. Another reason could be the phase fraction of the T2 dataset, which is substantially lower than the source dataset. This assumption will be discussed in the following Section 4. 3. Lastly, we consider it unlikely that the SEMs different distortion and noise level characteristics (see Section 2. 1) are causing this difference since the UDA framework can cope with different modalities and corresponding data augmentations were applied. Additionally, to improve over the individually trained UDA models, we also implemented a bagging strategy.
This consists in averaging the predictions of multiple models to give an improved segmentation. In our case, we use the models trained with the five different folds and achieve 85. 9%, 70,2% and 75. 0% mIoU, which results in 1. 2%, 2. 9% and 1. 7% increase over the best results presented in the T1, T2, and T3 result tables, respectively. The larger improvement for T2 can be attributed to its comparatively weak individual classifiers, making the bagging paradigm relatively more profitable. Lastly, as opposed to DG models, which are frequently biased towards a class (cf. Figures  4, 5 ), UDA leads to more balanced models. Consequently, this characteristic improves the estimation of phase fractions or other metrics, which do not require full details of the segmentation mask. As an example, predicted phase fractions on the different target datasets with DG and UDA are reported in Table  7.
Lastly, one should recall that the fairly small test set size renders the results very sensitive concerning any labeling inconsistencies. For these reasons, the impact of intra-rater reliability is presumably elevated. Another constraint was witnessed when trying to bridge the gap between SEM and LOM data (T3). Indeed, the learning process failed when feeding the model with bright-field LOM tiles, motivating us to invert their pixel values. As the segmentation architecture shares weights between source and target data, it might be difficult to learn filters, which perform well on both modalities while keeping internal features independent of the originating domain. Moreover, this AdaptSegNet 7 framework is built on the strong prior assumption that the source and target datasets are sharing the same label space distribution.
This poses a boundary condition for the segmentation model to give good predictions and fool the discriminator simultaneously. In case of pronounced label space deviations between the source and target datasets, the discriminator should hypothetically quickly learn how to differentiate the segmentation masks, hampering the transfer learning process. Such a label distribution shift could be due to different phase morphology or phase fractions. For this purpose, the phase fractions are provided in Table  1. Aside from generally low data quantity in the T2 domain, the lath bainite phase was not oversampled during image acquisition as opposed to the other domains. Therefore, selecting a suitable training tile subset to match the source domains' 53% mean lath-bainite phase fraction was unfeasible.
Taking the phase fraction histograms of the training tiles (based on expert-reviewed pseudo-labels) into consideration, the discriminator seemingly should learn the tendency that the images from the T2 domain show generally a smaller lath-bainite content. Initially, we considered this to be the primary reason for UDA being more beneficial for T3 compared to T2. However, an additional experiment where we varied the lath-bainite phase fraction of the LOM training dataset provided to the UDA framework invalidated this hypothesis. Specifically, we sampled another target 3 v train set with a lower mean phase fraction (φ train,3 v = 0. 28 similar to T2) and trained a model with it in the UDA framework. A common test set exempt from both LOM training sets was created for testing purposes consisting of 12 test images with a mean phase fraction of 40%.
This LOM test set phase fraction was chosen to be the average between the training phase fractions such that the influence of target train-test shifts could be excluded and domain phase fraction shifts during training could be investigated. The results showed that models trained with T3 reached 78. 0 ± 1. 0 mIoU, whereas those trained with T3 v scored 77. 4 ± 1. 8. Note that these results deviate from the results provided in Table  5  due to the distinct test set. This suggests that distinct phase fraction training data of the different domains does not hamper the UDA training. One thing to underline is that the discriminator receives predictions rather than actual annotations, which complicates the distinction based on phase fraction histogram separability. Nevertheless, the robustness of this adversarial process concerning space label non-conformity is auspicious for materials science tasks.
7  as a reference and starting point  [36] [37] [38] [39] [40]. All these studies rely on the GTA5-to-Cityscapes reference task, and some of them exploit specific characteristics of these datasets  37. Therefore, the approaches in these works are not directly applicable for our binary segmentation task but potentially relevant for other material science tasks. Nevertheless, some models could potentially improve over AdaptSegNet in our setting. One example is the ADVENT model of Vu et al. 38  , which makes use of entropy maps instead of segmentation maps as input for the discriminator. It encourages entropy minimization in the target domain by matching the source and target entropy distributions. This entropy minimization paradigm is borrowed from semi-supervised learning. Also inspired by semi-supervised techniques, Pan et al.
Conclusion and Outlook Amidst others  15, 16  , this study validates the benefit of pre-training when a low-data scenario is encountered. In this context, we show that a two-stage pre-training using ImageNet and an in-field SEM dataset also improves the generalizability of the trained models across domains. From the results, it is evident that models learn and forget relations, and generalizability is sacrificed for specificity upon prolonged training. The success of pre-training and fine-tuning motivates the demand for publicly available datasets in the material science field. Concerning model transferability, this works' significant contribution lies in the successful application of Tsai et al. 's UDA approach  7  with small datasets, making this technique promising for many applications.