dataesr/scientific-paragraphs-categorization huggingface dataset:
The choice of such an architecture is based on aF trial and error procedure that  we performed in order to find the best model that can handle all types of training databases used in this work. The strategy of selecting the number of hidden layers and the size of the convolution layers will be described in a future paper. We decided to do all our tests using the ML platform TensorFlow⁶ with the Keras⁷ interface. The reason is that these two options are open-source and written in Python. Although the calculation time is an important parameter constraining the choice of a network, we have decided not to take it into consideration while selecting the optimal network. The reason for that is that the calculation time depends mainly on the network's architecture which is not discussed in this article.
Two parameters are also constraining the calculation time, the number of epochs, and the batch size (related to the size of the TDB). Calculation time increases with increasing epoch number and decreases with increasing batch size. The main goal of this work is to find the optimal configuration for the parameters independently of the calculation time and the Network architecture. As a rule of thumb, using a Database of 70,000 spectra and 50 eigenvectors, it takes around 17 h to run the CNN over 2,000 epochs using 64 batches and a Dropout of 30%. These calculations are done on a Intel Core i7-8750H CPU × @ 2. 20 GHz 6 CPU.
Data augmentation Data augmentation is a regularization technique that increases the diversity of the training data by applying different transformations to the existing one. It is usually used for image classification (Shorten and Khoshgoftaar 2019) and speech recognition (Jaitly and Hinton 2013). We tested this approach in our procedure in order to take into  account some modifications that could occur in the shape of the observed spectra due to a bad normalization or inappropriate data reduction. We also took into account the fact that observed spectra are affected by noise and that the learning process should include the effect of this parameter. For each spectrum in the TDB, five replicas were performed. Each of these five replicas has different amount of flux values but they all have the same stellar labels T eff , g log , [M/H], and v i sin e.
Pooling Pooling layer is a way to down sample the features (i. e. reducing the dimension of the data) in the database by taking patches together during the training. The most common pooling methods are the average and the max pooling  Zhou and Chellappa (1988). The average one summarizes the mean intensity of the features in a patch and the max one considers only the most intense (i. e. highest value) value in a patch. The size of the patches and the number of filters used are decided by the user. The standard way to do that is to add a pooling layer after the convolutional layer and this can be repeated one or more times in a given CNN. However, pooling makes the input invariant to small translations. In image detection, we need to know if the features exist and not their exact position. That is why this technique has shown to be valuable when analysing images  (Goodfellow et al.
Loss functions The loss function controls the prediction error of an NN as explained in Section 4. It is an important criterion in controlling the updates of the weights in an NN, mainly during the backward propagation. The selection of the type of the loss function is decided depending on the types of output labels. If the output is a categorical variable, one can use the categorical crossentropy or the sparse categorical crossentropy. If we are dealing with a binary classification, binary crossentropy will be the normal choice for a loss function. Finally, in case of a regression problem like the one used in stellar spectra parameters determination, variants of mean squared error loss functions are used. In our work, we have tested the following functions: -Mean squared error: ( ) ∑ - = y y N i N i i 1 1 2 -Mean squared logarithmic error: ( ) ∑ = + + log N i N y y 1 1 1 1 ˆ2 i i -Mean absolute error: | | ∑ - = y y N i N i i 1 1 y being the actual label, y ˆthe predicted ones, and N the number of spectra in the training dataset.
Batches Instead of passing the whole training dataset into the NN, we can divide it in N Batches batches and iterate on all batches per epoch. In that case, the number of iteration will be the number of batches needed to complete one Batches are used in order to avoid the saturation of the computer memory and the decrease of iterations speed. However, the selection of the optimal batch number is not straightforward. Adopted values are usually 32, 64, or 128  (Keskar et al. 2016). One of the most important measures of the success for a deep neural network is how well it generalizes on some test data, not included in the training phase. In current deep neural networks, the loss function has multiple minima. Many experimental studies have shown that, during the training phase, the path to reaching a minimum is as important as the final value  (Neyshabur et al.
Effective temperature According to Figure  5 , the use of a relu or elu activation functions leads to a similar conclusion within a difference of few percents. And this could be applied independently of the number of epochs. As for the Optimizer, Adam and Adamax optimizers seem to be consistently accurate across all epoch numbers. The optimal number of batches is found to be between 32 and 64. The number of epochs is tightly related to the batch number, however, in case of 64 batches, the optimal number of epochs is found to be 2,000. The dropout factor is, as introduced in Section 4. 1. 5, a regularization technique that avoids overfitting. This means that the optimal value depends on the size of the training database. In the case of our 40,000 sample database, the optimal dropout is found to be between 10 and 60%.
Of course, the search will depend on the size of the training database, the spectral region, the spectral-type, the resolution, and so on. The optimal configuration that we found for T eff cor- responds to the following parameters: Activation function: relu. Optimizer: Adam. Batches: 64. Epochs: 2,000. Dropout: 30%. Loss function: mean squared logarithmic error. Kernel initializer: he_normal.
Database size and the role of augmentation In order to check the dependency of the performance of the CNN on the size of the training set, three databases are used. The first database (TDB1) contains 25,000 random synthetic spectra as explained in Section 5, the second database (TDB2) contains 40,000 random spectra, and the third (TDB3) contains 70,000 spectra, resulting from the same TDB1 parameter ranges. We have also checked the importance of using Data Augmentation as a regularization technique for deriving accurate parameters (see Section 4. 1. 1 for details). For each stellar parameter, we used the optimal CNN with the configuration that was derived in Sections 5. 1-5. 4. Each configuration was tested with TDB1, TDB2, and TDB3 with and without Data Augmentation. Figure  9  displays the average relative standard deviation for each stellar parameter with respect to the maximum values, for the training, validation, test, and observation sets.
In order to quantify these proxies for the uncertainties of the techniques, Table  3  collects the standard deviations for the four stellar parameters as a function of the training database. According to Table  3 , each parameter behaves differently with respect to the change of the databases. This is mainly due to the number of unique values of the parameter in the database. For that reason, [M/H] is well represented by TDB1 without data augmentation, whereas T g log eff and v i sin e require a larger database to be well represented. g log can be well represented with TDB3 with data augmentation, whereas T eff can be predicted with TDB2 with data augmen- tation. Finally, v i sin e can be predicted using TDB3 with data augmentation.
Accuracy for the optimal configuration After selecting the optimal configuration for each stellar parameter, the predicted parameters are displayed in Figure  10  as a function of the input ones for the training, validation, and the two sets of test datasets. All data points are located around the = y x line. The dispersion of the observation around that line is due to spectra with very low signal to noise. The accuracy that we found using our CNN architecture seems to be appropriate for A stars as they are comparable to most of the previous studies using classical tools  (Aydi et al. 2014)  or more complicated statistical tools  (Gebran et al. 2016 , Kassounian et al. 2019 ). The same is true for all parameters. In order to verify the effect of the noise on the predicted parameters, Figure  11  displays the variation of the accuracy of the predicted values with respect to the input SNR.
Extrapolating to other spectral-types In order to verify how universal the results are, we checked that the optimization of the code is not dependent on  wavelength and/or spectral-type, we also tested the procedure on FGK stars. To do that, we have calculated a TDB specific for FGK stars using the parameters displayed in Table  4. The wavelength range was selected to coincide with the one of  Paletou et al. (2015a). This range is sensitive to all the concerned stellar parameters. A database of 50,000 random synthetic spectra with known stellar labels is used in the training. About 20,000 test data, with and without noise, were calculated in the same range of Table  4  to be used for verification. The optimal NNs that were introduced in Section 5 were used again, as a proof of concept, for the FGK TDB.
This means that for a specific network and a specific stellar parameter, a study should be made to find the optimal configuration of hyperparameters. This is due to the contribution of the specific stellar parameter on the shape of the input spectrum. Using the PCA decomposition, we have reduced the size of the input parameters to only 50 points per spectrum while keeping more than 99. 5% of the information. This is recommended in case of large databases and wide wavelength range and could avoid the use of extra pooling layers in the network. This projection technique is not only applicable for AFGK stars but can also be used for cooler stars. Although the CNN architecture was not optimized, we were able, using a strategy of finding the best hyperparameters, to reach a level of accuracy that is comparable to other adopted techniques.
[M/H] = 0. 0 dex as well as the extra five modifications that were performed on this spectrum. We have decided to use a continuous SNR between 5 and 300 but different modifications could be tested. As an example, González-Marcos et al. (2017) adapted the SNR of the spectra used in the training dataset to the SNR of the spectra for which the atmospheric parameters are needed (evaluation set). They concluded that in case of T eff , only two regression models are needed (SNR = 50 and 10) to cover the entire SNR range.
For ARMADA our main interest is differential astrometry between two components of a binary system within the interferometric field of view (both components unresolved). Since closure phase is immune to atmospheric effects, and extra dispersion in differential phase can be fit with a polynomial, we are able to observe for ARMADA without the use of traditional calibrators by fitting to closure and differential phase as described in  Gardner et al. (2021). Our observational setup and calibration methods follow the procedures outlined in that paper. We used the MIRC-X data pipeline (version 1. 3. 3) to produce OIFITS files for each night, described in  Anugu et al. (2020). This pipeline and its documentation are maintained on Gitlab. 12 These data were reduced with the "spectral-differential" method of the MIRC-X pipeline for computing differential phase.
GRAVITY at VLTI To cover southern sky targets for the ARMADA survey, we use data from the GRAVITY instrument at VLTI  (Gravity Collaboration et al. 2017). ARMADA data are taken in service mode under the large program ID 1103. C-0477. For our service-mode program, GRAVITY combines the four 1. 5 m ATs in the K band with baselines up to 130 m. We utilize the large, astrometric, and medium configurations to best constrain the astrometry of our wide (∼100-200 mas) binaries. Our data are recorded in GRAVITY's high-resolution mode, with R ∼ 4000. This high spectral resolution is needed to achieve our desired wavelength knowledge for precision astrometry. Our data are taken with a Detector Integration Time (DIT) of 30 s, which prevents signal loss due to the fast-varying phase and visibility changes of wide binary stars.
Each pointing has eight frames (NDIT = 8), with an observing sequence OOSOOOSOOO (O = object, S = sky). With this sequence a single epoch on an ARMADA binary is completed in just under an hour when including overhead time. To make our large number of target observations feasible, we do not use calibrator stars (just as for the MIRC-X/CHARA data). Hence, our binary astrometry is again largely set by the variation of signal with wavelength for the closure phase. We used the standard reduction pipeline for GRAVITY data, 13 version 1. 5. 0, along with the suite of Python tools developed by the GRAVITY team for running the pipeline. When reducing the data, we set the maximum number of frames to 2, which means that each measurement of phase and visibility is 60 s of integration. Since the wavelength variation of visibility and phase sets the binary astrometry, we are able to fit directly to the uncalibrated closure phase.
This choice has no effect on the fitted astrometry within our measurement error bars. As described in  Gardner et al. (2021) , to find our best differential astrometry solution on a given night, we first perform a wide grid search in R. A. and decl. with step sizes of 0. 1 mas to find the minimum χ 2 solution. We then use the lmfit package in Python to narrow in on the best solution with a nonlinear least-squares fit  (Newville et al. 2016). We convert our astrometry solutions from differential R. A. and decl. to a separation and position angle (PA) east of north pointing from the primary to secondary (ρ, θ). After finding the best-fit ρ and θ, we need to compute the errors in position. In general, it is very difficult to accurately estimate the astrometric errors of individual data points.
Orbit Fitting Once we measure our binary positions for each night, we perform a Keplerian orbit fit to our data with our Python routines described in  Gardner et al. (2021). The Campbell elements (ω, Ω, e, i, a, T, P) describe the Keplerian motion of the brighter star of a binary system relative to the other and relative to the observer. Those symbols have their usual meanings, where ω is the longitude of the periastron, Ω is the PA of the ascending node, e is the eccentricity, i is the orbital inclination, a is angular separation, T is a time of periastron passage, and P is the orbital period. When including RV data, we also fit the semiamplitudes K and systemic velocity γ. The longitude of periastron ω is traditionally reported for the secondary when fitting to visual binary orbits alone.
The convention when combining RV orbits is to report ω of the primary, which is flipped by 180°. When we have RV data included in the orbit, we report the ω of the primary for these orbits. For purely visual orbits, there is a 180°ambiguity between ω and Ω. In these cases we report the case where Ω < 180°. When performing fits, we convert these parameters to the linear Thieles-Innes coefficients as described in  Wright & Howard (2009). We again use the lmfit package in Python for nonlinear least-squares fitting. When fitting a system of three or more components for ARMADA, we assume that the system is hierarchical with the wide companion orbiting the center of mass of the inner pairs. This means that our orbit model is simply a sum of the outer plus inner Keplerian orbits. Most of the outer orbits we present in this paper are significantly larger than the inner orbits (>200 times larger in orbital period, with the exceptions of HD 220278 and HD 185762), so this hierarchical model is a reasonable assumption.
However, this time-consuming procedure emphasizes the demand for frugal models in terms of labeled training data. Moreover, this work draws the focus on model transferability to a target domain in the context of low source data availability and unavailability of target domain annotations. These are typical boundary conditions in materials science. As a first step, having trained a model on a source dataset, we want to give some insight into the possibility of adopting it to a target domain (domain generalization experiments). In this case, we investigate whether pre-training with domain-extrinsic datasets and subsequent fine-tuning to the source domain can improve generalization to target domains in the low data regime. To this end, we compare different pre-training datasets with models trained from random initialization.
Related work Instances where deep learning was applied to solve materials science tasks, are fairly scarce. For instance, Azimi et al. 8  perform a microstructure segmentation task on SEM images of dual-phase steels using a fully convolutional deep neural network coupled with a super-pixel voting approach. Holm et al. investigate various tasks, amongst others microstructure segmentation, on their ultra-high carbon steel (UHCS) dataset using an end-to-end deep learning approach  9. Recently, Thomas et al. 4  published a study implementing a U-Net architecture for damage segmentation, detecting fatigue-induced extrusions and cracks on steel micrographs.
Fine-tuning pre-trained models When training with little data, the first prevalent practice in DL is fine-tuning a pre-trained model  10. This procedure is often used synonymously with transfer learning even though latter now encompasses many further methods. Pre-training a neural network is based on the assumption that its first layers learn similar features regardless of the trained task. Indeed, these layers act as feature extractors, detecting edges, corners, colors, or blobs. Thus, carrying over the weights of a model pre-trained on a large-scale dataset and fine-tuning them on an another task reduces the demand for training data in the latter task while accelerating convergence. Different strategies exist, ranging from full model fine-tuning with a small learning rate to freezing the initial layers of the model and fine-tuning only the last ones.
Utilizing readily available model weights from pre-training with ImageNet, owing to its apparent size and richness, has become the status quo for weight initialization in deep learning. Recently, He et al. 11  questioned the undifferentiated ImageNet weight usage, showing that it only helps in faster convergence but does not enable performance improvement. However, the low-data regime, i. e. , when little data is available for the final task, was shown to be exempt from this finding. Specifically, pre-training on ImageNet culminated in improved COCO object detection  12  only when less than 10k images were used for fine-tuning. However, in material science, this low-data regime is a lasting attendant circumstance of both materials diversity and annotation complexity, making pre-training a suitable strategy to infer robustness and to raise the performance of the trained models  13.
Nevertheless, pre-training dataset selection remains an unsettled issue. In other domains, this has been the subject of recent research  14, 15. The overarching trends seem logical -using a large pre-training dataset as close as possible to the target task dataset proves beneficial. However, the trade-off between data quantity and domain gap is undefined. Cheplygina et al. 14  condensed 12 studies from the medical field in a detailed review, comparing ImageNet with smaller in-field datasets. The conclusions are not consistent, and no clear trend could be identified. Romero et al. 15  studied the effect of different pre-training for chest X-ray radiographs classification, varying the number of target training images from 50 to 2,000. Pre-training was conducted using ImageNet or one of two X-ray datasets (220k chest images and 40k images of various body parts).
They show that pre-training is always helping in their low-data regime and emphasize that the chest dataset pre-training yields the best results. However, their miscellaneous body parts radiography dataset only performed comparably to ImageNet, showing the trade-off between data amount and domain gap for choosing a pre-training dataset. Gonthier et al. studied similar aspects for artwork classification, where pre-training on ImageNet outperformed another artwork pre-training dataset containing around 80k images, presumably ascribed to the low quantity of the artwork pre-training dataset  16. Interestingly, a gradual two-step pre-training, using first ImageNet and then the intermediate artwork dataset, led to further improvement. This suggests that successive pre-trainings could help to bridge domain gaps continuously.
Unsupervised domain adaptation Unsupervised domain adaptation (UDA) describes training a model with labeled data from a source domain and unlabeled data from a target domain to perform well on the latter. This is of major interest when the target domain labeling process is costly or when source domain annotations are readily available. One very good example is the well-known GTA5  17  or SYNTHIA  18  (source) to Cityscapes  19  (target) task. In both cases, source datasets provide synthetic and inherently annotated urban landscapes. A wide branch of the UDA methods is now improving an adversarial learning framework, which was initially proposed by Ganin et al. 20. The underlying idea is to force the model to adopt a domain-independent feature representation. This is achieved by passing images of both domains into the main model and feeding intermediate layer feature representations into a discriminator.
Dataset processing and descriptions Processing. The LOM images were assimilated to the native SEM datasets in terms of physical pixel size and field of view by scaling and cropping operations, cf. Figure  1a  and 1j. Subsequent processing was identical for all datasets. Images were mirror-padded to make them square so four tiles can be extracted per image. Following the results of our last paper  5  , the images were downscaled by a factor ×0. 5 before tiling, effectively increasing the context passed to the models' receptive field  23. The obtained training tiles are then of size 636×636 (512×512 plus an additional 62 pixel overlap extending into the adjacent tiles at each tile border). This overlap-tile strategy was introduced by Ronneberger et al. 24  to cope with memory restrictions and helps the model to circumvent tile border effects.
For segmentation loss computation, the overlap regions extending in neighboring tiles were discarded (i. e. , 512×512 center region was used solely). To align the domains' phase fractions as much as possible, few training tiles of the target domains were discarded, ultimately resulting in the tile numbers and phase fractions listed in Table  1. The source dataset was split in five folds for cross-validation. In contrast, the annotated portion in the target domains was too small to perform cross-validation. Therefore, only a single train and test set has been built for the target datasets. While training was performed with tiles, the model evaluation was conducted on full images. Bright-field LOM images (T3) pixel values were inverted to give them a dark background similar to SEM. Descriptions.