dataesr/scientific-paragraphs-categorization huggingface dataset:
Another approach is to align the feature distributions directly by minimizing a divergence measure. Choices for divergence measure include maximum mean discrepancy (MMD)  [23] , correlation alignment  [32] , or optimal transport  [33, 34]. Recently, JUMBOT  [34]  achieves stateof-the-art UDA results by using mini-batch unbalanced optimal transport to minimize the domain discrepancy of joint deep feature and label distributions. While the aforementioned methods achieve high performance on computer vision datasets, they do not handle the temporal dimension of SITS data.
AM Score A shortcoming of IS is that the highest score is achieved when p θ (y) is uniform  [47] , which corresponds to an even distribution of classes in the target domain. For SITS, where the class distribution is often highly imbalanced, this may cause IS to estimate a suboptimal shift. AM  [46]  addresses this issue by taking the actual class distribution C t into account: AM(δ t→s , θ, C t ) = E (x t ,τ t ) H p θ y|(x t , τ t + δ t→s ) + D KL (C t p θ (y)). ( 6 ) Algorithm 1: EstimateTemporalShift AM consists of two terms: the first term is an entropy term on the conditional distribution, and the second is the KLdivergence between the underlying class distribution C t and the marginal distribution. Lower values of AM indicate a better δ, as the model is confident in its predictions, and the actual class distribution of the data matches the predicted distribution of classes.
The TimeMatch Dataset The TimeMatch dataset  [29]  contains SITS from Sentinel-2 Level-1C products in top-of-atmosphere reflectance. Four Sentinel-2 tiles are chosen in various climates: 32VNH (Denmark), 30TXT (France), 31TCJ (France), and 33UVP (Austria). A map of the tiles is shown in Figure  3. We use all available observations with cloud coverage ≤ 80% and coverage ≥ 50% between January 2017 and December 2017. Figure  4  shows the resulting acquisition dates for the four tiles. We leave out the atmospheric bands (1, 9, and 10), keeping C = 10 spectral bands. The 20m bands are bilinearly interpolated to 10m. For ground truth data, we retrieve geo-referenced parcel shapes and their crop type labels from the openly available Land Parcel Identification System (LPIS) records in Denmark 1 , France 2 , and Austria 3.
Thus, the only difference between VRADA, CoDATS, and the DANN approach mentioned here is the architecture of the feature extractor, which in our case is based on the temporal feature extractor of PSE+TAE. PAN  [20]  learns domain-invariant features for crossregion adaptation by minimizing the MMD loss for features extracted by an RNN and self-attention-based crop classifier. Unfortunately, we were unable to gain access to the source code of PAN for comparison. As an alternative, we include the MMD comparison, which is similar to PAN, except the crop classifier is changed to PSE+TAE.
Implementation Details All experiments are implemented in PyTorch  [51]  and trains on a single NVIDIA 1080 Ti GPU. Our implementation is based on the source code of PSE+TAE  [50]. Pre-training on the source domain. To train models with the labeled source domain, we follow the original approach of PSE+TAE  [12]. We train for 100 epochs with the Adam  [52]  optimizer with an initial learning rate of 0. 001 and we decay the learning rate using a cosine annealing schedule  [53]. We use weight decay of 0. 0001, batch size 128, focal loss γ = 1. Inputs are normalized to [0, 1] by dividing by the max 16-bit pixel value 2 16 -1. The best source-trained model is selected using the source validation set. We augment the inputs by randomly sub-sampling 30 time steps and 64 pixels during training. The same setup is used for the target-trained model, using the target domain instead.
For the final evaluation, we do not sample time steps or pixels, and instead input all available time steps and pixels to the model. This ensures the test results are deterministic, and also slightly improves results by providing all available data to the model. TimeMatch. We use the same training setup as the sourcetrained model but instead train for 20 epochs with a lower initial learning rate of 0. 0001. We define an epoch as 500 iterations to fix the frequency in which the temporal shift is re-estimated. We use maximum temporal shift ∆ = 60 days, as we did not observe shifts greater than 2 months for our dataset in Europe. We set the trade-off hyperparameter λ = 2. 0 in Eq. 13, EMA keep-rate α = 0. 9999, and pseudo-label threshold τ = 0. 9. A sensitivity analysis of these hyperparameters is provided in Section 5.
This ensures the batch normalization  [57]  statistics are calculated separately for each domain, for improved adaptation. Existing UDA Methods. We re-implement the competitors MMD, DANN and CDAN+E following the domain adaptation library in  [58] , and JUMBOT from the original source code  [34]. All methods are initialized from a source-trained model, train for 20 epochs, similar to TimeMatch. We also tune their hyper-parameters on the task 32VNH→30TXT. The full details of the implementation and chosen hyperparameters can be found in our source code.
Main Results Table  1  shows the performance obtained with our approach and the re-implemented baselines and competitors. We report the mean and standard deviation of macro F1 scores, calculated from the results of three runs with different dataset splits. We observe a source-trained model transfers very poorly to a different target region, with an average loss of 43% in F1-score on target data, which strongly motivates UDA. However, existing UDA methods only slightly increase the performance of the source-trained model when applied to SITS. In comparison, our approach significantly outperforms all existing methods (+11% on average). This shows that addressing the temporal shift is crucial for the cross-region adaptation problem of SITS. The results of the target-trained model are the highest achievable performance if target labels were available.
Ablation Study To better understand how TimeMatch is able to obtain state-of-the-art results, we perform an ablation study on its components for the task 32VNH→30TXT. We report the results in Table  2. We first study the impact of the EMA training. Instead of the EMA, we set the teacher as a direct copy of the student (No EMA). We observe that training without the EMA introduces a significant drop in F1-score-though the performance is still better than the competing methods. Setting δ s→t = 0 disables the temporal shift of the source domain, and the student is trained with datasets with increased domain discrepancy. We observe a significant decrease in F1-score as a result. Disabling the balanced mini-batch sampler for the source domain also leads to a degradation of the performance. If the model is trained with class imbalanced source data, the teacher will make biased pseudo-labels for the samples from the target domain  [60].
Conclusion This paper presented TimeMatch, a novel cross-region adaptation method for SITS. Unlike previous methods that solely match the feature distributions across domains, TimeMatch explicitly captures the underlying temporal discrepancy of the data by estimating the temporal shift between two regions. Through TimeMatch learning, we adapt a crop classifier trained in a source region to an unlabeled target region. This is achieved by a learning algorithm that unifies temporal shift estimation with semisupervised learning, where pseudo-labels are generated for unlabeled, temporally shifted target data to train the classifier for the target region. Lastly, we presented the TimeMatch dataset, a new large-scale cross-region UDA dataset with SITS from four different regions in Europe. Evaluated on this dataset, TimeMatch outperforms all existing approaches by 11% in F1-score on average across five different adaptation tasks, setting a new state-of-theart in unsupervised cross-region adaptation.
Selection of instrumental variables In order to ensure that preselected ancestry informative markers fulfilled the first MR assumption (relevance), we used the software ADMIXTURE (version 1. 3) for supervised estimation of the individual proportions of Mapuche ancestry in sample I, relying on the reference panel of Aymara, Mapuche, European and African individuals  (Alexander et al. , 2009) , and retained as instrumental variables (IV) the genetic variants that showed a robust association (p < 5×10 -8 ) with the estimated proportion of Mapuche ancestry, adjusted for age and sex. The second MR assumption (independence) was assessed by a phenome-wide association study (PheWAS), excluding genetic variants associated (p < 5×10 -8 ) with potential confounders (menopause, educational level, diabetes, body circumference, smoking, alcohol consumption or gallstones) in MR-Base  (Hemani et al.
Two-sample MR, sensitivity and mediation analyses (two-step and multivariable MR) Using the previously selected IV and the respective sample I and sample II as described above, we performed MR on the association Mapuche ancestry → GBC risk as the primary analysis. In addition to radial MR, we (1) visually inspected the MR analyses were conducted using the R version of MR-Base, which provides convenient tools for the harmonization of the association statistics, including standardization of the effect alleles and removal of problematic palindromic genetic variants  (Hemani et al. , 2018). The R package 'RadialMR' was used for radial MR  (Bowden et al. , 2018). For general data processing we used Plink version 1. 9  (Purcell et al. , 2007) , and the R software environment for statistical computing and graphics (version 3.
describes the performed analyses with some intermediate results. Preselection of markers of Mapuche ancestry in a reference panel composed of Native American Mapuche (n = 28), Native American Aymara (n = 63), European (CEU and IBS, n = 206) and African (YRI, n = 108) individuals relying on the informativeness for assignment measure yielded 21,854 candidate markers. Of these, 985 genetic variants were robustly associated (p < 5×10 -8 ) with the individual proportion of Mapuche ancestry, which was previously estimated in a dataset with genome-wide genotype data from 1,861 genetically admixed Chileans. Exclusion of markers associated with established GBC risk factors (PheWAS p < 5×10 -8 ) complemented with LD pruning (r 2 < 0. 01) resulted in 430 instrumental variables preliminarily retained for the subsequent MR analyses.
Gallstone patients were patients who underwent cholecystectomy due to symptomatic gallstones. Populationbased controls were selected from the Chilean subgroup of CANDELA and from Chilean studies on COPD and Chagas disease, with GBC and gallstone incidences representative of the general Chilean population (Blandino et al. , 2022). Sample I and sample II for MR analyses on Mapuche ancestry → GBC risk and Mapuche ancestry → gallstone disease partially overlapped: 84 of the 412 controls and 91 of the 351 controls in the respective sample II also belonged to sample I. Sample II for MR analysis of the association Mapuche ancestry → BMI was based on 12,216 individuals from the Hispanic Community Health Study/Study of Latinos (HCHS/SOL, dbGaP accession number phs000810. v1. p1).
Since a causal effect of gallstones and BMI on GBC risk has recently been reported (Barahona Ponce et al. , 2021) , we investigated the mediating effects of gallstone disease and BMI as a surrogate marker for obesity, as potential mechanisms explaining the relationship between Mapuche ancestry and GBC risk. The unavailability of large prospective Chilean datasets with complete information on GBC, gallstone, BMI and individual genotypes precluded the implementation of formal mediation analyses, and we decided first to apply two-step MR to assess mediation (Zheng et al. , 2017). In the first MR step, IV for the proportion of Mapuche ancestry were used to assess the causal effect of the Mapuche ancestry proportion on the potential mediator (gallstone disease or BMI) by MR as described above. The second MR step was based on published findings, also based on genetically admixed Chileans, on the causal effect of the mediators on GBC risk.
Acknowledgements The authors gratefully acknowledge the data storage service SDS@hd supported by the  Ministry of Science, Research , and the  Arts Baden-Württemberg (MWK)  and the  German Research Foundation (DFG)  through grants  INST 35/1314-1 FUGG  and  INST 35/1503-1 FUGG , and the access to computing facilities provided by  Steven J. Lubbe  at  Northwestern University -Feinberg School of Medicine.
, 2021). The MVMR results, based on a relatively small sample size and weak instrumental variables, showed good agreement with two-step MR and also supported a mediating effect of gallstones between Mapuche ancestry and GBC risk; however, large prospective Chilean datasets with complete information on GBC, gallstones and individual genotype data are urgently needed to conduct formal mediation analyses and accurately quantify the magnitude of the direct and indirect effects of gallstones, which are highly relevant for more precise GBC prevention.
Adaptive optimal smoothing With at hand an estimate of the local regularity ς t 0 = d+H t 0 obtained from a learning set of N 0 curves, we aim at recovering N 1 new noisy trajectories of X from what we call the online dataset. One of the most popular smoother is the local polynomial estimator, see  [5]. This type of estimator crucially depends on a tuning parameter, the bandwidth, which should ideally be chosen according to the unknown regularity of the target function. One has to connect a definition of local regularity that is meaningful from the theory of stochastic processes to the usual definition of function regularity used in nonparametric curve estimation. Fortunately, in our framework, the parameter ς t 0 , which is understood as the local regularity of the process (X t : t ∈ J µ (t 0 )) in quadratic mean, see  (2) , is intrinsically linked with the regularity of the sample paths of the process.
Local polynomial estimation We assume that d ≥ 0 is an integer and H t 0 ∈ (0, 1). Let d and H t 0 be some generic estimators of d and H t 0 , respectively, and let ς t 0 = d + H t 0 be the corresponding estimator of ς t 0 = d + H t 0. We assume that d and H t 0 are independent of the N 1 from the online dataset, generated according to  (1). The estimator of ς t 0 could be used to smooth any curve Y [n 1 ] (n 1 = 1,. , N 1 ) from the online dataset. For the sake of readability, we omit the superscript [n 1 ] and we consider a generic curve from the online dataset: Y m = X(T m ) + ε m , 1 ≤ m ≤ M. For any u ∈ R, we consider the vector U (u) = (1, u,. , u d/ d!). Let K : R → R be a positive kernel and define: ϑ M,h = arg min ϑ∈R d+1 M m=1 Y m -ϑ U T m -t 0 h 2 K T m -t 0 h , where h is the bandwidth.
For simplicity, and without loss of generality we consider the same constant K 1 in Assumptions (LP3) and (LP4). Theorem 2. Assume that Assumptions (H1), (H2), (H4)-(H6) and Assumptions (LP1)-(LP4) hold true and let K(•) be a kernel such that, for any t ∈ R : κ -1 1 [-δ,δ] (t) ≤ K(t) ≤ κ1 [-1,1] (t), for some 0 < δ < 1 and κ ≥ 1. ( 14 ) There then exists a constant Γ 0 such that for any µ ≥ 1, E exp τ (µ) Xt 0 -X t 0 2 1/4 ≤ Γ 0 where τ (µ) = 1 log 2 (µ) µ log(µ) 2ς t 0 2ς t 0 +1. The bound on the exp( √ x)-moment of the | Xt 0 -X t 0 | seems a new result for local polynomial estimators. For our purposes, it will entail a sharp bound for R( X; t 0 ). More precisely, the price for considering a risk measure uniformly over the whole online dataset is very low, that is a multiplying factor as large as a power of log(N 1 ) in the risk bound we derive below.
The log(1 + N 1 ) factor is given by the maximum over the N 1 curves in the online dataset. The factor {log(µ)} 2ςt 0 /(2ςt 0 +1) is due to the concentration properties of M around its mean µ. This factor would not appear if M/µ is almost surely bounded and bounded away from zero. The factor log 2 (µ) comes from probability theory. The trajectories of a stochastic process X with local regularity H t 0 does not necessarily belong to C ςt 0 (J µ (t 0 )) but they are almost surely in any C ςt 0 -(J µ (t 0 )) for any 0 < < ς t 0. Finally, let us notice that Corollary 1 states that the estimator defined by (  7 ) satisfies (H3) for d = 0 and any 0 < H t 0 < 1. This leads us to the following result. Corollary 4. Assume d = 0 and let H t 0 be the estimator of 0 < H t 0 < 1 defined in Corollary 1.
When the variance of the errors also depends on the realizations X u , as described in Section 2. 4, in general it is no longer possible to consistently estimate σ 2 (X t 0 , t 0 ). Our simulation experiments indicate that the estimate (17) remains a reasonable choice. Finally, the constant involved in the definition of the bandwidth could be estimated by C obtained by plugging the estimates of the unknown quantities into the definition of C in  (15). Concerning the kernel, we use K(t) = (3/4) 1 -t 2 1 [-1,1] (t), that is the Epanechnikov kernel for which K 2 = 3/5 and |K(v)||v| ςt 0 dv = 3{(ς t 0 + 1)(ς t 0 + 3)} -1. An implementation of the method is available as a R package on Github at the URL adress: https://github. com/StevenGolovkine/denoisr.
The bandwidth is cal- culated as h n 1 = ( C/M n 1 ) 1/(2 ςt 0 +1) , 1 ≤ n 1 ≤ N 1 , with ς t 0 obtained from Algorithm 1. The constant estimate C is the same for all curves in the online set, that is that obtained with ς t 0 , L t 0 ( H t 0 ) and σ 2 t 0. We compare our approach with the classical crossvalidation (CV) (least-squares leave-one-out) method applied for each curve X [n 1 ] separately. For CV, we use the R package np  [11] , after rescaling the CV bandwidth to account for their different definition of the Epanechnikov kernel. At this stage, we want to point out that our smoothing method is much faster than any standard, trajectory-by-trajectory approach, such as CV. We report a time comparison in the Supplementary Material, and as expected, the ratio between the times needed for CV and for our approach is at least of the same order as N 1.
7, 0. 005). As expected, our local regularity estimation approach also performs well for smooth trajectories. Next, we present the results on the risk R( X; t 0 ). Figure  3  presents the boxplots of the risk R( X; t 0 ) defined in  (11)  in the case of piecewise constant local regularity, with three values of t 0 , each one in the middle of the interval of the changes of regularity are defined. The results are quite good. Part of the curves with lower regularity are harder to estimate and thus results in higher risks than the more regular parts. It appears that N 0 and µ do not have the same influence on the risk as the estimation of the local regularity, and this is in line with the risk bound in Theorem 3. Thus, going from 300 to 1000 sampling points leads to large improvement in terms of risk whereas going from 250 to 1000 curves in the learning dataset only results in little or no improvement.
Real data analysis: the NGSIM Study In this section, our method is applied to data from the Next Generation Simulation (NGSIM) study, which aims to "describe the interactions of multimodal travelers, vehicles and highway systems", see  [10]. This study is known to be one of the largest publicly available source of naturalistic driving data. This dataset is widely used in traffic flow studies from the interpretation of traffic phenomena such as congestion to the validation of models for trajectories prediction (see e. g. [4, 14, 24, 16, 12]  for some recent references). However, such data have been proved to be subject to measurement errors revealed by physical inconsistency between the space traveled, velocity and acceleration of the vehicles, cf. [18]. Montanino and Punzo  [17]  developed a trajectoryby-trajectory four-steps method to recover the signals from the noisy curves, and their methodology is now considered as a benchmark in the traffic flow engineering community for analyzing NGSIM data.
A value of the ratio r( X, X) less than 1 indicates smoothed values closer to the observations. For our illustration, we consider a subset of the NGSIM dataset, known as the I-80 dataset. It contains 45 minutes of trajectories for vehicles on the Interstate 80 Freeway in Emeryville, California, segmented into three 15-minute periods (from 4:00 p. m. to 4:15 p. m. ; from 5:00 p. m. to 5:15 p. m. and from 5:15 p. m. to 5:30 p. m. ) on  April 13, 2005  and corresponds to different traffic conditions (congested, transition between uncongested and congested and fully congested). In total, the dataset contains trajectories, velocities and accelerations for N 0 = 1714 individual vehicles that passed through this highway during this period, recorded every 0. 1s. The number M n of measurements for each curve varies from 165 to 946.
We focus on the velocity variable and rescale the measurement times for each of the 1714 velocity curves such that the first velocity measurement corresponds to t = 0 and the last one to t = 1. Figure  8  presents a sample of five curves from this data. It can easily be noticed that the velocities are quite erratic and their variation is not physically realistic, indicating the presence of a noise. Moreover, the data have been recorded at a moment of the day when traffic is evolving, it goes from fluid to dense traffic. Therefore, we consider that there are three groups in the data: a first group corresponding to a fluid (high-speed) traffic, a second one for in-between fluid and dense traffic, and a third groups corresponding to the dense (low-speed) traffic. To determine the three clusters, we fit a finite Gaussian mixture model to the vector of number of sampling points.
The model is estimated by an EM algorithm initialized by hierarchical model-based agglomerative clustering as proposed by Fraley and Raftery  [6]  and implemented in the R package mclust  [21]. The optimal model is then selected according to BIC. The three resulting classes have 239, 869 and 606 velocity trajectories, respectively. Plots of randomly selected subsamples of trajectories from each groups are provided in the Supplementary Material. The respective numbers of measures M n are plotted in Figure  9. The mean estimates µ obtained in the three groups are 218, 474 and 684, respectively, and the corresponding values K 0 , as defined as in Corollary 1, are 13, 17 and 20. Figure  10  presents the results of the estimation of ς t 0 for values of t 0 from 0. 2 to 0. 8, for each group. The evolution of ς t 0 is quite smooth, except for Group 1 (Figure  10b ).
On the pointwise risk For technical convenience, in our theoretical study, we only considered the case where the regularity estimator ς t 0 is applied with an independent sample. If one wants to smooth the curves in the learning set, one can use a leave-one-out method. That is, for each curve, one can estimate the local regularity without that curve, and smooth the curve with the estimate obtained. Our method for calculating H t 0 is very fast, and such a leave-one-curve-out procedure is feasible. This idea was used to analyze the NGSIM data. However, one could also simply smooth the learning set curves using the same local regularity estimates obtained from this dataset. Figure  5  presents the estimation of the risks R( X; 1/6), R( X; 0. 5) and R( X; 5/6) for piecewise fBm, with constant noise variance σ 2 = 0.