dataesr/scientific-paragraphs-categorization huggingface dataset:
Our algorithm We can finally explain our algorithm, which can be tersely summarized as follows: Make N copies of the original training set. For each copy, replace a fraction of the data with synthetically generated data and perform cross-validation to learn the error-rate vs. w curve. Use the average of all N curves to predict w. This algorithm, outlined in Table  6 , contains a subroutine presented in Table  5. Individual elements are motivated and explained in following subsections. The essence of the method we are proposing is in making N new training sets by using the algorithm in Table  5. These datasets will be used instead of the original training set to learn w. While each dataset may produce a noisy error vs. w curve (as in Fig. 22. right), the average of all such curves will be smoother, and it will more closely resemble the true noisy error vs.
w curve (as in Fig. 23. right). As shown in Table  5 , we begin in line 1 by randomly sampling portion of the original training objects with replacement. These objects will be included in the new training set and will be unmodified. After that, we randomly sample a portion of the original training set again. These objects are then distorted by adding a warping and are appended to the new training set. Using this algorithm, the new training sets have the same number of objects as the original set. Note that the sampling is performed in a stratified manner; otherwise, when working with small datasets, we run the risk of only adding warping to one class and possibly skewing the results. The sub-routine of making new training set is invoked over a number of iterations, as shown in line 2 of the main algorithm in Table  6.
24. middle). It is important to note, as a recent work indicates, that the endpoints can result in misleading DTW distance  (Silva, Batista, and Keogh 2017). Recall that DTW's constraints require it to match the pairs of beginning and end points, even though they may be a poor match. The MATLAB code to add warping in Table  7  contains a small modification of Table  3  to reflect these changes. Even though "without padding" still brings about reasonably good results, findings from our experiments presented in Section 4. 2 confirm that "adding padding" improves the performance. It may be possible to further improve our overall method if we find better ways to make more "natural" synthetic exemplars. We have experimented with several methods to generate synthetic time series  (Chawla et al. 2002; Forestier et al.
Given this, we should clearly err on the side of adding more warping. To demonstrate this, we consider the ShapeletSim dataset. We ran our algorithm (Table  5 ) on this dataset multiple times, changing only M, the amount of warping added to make synthetic exemplars (Table  5  line 4) from 0% to 30%. The lowest error achievable for this size subset of ShapeletSim is 26. 11%, and the error-rate obtained with the baseline method is 52%. As shown in Fig. 25 , adding too little warping hinders our ability to predict the best w, but once we have added at least 15% warping, we learn a setting for w that gives us the lowest error-rate. Given this observation, for simplicity, we hardcode the amount of warping to 20% for all experiments in this work. Finally, the parameter N, the number of new (partly synthetic) training sets needs to be determined.
This is a simple parameter to set; the more the better, but the gain comes with diminishing returns. N is hardcoded to a conservative 10 for all experiments presented in this work. Using hardcoded settings for all the datasets in the UCR Archive is an opportunity cost; an adaptive approach could be better. However, our strategy guards against over-fitting. Moreover, we see this work as proof that a more robust learning of w is possible, and it is not the final word on the matter.
Why 10-fold cross-validation Leave-one-out (LOO) is a common variant of cross-validation (CV) to tune the parameters, and it is the method used by the UCR Time Series Archive to learn the given that we perform 100 iterations for all w from 0-100. Note that we can further reduce this time by choosing to perform fewer iterations and a narrower w range. Our experiment results demonstrate that even 10 iterations offer statistically significant improvement over the baseline method, and the best w for a dataset, regardless of its size, does not exceed 60. This applies for datasets discussed in Section 4. 2. 1, which are framed around small training set problem, not the original UCR splits.
Related work The more general idea of creating synthetic data to mitigate the problems of imbalanced datasets  (Chawla et al. 2002)  or to learn a distance measure (Ha and Bunke 1997) is well-known. However, we are not aware of any other research suggesting a window size for improving DTW-based classification. We suspect that the dearth of study on this important problem is likely due to the community's lack of appreciation of the importance of w setting.
Datasets We use the UCR Time Series datasets for our experiments  (Chen et al. 2015). As of February 2018, the UCR Time Series Archive has 85 datasets from various domains, has served as the benchmark for the time series community, and is widely referenced in the literature. A more comprehensive version of the UCR Archive together with classification results of different algorithms is hosted by  Bagnall et al. (2018). As we have demonstrated in Fig. 22. left, our ability to learn w depends on the amount of training data. With enough data, the simple baseline method is effective, and we have little to offer. The ideas proposed in this work are most useful for smaller datasets. Some of the train/test splits in the UCR datasets have large enough training sets that our ideas do not offer any advantages.
Rather than ignoring these datasets, we will recast them to a smaller uniform size. We merge the original train and test set together, then randomly sample ten objects per class for training. The remaining objects are used for testing. As three datasets do not have enough ten objects per class, we exclude them from the experiment (the excluded are: OliveOil, 50words and Phoneme). Therefore, we are left with 82 datasets. These new splits are published in the paper supporting webpage (Supporting Page 2018) for reproducibility. Note that with these new splits, the training sets all have equal class distribution. However, this distribution may not be true for the test set.
5%. If the newly learned w results in test error-rate that is less than 1% different from the test error-rate obtained by the traditional method, we consider our method neutral. This can happen in two ways. Our method suggests the same value of w as the baseline method, or it recommends a different value of w, which offers similar accuracy. Given this nomenclature, we can say that of the 82 datasets tested, our method improves classification accuracy of twenty-four, with an average improvement of 3. 2%, and decreases the accuracy on only thirteen with a smaller average of 1. 6%. This statement can in turn be visualized with the linear plot in Fig. 26. - Below: Data sets that we hurt (13) Fig. 27 : Possible error-rate reduction (how close a method's error-rate to the optimal error-rate is) of the baseline method and our proposed method.
Another way to demonstrate how our proposed method outperforms the traditional method is to look at the possible room for improvement, which is the difference between the error-rate achieved by the learned w and the error-rate of the best w of a dataset (found by exhaustive search). The smaller the difference, the better the method is. This is illustrated in Fig. 27. While the results are visually compelling, we turn to statistical tests to ensure that the superiority of our method is statistically significant. Both the paired-sample t-test and the one-sided Wilcoxon signed rank test confirm that our method is better than the baseline method at the 5% significance level. Details are available on our website (Supporting Page 2018).
On time complexity It is important to clarify that we are optimizing the classification accuracy in trade-off for speed. However, we are only compromising training time here. The test time is not affected. Instead of running cross-validation one time as the baseline method, we would need to do that multiple times and average the results of these independent runs. So, if we decide to use ten iterations, the time it takes to learn the right w will be ten times slower than the traditional method (the resampling and adding warping to construct a new training set is linear and inconsequential). In this region our method wins Once the correct setting for w has been learned, we can readily use it for testing. This use of multiple random samples might seem like a computational burden but recall that many datasets in the UCR Archive took days, weeks or even months to collect, so spending a few more seconds or minutes on training the model to improve classification accuracy is well worth the relatively small increase in computational effort.
Case Study: Fall Classification We conduct a case study in the context of fall classification. We do not claim any expertise in this domain, and we only have a superficial idea of how the data was collected. This is exactly the purpose of this case study. We wish to demonstrate that our ideas can be easily applied to any dataset/domain with minimum effort and show the potential for significant gains in accuracy. The accuracy may be improved by several other (mostly orthogonal) methods; for example, by carefully truncating data  (Silva, Batista, and Keogh 2017) , averaging exemplars  (Petitjean et al. 2015) , and discarding data  (Xi et al. 2006 ). However, we believe our method offers an unusually large "bang-for-the-buck. " Falls are a common source of injury among the elderly. A fall generally has few consequences for the young, but it can lead to fatal consequences to the elderly.
According to the US Centers for Disease Control and Prevention, in the USA alone, an older adult is hospitalized due to a fall every 11 seconds, with one such individual succumbing to their injuries every 19 minutes. The total cost of fall injuries mounted to $34 billion in 2013 in the US alone (National Council on Aging 2017). The type of fall is highly predictive of the extent of the injuries that the victim sustains (Brain Injury Society 2016). Thus, knowing the cause or manner of a fall may assist timely and relevant medical intervention post-fall, as well as help prevent more fall in the future. The dataset we consider was kindly shared by  Albert et al. (2012). It was collected with a built-in phone accelerometer, which was attached the volunteer subjects' lumbar by a belt strap, positioned such that the accelerometer x, y, and z axes were directed upward, left, and behind the subject, respectively.
All falls were carried out onto a pad in a controlled lab environment. We only consider a small subset of the data and only the x-axis acceleration to demonstrate the utility of our method. Each example in our dataset is 400 data points long, representing a 20 second fall event at the sampling rate of 20Hz (we re-sample the subsequences of uniform length if some are slighter shorter or longer). Fig. 28  displays four examples of a trip fall. The data can be considered weakly labeled. The fall does not span the entire 20-second session, but it can be shifted in the time axis by an arbitrary amount ("arbitrary" to us, as we did not collect the data). Visual inspection suggests that this dataset needs warping invariance, and our algorithm helps determine the appropriate amount of warping to allow, as shown in Fig.
29. The baseline method leads us to use Euclidean distance (w = 0), which gives a classification accuracy of only 64%. However, our method suggests w = 9, reducing the error-rate from 36% to only 28. 5%. The best warping window width for this dataset is w = 7, which corresponds to a 25. 23% error-rate. This result is less impressive than the one published in  (Albert et al. 2012) , but note that we intentionally frame our problem around limited cross-subject training data and we perform classification using only a single dimension. 5.
Conclusion In this work, we have shown that w, the maximum amount of warping allowed by DTW, is a critical parameter for both the classification and clustering of time series under the DTW distance. For most datasets, if this parameter is set poorly, then nothing else matters; it is impossible to produce high-quality results. In many cases, a more careful setting of the value of w can close most or the entire performance gap gained by other more complicated algorithms recently proposed in the literature. For clustering, we have further proposed the first semi-supervised technique designed to discover the best value for w. Our approach is unique since human involvement is not required up-front as it is in other semi-supervised clustering algorithms. Instead, we seek user annotations after the clustering process, and we devise a scoring scheme to ask for only the labels that really matter.
This gives our algorithm the desirable anytime algorithm property. We have also forcefully demonstrated that the choice of warping window width w is critical for accurate DTW-based nearest neighbor classification of time series and proposed a resampling method to learn w in this context. Our method is parameter-free (or equivalently, we hardcoded all parameters). However, experimenting with adaptive parameters may allow others to improve upon our results. We have tested our algorithms on more than one hundred datasets from diverse domains, showing that it offers statistically significant improvements. We note that the ideas we have proposed are very simple. This is not an accident. We hope that the reader sees this simplicity as the strength it is intended to be, not as a weakness; simple ideas are more likely to be widely adopted and widely used.
While the absence of evidence is not evidence of absence, the extensive nature of our experiments, which failed to find a single dataset which requires a value of w greater than 20 for either clustering and classification of the UCR Time Series Archive data, suggests that these efforts are likely to be fruitless. Future work includes a more theoretical treatment of the issues at hand and determining if the basic framework can be extended to other distance measures with tunable parameter(s)  (Beecks, Uysal, and Seidl 2010; Assent, Wichterich, and Seidl 2006; Lee et al. 2008; Vlachos, Kollios, and Gunopulos 2002). Finally, we have released all our code and data in a public repository (Supporting Page 2018), to allow others to confirm, extend, and exploit our ideas.
We begin by formalizing the task at hand: Problem Statement: Given a labeled time series training set D; find the value of w that maximizes the classification quality on an unlabeled test set. Where ties exist, report the smallest w. We evaluate the classification quality by the measure of accuracy. Maximizing accuracy means minimizing the classification error-rate. Readers may argue that some other performance measures, such as the F-measure, are more suited. The F-measure penalizes false positive and false negative equally, making it a fairer metric for However, the uneven distribution of classes is not an issue here, since all the datasets we consider are stratified sampling. We are more interested in learning the appropriate value of w for the maximal classification accuracy of the test set, given that we only have limited training examples to learn from.
(Ng 1997)he advantage that no data is wasted. However, as noted in(Ng 1997), LOO can be more susceptible to over-fitting. This is because the models trained in each iteration are only slightly different (since the training set differs in only one object each time). Moreover, the entire purpose of creating N training sets to learn w is to increase the variance of the results. LOO is deterministic, but (with shuffling) K-fold CV (when K is less than size of training set) is not. On the other hand, if we set K = 2, we are learning from a dataset that is only half the size of the dataset we have. As we explained in Section 4. 1. 5, for small training sets, this is likely to result in learning a pessimistic value of w, which is much too large. Given these two constraints, we propose to use 10-fold CV throughout this work.
6 Acknowledgements This material is based upon work supported by the  Air Force Office of Scientific Research, Asian Office of Aerospace Research and Development (AOARD)  under award number  FA2386-16-1-4023. The  Australian Research Council  under grant  DE170100037  and the  UK Engineering and Physical Sciences Research Council ( EPSRC)  under grant number  EP/M015807/1  have also supported this work. Finally, we acknowledge the funding from  NSF   IIS-1161997 II  and  NSF IIS-1510741. We also wish to take this opportunity to thank the donors of the data to the  UCR Time Series Archive.
It provides a good tradeoff between low-variance LOO and the biased-to-large-w 2-fold CV. warping window size. error-rate reduction 0. 1 0. 2 0. 3 ShapeletSim Possible error rate reduction with our method Possible error rate reduction with UCR-method Possible 0 0 5 10 15 20 25 30 Different amounts of warping (%) Finally, it may appear that performing 10 repetitions of 10-fold CV will be computationally expensive. However, recall that the datasets in question are small by definition. Additionally, we can accelerate the entire process by embedding the current state-of-the-art DTW lower bounding and early abandoning techniques. Even without these techniques, our entire learning algorithm only takes 23 minutes for Gun_Point,.
The detection of anatomical landmarks in bioimages is a necessary but tedious step for geometric morphometrics studies in many research domains. We propose variants of a multi-resolution tree-based approach to speed-up the detection of landmarks in bioimages. We extensively evaluate our method variants on three different datasets (cephalometric, zebrafish, and drosophila images). We identify the key method parameters (notably the multi-resolution) and report results with respect to human ground truths and existing methods. Our method achieves recognition performances competitive with current existing approaches while being generic and fast. The algorithms are integrated in the opensource Cytomine software and we provide parameter configuration guidelines so that they can be easily exploited by end-users.
As such studies could typically involve hundreds or even thousands of individuals and tens of landmarks, the need to manually position the landmarks prior to such analysis is a very limiting factor. There is therefore a strong need for (semi-)automated landmark detection methods in biology. In computer vision, the problem of landmark localization has been extensively studied in faces  10, 11. Methods for face analysis can however not be easily transposed to biological images, because of their very different and variable nature. The small size of ground-truth datasets typically available in biology also requires to design more data-efficient approaches. In biology, the landmark structure is also very different than in face images. Indeed, the number of landmarks of interest is typically small (∼20 landmarks) and the images typically large (∼ × 1500 1500), which makes the landmarks more spaced apart than in face images.
It is based on the extraction of multi-resolution features and the use of generic tree-based ensemble machine learning methods, namely (Extremely) Randomized Forests  16, 17. The choice of this latter family of machine learning tools is motivated by their many successful applications in computer vision in general and biomedical image analysis in particular  18. Random decision forests have been exploited for example in various image classification tasks  [19] [20] [21]  , for organ segmentation in medical images  15, 22  , but also for landmark detection in radiology  [13] [14] [15]. Our contributions in this paper are as follows: • We propose a novel generic learning-based approach for landmark detection. • We thoroughly study the effect of its parameters on three diverse bioimage datasets (for human cephalometric radiographs, zebrafish skeletogenesis, and Drosophila wing developmental studies).
From this analysis, we derive guidelines for choosing these parameters on new problems. • We compare our method with several landmark detection algorithms from the literature, both on the same three datasets and on two cephalometric challenges. These comparisons show that our approach yields competitive results in terms of accuracy, with lighter models and lower prediction times. • We provide an open-source implementation of these algorithms through the Cytomine platform  23  that further implements proofreading tools to combine automatic detection and manual refinements. • As an important side contribution, we provide an easy access to the datasets used in this study with the hope that the landmark detection problem will gain more interest in bioimage informatics and machine learning research.
Methods Materials. We tested our method on three datasets summarized below. A more complete description is presented in the Supplementary Materials. An illustration of the landmarks is given in Fig. 1  with one image per dataset and their corresponding landmarks. • CEPHA, a dataset of 100 lateral human cephalometric radiographs. This dataset has been previously described in Wang et al. 24, 25. 100 cephalometric X-ray images were collected from 100 patients aged from 6 to 60 years old. Image resolution is 1935 by 2400 pixels. 19 landmarks were manually marked and reviewed by two experienced medical doctors for each image. Additional images were provided during two international challenges  24, 25. Some of the landmarks corresponds to visual edges (7, 8, 13) while some others corresponds to morphological locations with less visual information (1, 4, 19).
• DROSO, a dataset of 138 colored images of Drosophila wings. • ZEBRA, a dataset of 113 ventral views of head skeleton of zebrafish larvae. The image resolution is 2576 by 1932 pixels. 25 landmarks were manually marked and reviewed by the same expert. Most landmarks of this dataset corresponds to locations with little visual information (1, 2, 5, 15, 18, …).
, about 4,000,000 pixels for our images). To generate a more balanced training sample, we select π(R)/(s) 2 pixels within a radius R to the landmark in each training image, where s is a user-defined spacing parameter allowing to control the number of pixels extracted. Pπ(R)/(s) 2 additional pixels are then randomly selected outside this radius, where P a user-defined parameter. In practice, one can expect in many medical and biological applications that the same landmark will be located in close positions from one image to another (see Fig. 3  for an illustration on one of our datasets). At prediction stage, this information can be used to constrain the search for the landmark position to pixels that are not too far from the average position of the landmark in the training images. When this constraint is exploited at prediction stage, it is natural to avoid putting in the training sample pixels that are too far away from the landmark position.
for an illustration). This subsampling contrasts with Stern et al. 26  where pixels were sampled in the whole image during the training and the prediction phase. In the experiments, we will show that the problem of robustness to rotations is not important on our three datasets because the deformations stay small. On other datasets with bigger rotations, we would suggest to initially use a 2D registration algorithm such as presented in Reddy et al. 30  in order to roughly align the images between each other before analysis.