dataesr/scientific-paragraphs-categorization huggingface dataset:
The resolution of the Pliades optical image is 50 cm, and its horizontal accuracy is theoretically between 1 and 3 meters, depending on the usage of ground control points on the considered area. These two characteristics of the Pliades imagery data, especially the horizontal accuracy, are the major factors causing its registration with the LiDAR data (both 2011 and 2017) to be not as accurate as the registration between the LiDAR data and the aerial image (2016). When regarding the resulting average discrepancy (i. e. 0. 99 and 0. 82 meters) and taking into account the spatial resolution of the datasets (i. e. 50 cm for the Pliades imagery data and 70 cm or 35. 4 cm for the LiDAR point spacing), one may interpret that these results are not good enough. However, as the horizontal accuracy of the Pliades imagery dataset varies between 1 to 3 meters as aforementioned, we would argue that such average registration discrepancy results are highly desirable.
Overall, the proposed coarse-to-fine method yields a very significant reduction of the spatial discrepancy between datasets, namely 63% to 67% for the registration of LiDAR data and orthorectified aerial imagery, and 98% for the registration of LiDAR data and Pliades imagery. 2) Based on check pair line segments: The authors of  [16]  proposed an evaluation based on distances between line segments that are manually sketched from the two datasets. Compared with check point-based evaluation, this evaluation relies on a higher geometrical basis (i. e. line segments compared with points) to assess the registration. The distance proposed in  [16]  between two line segments p (with A and B are its end-points) and q is defined as follows, d(p, q) = 1 2 (d A + d B ) (13) where d A and d B respectively denote the distances from A and B to the line segment q.
They are then manually matched, yielding check pair line segments. The source for these line segments are mainly the building straight boundaries. Then, the Hausdorff distance between each pair is computed. A smaller distance indicates a more accurate  Distance measure  [16]  Modified distance measure  As we can see from Table  V , the discrepancy between the datasets measured based on manually sketched line segments is significantly reduced after each of the registration, i. e. the coarse and the fine registration. On the one hand, the proposed registration method ultimately yields an average discrepancy of 0. 63 and 0. 64 meters between the LiDAR data (2017 and 2011) and the orthorectified aerial imagery (2016). On the other hand, for the registration of LiDAR data (2017 and 2011) and the Pliades multispectral imagery (2015), the resulting check pair line-based discrepancy value is 1.
66% from Table  IV ), whereas the second percentage is computed from check pair line-based evaluation result (i. e. 24. 62% and 36. 98% from Table  V ). On the other hand, for the registration between the airborne LiDAR data and the orthorectification aerial image, these average gains are only 15. 8% and 4. 15%. Both spatial discrepancy assessments and all these mentioned elements show that the results yielded by our proposed method are relevant. These presented assessments have also shown and validated the versatility of the proposed method, through the differences between the registered datasets and the complexity of the test areas. However, it should be noted that it is virtually impossible for a registration method to perform well on any other scene without an adaptation or re-parametrization.
V. CONCLUSIONS AND PERSPECTIVES This paper has presented and evaluated a coarse-to-fine registration method between airborne LiDAR data and optical imagery. It is dedicated to overcome the challenges associated with the difficult context, where the two datasets are not acquired from the same platform, neither from the same point of view nor having the same spatial resolution and level of detail. In the literature, even one or several of these constraints have been shown problematic for carrying out a registration method (see Section II). To the best of our knowledge, there is currently no solution able to achieve the registration between airborne LiDAR and optical imagery under such constraints altogether. As a matter of fact, the proposed registration method has been evaluated according to its own quality, before and after the registration.
Indeed, it is not compared with existing methods because they were not designed to address the considered context. Nevertheless, we reconsidered the subjective accuracy suggestion related to a sub-pixel level of accuracy for a registration (see I-B5). Instead, we rely on an objectively quantitative accuracy which is that, if the resulting spatial discrepancy less than 50 cm, then the registration will be considered accurate. In this regard, the proposed registration method has achieved a highly desirable accuracy. The proposed method can be summarized as follows. First, a coarse feature-based registration is carried out based on the extraction and matching of building candidates on the two datasets, reducing significantly the spatial shift between them. Here, it should also be noted that this building-based approach certainly does not limit the usability and versatility of our method, since urban scenes with buildings (even very sparse) are available most of the time  [68].
Then, a fine registration based on the maximization of MI or NCMI (both measures have been performed separately) is carried out to determine the optimal camera pose, granting the datasets to be precisely aligned. It involves a process of super-resolution of LiDAR data to generate high-resolution images of altitude and intensity values. This approach neutralizes the difference of spatial resolution and level of detail between datasets, enabling the MI-based and NCMI-based fine registration. The fine registration also involves in dividing the considered area into many equal patches. For each patch a local transformation model is estimated. This approach allows reducing significantly the computational cost of the fine registration. Lastly, a smoothing of the patch-based transformation models is carried out to resolve the conflicts and discontinuities between them.
It involves an IDW average of camera pose parameters from neighboring patches. As one can realize, many elements of the proposed method are intended as the solution to the challenges associated with the considered context. First, in order to address the spatial shift between datasets caused by the differences of points of view and fields of view, a coarse registration is carried out. It relies on using buildings as primitives, which is a relevant choice of primitive considering the low density of airborne LiDAR point cloud around vertical surfaces. Then, the differences of spatial resolution and level of detail between datasets have been dealt with by the SR approach. An areabased fine registration using MI or NCMI measurement is carried out to finely tune the optimal local transformation model.
Overall, as highlighted by the comprehensive spatial discrepancy assessments, the proposed method has achieved a very high registration accuracy. It is especially desirable when taking into account the difficulties of the considered context, and the horizontal accuracy of the datasets. It is suggested that only one registration approach is not sufficient to register the data accurately from heterogeneous sensors, even when they are rigidly fixed to the same platform  [14]. In this paper, we proposed a coarse-to-fine registration method consisting of two steps of registration. It reinforces the relevance of a coarse-to-fine approach for registering an optical aerial/satellite imagery with an airborne LiDAR dataset. Nevertheless, it can be anticipated that the proposed approach could have limitations to operate in an environment lacking of man-made objects providing reliable primitives, such as forest and desert areas.
ACKNOWLEDGMENT The authors would like to thank  Centre GéoStat ,  Université Laval , as well as  Québec City, Communauté Métropolitaine de Québec (QC, Canada ), and  Centre National d'Etudes Spatiales (France)  for providing the datasets used in this work. We also would like to thank  Dr. Hongchao Ma ,  School of Remote Sensing and Information Engineering, Wuhan University, China  for his/her help on the usage of the check-pair line evaluation.
In the present work, only 14 human key-points are chosen from the ones extracted with the Cascaded Pyramid Network (CPN)  [14]. In this paper, we propose a real-time and context-invariant approach based on 2D pedestrian body poses to address the Crossing/Not Crossing (C/NC) prediction in realistic driving conditions. We evaluate the proposed approach on the Joint Attention in Autonomous Driving (JAAD)  [15, 16]  public data set, a standard benchmark in the matter of pedestrian behaviors prediction. To proceed, we collect human key-points for all video frames in the JAAD data set thanks to skeleton IDs and associated spatial coordinates defined in JAAD annotations. Thereafter, we develop SPI-Net: a representation-focused multi-branch network that combines Cartesian features and location-invariant geometric skeleton features.
In the second branch, a 2D convolutional network, we represent a skeleton sequence as a pseudo-image. This allows us to extract spatio-temporal features using standard computer vision deep-learning methods. We then perform a late fusion on those two branches and fine-tune the entire approach in order to evaluate the model. According to experiments, SPI-Net achieved 94. 4% accuracy in pedestrian crossing prediction. To the best of our knowledge, SPI-Net is at the moment, the state-of-the-art for the C/NC task on the JAAD data set. The paper is organized as follows: Section 2 outlines some existing approaches from the literature for skeleton-based action recognition and pedestrians' intentions prediction. Section 3 presents an overview of SPI-Net. Section 4 describes the experiments setups and shows the results of SPI-Net on the JAAD data set.
Geometric Deep-Learning The evolution over time of the skeleton of the human body can be considered in the form of a dynamic graph. So far, research in deep-learning for action recognition on skeletal data has focused mainly on Euclidean data. The non-Euclidean nature of data in graph format makes the use of basic operations, such as convolution, difficult to perform. However, convolutions have by definition the ability to extract local spatial features and could use the skeleton data structure in graph format for the classification of human actions. Such ability fits perfectly to Graph-type data structures since they are, by definition, locally connected structures: the set of neighbors of a node. In this way, representing the skeleton in the form of a graph can have the advantage of not exploiting non-existent neighborhood links between joints, but of preserving coherent spatial semantics for the skeleton.
x = x x max ; y = y y max (1) Subsequently, obtained pedestrian pose sequences are defined as: s = (s 1 , s 2 ,. , s T ) ∈ R T×K×d , where T is the sequence duration, K is the count of key-points, and d is the dimension of each key-point. All sequences of skeletons are then sampled by a sliding window to keep a fixed size in the form of a 3-dimensional (T, K, d)-shaped tensor where T = 300, K = 14, and d = 2. The majority of the extracted sequences are smaller than the fixed T size of the sliding window, therefore sequences with less than T frames are padded with zeros. Finally, all processed data are introduced as a complete sequence to the SPI-network.
SPI-Net Architecture The network architecture of SPI-Net is shown in Figure  4. In the following, we explain our motivation for designing input features and network structures of SPI-Net. A large majority of current research in skeleton-based action recognition and pedestrian intention prediction focuses mainly on the sequential modeling part of the problem. Moreover, the architectures of the approaches presented for action recognition and intentions prediction have become more and more complex over the years. We think this is because those approaches rely heavily on deep-learning networks to learn informative representations of data itself by adding hidden layers in the architectures. In this work, we propose to go back to "It is all about embedding and standardization in machine-learning" : once one finds a way to standardize and represent data in a more adequate way, any classifier might be able to obtain good results as long as the input data is informative.
By normalizing the input data, creating global-motion features and location-viewpoint invariant features or enforcing certain constraints towards the data representation of designated hidden layers, we send informative-representation ready data to the classification network. It allows us to rely on less hidden layers to learn informative representations of data and therefore reduce the complexity of the network compared to other approaches. Since we choose to rely on a reduced number of hidden layers, we can focus on the inference time of our model, which is mandatory since we take the model speed as one of our priorities. Finally, we show with SPI-Net's architecture composed of Dense layers and 2D convolutions that combining sequential modeling and taking into consideration the representation and the normalization of an action can be quite effective for the overall accuracy of a network designed for the C/NC task for the JAAD data set.
Results on JAAD Data Set In ablation studies, we first explore how each SPI-Net branch contributes to the intention prediction performance. We, therefore, explore how the LDA constraint for the Geometric branch or the spatial joint reordering trick impact the intention prediction performance on JAAD. Therefore, both Geometric and Cartesian branches results are presented in Tables  1  and 2 , Figure  8. The crossing prediction results of the overall SPI-Net approach on the JAAD data set are then presented in Table  3. Finally, more details about each branch and SPI-Net are listed in their respective confusion matrices for the C/NC task in JAAD data set in Table  4. Table  1. Intention prediction accuracies of the Geometric branch alone, for different encodings of the sequences of inter-keypoints distances.
One of the main reasons, to this day, has been the lack of reliable data sets available  [89]. However, our pipeline makes it easy to keep up with the state-of-the-art in this field without completely disrupting the SPI-Net approach for intention prediction. Compared to image-based approaches, if major advances are made in the computer vision field and more specifically for pose estimation, SPI-Net could still be relevant. • Temporal tracking of pedestrians: In the real world, there are usually more pedestrians on the streets passing and occluding each other, which requires sophisticated mechanisms not only for their detection but for their temporal tracking without mixing their identity over time. In order to compare SPI-Net to the literature on JAAD, our current approach completely omits such issue and relies on the ground truth spatial coordinates and individual IDs of each pedestrian provided by the data set.
• Data set size: A recurrent barrier to using deep-learning is small data sets. Even though JAAD is one of the most complete data sets for pedestrians intents, the number of instances present in the data set is still undersized to use the generalization ability of neural networks to its finest. In the present work, we had to focus a lot on regularization techniques present in the literature to avoid over-fitting. It is, therefore, necessary to extend the total number of instances for such task. Our model is directly extensible to other input formats with different 2D or 3D skeletal data structures: the proposed approach can therefore be applied to a broader family of applications that discover the intentions of moving subjects. However, to use the generalization ability of neural networks to its finest on such small data sets, future research will focus on proposing a tool to enrich the existing databases on human skeletal dynamics by combining both Geometric features and Cartesian features in order to generate skeleton dynamics that are coherent both spatially and sequentially.
Our SPI-Net approach has achieved remarkable results: 94. 4% accuracy, i. e. , 2. 5% more than the current state-of-the-art for the Crossing or Not Crossing prediction task on the JAAD data set. Furthermore, since we choose to rely on a reduced number of hidden layers, we can focus on the inference time of our model, which is mandatory since we take the model speed as one of our priorities: SPI-Net speed can reach around one inference every 0. 25 ms on one GPU (i. e. , RTX 2080ti), or every 0. 67 ms on one CPU (i. e. , Intel Core i7 8700K), which makes it highly effective for the task of predicting discrete intentions of pedestrians and directly applicable to embedded devices with real-time constraints.
Brain tumors, such as low grade gliomas (LGG), are molecularly classified which require the surgical collection of tissue samples. The pre-surgical or non-operative identification of LGG molecular type could improve patient counseling and treatment decisions. However, radiographic approaches to LGG molecular classification are currently lacking, as clinicians are unable to reliably predict LGG molecular type using magnetic resonance imaging (MRI) studies. Machine learning approaches may improve the prediction of LGG molecular classification through MRI, however, the development of these techniques requires large annotated data sets. Merging clinical data from different hospitals to increase case numbers is needed, but the use of different scanners and settings can affect the results and simply combining them into a large dataset often have a significant negative impact on performance.
This calls for efficient domain adaption methods. Despite some previous studies on domain adaptations, mapping MR images from different datasets to a common domain without affecting subtitle molecular-biomarker information has not been reported yet. In this paper, we propose an effective domain adaptation method based on Cycle Generative Adversarial Network (CycleGAN). The dataset is further enlarged by augmenting more MRIs using another GAN approach. Further, to tackle the issue of brain tumor segmentation that requires time and anatomical expertise to put exact boundary around the tumor, we have used a tight bounding box as a strategy. Finally, an efficient deep feature learning method, multi-stream convolutional autoencoder (CAE) and feature fusion, is proposed for the prediction of molecular subtypes (1p/19q-codeletion and IDH mutation).
g. , 3D medical images). Although convolutional neural networks (CNNs) are frequently used for characterizing visual objects in computer vision, deep autoencoder (AE) is often adopted as well. Deep AE is another type of deep learning method for characterizing images, however, the principle of AEs is different from that of CNNs and is based on applying codebooks (encoder and decoder) and generating codes. Additional advantages of AEs can include, e. g. , noise robustness and feature reduction (depending on codebook size). It can also be used for both supervised and unsupervised learning. Further, the trained encoder part of the convolutional AE (CAE)  [17, 18]  could also be used as a CNN. Such a setting can be benefited by first applying pre-training using a CAE for learning the manifold of dataset in a self-supervised way, followed by further refining the learning of network by learning complex features through supervised refinement.
Such an approach has shown an improved generalization performance as compared to training the networks from the scratch with a small dataset  [19]. Observing these advantages of CAEs along with considering our application, where MRI data could be noisy, we decided to select CAEs as the method for deep learning of brain tumor features in our study. One practical challenging issue of using clinical dataset for glioma subtype classification is that the available medical datasets are often rather small, as they are usually collected by a local hospital from a region of a country. It might be desirable to learn a model on a specific subset of data  [20, 21]. For example, a hospital may require a model to be deployed that might perform well only on the hospital's patient population. However, using the limited data from a single hospital might not be well enough to learn an accurate model causing generalization problem and achievements made for one hospital is not true progress unless it can be disseminated to other settings as well.
Recently, a new data augmentation technique and its variations have gained popularity, known as Generative Adversarial Networks  [22]. The GAN frameworks have been used in various medical imaging applications  [23, 24]. Most studies have proposed image-to-image translation such as label-to-segmentation  [23] , segmentation-to-image  [25]  or cross-modality translation  [24, 26]. Inspired by the above, we have decided to investigate deep convolutional GAN for augmenting synthetic training data in addition to existing data to improve the generalization performance (i. e. , on the test set). Another very challenging issue encountered in the real clinical application is that when there are many small glioma datasets, simply merging them into one dataset would not lead to significant increase of the generalization performance (i.
However, for medical image datasets, especially for glioma datasets, such studies are in their infant stage. A particular challenging issue is whether one may obtain an effective domain mapping method that is able to map between MRI datasets, in the meantime, retaining the molecular-subtype information after the mapping. Our work is mainly motivated by the following issues: molecular marker-information in low grade gliomas (LGGs) are rather recently integrated and most datasets are small; the mismatches that arise when multiple datasets from different sources are combined together to enlarge the data size. Considering these challenges, our work is focused to propose a robust method by domain mapping to overcome the scanner dependent mismatches that preserves the molecular structural originality of gliomas.
In this paper, we propose a novel approach based on CycleGAN  [33]  and multistream convolutional autoencoder framework  [34]  as a classifier. Although CycleGAN has been applied for non-medical applications  [33]  and cross-modality translation of MRIs  [26] , to the best of our knowledge this is the first work used for domain mapping that retains molecular-subtype information in low grade gliomas. Moreover, the data used in this work is raw clinical data for the prediction of 1p/19q codeletion and IDH genotype without annotations (tumor segmentation masks) obtained from multiple hospitals. The main contributions of this paper include:.
• Extensive empirical tests and performance evaluation on the effectiveness of the proposed scheme and comparison with some state-of-the-art methods. It is worth mentioning that although a part of this work has been presented in  [34]  which was based on classification of low and high grade gliomas, however, this paper applies to classify molecular subtypes in LGGs instead of tumor grading. Furthermore, this paper differs in terms of: mapping multi-source data to a common domain; dealing with the clinical MRIs that are not uniform in size in all 3D directions; avoiding laborious task of defining soft tissue boundaries; and lastly, including empirical tests and evaluation on two clinical datasets from different hospitals.