dataesr/scientific-paragraphs-categorization huggingface dataset:
In any case, as the training data should contain a vast majority of normal data, the model is actually trained to learn the normal behaviour of a system and then used in the test phase to identify anomalies. On the other hand, with unsupervised learning techniques, the only assumption is that a very small fraction of the overall data is anomalous data or that anomalies lie far away from most data instances or in low-density regions. A final consideration is that even though anomaly detection is often based on unsupervised learning, Erhan et al. [3]  explain how unsupervised methods can be of significant help in building supervised predictive models. Chandola et al. [2]  identify three main types of anomalies: • Point anomalies. A data point that differs significantly from the rest of the data points in the dataset considered.
• Collective anomalies. When a group of data in a dataset is an anomaly as a whole, but the individual instances in that group (or subsets of them) might be not on their own. In time series, this would correspond for instance to a situation or condition persisting over an abnormal long time. Collective anomalies can only be detected in datasets where data is related someway, i. e. , sequential, spatial or graph data. Detection techniques for contextual and collective anomalies are particularly relevant in our survey since they are applicable to time series data. The adoption of a particular method depends on the nature of the anomaly, the characteristics of data (existence of labels, number and types of data attributes, data volume) and the expected output (label or score, need for result interpretability).
Motivation and Organisation of the Survey The complexity of aviation systems and traffic operations makes the use of model-based anomaly detection techniques difficult due to insufficient model fidelity and over-simplified assumptions. Indeed, a significant research effort have been dedicated to the development of data-driven approaches, which have notably benefited from significant advances in machine learning and the availability of massive amounts of sensor-generated data. However, some of the classical statistical and machine learning techniques for anomaly detection do not scale well with large datasets or perform poorly with high-dimensional data, which is usually the kind of data available in aviation. In this context, recent advances in the deep learning field should significantly improve the performance of anomaly detection with large-scale high-dimensional data.
This standardized outlier score allows for comparisons over one dataset and even over different datasets. The probabilistic approach of LoOP is also more reliable and tolerant to noise than LOF, where an inappropriate choice of parameter k can cause unstable results. Although LOF works better than kNN with datasets of varying density, both techniques do not scale well with large and high-dimensional data as they need to compute pairwise distances between data points and determine the nearest-neighbours. In fact, the computational complexity not only can be high during the training but also during the testing phase. See  [1, 2]  for a more exhaustive survey on the multiple improved variants reducing the quadratic complexity of the basic techniques. For instance, the Orca program  [17]  is an example of an improved variant often cited in aviation papers as part of the literature review in anomaly detection.
PCA models are designed to be trained on a training set: then the fitted linear transformation can be efficiently applied to any volume of further samples. Although the fitted transformations can be applied on datasets of any size, their high computational complexity make them unsuitable to be trained on very large datasets. The application of subspace-methods as dimensionality reduction techniques are particularly useful when applied to high-dimensional data. Several applications of subspace-methods to aviation exist (see Section 4), including an improved faster method based on Kernel PCA  [46]  and Functional PCA  [52].
Surprisingly, the results of their research show that VAR significantly outperforms LSTM, which according to the authors could be explained by the fact that the LSTM capability of capturing long term dependencies may not be necessary. Malhotra et al. [57]  present a RNN model with several layers of stacked LSTM which is trained with normal data. The trained model is then used as a predictor over a number of multiple time steps and the residuals computed over the training set are modeled as a multivariate Gaussian distribution. The probabilities on the residuals can thus be computed and a threshold to discriminate the anomalies is determined by maximising the F β score over a validation dataset. More recently, Ergen et al. [59]  present a hybrid framework for variable length sequences based on LSTM and the use of either OC-SVM  [43]  or SVDD  [77]  as anomaly detectors.
The novelty in the approach comes from the fact they jointly optimise the parameters of both LSTM and the anomaly detector by developing specific gradient based training methods. The experiments on several real and simulated datasets show significant performance improvements over the traditional OC-SVM and SVDD methods. It is also interesting to note that some approaches use RNN as part of an autoencoder (e. g. , LSTM encoder-decoder (LSTM-ED)  [63, 78] ) or generative architecture (e. g. , GAN-AD  [68] ). See Sections 3. 3 and 3. 4 for further details.
The resulting signature matrices can then be fed to a CNN for pattern extraction. CNN-based anomaly detection methods have been mainly applied to intrusion detection  [60, 61]  by preprocessing data samples with float and integer attributes into an image form convenient for CNN processing. In a more recent study, Kwon et al. [62]  assess several CNN architectures for anomaly detection using different network traffic datasets by comparing their performance to other techniques including Variational Autoencoders (VAE), Fully Connected Networks (FCN)  [84]  and LSTM. Their results indicate that CNN perform better than VAE, but worse than FCN and LSTM. While the use CNN for anomaly detection is an active area of research, several hybrid architectures integrating CNN exist to perform anomaly detection.
g. , Stacked Denoising Autoencoders. In the case of AE using RNN, it is worth mentioning the research by Malhotra et al. [63]. The authors propose a reconstruction-based approach for anomaly detection in time series based on LSTM Encoder-Decoder (LSTM-ED) models, which has been previously used for machine translation  [75]. A LSTM-ED is trained only with normal time series data so that higher reconstruction errors should be obtained for anomalous sequences. A normal distribution is fitted on the reconstruction errors computed over a subset of the validation data. The estimated mean and covariance is then used to compute an anomaly score for each point in the time series. The threshold to determine whether a point is anomalous or not is computed so that it maximises F β score over a validation dataset.
Domain-Based Approaches If SequenceMiner  [18]  is one of the few anomaly detection methods specifically designed for the processing of discrete sequences, MKAD developed by Das et al. [44]  is one of the first methods designed to effectively detect operationally significant anomalies with heterogeneous sequences of both discrete and continuous variables. Based on kernel functions and OC-SVM, MKAD can identify operational situations in FOQA data such as go-around operations, unusually high airspeed flights, flights impacted by gusty winds and abnormal approaches. More recently, Das et al. [104]  applied MKAD to detect anomalies in the approach phase but this time with a much larger set of flights of the same fleet and aircraft type. In the paper, the authors report exclusively on two anomalous situations correctly identified by MKAD corresponding to two significant operational events: high energy approaches and turbulent approaches.
Distance-Based Approaches One of the first attempts in the field was the research by Budalakoti et al. [18]  who address the problem of anomaly detection in a set of sequences of switches used by the pilot and co-pilot to maneuver an aircraft. Their method (SequenceMiner), based on a clustering approach, is able to detect anomalous switching behaviour linked to the loss of autopilot mode awareness by the flight crew. Li et al. [19]  apply a cluster-based anomaly detection (ClusterAD) method based on DBSCAN to detect anomalies in a FOQA dataset of an airline for 365 B777 take-off and approach operations. The anomalous operational situations correctly identified by ClusterAD include high/low energy approaches, unusual pitch excursions, abnormal flap settings and high wind conditions. One of the advantages of ClusterAD compared to MKAD is that it can automatically identify multiple types of flight operation patterns (different nominal operations) corresponding to the identified clusters.
Following up this research, Li et al. [20]  present a method based on DBSCAN called ClusterAD-Flight, which is able to detect abnormal flights during take-off or approach as whole. In this work, more extensive tests are conducted with an additional dataset of 25,519 A320 flights. Results show that both ClusterAD-Flight and MKAD are able to identify more operationally significant anomalies than exceedance-based methods. ClusterAD-Flight performs better with continuous parameters, whereas MKAD is more sensitive toward discrete parameters. The latest research by Li et al. [21]  is on an improved ClusterAD approach called ClusterAD-DataSample. However, as this method is based on a GMM, we cover it as part of the statistical approaches in Section 4. 1. 4. Compared to MKAD and ClusterAD which are able to process hundreds to tens-of-thousands of flights, Oehling et al.
[16]  propose an approach able to scale to very large datasets as the ones used in the production environments of big airlines. The approach, based on the Local Outlier Probability (LoOP) method, is applied to an airline dataset of 1,2 million flights in order to detect anomalies related to safety events. The top outliers identified by their approach are reviewed by the airline pilots in order to assess their safety-relevance. The results of the research show that their method is able to reduce the number of undetected safety-relevant events compared to the current exceedance based approaches implemented in FDM systems. Churchill et al. [22]  present a hierarchical clustering method to group in space and time aircraft trajectories in the airport surface. The goal is the identification of statistically anomalous taxi paths, which may be unplanned and unexpected by the controllers and thus could represent a safety risk.
Reconstruction-Based Approaches Zhang et al. [107]  point out two known issues when the classical Kernel PCA algorithm  [46]  is applied to a large dataset for anomaly detection: it is computationally expensive (O(n 3 ) where n is the size of the dataset) as well as hard to adapt as parameters such as the number of principal components and the confidence for the confidence limit needs to be set before anomaly detection. Thus, the authors develop an optimized GPU implementation where the previous parameters are computed automatically. The improved algorithm is applied to synthetic datasets  [44]  and compared to the OC-SVM  [43]  technique. The results show significant speed increases and a detection efficacy close to the OC-SVM one. Jarry et al. [52]  propose a method based on FPCA to identify atypical approaches and landings both in post-operational analysis and on-line.
The results show the method can effectively identify atypical flights although the results can be very sensitive to the size of the sliding window. Janakiraman and Nielsen  [56]  propose an unsupervised anomaly detection approach based on ELM. This approach developed by NASA is an alternative to MKAD for the identification of safety risks in very large aviation datasets. The performance of the three ELM variants are evaluated and compared to MKAD on a dataset of over 40,000 flights corresponding to landing operations at Denver airport. While the results of the ELM-based approach are comparable to MKAD in terms of detection accuracy, the training of ELM models are faster by two orders of magnitude. Olive et al. [108]  present a method based on autoencoders to analyse flight trajectories, detect unusual flight behaviours and infer ATC actions from past Mode S data.
[115]. The key step in the approach is to compute a distance matrix between flights defined in terms of residuals of modeling one flight's data using another flight's VARX model. Once the distance matrix is built, a LOF method  [13]  is applied to identify the anomalous flights. The evaluation results on a large FOQA dataset (over a million flights) show a good performance into detecting already known safety events as well as previously undetected ones compared to state-of-the-art algorithms like MKAD. In another framework also based on VAR modelling and adapted to online anomaly,  Melnyk et al. [39]  represent each flight with a semi-Markov switching vector autoregressive (SMS-VAR) model. With this approach, each phase of a flight determined by the set of pilot switches is represented by a different VAR process  [115]  and a semi-Markov model (SMM)  [116]  is used for the dynamics of flight switches.
Anomaly detection is based on the dissimilarities between the one-step ahead model's predictions and observed data. The framework is extensively evaluated on both a synthetic and an airline FOQA dataset and the achieved performance is similar or slightly better than the MKAD one. Nanduri and Sherry  [58]  present a regression-based approach applied to simulated FOQA-like data  [117]  corresponding to 500 approaches into San Francisco airport. Several different kinds of RNN architectures (GRU and LSTM) are tested and compared with MKAD. In all cases, the RNN-based models are able to detect more anomalies than MKAD. The authors explain the superiority of RNN-based approaches by the fact that they do not have the limitations of MKAD: the need for dimensionality reduction which results in loss of information and poor sensitivity to short duration anomalies, and its inability to detect anomalies in latent features.
Unfortunately, the authors give few details on how the anomaly threshold based on the residuals (prediction errors) was chosen. ClusterAD-DataSample by Li et al. [21]  is a method based on a GMM which is capable of instantaneously detecting abnormal data samples during a flight rather than abnormal flights as a whole during a specific flight phase. The method is tested with a real dataset of 10,528 A320 flights and compared with exceedance-based methods. Then, it is compared with MKAD and ClusterAD-Flight with a second dataset of 25,519 A320 flights (already used in  [20] ). The results indicate ClusterAD-DataSample performs better in detecting known unsafe events (detected with exceedance-based methods), but the authors point out the need for further evaluation of the performance with detecting unknown issues.
Jiang et al. [37]  propose a method based on independent component analysis (ICA) for online monitoring of air traffic congestion. Based on the complex networks topology, a model is trained with a dataset of smooth situations. Any new situation is then compared to the reference 'normal' representation by analyzing the change of statistics. As the confidence limits cannot be determined directly from a particular approximate distribution, a kernel density estimation (KDE) is used to set the control limits. 4. 1. 5. Temporal-Logic Learning Based Approaches Deshmukh et al. [103]  propose a temporal logic based anomaly detection algorithm (TempAD) applicable to trajectories in the terminal airspace. The algorithm, based on a temporal-logic learning approach  [71] [72] [73] , can learn human-readable mathematical expressions from data which facilitates the feedback and interaction with operational experts.
• NASA DASHlink open database originally designed and collected by Balaban  [118]   Although prognostics and RUL estimation is a core function of PHM, it falls out of the strict anomaly detection scope and hence it is not covered in this survey (two recent reviews by  Elattar (2016)   [120]  or  Lei (2018)   [121]  already address this topic). Instead, we focus our review on the application of data-driven techniques aimed at detecting anomalous behaviour in aviation systems with the goal of identifying faults after their occurrence or anticipating potential failures as part of the condition monitoring process in PHM  [122]. Effective anomaly detection techniques to predict incipient failures from historical data is important to estimate time-to-failure and help schedule maintenance activities.
More recently, Nicchiotti et al. [124]  (2018) leverage closed commercial aircraft maintenance operational data and apply SVM based methods and PCA as a tool to reduce dimensionality in order to predict such unscheduled maintenance operations. Deep autoencoders  [65]  and convolutional denoising autoencoders  [85]  (2019) have been used for fault detection and anomaly detection, both on the NASA open database and on a dataset of Customer Notification Reports sent over ACARS to airlines to help them detect engine faults. Recurrent Neural Networks autoencoders have also been used on time series  [125]  in order to find a proper embedding or representation of time series that is in turn used for predicting a RUL estimation on the turbofan dataset. More recently, Zhao et al. compare in  [126]  different approaches of feature selection mechanism based on dimensionality reduction.
This is not an issue for applications where models can be trained offline, but some real-time safety monitoring applications may require some kind of incremental or very fast online training. In aviation, among the traditional approaches, the domain-based MKAD  [44]  developed by NASA is still one of the state-of-the-art methods for the detection of operationally significant events in flight data. However, its computational complexity is quadratic with respect to the number of training examples, which makes it unsuitable for very large datasets and certain applications. ClusterAD methods are also among the most widely applied, in spite of having the same performance issues than MKAD. For faster learning with large datasets, ELM  [56]  or LoOP  [16]  based anomaly detection seem to be two good alternatives to MKAD.
Acknowledgments Arina Buzdalova  was supported by the  Government of Russian Federation  (Grant  08-08 ). Carola Doerr was supported by a public grant as part of the  Investissement d'avenir project , reference  ANR-11-LABX-0056-LMH , and by the  Paris Ile-de-France Region. We also acknowledge support from  COST Action   CA15140  ' Improving Applicability of Nature-Inspired Optimisation by Joining Theory and Practice (ImAppNIO) ' supported by the  European Cooperation in Science and Technology.
The time series reveals that temperature anomalies are mostly coherent throughout the entire water column (e. g. , late 2015), although ∼30% of the time the anomalies above and below 100 m have opposite signs (e. g. , early 1997). XBT data also revealed an unprecedented warming of the Florida Current during 2014-2015, which followed a relatively cold period in 2010-2013. During the 2014-2015 event, the entire water column in the Florida Straits was ∼0. 5 • C warmer than average conditions. As discussed in section regional sea level changes, these changes are key drivers of coastal sea level anomalies in the region. Temperature changes in the Florida Current are also found to be uncorrelated with changes in the intensity of its flow  (Domingues et al. , 2018). These phases of warming and cooling of the Florida Current have important impacts on regional sea level changes along the US Southeast Coast.
Brazil Current The Brazil Current (BC) is the WBC of the South Atlantic subtropical gyre. There are two XBT transects that cross the BC: AX18 (Buenos Aires to Cape Town) at 34 • S and AX97 (Rio de Janeiro to Ilha da Trindade) at 22 • S. Started in 2002 and 2004, respectively, AX18 and AX97 are the longest continuous efforts to assess the structure and variability of the BC. The BC is of key importance in closing the mass budget in the South Atlantic, since it is the WBC that closes the subtropical gyre, transporting waters from subpolar regions, thus constituting an integral part of the AMOC. Until the implementation of these two XBT transects, most of the BC observations relied on sparse cruise data, short period mooring deployments, or models. A recent study  (Lima et al. , 2016)  used geostrophic velocity fields constructed from AX97 data to show that models generally misrepresent the structure of the variability of this current, simulating it as too deep, and too wide.
(2016)  (Figure  7 ). Studies that combine synergistic measurements of HD XBT data with altimetry and Argo observations are conducive to understanding the along-current variability of the EAC, resolving both the major jets and the EAC recirculation, and improving estimates of the basin-scale transports of mass, heat, and freshwater in the shallow South Pacific MOC. A pilot project that will merge data from the XBT network with multidisciplinary data from Argo floats, satellites, gliders, and ocean moorings is presently underway to connect ocean dynamics and productivity in the EAC and over the continental shelf.
Kuroshio Current The Kuroshio Current, the WBC of the North Pacific gyre, is sampled by XBT transect PX40 (Honolulu to Yokohama) that began in 1998. This transect is often combined with XBT transects PX37 (San Francisco to Honolulu) and PX10 (Honolulu to Guam) to estimate the complete trans-basin mean heat and freshwater transports in the North Pacific  (Uehara et al. , 2008; Douglass et al. , 2009 Douglass et al. , , 2010;; Auad et al. , 2011; Nagano et al. , 2012 Nagano et al. , , 2016 ). An analysis of the total heat budget of the North Pacific Ocean, including heat storage, air-sea flux, and heat transport by the ocean circulation, was carried out using HR XBT data and an ocean data assimilation model  (Douglass et al. , 2009 (Douglass et al. , , 2010)). The mean offset between the northward heat transport from XBT data and that estimated from the model is due to the low model resolution near the WBC and to a meridional offset in the simulated position of the North Equatorial Current.
XBT observations collected along AX25, when jointly analyzed with temperature and salinity climatological fields and nearby observations from Argo floats and satellite altimetry, show that the Subantarctic Front (SAF) and the Antarctic Polar Front transport together account for over 80% of the total ACC transport at this longitude  (Swart et al. , 2008). The year-to-year changes of frontal transports were driven by local winds associated with the Southern Annular Mode  (Domingues et al. , 2014). However, local winds were not directly linked to meridional excursions of these fronts in this region  (Sallée et al. , 2008). XBT data have shown that the location and transport of the various frontal regions along the AX25 transect do not have a strong annual cycle. The SAF transport, for example, is related to the local wind field and so exhibits a biannual period.