dataesr/scientific-paragraphs-categorization huggingface dataset:
Computational Complexity of Comparing Neighbourhoods The full neighbourhood of a given instance consists of every other instance in the dataset. Comparing the ordering of the full neighbourhood is quite expensive, as it requires a O(n log(n)) comparison sort, which makes the complexity of evaluating the cost of a single GP individual O(n 2 log(n)). In GP-MaL we introduced a subsampling technique, which computes only a partial ordering of a selection of neighbours. The same technique is used in GP-MaL-MO: we choose the first k neighbours, followed by k of the next 2k neighbours (spaced evenly), k of the next 4k neighbours, and so on. This gives a sub-linear complexity of O(log(n) log(log(n))), which scales much more reasonably on large datasets. The reader is referred to our earlier work for more extensive discussion of this technique  [22].
Choice of Multi-Objective Algorithm We use the Multiobjective Evolutionary Algorithm Based on Decomposition (MOEA/D)  [45]  algorithm in this work for two main reasons. Firstly, we found it is particularly important to have strong diversity preservation as the EC process will naturally find it "easier" to generate non-dominated solutions with fewer trees (and higher cost) as it is difficult to find a large number of trees that can work well together, with a low cost value. MOEA/D is particularly well suited to diversity preservation due to its decomposition approach. The second reason is that MOEA/D also strongly encourages crossover to occur between solutions that represent similar trade-offs between the two objectives (i. e. similar sub-problems) through its use of a neighbourhood. This is again important to this work, as exchanging information between individuals with similar t values is inherently more productive than those with dissimilar values of t, given that they are capable of preserving similar amounts of the dataset structure.
UMAP is often regarded as the state-of-the-art manifold learning technique, and a spiritual successor to the widely-known t-Distributed Stochastic Neighbour Embedding (t-SNE) method  [27]  that we compared to in our initial work. -A single hidden layer auto-encoder (AE)  [18] : auto-encoders are a specialised neural network used for (unsupervised) representation learning. In this work, we use a simple auto-encoder with a single hidden (embedding) layer that has the number of nodes equal to the number of desired created features. While more advanced AEs exist, they are generally outperformed by methods such as UMAP and are very difficult to interpret. Each of the above methods are somewhat less expensive than GP-MaL-MO for the sizes of datasets tested: for example, MDS and LLE are approximately O(n 2 ) (depending on implementation), whereas UMAP is O(n 1.
We apply GP-MaL-MO and each of the baseline manifold learning methods to each classification dataset without the labels present, producing a learned lowdimensional manifold. We then measure the classification accuracy attained on this transformed data by using 10-fold cross-validation with both the k-Nearest Neighbour (kNN) and Random Forest (RF) classifiers. kNN is used as an example of a very simple, distance-based classification algorithm. RF, in contrast, is much more sophisticated (using an ensemble decision-based approach) and is widely used for its high classification accuracy and applicability to a wide range of datasets  [44]. We use the standard default implementations of this classifiers in the scikit-learn package  [34] , with k = 3 for kNN, and 100 base estimators for RF. The 11 datasets used are summarised in Table  2.
Experiment Settings GP-MaL-MO was run 30 times with different initial seeds, due to its stochastic nature. The parameter settings of GP-MaL-MO are shown in Table  3. These parameters are reasonably standard for multi-objective GP. We used a population size of 100 as a large population is not as necessary to capture all trade-offs when one of the objectives is discrete; instead, we used a reasonably large number of generations (1000) to allow the front to be well-optimised. The "standard" mutation and add/remove mutation each had a 15% probability of occurring, leaving crossover to occur the other 70% of the time. The maximum number of trees was capped at half the number of features in the dataset. Arguably, fewer trees may be sufficient, but we wanted to test if GP-MaL-MO would be able to use a large maximum number of trees effectively, by finding a good (and smaller) number of trees to use.
Results We will evaluate the performance of GP-MaL-MO in two stages: first we will discuss the approximated Pareto fronts attained by GP-MaL-MO, and how this front maps to classification accuracy; we will then compare GP-MaL-MO to the baseline classifiers. The orange line represents the 10-fold test accuracy achieved by RF when using all the original features of the dataset. This represents, in some sense, the optimal performance that can be achieved only if GP-MaL-MO successfully preserves all necessary structure from the high-to low-dimensional space. The secondary (top) x-axis on the second plot shows how large the number of trees used is compared to the original feature set size. The test performance of the GP-MaL results are also shown on the second plot, using an upwards/downwards facing triangle for RF and KNN, respectively.
Wine: On the simplest dataset, GP-MaL-MO achieves over 90% test accuracy even when only using a single tree -the Wine dataset is quite well-separated and has a small number of features, which means one tree is enough to preserve sufficient  structure. Accuracy improves slightly with additional trees, but effectively plateaus at about four features (≈70% dimensionality reduction). Compared to using all features, four created features gives only about 2% lower RF test accuracy. The training accuracy of RF is nearly 100% even with a single created feature. Given the ensemble approach of RF, it is easy for it to achieve perfect training performance -this same pattern is seen in the remaining datasets. Vehicle: On the the Vehicle dataset, increasing the number of trees gives a much more meaningful increase in classification performance than on Wine.
This is matched by large succesive drops in cost from one to four trees. Using half as many created features as original features gives a RF test accuracy within a few percentage points of using the full original feature set. Image Segmentation: GP-MaL-MO produces a very similar Pareto front on this dataset as on Vehicle, but with an even stronger relationship between decreasing cost and increasing classification accuracy. As cost begins to decrease more slowly at t = 4, test accuracy also begins to level off. By the time ten trees are used, RF test accuracy is almost identical to the accuracy achieved on the original feature set. Ionosphere: The test accuracy for KNN on the Ionosphere dataset begins to decrease when more than six trees are used. RF, in contrast, continues to slowly increase in accuracy until around 13 trees.
This is perhaps due to the "naive" approach of KNN -by using only raw distances to perform classification, KNN may become overly sensitive at a high number of trees on this dataset. Dermatology: The Dermatology dataset shows a similar pattern to the Ionosphere dataset, except that both RF and KNN are able to achieve 90% test accuracy when only a single tree is used. Movement Libras: The plots for the Movement Libras dataset show clear diminishing returns once a certain number of trees are used (t ≈ 15). Very little further improvement is gained at higher values of t, which highlights one of the key benefits of GP-MaL-MO: from examining the approximate Pareto front, we can clearly see the majority of the manifold structure has been preserved by t = 15, suggesting this is an appropriate number of created features on this dataset.
Madelon: The Madelon dataset produces remarkably different patterns to any of the datasets used in this paper. The approximated Pareto front does not appear as a log curve, but rather is essentially linear across most of the front. Even more notably, the classification performance on Madelon does not plateau -rather, there is a sharp spike in test accuracy around five trees and then it quickly drops. Madelon is an artificial dataset designed for testing feature selection algorithms. Of its 500 features, only five are actually useful. The remaining features consist of 480 "useless" (noisy) features; 10 repeated features; and 5 redundant features. The spike in classification performance at five trees corresponds to GP-MaL-MO discovering the five useful features  3  , and producing a low-dimensional manifold consisting solely of them (or their redundant features).
At more than five trees, GP-MaL-MO is likely attempting to preserve the "structure" of the noisy features, which leads to KNN and RF struggling to classify accurately. This pattern shows the power of GP-MaL-MO to preserve the most important structure (the useful features) of a dataset at a small number of trees -despite there being 500 features available, GP-MaL-MO is able to reliably detect the five best ones. MFAT, MNIST 2-class, Yale, and COIL20: The final four datasets, which have the highest dimensionality, all have quite similar patterns and so we will discuss them  together. On all three datasets, cost decreases substantially until around 15-30 trees, at which point it decreases only very slowly as additional trees are added. On MFAT, MNIST, and COIL20, the approximated Pareto front is not entirely concave -this is due to the front being averaged over 30 runs: not all runs produce all numbers of trees.
On all three datasets, classification accuracy begins to plateau at around 30-45 trees. At this point, test accuracy on RF is nearly the same as when using all features in the original feature set, despite the number of trees being only about 5% of the number of original features. On the Yale and COIL20 datasets, which have 1,024 features, GP-MaL-MO never uses more than ≈ 155 features, which represents a dimensionality reduction of over 80%. This is a particular benefit of a MO approach: GP-MaL-MO eliminates individuals with higher numbers of trees as they are dominated by other individuals who have fewer trees at the same (or lower) cost.
General Findings Across all the datasets tested (except Madelon), GP-MaL-MO was able to produce a well-formed approximate Pareto front that gives a clear trade-off between complexity (number of trees) and cost (i. e. approximate classification performance). There was a very strong correlation between cost and test accuracy: the test accuracy plot is essentially an inverse of the approximate Pareto front. Another pattern found across all the tested datasets is the diminishing returns achieved when increasing the number of trees. Above about 30-45 trees, there is very little further decrease in test accuracy as the number of trees rises. On the datasets with highest dimensionality, this means that up to a 95% reduction in dimensionality can be achieved without much loss of manifold structure.
At a high number of created features, the MaL methods mostly converge, as it becomes increasingly trivial to maintain all structure in the dataset. For each method in each plot, we include the maximum test accuracy in brackets in the legend (across all numbers of created features, not just up to 20) to show the maximum potential of that method. We also include an orange line in each plot that gives the test accuracy of the classifier when using all features: this represents a type of upper-bound on performance. Wine: On the Wine dataset, the five methods give mostly comparable results, except for AE, which struggles to perform competitively at a small number of created features. On a single created feature, PCA and MDS provide about 10% lower test accuracy using both the KNN and RF classifiers than the other MaL methods.
On RF, LLE also provides clearly lower performance than UMAP and GP-MaL-MO for smaller numbers of created features. Vehicle and Image Segmentation: UMAP achieves a relatively high accuracy of ≈ 60% with a single created feature on the Vehicle dataset, whereas the other methods achieve ≈ 40%. By the time six created features are used, the other methods achieve similar performance to UMAP, and then GP-MaL-MO and PCA outperform UMAP at 8-9 created features. A similar pattern occurs on the Image Segmentation dataset, where UMAP performs very well at small numbers of created features, but is matched or slightly outperformed by other methods at higher numbers of features. GP-MaL-MO has a higher maximum accuracy than UMAP, on both these two datasets, achieving 2-3% higher using the RF classifier.
GP-MaL-MO clearly has the ability to provide high classification accuracy given enough features. The LLE method also starts out quite strongly, but has a much lower maximum accuracy than the other methods. Ionosphere: GP-MaL-MO, PCA, AE, and MDS outperform UMAP and LLE on the Ionosphere dataset, with around 5% higher maximum accuracy. LLE has a similar pattern to on Image Segmentation, where it has comparatively strong performance for few features, but then improves little more with additional features. Dermatology: All the methods are able to achieve very good accuracy with six or more created features on the Dermatology dataset. GP-MaL-MO and UMAP both have strong performance with only a single created features, while the other methods start off worse. UMAP is clearly suited to this dataset, achieving near-maximum accuracy with a single feature.
Movement Libras: The LLE method lags behind on both classifiers on the Movement Libras dataset. UMAP again starts off with strong performance at low numbers of features, but is ultimately outperformed by GP, MDS, and/or PCA at higher dimensionality. Madelon: The Madelon dataset again produces very distinctive patterns. Both PCA and GP-MaL-MO are able to identify the five meaningful features, whereas the other methods seem to be greatly affected by the noise in the dataset. UMAP is slightly more successful than MDS and LLE, but all three methods struggle to achieve much more than 60% accuracy. AE has poor performance when using only a few created features, but increases reasonably consistently as the number of created features rises. It is not unexpected that PCA would perform well. PCA computes the next direction of maximum variance for each successive created feature; the directions of highest variance are very likely to be along the five meaningful features in the dataset.
MFAT: The UMAP and LLE methods appear to be very well-suited to the MFAT dataset, achieving nearly 98% accuracy with one and two created features respectively. The other four methods each follow a similar pattern, starting out with quite poor accuracy before reaching similar accuracy to UMAP and LLE at around ten created features. MNIST 2-class: As on the MFAT dataset, UMAP and LLE achieve very good results with only one or two created features. The remaining methods all achieve over 90% classification accuracy by the time that three created features are used, and then slowly converge to around 95% accuracy as additional constructed features are used. GP-MaL-MO appears to be slightly worse than the other methods on the KNN classifier (around 1% less accurate), but is nearly indistinguishable when RF is used.
Yale: In contrast to MFAT and MNIST, on the Yale dataset the LLE method has arguably the worst performance at a low number of created features. UMAP again starts off quite strongly at one or two features, and then quickly levels off. With the RF classifier, the maximum performance of UMAP is 69. 67%, whereas GP-MaL-MO and MDS achieve the best accuracy of 80. 67%. GP-MaL-MO produces much better results with the RF classifier than with KNN on this dataset -this is consistent with the results achieved when using all features, which suggests that this dataset benefits from the use of a more sophisticated classification algorithm. COIL20: On the final dataset, UMAP and LLE again both have the best performance at a small number of created features. GP-MaL-MO starts off with poor performance, but has the best overall accuracy with a sufficient number of features on RF, at 98.
Discussion There are a number of common patterns across the 11 datasets, which provide a number of interesting findings and suggestions for future work. The proposed method, GP-MaL-MO, is always able to achieve near-maximal test accuracy given a sufficent number of features, using the RF classifier. Other methods such as UMAP and LLE often have high relative performance for a small number of features, but then plateau as the number of created features increases. On most datasets, PCA and MDS provide quite similar results to GP-MaL-MO. However, PCA necessarily uses all the original features in each constructed feature; MDS provides only a low-dimensional embedding and not a way of mapping this to the original feature space. While AE does produce a mapping, it has the worst performance of the MaL methods across nearly all of the datasets.
Further Analysis To better understand the potential of GP-MaL-MO for performing manifold learning, we will analyse some sample learned individuals in this section. Of the 11 datasets considered in this paper, the COIL20 dataset has the largestequal number of features (1,024) and the highest number of classes (at 20). As such, it can be considered to be the "biggest" problem: that is, the one that can benefit most greatly from effective manifold learning. An example GP individual trained on the COIL20 dataset, shown in Fig. 23 , has 29 trees (3% of the original dimensionality) and a very low cost of 0. 035. On 10-fold cross validation, it achieves 99. 1% test accuracy using RF, and 96. 6% test accuracy using KNN (higher than when using all features). Even with this impressive accuracy, the evolved trees are surprisingly simple: 23 of them are single selected features, and the other 6 use only 24 additional unique features from the original feature set.
These 29 trees use only 47 out of 1024 original features (4. 6%), which is clearly a significant reduction in dimensionality. The three most commonly used features are f808 (9 times), f151 (6 times) and f541 (4 times). These three pixels in the original COIL20 images are likely to be particularly characteristic in separating the 20 classes. On the MFAT dataset, a very simple but powerful individual was found containing 12 trees, as shown in Fig. 24. The embedding produced by this individual has a cost of 0. 033 and achieves 94. 75% test accuracy on RF, and 95. 8% using KNN. Only 13 of the 649 features are used, and the most complex of the trees uses only six functions and three unique features. This model could be used very efficiently on future data (without retraining), as it it requires very little computation to map the 649-dimensional feature space into the 13-dimensional embedding.
This model could also be potentially used to give significant insight into the MFAT dataset, by further examining the 13 features used to see how they separate the data into classes. Both these two example individuals show the potential of GP-MaL-MO to not only produce representative embeddings at relatively low dimensionality, but also for the evolved mapping to be interpretable and computationally simple. This is a benefit of GP-MaL-MO that is not applicable to any of the other baseline MaL methods. In future work, we plan to investigate this further, by introducing parsimony pressure or a third conflicting objective that minimises the size/complexity of the learned GP trees.
We also further refined our previous measure of manifold quality. ReLU f298 (3) Comprehensive testing of GP-MaL-MO showed that it was able to automatically find clear trade-offs between increasing manifold quality and dimensionality. The specialised crossover and mutation operators allowed the proposed method to scale upwards of 100 trees, while the multi-objective algorithm prevented the maximum number of trees being larger than necessary. GP-MaL-MO was shown to be competitive with a range of existing manifold learning methods across a variety of datasets, despite using a more indirect mapping-based approach and using only a subset of the feature set in every tree. Further analysis showed that GP-MaL-MO was able to create interpretable and very efficient models that used a minimal number of features and functions within GP trees.
Figures 1 to 11  show the results of GP-MaL-MO for the 11 datasets, ordered by increasing dimensionality. For each dataset, there are two plots. The first plot shows the mean approximated Pareto front (over the 30 runs), with the number of trees on the x-axis and the cost on the y-axis. The second plot shows the mean 10-fold crossvalidation accuracy across the 30 GP-MaL-MO runs for the two classification algorithms. RF and KNN are represented by a green and blue line respectively, with dotted lines used for training performance and solid lines for test performance. The orange line represents the 10-fold test accuracy achieved by RF when using all the original features of the dataset. This represents, in some sense, the optimal performance that can be achieved only if GP-MaL-MO successfully preserves all necessary structure from the high-to low-dimensional space.
5. 1 1 Figures 1 to 11  show the results of GP-MaL-MO for the 11 datasets, ordered by increasing dimensionality. For each dataset, there are two plots. The first plot shows the mean approximated Pareto front (over the 30 runs), with the number of trees on the x-axis and the cost on the y-axis. The second plot shows the mean 10-fold crossvalidation accuracy across the 30 GP-MaL-MO runs for the two classification algorithms. RF and KNN are represented by a green and blue line respectively, with dotted lines used for training performance and solid lines for test performance. The orange line represents the 10-fold test accuracy achieved by RF when using all the original features of the dataset. This represents, in some sense, the optimal performance that can be achieved only if GP-MaL-MO successfully preserves all necessary structure from the high-to low-dimensional space.
Study Population We conducted a retrospective study at Amiens and Nancy University Hospitals in France. Consecutive patients with CD treated with anti-tumor necrosis factor agents (infliximab and adalimumab) or vedolizumab between April 1, 2008 and December 31, 2015 were identified by using existing databases of these 2 centers. The inclusion criteria were (1) adult patients with CD in clinical remission, (2)  available colonoscopy before the initiation of biologics, and (3) a second colonoscopy with a CDEIS available and <4. The patients' charts were reviewed for demographic information, disease duration, disease location, and phenotype according to the Montreal classification, previous and concomitant CD-related medications, smoking status, and any history of CD-related surgery. It is noteworthy that in these 2 centers, CDEIS scoring is part of routine practice.
Implications for patient care Patients with CD and complete mucosal healing, based on endoscopy scores, require fewer surgeries and hospitalizations and have a significantly reduced risk of treatment failure compared with patients with only partial mucosal healing. hospitalization expressed as hazard ratio (HR) (95% confidence interval [CI]). All variables identified with P <. 10 were included in a multivariable model. The analysis was performed with SAS software version 9. 4 (SAS Institute, Cary, NC). The protocol was approved by the CNIL committee (Comite Consultatif sur le Traitement de l'Information en matière de Recherche dans le domaine de la Sante N T. 194 et 1404720).