dataesr/scientific-paragraphs-categorization huggingface dataset:
This might have a negative impact on the bump hunting algorithm performance. Black Boxe datasets Here we discuss the results obtained for the black box dataset provided for the LHC Olympics challenge. Figure  14  shows the Euclidean distance distribution obtained for each black box. Compared to what was obtained with the R&D background, the distributions seem larger and globally shifted to the right. This is most likely due to the difference between the R&D background and the background generated in the black boxes. This fact shows that the method used is quite sensitive to the modeling of the background. Figure  15  shows the shaping function obtained using 100k events from each black box dataset. A preliminary fit was made to each of the distribution. Since the fit is suboptimal this might lead to the appearance of fake bump or fake deficit during the BumpHunter scan.
Finally Fig. 16  shows the results obtained with BumpHunter for all black boxes. As foreseen with the poor fit of the shaping functions, the constructed reference backgrounds do not fit well the data after cut on the Euclidean distance. In this condition and at the current stage of this work we can not really evaluate a meaningful p-value for a potential signal. If the results were good on the R&D dataset, it seems that the method is more challenging to apply without a good modeling of the background shape.
Lessons Learned The LHC Olympics challenge has been a good opportunity to test the potential of the GAN-AE algorithm that we have been developing. This shows the potential of this method with the good results on the R&D dataset, but also its limits. The results obtained revealed the sensibility of GAN-AE to the modeling of the background and to the correlation of the distance distribution with the dijet mass, despite the use of DisCo term. In addition, the fact that no background simulation that fits the black boxes data were available made the use of the BumpHunter algorithm difficult to apply.
Parallel and independent to our development and application of our conditional density estimation method, a similar one was applied in  [63] , to great results on the R&D dataset. The R&D dataset  [74]  was used for constructing and testing the method, while the first of the 'black boxes'  [38]  was the basis of our submission to the winter Olympics challenge. As the up to 700 particles given for each event are likely the result of hadronic decays we expect them to be spatially clustered in a number of jets. By focusing on the jet summary statistics rather than the particle data from an event we are able to vastly reduce the dimensionality of the data space. We note that this form of dimensionality reduction requires a small amount of prior knowledge and understanding of the data, and the assumption that the detected jets contain the anomaly, and other data-agnostic dimensionality reduction methods could instead be used.
While we found a larger than average reconstruction error for signal events, and latent space parameters to be noticeably different between background and signal events, on the R&D dataset, these autoencoder-based variables introduced more noise in the density estimation than the physics-based parameters, so they were not used in our final submission. Simple investigations of the dataset showed that it was smoothly distributed, and no anomalies were apparent by eye. We trained the conditional GIS on all events, and evaluated the anomaly score α for each datapoint. On the R&D set we found that point estimates of the conditional densities resulted in a larger noise level than convolving the conditional density with a Gaussian PDF of width σ = ∆ (1-PDF convolution for the background), discretely sampled at 10 points, so used the Gaussian-convolved probability estimates.
In this work we consider two fairly general bases of jet substructure observables, the so called mass-basis and the Lund-basis. In the mass basis we only include splittings from subjets of mass above 30 GeV. In the Lund basis we only include splittings from subjets which lie in the primary Lund plane. We emphasise that the resulting two data representations do not only differ in the observables included, but also in the set of splittings kept for each jet due to the different declustering cuts. In our current setting, the number of considered jets in an event is fixed to two (of highest p T ). 14 fter choosing a suitable data representation and binning, the procedure is as follows: We first split the dataset into overlapping invariant mass bins. In each bin, we perform a hyper-parameter optimization using perplexity to find the best LDA model.
This summed message h i is the bottleneck or encoded representation for the ith particle. The decoder is also an edge convolution layer, containing a network φ d with layers of sizes  (4, 32, 32, 4)  and ReLU activation functions, except for the final layer, which reconstructs each particle's momentum. We note that the architecture itself is insensitive to the ordering of the input particles. PyTorch Geometric supports variable-size input graphs so there is no need for zero-padding. The model is trained on the QCD background dataset with two different loss functions. The first is the mean squared error (MSE) between the input and output particles. This choice of loss function violates the permutation invariance of the algorithm because the particles must be reconstructed in the same order as they are input to achieve a small value of the loss function.
Results on LHC Olympics First, we studied our algorithm on the R&D dataset. As the truth information is provided, we can create a receiver operating characteristic (ROC) curve to determine the effectiveness of the PGAE to identify a signal (W → XY, X → qq, and Y → qq with m W = 3. 5 TeV, m X = 500 GeV, and m Y = 100 GeV) that it did not observe during training. The ROC curves for both the MSE and Chamfer loss functions are shown in Fig. 25. Although the MSE loss is not permutation invariant, we find it provides better discrimination for a new unseen signal. To evaluate our model's performance for anomaly detection, we perform a resonance search (or "bump hunt") in the dijet invariant mass m jj , computed from the two jets with highest p T in the event. We perform this dijet search in black box (BB) 1, which contains a resonant dijet signal at m jj ∼ 3.
8 TeV, and BB 2, which contains no signal. We require both of the jets to be "outliers," which we define as jets with a reconstruction loss exceeding a threshold corresponding to the 90% quantile of the loss distribution for the leading two jets in the corresponding evaluation dataset. We note that because our algorithm is jet-focused, it is straightforward to generalize this search to multijet events. For the background prediction in the signal-enriched outlier region, we perform a simplified analysis using the shape of the data in the background-enriched nonoutlier region. Specifically, we fit the ratio of the nonoutlier-to-outlier dijet mass distribution with a fourth-order polynomial to derive a transfer factor (TF). We take nonoutlier data distribution weighted by the TF as an estimate of the expected background in the outlier region.
Lessons Learned Graph neural networks, like our proposed particle graph autoencoder, are promising methods for anomaly detection. However, further work is needed to define a permutationinvariant loss function for use with such architectures that is more performant for anomaly detection. In addition, a more generic resonance search procedure, such a multimensional fit in the trijet, dijet, trijet, and single-jet mass distributions possibly using methods like Gaussian process fitting  [95] , would be appropriate to use in combination with this algorithm. In our experience, the R&D dataset was extremely helpful in preparing our anomaly detection algorithms and gauging whether the algorithm we were developing was on the right track. In the future, more extensive R&D datasets, together with additional black  boxes with different signals, may be useful.
3. 13. L manif old = ||x -g(g -1 (x))|| 2 (3. 12) L density = log p u (g -1 (x)) (3. 13) Regarding the preprocessing steps, the LHC Olympics datasets have been clustered and the following features have been selected for each of the two leading jets: p T , η, E, m, τ 3 /τ 2 , τ 2 /τ 1 , where τ n is the n-subjettiness. For these 12 features, the best performing manifold size was 8. This model offers the possibility to calculate both the density on the manifold and the reconstruction error from the projection on the manifold. We tried to use both of those metrics in order construct a robust anomaly score as in Eq. 3. 14. This metric performs the anomaly detection task better on the R&D dataset than its components and better than a basic normalizing flow model trained on the same data, judging by the ROC curves in Fig.
29. R exp (x) = ||x -g(g -1 (x))|| 2 1 + p u (g -1 (x)) (3. 14) While experimenting with this anomaly score, it became apparent that it generates a bias towards events with high dijet mass (m jj ). In order to decouple R exp from m jj we included the marginal likelihood of m jj , that was modeled using Kernel Density Estimation (KDE), as a term into the anomaly score. The resulting metric, denoted R m jj , uses the ratio between the likelihood on the manifold and marginal m jj likelihood as in Eq. 3. 15. R m jj (x) = ||x -g(g -1 (x))|| 2 1 + pu(g -1 (x)) p KDE (m x jj ) (3. 15) Translating the performance obtained on the R&D data to the black boxes proved to be a big challenge. The small differences in modeling from a black box to another are often enough to introduce significant biases. The only apparent solution seems to be training and applying the method on the same dataset.
Results on LHC Olympics The R&D dataset was heavily used for benchmarking different approaches, Fig. 29  shows the anomaly detection performance of different metrics on the R&D dataset. In order to evaluate the performance of this method in the absence of pure background training data, a small fraction (∼ 1%) of signal was introduced into a subsample from the R&D dataset. The resulting data sample was used both for training and evaluation of the model. Several cuts have been applied on R m jj while trying to find any indication of a resonance in the m jj spectrum. Although less apparent, there is still a bias towards identifying higher m jj events as being anomalous. The right plot in Fig. 3. 8. 2 shows the m jj distribution for events above the 50 th percentile of R m jj vs events above the 70 th percentile of R m jj.
In order to remove some amount of correlation between the jet masses and m JJ in background QCD events, we rescale the jet masses before they are input into the classifiers m J → m J = m J -30 GeV m JJ + 30 GeV 3000 GeV. (4. 2) The key part is the rescaling by dividing by m JJ , since the m J distributions have a strong scaling with m JJ. The additional offset by 30 GeV is not important, but was judged by eye to result in smaller correlation between m J and m JJ. By construction, this strategy is sensitive only to signals that result from the decay of a heavy resonance into two significantly lighter particles which each decay into largely hadronic boosted final states. This still covers a broad range of phenomenological possibilities, as the space of possible jet substructures is large. This has the potential to be sensitive to the the signal in the R&D dataset and BB1, but not to that in BB3.
We attempted to apply a modified form to the signal in BB3 without success, as briefly described at the end. The statistical independence of training and test sets is critical, and in order to retain as much statistical power as possible we perform a nested cross-validation procedure to select signal-like events. A detailed explanation follows. There are four training loops including the scan over m hyp , the additional ones running over loops indexed by the labels k, l, i. k We split the entire dataset (including events outside the signal and sideband regions) randomly into five k-folds, and when searching for a signal in the kth fold we train a classifier using the remaining k -1 folds. Given a pre-determined threshold efficiency sel , that fraction of highest scoring events is chosen from the kth fold as judged by the classifier trained on the other folds.
In this attempt, the top ten jets in each event have recorded their 4-momenta which act as the inputs to the classifiers (with zero-padding in the case of fewer jets in an event), and the total invariant mass of the system acts as the resonant variable. The jet-momenta are rescaled by this mass in an attempt to avoid correlations. The classifiers are trained simultaneously on BB3 data and also on QCD simulation from the R&D dataset, but in this second dataset the sideband and signal region labels are reversed ('tossed'). If the simulated background is similar to the true background, then this training strategy penalizes attempts to learn background correlations. Nonetheless, all attempts to apply this strategy and the original CWoLa SALAD strategy on BB3 led to heavy sculpting of the background m JJ distribution.
Machine Learning Setup There are two techniques that show great potential at model-independent anomaly detection: Classification Without Labels (CWoLa)  [67, 68, 107]  and deep autoencoders  [80, [108] [109] [110] [111]. These techniques have two important advantages over supervised methods. First, they are model independent and therefore allow to extend the sensitivity of current new physics searches to model-agnostic BSM scenarios. Second, they can learn directly from real data and thus do not rely on simulations that may suffer from potentially large mismodeling effects. In this section, we provide a comparative study between CWoLa and an autoencoder (AE) using a similar signal than the one released in the R&D dataset: two jets with masses m j 1 = m j 2 = 500 GeV. We examine the ability of the two methods to identify the new physics signal at different cross sections and to increase the significance of the signal excess.
CWoLa is expected to reach an excellent performance for large amounts of signal, while the AE should show a robust performance in the limit of low signal statistics. Therefore, these two approaches may have an intersection in performance at different cross sections that would be of great interest for real experimental searches. The R&D dataset represents a general new physics scenario where a signal is localized in one known dimension of phase space (in this case, the dijet invariant mass m JJ ) on top of a smooth background. In this scenario, CWoLa and the AE can be trained to exploit the information in the substructure of the two jets to gain discriminating power between the signal and background events. From the full dataset, we select all of the events in the range m JJ ∈ [2800, 5200] GeV and split them uniformly in log(m JJ ) in 15 bins.
After standardizing all the input features, we divide each bin of the full dataset in five parts to build five samples of events of equal size. First, four of these samples are used to perform four rounds of training and validation, using three different subsets for training and one for validation each time, and the other sample is saved for testing. For each cross-validation round, ten neural networks are trained for 200 epochs on the same training and validation data using different initializations. The performance of each classifier is measured on validation data according to the metric val. This metric is defined as the true positive rate for classifying signal region events as such, calculated at a threshold with a false positive rate z = 0. 5 % for incorrectly classifying sideband region events.
We found that one can incorporate the assumption of a resonant signal by requiring signal-like events fall in particular dijet mass window and scanning this window over the full range during a search (as in  [67, 68] ). This requirement helps to better isolate resonant signals and improves the performance of the resulting classifier. Our implementation of the TNT based anomaly search used jet images as the inputs for both the autoencoders and the TNT classifiers and CNN based model architectures trained with the Adam optimizer. We chose a latent size of 6 for the autoencoder based on results of the previous studies in the literature  [80, 108]. Based on results on the R&D Dataset we found that the second iteration of the Tag N' Train technique generally reached the plateau performance and so we used 2 iterations in our search.
No optimization of the model architectures and optimizer hyperparameters was attempted. A rough optimization of the selection of signal-like and background-like samples in the TNT technique was performed using the R&D dataset. In the first iteration, we used the 40% of events with the lowest autoencoder reconstruction losses as the background-like sample and the 20% with the highest as signal-like sample during the first iteration. In the second iteration, we once again used the 40% of events with the lowest scores as the background-rich sample, but tightened the signal-like cut to the top 10% of events. On the R&D dataset we found the performance was quite insensitive to the exact background-like cut used (as the resulting sample was always nearly pure background) and moderately sensitive to the signal-like cut used.
Results on LHC Olympics On the R&D dataset we compared the performance of the Tag N' Train classifiers to autoencoders and the CWoLa hunting  [67, 68]  method for various amounts of signal in the dataset (9%, 1%, 0. 3% and 0. 1% of the total dataset respectively). We generally found the Tag N' Train approach to be competitive with these other methods. For the 1% signal test, TNT produced a classifier that is somewhat worse than the one produced with TNT with an additional dijet mass cut (TNT + M jj ), but still had significantly improved performance with respect to the autoencoder. For the 0. 3% and 0. 1% signal tests, there was too little signal for the TNT classifier to learn from, and TNT performs significantly worse than the autoencoder. The TNT + M jj classifier performs similarly to the one trained using CWoLa hunting for the 3 tests with larger signal.
For the 0. 1% test the TNT + M jj classifier is able to achieve better performance better than that of the CWoLa hunting method, but does not improve with respect to the autoencoders approach. More details along with ROC curves are in the TNT paper  [113]. When applying the Tag N' Train search to Blackbox 1 we found a resonance at around 3800 ± 50 GeV with a local significance of 4σ. The bump-hunt plot for one of the subset is shown in Fig. 39. We had difficulty characterizing the nature of the signal as that was not extensively tested on the R&D dataset. We reported that one of the resonance's daughters had a mass of 270 ± 40 GeV (this was meant to be the lighter daughter) and did a very rough guess of the total number of signal events present. When doing an initial run over Black boxes 2 and 3 we did not see any significant evidence of a signal and we did not revisit Blackboxes 2 and 3 once the results of Blackbox 1 was revealed.
However we had not extensively studied this effect, making it difficult to extract the signal jet masses from the distributions of most signal-like events. We think with more deliberate study of these effects and/or using more interpretable model architectures, characterizing basic aspects of the signal (number of prongs, jet masses, etc. ) should be possible. What poses a more significant challenge is trying to estimate the signal cross section (total amount of signal present in the dataset) with an anomaly detection search that features a cut meant to isolate signal events. One can always set a lower bound based on the estimated number of signal events in the final fit, however because these events are selected with quite a low selection efficiency it will usually be a poor lower bound. Without specifying a particular model, one cannot know the signal efficiency of the selection imposed so it is difficult to estimate how far this lower bound is from the true amount of signal.
An approach that could be taken would be to try to calibrate the sensitivity of a technique in mock experiments on simulated datasets where the amount of signal is known. However it is likely that such a calibration, a mapping from observed p-value to total amount of signal present, depends on the nature of the signal and will not be universal. Some signals (e. g. those containing more exotic substructures) may be easier to find than others. Thus such a procedure would face difficult to estimate modeling uncertainties even if performed after signal characterization has been attempted.
Results on LHC Olympics The R&D dataset was used for the results presented in this section. The first step of the Dctr reweighting procedure is to train a classifier to distinguish the 'data' (Pythia) from the 'simulation' (Herwig) in a sideband region. The next step for Salad is to interpolate the reweighting function. The neural network is trained conditional on m jj and so it can be evaluated in the SR for values of the invariant mass that were not available during the network training. Note that the signal region must be chosen large enough so that the signal contamination in the sideband does not bias the reweighting function. Figure  40  shows a classifier trained to distinguish 'data' and 'simulation' in the signal region before and after the application of the interpolated Dctr model as well as τ 21.
Results on LHC Olympics The R&D dataset is used for the results presented here. For anomaly detection, the dijet invariant mass m JJ is used as the resonant feature. The classification features used are the invariant mass of the lighter jet, the mass difference between the two leading jets, and the N -subjettiness τ 21 of the two leading jets. As a benchmark, 1500 signal events corresponding to a fitted significance of about 2σ are injected into the data for training. For evaluation, the entire signal sample (except for the small number of injected events) is used. In order to demonstrate the breakdown of CWoLa in the presence of dependencies between the classification features and the resonant feature, we strengthen the dependence between the jet masses m J and invariant dijet mass m JJ by setting m J → m J + 0.
Method For the LHC Olympics challenge we opted for a semi-supervised approach. This was partly motivated by lack of time, and partly by the way the challenge itself was set up. Indeed, previous to the releasing of the blackboxes, the organisers had provided warm-up examples including signal and background labels. At the end we focused on a mixture of neural networks, with convolutional layers, and Boosted Decision Trees (BDTs). This hybrid approach was based on previous studies by one of the authors  [127] , which proposes to use a two step "pipeline" to assign event-by-event probabilities in categories signal or background. This model uses event input in two forms; raw data as an image as well as high level features (kinematic variables). We train the model for the labelled background and signal data sets (R & D data set).
For this analysis, we consider the first two leading jets. If there were only one jet, then all the kinematics of the second jet were set to zero values. We cluster again the constituents of the fat jets with R=0. 4 and k T algorithm with minimum p T condition of 20 GeV. With these jets we construct the following observables : p j 1 T , m j 1 , η j 1 , φ j 1 , E j 1 , p j 2 T , m j 2 , η j 2 , φ j 2 , E j 2 , δη, δφ, m/E j 1 , m/E j 2 , m jj , P A T (j 1 j 2 ), δR j 1 12 , δR j 1 13 , δR j 1 14 , δR j 1 23 , δR j 1 24 , δR j 1 34 , δR j 2 12 , δR j 2 13 , δR j 2 14 , δR j 2 23 , δR j 2 24 , δR j 2 34 , n subjets1 ,n subjets2. Some of the observables are constructed from the fat-jets kinematics and some of them are from their sub-jets. After we have a trained CNN model for the classification of the image dataset, we include in the tabular data the predicted scores from the CNN model for a given event.