dataesr/scientific-paragraphs-categorization huggingface dataset:
The log of the MATISSE observations is given in Table  2. The raw data were processed using the version (1. 5. 5) of the MATISSE data reduction software 2. The steps of the data reduction process are described in  Millour et al. (2016). The MATISSE absolute visibilities are estimated by dividing the measured correlated flux by the photometric flux. Ther-mal background effects affect the measurement in the M and N bands significantly more than in the L band. Thus, we use only the L band visibilities in our modeling and analysis, while the L and M photometries are used for deriving the observed total flux (see Sect. 3. 2). In the N band, the total flux of Car (≈17 Jy) is at the lower limit of the ATs sensitivity with MATISSE for accurate visibility measurements (∼ 20 Jy, as stated on the ESO webpage of the MATISSE instrument).
Calibration of the squared visibility in the L band Calibrators were selected using the SearchCal tool of the JMMC  3  , with a high confidence level on the derived angular diameter (χ 2 ≤ 5, see Appendix A. 2 in Chelli et al. (  2016 )). Moreover, calibrator fluxes in the L and N bands have to be of the same order than for the science target (i. e. ≈10 to 100 Jy in L and N) to ensure a reliable calibration  (Cruzalèbes et al. 2019 ). These flux requirements induce the use of partially resolved calibrators at the 132-m longest baseline. The L band uniform-disk (UD) angular diameters for the standard stars as well as the corresponding L and N band fluxes and the spectral type are given in Table  3. We analyze these calibration stars with a special care. The first calibrator of this night, q Car, was observed in the frame of a different program and was not used in the interferometric visibility calibration process.
These models are indeed suitable for deriving precise synthetic photometry  (Castelli 1999) , in comparison with MARCS models giving rough flux estimates 7. The gray line corresponds to the best SPIPS model, which is composed of the latter model without CSE plus an IR excess model. In the angular diameter panels, the gray curve corresponds to limb-darkened (LD) angular diameters. For stars with solar metallicity, when effective temperature is low enough, CO molecules can form in the photosphere and absorb light in the CO band-head at 4. 6 µm  (Scowcroft et al. 2016 ). This effect is observed in the Spitzer I2 IRAC dataset. In this case, these data were ignored during the fitting of SPIPS. Panels presenting a horizontal blue bar contain only one data with undetermined phase of observation, thus these photometric mid-infrared bands were not used in the IR excess reconstruction in Sect.
6 µm (Scowcroft et al. 2016 ). This effect is observed in the Spitzer I2 IRAC dataset. In this case, these data were ignored during the fitting of SPIPS. Panels presenting a horizontal blue bar contain only one data with undetermined phase of observation, thus these photometric mid-infrared bands were not used in the IR excess reconstruction in Sect. 2. 1 and 2. 2.
Acknowledgements. The authors acknowledge the support of the  French Agence Nationale de la Recherche (ANR) , under grant  ANR-15-CE31-0012-01  (project  UnlockCepheids ). The research leading to these results has received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation program  under grant agreement No  695099  (project  CepBin ). This research was supported by the  LP2018-7/2019  grant of the  Hungarian Academy of Sciences. This research has been supported by the  Hungarian NKFIH  grant  K132406. This research made use of the SIMBAD and VIZIER 5 databases at  CDS, Strasbourg (France)  and the electronic bibliography maintained by the  NASA /ADS system. This research also made use of Astropy, a community-developed corePython package for Astronomy (Astropy Collaboration et al.
A new paradigm for data-driven, model-agnostic new physics searches at colliders is emerging, and aims to leverage recent breakthroughs in anomaly detection and machine learning. In order to develop and benchmark new anomaly detection methods within this framework, it is essential to have standard datasets. To this end, we have created the LHC Olympics 2020, a community challenge accompanied by a set of simulated collider events. Participants in these Olympics have developed their methods using an R&D dataset and then tested them on black boxes: datasets with an unknown anomaly (or not). This paper will review the LHC Olympics 2020 challenge, including an overview of the competition, a description of methods deployed in the competition, lessons learned from the experience, and implications for data analyses with future datasets as well as future colliders.
In particular, single events often contain no useful information -it is only when considering a statistical ensemble that an anomaly becomes apparent. This is a contrast between anomaly detection that is common in industry ("off manifold" or "out-of-sample" anomalies) and that which is the target of searches in high energy physics ("over-densities"). Furthermore, HEP data are systematically different than natural images and other common data types used for anomaly detection in applied machine learning. In order to test the resulting tailored methods, it is essential to have public datasets for developing and benchmarking new approaches. For this purpose, we have developed the LHC Olympics 2020 challenge and corresponding datasets  [31]. The name of this community effort is inspired by the first LHC Olympics that took place over a decade ago before the start of the LHC  [32] [33] [34] [35].
As stressed above, calibrating the background prediction is an essential aspect of BSM searches and so we have restricted this search to high energy hadronic final states where sideband methods can be used to estimate the background. We provide lists of the detectorreconstructed final state particles in order to allow contestants to test methods that can process low-level event information. To aid in method development and testing, we also provide a simulated dataset (with no anomalies) and a benchmark signal dataset. These two are combined to form the R&D dataset, and provided in addition to the black box datasets. The goal of this paper is to describe the Winter  [36]  and Summer  [37]  Olympics 2020 competitions. Well over one hundred researchers participated in these events, with over a dozen teams submitting their results for the black boxes.
This paper is organized as follows. Section 2 introduces the LHC Olympics competition, including the R&D and black box datasets. A brief description of methods deployed in the competition are provided in Secs. 3, 4, and 5. Each contribution includes an in-troduction to the method, a concise statement of the results, as well as lessons learned before, during, and/or after the challenge. The results and lessons learned are synthesized in Sec. 6. Implications for data analyses with future datasets and as well as future colliders are discussed in Sec. 7 and the paper concludes in Sec. 8.
Dataset and Challenge The portal for the LHC Olympics dataset can be found at the challenge website  [31]. The datasets described below are all publicly available and downloadable from Zenodo  [38]. Contestants entered their results in a Google form. On the form, participants were asked to state: • The black box number  (1) (2) (3)  corresponding to their submission. • A short abstract describing their method. • A p-value associated with the dataset having no new particles (null hypothesis). • As complete a description of the new physics as possible. For example: the masses and decay modes of all new particles (and uncertainties on those parameters). • How many signal events (with the associated uncertainty) are in the dataset (before any selection criteria). Additionally, contestants were encouraged to submit plots or a Jupyter notebook  [39].
R&D Dataset The R&D dataset consisted of one million SM events each comprised of two jets produced through the strong interaction, referred to as quantum chromodynamics (QCD) dijet events, and 100,000 Z → XY events, with X → q q and Y → q q, as shown in Fig. 1  for the topology. The masses of the new BSM particles Z , X, and Y are 3. 5 TeV, 500 GeV and 100 GeV, respectively. The events were produced using Pythia 8. 219  [43, 44]  and Delphes 3. 4. 1  [45] [46] [47] , with default settings, and with no pileup or multiparton interactions included. They are selected using a single large-radius (R = 1) anti-k T  [48]  jet trigger with a p T threshold of 1. 2 TeV. The signal model was discussed in Ref. [11]  and has the feature that existing generic searches for dijet resonances or targeted searches for diboson resonances may not be particularly sensitive.
For example, existing searches may register a low significance (< 2 σ) while automated methods may be able to identify the signal with a high significance. These events are stored as pandas dataframes  [49]  saved to compressed HDF5  [50]  format. For each event, all Delphes reconstructed particles in the event are assumed to be massless and are recorded in detector coordinates (p T , η, φ). More detailed information such as particle charge is not included. Events are zero padded to constant size arrays of 700 particles, with the truth bit appended at the end to dictate whether the event is signal or background. The array format is therefore (N events =1. 1 M, 2101). q q X Y q q q q Z Table  1. Pythia settings for the different datasets. For R&D the settings were the Pythia defaults while for BB1 and BB3 they were modified.
Black Box 1 This box contained the same signal topology as the R&D dataset (see Fig. 1 ) but with masses m Z = 3. 823 TeV, m X = 732 GeV and m Y = 378 GeV. A total of 834 signal events were included (out of a total of 1M events in all). This number was chosen so that the approximate local significance inclusively is not significant. In order to emulate reality, the background events in Black Box 1 are different to the ones from the R&D dataset. The background still uses the same generators as for the R&D dataset, but a number of Pythia and Delphes settings were changed from their defaults. For the Pythia settings, see Table  2  1. For the Delphes settings, we changed EfficiencyFormula in the ChargedHadronTrackingEfficiency module, ResolutionFormula in the ChargedHadronMomentumSmearing module, and HCalResolutionFormula in the hadronic calorimeter (HCal) module.
Individual Approaches The following sections describes a variety of approaches to anomaly detection. In addition to an explanation of the method, each section includes a set of results on the LHC Olympics datasets as well as a brief description of lessons learned. We have grouped the various methods into three loose categories: Unsupervised (Sec. 3), Weakly Supervised (Sec. 4), and (Semi)-Supervised (Sec. 5). Supervision refers to the type of label information provided to the machine learning algorithms during training. Unsupervised methods do not provide any label information and learn directly from backgrounddominated data. Typically, these methods try to look for events with low p(background). (Exceptions exist, see e. g. ANODE in Sec. 3. 2 and GIS in Sec. 3. 5 which use likelihood ratios.
) Weakly supervised methods have noisy labels. 3 Many of these approaches operate by comparing two datasets with different amounts of a potential signal. The labels are noisy because instead of being pure 'signal' and 'background', the labels are 'possibly signal-depleted' and 'possibly signal-enriched'. The goal of these methods is to look for events with high p(possibly signal-depleted)/p(possibly signal-enriched). Supervised methods have labels for each event. Semi-supervised methods have labels for some events. Methods that are labeled as (Semi-)Supervised use signal simulations in some way to build signal sensitivity. These three categories are not exact and the boundaries are not rigid. However, this categorization may help to identify similarities and differences between approaches.
Within each category, the methods are ordered alphabetically by title. Furthermore, the results on the datasets can be grouped into three types: (i) blinded contributions using the black boxes, (ii) unblinded results or updates on blinded results (and thus, also unblinded) on the black boxes, and (iii) results only using the R&D dataset. All three of these contribution types provide valuable insight, but each serves a different purpose. The first category (i) corresponds to the perspective of a pure challenge that is analogous to a real data analysis. The organizers of the LHCO challenge could not participate in this type of analysis. Section 6. 1 provides an overview of the challenge results. The LHC Olympics datasets have utility beyond the initial blinded challenge as well and so contributions of type (ii) and (iii) are also important.
When compared to p T -sorted constituent ordering, the k t -sorted sequence consistently travels back and forth between each prong, making their existence readily apparent and easy to model. As a result, a significant boost in performance is observed. The loss function L(t) for each constituent, defined in Eq. 3. 2, is very similar to that of an ordinary VAE. It consists of a mean-squared-error (MSE) loss between input constituents and generated output constituents as a reconstruction loss, as well as a weighted KL-Divergence from the learned latent space prior to the encoded approximate posterior distribution. Since softer constituents contribute less to the overall classification of jet substructure, each KL-Divergence term, computed constituent-wise, is weighted by the constituent's p T -fraction with respect to the jet's total p T , averaged over all jets in the dataset to avoid correlations with constituent multiplicity.
The weight coefficient of the KL-Divergence term is enforced as a hyperparameter, and has been optimized to a value of 0. 1 in dedicated studies. L(t) = MSE + 0. 1 × p T (t)D KL (3. 2) After a jet is fully processed by the VRNN, a total loss function L is computed as the average of the individual constituent losses over the jet: L = ΣL(t) N. The architecture is built with 16 dimensional hidden layers, including the hidden state, with a two-dimensional latent space. All hyperparameters used are determined by a hyperparameter optimization scan. The model is trained on the leading and sub-leading jets of each event, where events are taken from the LHC Olympics datasets. After training, each jet in the dataset is assigned an Anomaly Score, defined in Eq. 3. 3, where D KL is the KL-Divergence from the learned prior distribution to the encoded posterior distribution.
Results on LHC Olympics The performance of the VRNN was first assessed with the LHC Olympics R&D dataset, which includes a known signal of a beyond-the-Standard-Model Z boson with a mass of 3500 GeV which decays to two hadronically decaying X and Y particles, each reconstructed by a R = 1. 0 jet. This study was used as a validation of the method, with a goal of directly investigating the ability of the Event Score selection to reconstruct the Z mass. Therefore, no selections beyond those described in Section 3. 1. 1 are applied. The VRNN was trained over a contaminated dataset consisting of 895113 background events and 4498 signal events, corresponding to a signal contamination level of 0. 5%. A selection on the Event Score is applied as the sole discriminator, and the invariant mass m JJ of the two jets is then scanned to assess the prominence of the Z mass peak.
Furthermore, the behavior of the Event Score consistently a↵ects the distributions of both Black Box 2 and the Background dataset. It is important to note that the model was trained independently on each dataset, and the resulting Event Scores are from entirely unique sets of network weights. Figure  6  shows our results for Black Box 3. Our model is specifically sensitive to boosted final states, and as a result, we are insensitive to the signal present in this Black Box.
is chosen to significantly enrich the fraction of anomalous jet events over the background, while retaining enough statistics in the background to display its smoothly falling behavior. Figure  4  shows the dijet invariant mass distributions before and after the Event Score selection, along with the local significance of the signal computed in each bin using the Bi-nomialExpZ function from RooStats with a relative background uncertainty of 15%  [59]. Applying this selection dramatically increases the significance of the excess from 0. 18σ to 2. 2σ without significantly sculpting the shape of the background. Once the method was validated in the R&D dataset, it was applied to Black Box 1, with a re-optimized tighter selection on the Event Score of 0. 75, as well as a requirement on the pseudorapidity of the leading and sub-leading jets to be less than 0.
75, to ensure that central, high momentum transfer events are considered. Figure  5  shows the dijet invariant mass for both the Black Box 1 and Background datasets. The Event Score selection reveals an enhancement in m JJ just below 4000 GeV. This is consistent with the Black Box 1 signal, which is a new Z boson with a mass of 3800 GeV decaying to two new particles, each decaying hadronically. The same method applied to Black Box 2, shown in Fig. 6 , results in no significant  excesses in the invariant mass distribution. Additionally, the effect of the Event Score selection on the m JJ shapes is similar between the Black Box 2 and Background datasets. Black Box 2 does not contain any beyond-the-Standard-Model events, and therefore these results are consistent with a QCD-only sample. It is important to note that the model was trained independently on each dataset, and the resulting Event Scores are from entirely unique sets of network weights.
Lessons Learned This challenge presented a highly useful avenue for the development of our model. Results from the R&D and Black Box dataset analyses indicate that the VRNN is capable of identifying anomalies via sequence modeling, as we have shown in the context of searching for anomalous substructure within boosted hadronically decaying objects. We learned that the pre-processing method is hugely influential on the performance of the model, in particular the choice of k t -ordered sequencing. We feel that this is a generalizable conclusion from our study which can be applied to the understanding and use of jet substructure in future analyses. Given these lessons, there are a variety of future opportunities with this application of the VRNN architecture to jet-level anomaly detection. Since the VRNN takes constituent information as input and learns jet substructure without explicit reliance on high level variables, it is expected to have less correlation with jet mass than standard substructure variables such as n-subjettiness.
Method The BuHuLaSpa method assumes that the LHCO event data was generated through a stochastic process described by an underlying probabilistic generative model with continuous latent variables. We use neural networks as approximators to the likelihood and posterior distributions of the model, and use the variational autoencoder (VAE) architecture as a means of optimising these neural networks. For each event in the dataset we cluster the hadrons, select the resulting two leading p T jets, and order these by mass, m j 1 > m j 2. The data representation we use for the LHCO consists of the following observables for each jet: jet mass m j , the n-subjettiness observables τ 2 /τ 1 and τ 3 /τ 2 , and an observable similar to soft drop defined by clustering the jets with the C/A algorithm, then de-clustering them branch by branch, and summing the ratios of parent to daughter subjet masses along the way, stopping at some pre-defined mass scale which we have chosen to be 20 GeV.
This novel way of inferring on what the network has learned could lead to new approaches to bump hunting with machine learning at colliders, or even more broadly to machine learning applications in highenergy physics. Optimization and classification Using the R&D dataset we investigated how best to train the VAE, and then applied what we learned here to the analysis on the black box datasets. After an extensive scan over the hyper-parameters of the model, and monitoring the behaviour of the network throughout the training, we have have come to the following conclusions regarding optimization and classification: • The Adagrad and Adadelta optimizers consistently outperform momentum-based optimizers like Adam and Nadam, which we expect is due to the smoothing of gradients in the latter which in effect reduce the sensitivity of the gradient updates to outliers in the data.
Results on LHC Olympics For the blackbox datasets and the R&D dataset we trained the VAE networks on the whole event sample, without any cuts or binning in invariant mass, and followed the early stopping procedure outlined above. In Fig. 28  we show an example of a ROC curve obtained by training on the R&D data with an S/B of 0. 1%. In Fig. 29  we show a bump in the invariant mass spectrum in the Black Box 1 data after applying a classifier trained with this method. The bump is at a mass of ∼ 3. 8 TeV and if we study the jet mass (Fig. 3. 8. 2) and τ 2 /τ 1 distributions of the events that pass the cuts we clearly see that they correspond to events with jet masses ∼ 750 GeV and ∼ 400 GeV, with τ 2 /τ 1 values from the lower end of the spectrum. Our analyses of the Black Box 2 and Black Box 3 data did not result in any clear signals in the data.
Lessons Learned The first interesting lesson learned through this analysis was that the choice of the optimizer can play an important role in different machine-learning tasks. While in standard classification tasks the momentum-based optimizers such as Adam perform very well, we found when using a VAE for anomaly detection this was not the case. Instead, when the VAE is tasked with learning an effective latent representation of the dataset, including a small subset of anomalous signal events, it performs much better when using either the Adagrad or Adadelta optimizers. The reason for this appears to be that the momentum updates in the Adam optimizer tend to smooth out the effects of anomalous events in the   gradient updates, in turn ignoring the signal events in the data. This may also be the case for other anomaly detection techniques, but has not been tested here.
Unfortunately, such reference is not always available, as it is the case for the LHC Olympics black-box dataset. Thus, in order to use BumpHunter, one must first extract a background model for the data. Another point that has to be taken into account is the fact that, despite the use of the DisCo term, the dijet mass spectrum is not totally independent from the reconstruction error. Thus, simply rescaling the full dataset precut to fit the mass spectrum postcut will not work. One way to do this is to use a small subset of the data to compute a shaping function. The objective of this function is to capture how the mass spectrum behaves when a cut on the reconstruction error is applied. This function is computed bin per bin on the dijet mass histogram by doing the ratio of the bin yields postcut and precut.
Results on LHC Olympics The results shown were obtained with an AE trained with the GAN-AE algorithm on 100k events from the R&D background. Note that before the training and application, cuts were applied on the dijet mass at 2700 GeV and 7000 GeV. R&D dataset Here we discuss the result obtained on the R&D dataset. The trained AE have been tested on 100k background events (not used during the training), as well as on the two signals provided. Fig. 13  shows the Euclidean distance distributions (left) and the corresponding ROC curves (right). This result illustrates the potential of the GAN-AE algorithm to obtain a good discrimination between the background and signals, event though only the background was used during the training. However, if the obtained AUC is good, it also appears that the Euclidean distance is still very correlated with the dijet mass.