dataesr/scientific-paragraphs-categorization huggingface dataset:
The AdaptiveConcatPool2d layer uses adaptive average pooling and adaptive max pooling and concatenates them both. Such procedure provides the model with the information of both methods and improves the overall performance of the CNN. We also make use of the label smoothing methodology  [128]  as well as the MixUp  [129]  training method. Due to the high imbalance between the number of signal and background events in the full R & D data set, we generate a smaller sample with 93390 images, with the same.
Results on the LHC Olympics In figure  48 , we demonstrate the performance of our algorithm in recovering the jet mass distributions for dijet events from the R&D dataset, using jet mass as our observable. We learn a model with 3 topics, corresponding to p X , p Y , p QCD , respectively. To generate these figures, we consider a signal fraction of 10%, and 1% respectively, solve the topic model, and then re-weight the component distributions by subtracting the overall background distribution and renormalizing. As the algorithms used to optimize the model returns extremal points in the polytope of all feasible solutions, the solution forces the components. Anomalous components at 10% signal, Background component at 10% signal, Anomalous components at 1% signal, Background component at 1% signal The components retrieved from factorized topic modeling of the LHC Olympics R&D dataset, using jet mass as our observable.
1% signal is still under investigation -as we currently formulate it, performance degrades considerably, mostly likely due to our choice of histogram bins. ) Even in the presence of this noise, our model has significant discriminative power. In particular, the model can infer which process any event was generated from using the likelihood ratio: L(x 1 , x 2 ) = f (signal) • p signal (x 1 , x 2 ) f (background) • p background (x 1 , x 2 ) (5. 8) = f (X, Y ) p X (x 1 ) p Y (x 2 ) + f (Y, X) p Y (x 1 ) p X (x 2 ) f (QCD, QCD) p QCD (x 1 ) p QCD (x 2 ). (5. 9) Using this likelihood ratio as a discriminant, we can test the ability of our model to classify events relative to the ground truth in the dataset. In both cases, the model performs well, with AUCs of 0. 88 and 0. 81, respectively.
Lessons Learned Leveraging the leading-order factorization theorem in eq. (  5. 2), we designed a statistically constrained, non-parametric, generative model to disentangle anomalous jet components from the LHC Olympics R&D dataset. For large enough signal fractions, our minimization algorithm finds a robust solution to Problem 5. 1, though performance degrades at lower signal fractions. Since the input to our model is simply a 2-dimensional histogram, an interesting direction for future research could be to use this as a drop-in replacement for density estimation steps in other anomaly detection methods. More crucially, we see this model as a proof-of-concept for the idea of encoding physical constraints on scattering processes into a statistical language.
QUAK: Quasi-Anomalous Knowledge for Anomaly Detection 26 Deep-learning-based anomaly detection within physics has largely focused on searching for anomalous signatures in the complete absence of a signal prior. In this scenario, two fundamental approaches have been considered: • Isolate two distinct datasets, one which may contain a signal, and one which does not; try to find a deviation between the two. • Embed our knowledge of known physics processes into a simulation or a deep learning algorithm and look for events with a low likelihood of being a known physics process. In the first approach, colloquially referred to as classification without labels (CWoLa), conventional discrimination algorithms are used to separate the two datasets  [63, 67, 68, 107]. Care must be taken to ensure that selection biases are mitigated so that the only discernible difference within the discrimination algorithm is the presence of an unknown physics signal.
The second approach attempts to embed a complete knowledge of physics processes within a selected region into a likelihood discriminant. An excess of events with a low likelihood of being from the selected region constitutes a new physics signature. Complete knowledge of all expected physical processes within a large, high dimensional dataset can become quite complicated and can lead to reduced sensitivity  [80, 108, 109, 141]. This method broadly comprises models that utilize deep learning autoencoders. When comparing the two approaches, the CWoLa approach is often more sensitive, provided a signal region is present. This increase in sensitivity results from the fact that an implicit signal assumption is placed on this type of anomaly search: a signal is localized to be within a specific kinematic region.
Results on LHC Olympics To perform the anomaly search, we first construct high-level jet features and then feed these inputs into the network for training. The high-level features consist of n-subjettiness ratios ranging from 1-prong to 4-prong and the raw jet mass of the individual jets  [76, 149]. Training and testing are performed with 12 variables for each event, 6 variables for each jet (4 n-subjettiness ratios, a total number of tracks, and the jet mass). In the construction of the unsupervised probabilistic model, an optimized scan for the studies with the LHC Olympics dataset converged on variational autoencoders (VAEs)  [150]  with normalizing flows  [151]  for improved posterior approximation. Among a wide variety of normalizing flow architectures, we find Masked Autoregressive Flow  [152]  yields optimized results.
Results on LHC Olympics The model performs well on the R&D data set, which is unsurprising for a supervised learning model, but it has relatively poor performances on the black boxes. In Black Box 1, it identifies some events as signals but with relatively low confidence, i. e. the output scores given by the model are not as high as given by the R&D data. Compared to the actual signals presented in the data sets, the number of events identified as signals by the model is relatively large, with a higher average mass. It is possible that the model does not actually capture any real signal and incorrectly labels events as signals or backgrounds. In Black Box 2 and 3, the output results are similar to the results when running the model over the pure background data set. In retrospect, this happens perhaps for a good reason.
Discussion The important thing in the Olympic Games is not to win, but to take part; the important thing in Life is not triumph, but the struggle; the essential thing is not to have conquered but to have fought well. 28 he results of the LHCO are to be understood in a similar way. The goal is not to declare one superior method, but to foster the development of novel tools for the unsupervised detection of anomalies. With this in mind, we now turn to a discussion and comparison of the various algorithms' performance on the LHCO2020 Black Box datasets. Knowing which algorithms achieved accurate results in blinded and unblinded tests is important information, as it will provide crucial feedback for the further refinement and improvement of each approach. Also, it is important to keep in mind that an approach which did not perform well in this challenge may have its strengths elsewhere and may turn out to be better suited for a different part of phase space.
Overall Results In the following we will review the results submitted during the two LHC Olympics sessions as well as additional contributions received for this paper. As approaches were allowed to change and improve between the sessions and in preparation of this document, we chronologically walk through results at these three stages. As discussed in Sec. 2. 2, the signal in Black Box 1 consists of 834 anomalous events with the same topology as the R&D dataset and masses m W = 3. 823 TeV, m X = 732 GeV and m Y = 378 GeV and were unblinded during the LHCO session at the 2020 ML4Jets workshop  [36]. Nine blind approaches were submitted and are summarised in Fig. 51 : ResNet + BDT (Sec. 5. 1), PCA (Principal component analysis used as an outlier detector), LSTM (Sec. 5. 4. 1), High-level features AE (encoding kinematic and substructure variables using an autoencoder, selecting signal as events with high reconstruction MSE loss), Tag N Train (Sec.
Overall Lessons Learned This large and diverse number of submissions on the blinded and unblinded datasets is very encouraging. Even better, the resonance in the first black box was successfully detected multiple times even before unblinding. Of the three methods finding a mass resonance mass closest to the true value, two were based on building a signal-to-background likelihood ratio (Tag N Train, Density Estimation) while one used a signal likelihood (LSTM), and likely benefitted from the same topology between the provided development signal and the first black box. However, there still is substantial room for improvement for anomaly detection in the realm of particle physics. First, no confident statement of the absence of signal for the second black box could be made, with a number of false positives at high values of the invariant mass.
Second, the resonance in Black Box 3 was not detected. The structure of this signal was different from the shared topology of the development data and Black Box 1 which was likely to cause issues for models too finely tuned on these signals. Furthermore Black Box 3 featured two different decay modes which need to be combined to achieve a significant observation. Finally, substructure offered a less useful handle here as one decay mode involved the direct production of a pair of gluon jets. Despite all this, the signal in Black Box 3 still decayed as hadronic resonance with a well-defined mass-peak and visible particles in the final state. Future developments therefore will need to both improve the sensitivity as well as the statistical interpretation of anomaly detectors. Beyond the reported results on the black box datasets, we also observe the use of the initial dataset for the development of new approaches.
Strategies that directly compare data and background simulation are promising for final states where the background model is accurate and when the uncertainty is well-known. A key challenge for these approaches is scaling up to high-dimensional features where the full systematic uncertainty covariance matrix may not be known. This is a general challenge that is also faced by model-dependent approaches, where signal model uncertainties in many dimensions may not be well constrained. Another independent dimension to consider is when in the data processing pipeline the anomaly detection happens. The LHC Olympics is designed as an offline analysis, where standard trigger algorithms are used to collect the data. There is significant unexplored phase space from existing triggers, but there is also a vast phase space that is deleted in real time before it can be explored.
The development of online anomaly detection will be a significant innovation to complement offline analysis. Recent innovations have shown that machine learning inference can be fast enough to fit within the strict trigger latency requirements (see e. g. [157, 158] ). However, the same algorithms applied offline may not be applicable online. For example, offline methods can make multiple passes through the dataset in order to identify anomalous regions of the phase space. In contrast, the trigger only sees collision once before a decision to save the event or not must be made. Even if a method could identify anomalous events within the required bandwidth, this is only a partial solution because strange collisions are only useful if we can quantify their level of strangeness. This is one key difference between anomaly detection in high energy physics and the typical anomaly detection protocols developed in industry; we are almost never able to declare a discovery with a single collision.
If an anomaly is found before the full high-luminosity LHC (HL-LHC) dataset is recorded, then a targeted search could be conducted using an independent dataset. What if the anomaly is only identified using the full HL-LHC dataset? What post-hoc analysis can and should be done? It is also important to ensure sensitivity to complex signals, where there may be multiple possible final states (as exemplified by Black Box 3). In the absence of a signal detection, there is a significant challenge to quantify the sensitivity of a search. For a model-dependent search, quantifying what was not seen is relatively straightforward for a given model, one can provide an upper limit on the cross section. However, model agnostic methods are sensitive to many models all at once and it is challenging to define the sensitive volume.
• Without a platform like Kaggle that offers broad exposure and a monetary prize, few ML experts outside of HEP participated in the LHC Olympics. Additionally, accessibility to non-experts could be improved. Code to read in the data and cluster jets were provided to participants, but given that nearly every group performed additional dimensionality reduction first suggests that additional information could have been useful. • One of the biggest difficulties with selecting the Black Boxes was that the anomalies should be easy enough to find that the challenge is doable, but not too easy that one could find them without new methods. Some checks were performed before releasing the Black Boxes, but with significant work, this could have been more robust and streamlined. There are many possibilities for variations on the LHC Olympics 2020.
A set of datasets were produced to emulate the typical setting where data are unlabeled but there is a corresponding labeled dataset for research and development. In the LHC Olympics, three black boxes were the analog of collider data, each with a different SM background simulation and a different potential anomaly. Many teams developed and implemented a variety of techniques on these datasets covering at least 18 different methods (some submissions compared multiple distinct methods). In addition to results with the R&D dataset, many teams deployed their techniques on the black boxes. At the Winter and Summer Olympics workshops, teams presented their results on these boxes before even knowing the nature of the signal in the datasets analyzed. While some strategies were closer to the correct answer than others, every team followed the scientific method and gained valuable insight and experience.
Acknowledgments We thank the organizers and participants in the ML4Jets2020 workshop hosted at  New York University  and at the anomaly detection workshop hosted (virtually) by the  University of Hamburg  for many interesting discussions at the Winter and  Summer Olympics , respectively. B. Nachman  and  G. Kasieczka  are grateful to the  NHETC Visitor Program  at  Rutgers University  for the generous support and hospitality during the spring of 2019 where the idea for the LHC Olympics 2020 was conceived. A. Kahn ,  J. Gonski ,  D. Williams , and  G. Brooijmans  are supported by the  National Science Foundation (NSF)  under Grant No. PHY-2013070. I. Ochoa is supported by the fellowship  LCF/BQ/PI20/11760025  from  "la Caixa" Foundation  ( ID 100010434 ) and by the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie Sk llodowska-Curie  grant agreement No  847648.
B. Bortolato ,  B. Dillon ,  A. Matevc ,  J. Kamenik ,  A. Smolkovic  acknowledge the financial support from the  Slovenian Research Agency  (research core funding No. P1-0035  and  J1-8137 ). D. A. Faroughy is supported by  SNF  under contract  200021-159720. M. Szewc would like to thank the  Jozef Stefan Institute  for its enormous hospitality. P. Komiske,  E. Metodiev ,  N. Sarda , and  J. Thaler  are supported by the  Office of Nuclear Physics of the U. S. Department of Energy (DOE)  under grant  DE-SC-0011090  and by the  DOE Office of High Energy Physics  under grant  DE-SC0012567. N. Sarda  was additionally supported by the  QCRI-CSAIL Computer Research Program. P. Komiske ,  E. Metodiev ,  N. Sarda , and  J. Thaler  are grateful to  Benjamin Nachman  and  Justin Solomon  for helpful conversations.
B. Nachman  and  J. Collins  were supported by the  DOE  under contracts  DE-AC02-05CH11231  and  DE-AC02-76SF00515 , respectively. P. Martín-Ramiro  acknowledges Berkeley LBNL, where part of this work has been developed. P. Martín-Ramiro  further acknowledges support from the  Spanish Research Agency (Agencia Estatal de Investigación)  through the contract  FPA2016-78022-P  and  IFT Centro de Excelencia Severo Ochoa  under grant  SEV-2016-0597. P. Martín-Ramiro  also received funding/support from the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie Sk llodowska-Curie  grant agreement No  690575  ( RISE InvisiblesPlus). S. Tsan ,  J. Duarte ,  J. -R. Vilmant , and  M. Pierini  thank the  University of California San Diego Triton Research and Experiential Learning Scholars (TRELS) program  for supporting this research, CENIC for the 100 Gpbs networks, and  Joosep Pata  for helpful discussions.
Dijet invariant mass distributions before (left) and after (right) a selection on the Event Score from the Black Box 2 dataset. No signal is present, and the dataset shown consists entirely of multijet background events. Black Box 2: Dijet Mass, No Event Score Cut Black Box 2: Dijet Mass, EventScore > 0. 75 10 3 Black Box 2 Background 10 3 Black Box 2 Background 10 4 10 5 10 4 10 6 10 7 10 5 2000 3000 4000 5000 M JJ [GeV] 6000 7000 8000 2000 3000 4000 5000 M JJ [GeV] 6000 7000 8000 3000 Black Box 3: Dijet Mass, No Event Score Cut 4000 5000 6000 7000 8000 M JJ [GeV] Figure 6. 2000 10 7 10 6 10 5 10 4 10 3 Black Box 3 Background 2000 10 5 10 4 Black Box 3: Dijet Mass, EventScore > 0. 75 3000 4000 5000 6000 7000 8000 M JJ [GeV] 10 3 Black Box 3 Background As the model described here is specifically sensitive to substructure within a large-R jet, it is insensitive to the signal present in this Black Box.
Authors: Louis Vaslin and Julien Donini. All the scripts used to train and apply the GAN-AE algorithm are given at this link: "https://github. com/lovaslin/GAN-AE LHCOlympics". The implementation of the BumpHunter algorithm used in this work can be found at this link: https://github. com/lovaslin/pyBumpHunter. In near future, it is planed that this implementation of Bum-pHunter becames a official package to be included in the scikit-HEP toolkit.
Vascular smooth muscle cells (VSMCs) provide vital contractile force within blood vessel walls, yet can also propagate cardiovascular pathologies through proliferative and pro-inflammatory activities. Such phenotypes are driven, in part, by the diverse effects of long non-coding RNAs (lncRNAs) on gene expression. However, lncRNA characterisation in VSMCs in pathological states is hampered by incomplete lncRNA representation in reference annotation. We aimed to improve lncRNA representation in such contexts by assembling non-reference transcripts in RNA sequencing datasets describing VSMCs stimulated in vitro with cytokines, growth factors, or mechanical stress, as well as those isolated from atherosclerotic plaques. All transcripts were then subjected to a rigorous lncRNA prediction pipeline.
We substantially improved coverage of lncRNAs responding to pro-mitogenic stimuli, with non-reference lncRNAs contributing 21-32% for each dataset. We also demonstrate non-reference lncRNAs were biased towards enriched expression within VSMCs, and transcription from enhancer sites, suggesting particular relevance to VSMC processes, and the regulation of neighbouring protein-coding genes. Both VSMC-enriched and enhancer-transcribed lncRNAs were large components of lncRNAs responding to pathological stimuli, yet without novel transcript discovery 33-46% of these lncRNAs would remain hidden. Our comprehensive VSMC lncRNA repertoire allows proper prioritisation of candidates for characterisation and exemplifies a strategy to broaden our knowledge of lncRNA across a range of disease states.
The datasets used to create such annotations cannot represent all possible biological and pathological settings so reference annotations are inherently incomplete. In addition, low abundance and lack of poly-adenylated tails for some lncRNAs means many are difficult to detect in transcriptomic datasets, which are polyA-enriched or of insufficient sequencing depth. A tendency for cell-specific expression also hinders their detection in samples containing heterogenous mixtures of cell types, such as tissues. Accordingly, several efforts have aimed to expand human lncRNA annotations beyond reference annotations. For example, building transcripts de novo from sequencing data was found to yield a substantially greater number of lncRNAs of interest in contexts, such as erythropoiesis  [16] , formation of CD8+ memory T cells  [17] , and psoriatic skin tissue  [18].
However, no novel transcript discovery methods have yet been applied to VSMCs responding to stimuli that induce proliferative or pro-inflammatory phenotypes. This means a subset of lncRNAs with potentially crucial roles in VSMC-directed tissue remodelling could remain hidden. To obtain a more complete representation of the lncRNAs expressed in pathologically active VSMCs, we obtained three published RNA sequencing (RNAseq) datasets describing in vitro stimulation of VSMCs into proliferative, migratory, or pro-inflammatory phenotypes or isolated from diseased tissue in vivo. We then combined a novel transcript discovery approach with a stringent lncRNA annotation pipeline, thereby markedly improving the coverage of stimuli-responsive, VSMC-enriched and enhancer-transcribed lncRNAs within VSMC pathology.
A Bioinformatic Approach to Provide a More Complete Annotation of lncRNAs Expressed in VSMCs in Basal and Pathological Conditions Several LncRNAs are known to be involved in VSMC phenotypic transitions occurring in vessel wall remodelling  [10] [11] [12]. However, a full accounting of lncRNAs expressed in these transitions does not yet exist. Accordingly, to gain in-depth representation of the lncRNAs expressed in pathological VSMCs, we applied a transcript discovery pipeline to published VSMC RNAseq datasets selected based on specific criteria (Figure  1a ). We focused on high depth, paired-end total RNA sequencing datasets to identify all lncRNAs and their gene structures (including non-polyA tailed and/or lowly expressed lncRNAs) and we selected two in vitro datasets fulfilling these criteria.
The first dataset describes primary human saphenous vein SMCs (svSMCs) either quiesced in 0. 2% FBS or treated with interleukin-1α (Il-1α) and/or platelet-derived growth factor-BB (PDGF-BB)  [10]. The second dataset describes primary aortic (aoSMCs) or coronary artery (caSMCs) VSMCs, plated in 5% FBS media onto soft or stiff culture matrices  [21]. These conditions model a convergence of pro-inflammatory and pro-mitogenic signals, or mechanical stretch in the vessel wall, both of which promote proliferation and disruption of contractility. We also selected an in vivo dataset describing VSMCs isolated and sequenced directly from enzymatically digested carotid plaques derived from symptomatic or asymptomatic patients, defined as such based on lumen size and occurrence of cardiovascular events prior to surgery  [22].