Anomaly Detection with Multiple Reference Datasets

Mayee F. Chen
Department of Computer Science
Stanford University
Stanford, CA 94305
mfchen@stanford.edu

Benjamin P. Nachman
Physics Division
Lawrence Berkeley National Laboratory
Berkeley, CA 94720
bpnachman@lbl.gov

Frederic Sala
Department of Computer Sciences
University of Wisconsin-Madison
Madison, WI 53706
fredsala@cs.wisc.edu

Abstract
An important class of techniques for resonant anomaly detection in high energy
physics builds models that can distinguish between reference and target datasets,
where only the latter has appreciable signal. Such techniques, including Classification Without Labels (CW O L A) and Simulation Assisted Likelihood-free Anomaly
Detection (SALAD) rely on a single reference dataset. They cannot take advantage
of commonly-available multiple datasets and thus cannot fully exploit available
information. In this work, we propose generalizations of CW O L A and SALAD for
settings where multiple reference datasets are available, building on weak supervision techniques. We demonstrate improved performance in a number of settings
with real and synthetic data. As an added benefit, our generalizations enable us toprovide finite-sample guarantees, improving on existing asymptotic analyses.

1

Introduction

Due to the vast parameter space of Standard Model extensions and to the lack of significant evidence
for new particles or forces of nature, a new model-agnostic search paradigm has emerged. Manyof these anomaly detection (AD) strategies are enabled by machine learning (see e.g. [11]) andthe first results with collision data are now available [3, 1]. One way to characterize AD methods
is based on their physics assumption of the new phenomena [12]. Strategies that assume the newphysics is “rare” [14] estimate (explicitly or implicitly) the data density and focus on events withlow density. In contrast, techniques that assume the new physics will manifest as an overdensity
in phase space use likelihood ratio methods to compare a reference dataset to a target dataset. The
latter approach has been extensively studied in the context of resonant anomaly detection, where
one reasonant feature (usually a mass) is used to create a sideband region (reference dataset) nearly
devoid of any anomalous events and a signal region (target dataset) that may contain anomalies. The
reference dataset is used to estimate the presence of anomalies in the target dataset via interpolation.
Many existing approaches are defined using one reference dataset and one target dataset. However,in practice one can have access to or construct multiple references. First, there may exist multiple
resonant features that can be used to construct sideband and signal regions. For instance, whena particle decays into two new particles, the decay products can be used to construct all threeintermediate resonances, a setting present in the LHC Olympics Dataset [13]. Second, for thesame dataset, multiple, independent Standard Model simulators can produce a reference dataset (e.g.
Pythia [21] and Herwig [4]). Using multiple reference datasets may improve performance, but it is
Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

not clear how to incorporate all of their information when using existing methods designed for asingle set.
We explore two generalizations of resonant AD to multiple reference datasets. First, we consider
Classification Without Labels (CWoLa) [16, 6, 7], in which the reference is simply the sideband
region—a form of weak supervision where the noisy label of “signal” is assigned to events in thesignal region and the noisy label of ‘background’ to events in the sideband region. We propose
a new method, Multi-CWoLa, that builds multiple reference datasets by constructing signal andsideband regions along different resonant features. We consider a point’s membership in eachfeature’s signal region as a noisy vote for anomaly, learn weights on each vote and aggregate them toproduce an overall prediction. We demonstrate M ULTI -CW O L A’s performance on the LHS Olympics
Dataset [13].
Second, we study Simulation Assisted Likelihood-free Anomaly Detection (SALAD) [2]. In thismethod, a reweighting function between a reference simulation dataset and a target dataset is learnedin the sideband conditioned on the resonant feature. The simulated events in the signal region are
reweighted by interpolating this function and then are used to distinguish anomalies in the target
dataset. We extend this to the case of multiple simulated datasets, each of which may make different
approximation choices and thus provide complementary accuracy when using SALAD. We introduce
M ULTI -SALAD, which combines the simulated datasets accordingly and then reweights, with the
key finding that combining data helps when each simulator approximates different components of the
background well. We demonstrate M ULTI -SALAD’s performance on synthetic data.
Finally, we study the finite sample guarantees of our proposed methods. Many resonant AD methods
have optimality guarantees in some asymptotic limit, but there is no first-principles understanding of
the methods’ performance with finite samples. We draw on results from statistical theory to begin
a formal study of resonant AD methods with limited data. Our results lay a foundation for future
investigations into the finite sample properties of AD and related methods.

2

BackgroundWe have an input space of discriminating features x ∈ X and k resonant features m =[m1 , . . . , mk ] ∈ Rk . Associated with a point (x, m) is an unknown label y ∈ Y for Y = {0, 1} (background vs. signal). Points (x, m, y) are drawn from a distribution P with density p(·). For a resonant
feature mi ∈ R, an interval Imi ∈ R is used to define a signal region SR = {(x, m) : mi ∈ Imi }and a sideband region SB = {(x, m) : mi ∈/ Imi }. We assume that the sideband region contains
little to no signal, i.e., p(y = 1|(x, m) ∈ SB) ≈ 0. Our goal is to construct a predictor f : X → Y,
outputting a value ŷ given (x, m), to perform anomaly detection.

3

M ULTI -CW O L A: Learning from Multiple Resonant Features

We introduce M ULTI -CW O L A, an approach to anomaly detection that uses multiple reference
datasets and is built using principles from the area of weak supervision [19, 10] .
Standard CW O L A We have one unlabeled dataset D = {(xi , mi )}ni=1 with one resonant feature(k = 1) that we want to use to learn f . We use m to construct the signal and sideband regions,
DSR , DSB ⊂ D, where DSR = D ∩ SR and DSB = D ∩ SB, with distributions pSR and pSB
respectively. With the intuition that there are more anomalies in the signal region, we express each
distribution as a mixture of signal and background components with weight 0 ≤ ηSR , ηSB ≤ 1:
pSR (x) = ηSR p(x|y = 1) + (1 − ηSR )p(x|y = 0), and pSB (x) = ηSB p(x|y = 1) + (1 −
ηSB )p(x|y = 0).
SR (x)
Under this construction, the density ratio of the mixtures ppSB(x) can be written in terms of the ratio ofpSR (x)ηSR r(x)+1−ηSRthe signal and background components, r(x) = p(x|y=1)p(x|y=0) , as pSB (x) = ηSB r(x)+1−ηSB . AssumingηSR > ηSB (e.g. more signal in the signal region), the mixture ratio is monotonically increasing in
SR (x)
r(x). Therefore, we train a classifier f to learn ppSB(x) by distinguishing between DSR and DSB ,and this f provides information about r(x) and thus can be used for anomaly detection.

2

M ULTI -CW O L A Method Intuitively, CW O L A uses the resonant feature m as a noisy label that
identifies the signal vs sideband region and then trains a classifier using these through the mixture
component membership. This idea leads to a simple question—if more than one such feature isavailable (k > 1), how can the multiple noisy labels best be utilized? We tackle this question using
principles from weak supervision [19, 20, 10].
In our approach, we split D along each resonant feature mi to produce multiple pairs of datasets
DSBi and DSRi for each i ∈ [k] based on membership in Imi . A straightforward way to use alldatasets (DSB1 , DSR1 ), . . . , (DSBk , DSRk ) is to apply standard CWoLa k times to train k classifiers
that we can then ensemble. Instead, in M ULTI -CW O L A, we directly aggregate the noisy membershiplabels provided by thresholding resonant features and training a model on the aggregated label.
Specifically, suppose that the mixture membership for a datapoint (x, m) is given by the label
Mi (m) ∈ {0, 1}, where Mi (m) = 0 if x ∈ DSBi and 1 otherwise. To aggregate all noisy labels
M(m) = {M1 (m), . . . , Mk (m)}, we draw on weak supervision, a recent class of techniques thatcombines the outputs of multiple weak sources using learned weight parameters and produces
estimates on the true label without requiring any labeled data. That is, we can aggregate M(m)according to each resonant feature’s quality, without needing to know the true presence of ananomaly. In particular, we first learn the probabilistic graphical model for the binary distributionp(y|M(m)) [10]. Since y is unknown, we learn the parameters of the distribution using latent variable
estimation on D. Then, weak labels ŷ are produced from our estimated distribution p̂(y|M(m)) forx ∈ D. Finally, an end model fˆ is trained with (x, ŷ) to perform anomaly detection. See Algorithm 1
in Appendix C for explicit algorithm statements.
Theoretical Results Assuming p(y, M(m)) can be parametrized as a class of graphical models(see Equation (1)), M ULTI -CW O L A offers finite-sample generalization guarantees. Suppose thedownstream model fˆ trained on ŷ belongs to class F. Define a loss function ℓC : Y × Y → R and let
the expected loss of f be LC (f ) := EPdata [ℓC (f (x), y)] on true labels. Then, the optimal classifier isf ⋆ = argminf ∈F LC (f ), which is achieved with unlimited labeled data. Let the empirical loss of f onPnŷ be L̂C (f ) := n1 i=1 ℓC (f (xi ), ŷi ). Then, the classifier we learn is fˆ = argminf ∈F L̂C (f ). Notethat this construction is different from the standard empirical risk minimization (ERM) loss on labeleddata and thus L̂C (f ) does not asymptotically equal LC (f ). We aim to minimize the generalization
error LC (fˆ) − LC (f ⋆ ). In Theorem 1 (Appendix D), we present a bound on LC (fˆ) − LC (f ⋆ ). We
find that there are four quantities controlling the bound: 1) the Rademacher complexity of F, which
describes the model’s expressivity and can be readily computed for a variety of classes (e.g., decision
trees, two-layer feedforward networks); 2) the noise from learning with n points; 3) the noise fromusing ŷ rather than true y; and 4) the gap between p(y|m) and p(y|M(m)) (e.g. how much we lose
about the resonant features by only modeling mixture membership). Surprisingly, we find that 2) and3) both scale in O(n−1/2 ) and go to 0 for large n.
Evaluation In Figure 1 (Left), we compare M ULTI -CW O L A with standard CW O L A as well as
two other baselines. We use simulation data from the LHC Olympics Dataset [13]; in particularfrom Pythia 8 [21], where the signal is boson decay and the background is generic 2 → 2 parton
scattering. This dataset contains 5 features; in the standard CW O L A setup, we use one thresholded
resonant feature (k = 1) and use 4 discriminative features as x. For M ULTI -CW O L A, we havegenerated k = 3 mixtures by varying how the 3 resonant features (the jet masses in addition to thedijet mass) are thresholded and use 2 discriminative features as x. We have two other baselines thatutilize 3 resonant features: CW O L A-intersect defines the signal region as the intersection of theresonant features’ signal regions, e.g. SR = SR1 ∩ SR2 ∩ SR3 , but this can be overly conservative.
CW O L A +x thresholding keeps one resonant feature as the noisy label ŷ = M1 (m), and includes
the remaining thresholded features as discriminative features {M2 (m), M3 (m), x}. We vary the
number of samples available on a logarithmic scale from n = 59 to 6003 and plot the AUC averaged
over 5 runs per sample size. We find that M ULTI -CW O L A offers a higher AUC and lower variance,
especially when there is limited data.

4

M ULTI -SALAD: Learning From Multiple Simulations

We introduce M ULTI -SALAD, an AD approach that uses multiple simulation datasets.
3

Rejection (1/False Positive Rate)105

Sim 1 SALADSim 2 SALAD
Multi-SALAD
SALAD-Switch

104
103
102
101
100
0.0

0.2

0.4
0.6
0.8
Signal Efficiency (True Positive Rate)

1.0

Figure 1: Left: Comparison between CW O L A and M ULTI -CW O L A. Using multiple mixed samples
helps performance across a range of dataset sizes. Access to multiple weak sources enables better
accuracy and lower variance compared to the single-feature version. Right: Signal efficiency torejection of M ULTI -SALAD versus other baselines (weighted and unweighted).

simStandard SALAD We have a background simulation dataset Dsim = {(xi , mi )}ni=1with yi = 0
sim
for all i in addition to one true dataset D. D
is drawn from some distribution Psim with density
psim . While CWoLA learns the likelihood ratio between the signal and sideband regions of D alone,
SALAD utilizes Dsim as well. Note that if psim is equal to pdata (·|y = 0), we could directly traina model to distinguish between D and Dsim in the signal region to get a classifier that could detect
anomalies. However, since Dsim may not match the true background data, we instead first need tolearn a reweighting function to capture the differences between Dsim and D’s background data, andthen we train a model to distinguish between D and the reweighted Dsim in the signal region.

Formally, given fixed SR and SB for both datasets, the method can be broken into two steps. 1)
p(x,m|y=0)
A classifier ĝ is trained to learn the weight w(x, m) = psim
(x,m|y=0) by distinguishing betweensimsim
DSB = D∩ SB and DSB . 2) Using a loss function LS weighted using the estimated ŵ(x, m)
sim
sim
applied to DSR= Dsim ∩ SR, a classifier ĥ is trained to distinguish between DSR and DSR. If theestimate ŵ(x, m) is exactly equal to w(x, m) (e.g. ĝ is Bayes-optimal), then the second step will bep(x)equivalent in expectation to learning the ratio p(x|y=0), from which one can detect anomalies.
M ULTI -SALAD Method Now we have multiple simulation datasets D1sim , . . . , Dksim . One approach
would be to maintain distinctions among simulations by reweighing each pair to learn k weightsimfunctions wi (x, m), and then using one overall loss function that weights points from each DSR,iwith wi . However, it has been shown that importance reweighting, despite working in expectation,
can be highly unstable and result in poor performance of tasks on the target data D [9]. To understandwhy, [8] showed that the generalization error of an empirical loss function with importance weights
w depends on the magnitude and variance of w. Applied to our setting, it suggests that the morep(x)inaccurate the simulation is, the less the reweighted loss recovers the true p(x|y=0), and the modelsim
may instead pick up on differences between DSR and the reweighted DSR
that are noise rather than
the anomaly. As a result, aggregating individual SALAD outputs can be equivalent to ensembling
many poor classifiers.
Given these observations, M ULTI -SALAD uses multiple simulation datasets in a very simple yettheoretically principled way: control the magnitude of the overall w by combining all the Disim toe sim whose distribution best approximates the true backgroundproduce one large simulation dataset De sim and D. Note that this approach both improvesp(x|y = 0), and then use standard SALAD with D
sample complexity and can “suppress” a simulation that on its own has high w, while the approachof learning k weight functions would not offer such improvements. In Algorithm 2, we write thisprocedure out where we simply concatenate all Disim together. However, with domain knowledge one sim by sampling
the strengths and weaknesses of each simulation across features, one could produce D
accordingly from each. We leave this direction for future work.
4

Theoretical Results We now present a finite sample generalization error bound on M ULTI -SALAD
e sim belonging tothat also applies to SALAD. Define nSR as the number of points from D and DSB
the signal region, and n
as the number of points belonging to the sideband. Let nSR
sim be thesim
e belonging to the signal region. To measure the generalization error, recall
number of points in Dp(x,m|y=0)w(x, m) = psim
(x,m|y=0) and let ŵ be the classifier ĝ’s estimate. We denote h as the reweightedclassifier. Let h⋆ = argminh∈H LS (h, w) and let ĥ = argminh∈H L̂S (h, ŵ). We aim to boundLS (ĥ, ŵ)−LS (h⋆ , w). Define W = maxx,m w(x, m) as the maximum ratio betweeen the simulationand true background. In Theorem 2 in Appendix D, we show an upper bound on generalization error−1/2
LS (ĥ, ŵ) − LS (h⋆ , w) that scales in (nSB )−1/2 , and (nSR
, where the former comes from thesim )initial reweighting step while the latter comes from the weighted classification step. The bound isalso dependent on the Rademacher complexities of both classifiers g and h used. Finally, our result
demonstrates that a larger W increases the generalization error bound.
Evaluation To demonstrate how M ULTI -SALAD can improve over using only one simulationand over using simulations separately, we consider a synthetic experiment with two simulation
datasets.1 The anomaly distribution is only slightly different from the background data, which follows
a symmetric Gaussian mixture. On the other hand, each simulation has a distribution that matches
only one mixture component of the background data. We assume that the signal and sideband regions
are the same over x (e.g. m and x are independent). A visualization is shown in Figure 2 and details
are in Appendix E.
Intuitively, the anomaly is only slightly different from the background data, which makes it importantto learn a good reweighting function from the simulations. Because each simulation alone diverges
greatly from the data for one mode, each individual reweighting may not approximate the trueP(·|y = 1) well, as suggested by Figures 3 and 4. On the other hand, if we combine both simulation
datasets together, the aggregate distribution has smaller weights with lower variance, which can allow
for more accurate reweighting. In Figure 1 (Right), we present the signal efficiency to rejection rateof four methods: 1) M ULTI -SALAD; 2) SALAD on the first simulation; 3) SALAD on the second
simulation; and 4) SALAD-S WITCH, where we learn k separate wi functions and switch among
them in the reweighted loss function. Table 1 contains the accuracy and AUC scores for each method.
Averaged over 10 random seeds, M ULTI -SALAD outperforms other methods. The signal efficiency
to rejection rate for each of the 10 runs is available in Appendix E.

MethodAccuracy
AUC

Simulation 1
None
SALAD
43.8±2.262.5±8.828.5±4.2
80.7±14.5

Simulation 2
None
SALAD
42.7±3.664.3±12.3
27.4±4.578.7±18.2

None50.0±0.0
15.4±5.3

Simulation 1 and 2
SALAD-S WITCHM ULTI -SALAD
54.3±6.2
64.8±9.3
74.7±17.0
90.8±10.2

Table 1: Accuracy and AUC scores (%) for M ULTI -SALAD on two simulation datasets. We compare
to SALAD-S WITCH (different reweighting), as well as standard SALAD on individual simulations
and no reweighting. Performance is averaged over 10 random runs with one standard deviation
reported.

5

Conclusion

We extend two resonant AD approaches to incorporate multiple reference datasets. For M ULTI CW O L A, we draw from weak supervision models to handle multiple resonant features. For M ULTI SALAD, we combine multiple simulation datasets to best approximate the background process.
Future work includes 1) exploring M ULTI -SALAD’s applicability on real data and algorithms forsampling from simulation datasets 2) extending M ULTI -CW O L A to model more complex relationships among resonant features and 3) using such approaches together over multiple simulations andresonant features, effectively utilizing as much information as possible.
1

We find that the differences between the simulations in the LHC Olympics are not enough to see a noticeable
gain from M ULTI -SALAD over SALAD.

5

6

Acknowledgements

BN was supported by the Department of Energy, Office of Science under contract number DEAC02-05CH11231. FS is grateful for the support of the NSF under CCF2106707 and the Wisconsin
Alumni Research Foundation (WARF). We gratefully acknowledge the support of NIH under No.
U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume
to Velocity), and 1937301 (RTML); ARL under No. W911NF-21-2-0251 (Interactive Human-AI
Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-12480: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275(NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi,
BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the
HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), andmembers of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Governmentis authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any
copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed inthis material are those of the authors and do not necessarily reflect the views, policies, or endorsements,
either expressed or implied, of NIH, ONR, or the U.S. Government.

References
decaying into a Higgs boson and a generic new[1] Anomaly detection search for new resonances√
particle X in hadronic final states using s = 13 TeV pp collisions with the ATLAS detector.
Technical report, CERN, Geneva, 2022. All figures including auxiliary figures are available athttps://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2022-045.
[2] Anders Andreassen, Benjamin Nachman, and David Shih. Simulation Assisted Likelihood-free
Anomaly Detection. Phys. Rev. D, 101(9):095004, 2020.
[3] ATLAS Collaboration. Dijet resonance search with weak supervision using 13 TeV pp collisions
in the ATLAS detector. 2020.
[4] Johannes Bellm et al. Herwig 7.0/Herwig++ 3.0 release note. Eur. Phys. J. C, 76(4):196, 2016.
[5] Francois Chollet et al. Keras, 2015.
[6] Jack H. Collins, Kiel Howe, and Benjamin Nachman. Anomaly Detection for Resonant New
Physics with Machine Learning. Phys. Rev. Lett., 121(24):241803, 2018.
[7] Jack H. Collins, Kiel Howe, and Benjamin Nachman. Extending the search for new resonanceswith machine learning. Phys. Rev., D99(1):014038, 2019.
[8] Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importanceweighting. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors,
Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[9] Sanjoy Dasgupta and Philip M. Long. Boosting with diverse base classifiers. In Bernhard
Schölkopf and Manfred K. Warmuth, editors, Learning Theory and Kernel Machines, pages
273–287, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.
[10] Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher R/’e. Fast and three-rious: Speeding up weak supervision with triplet methods. In
International Conference on Machine Learning, 2020.
[11] HEP ML Community. A Living Review of Machine Learning for Particle Physics.
[12] Georgia Karagiorgi, Gregor Kasieczka, Scott Kravitz, Benjamin Nachman, and David Shih.
Machine Learning in the Search for New Fundamental Physics. 12 2021.
[13] Gregor Kasieczka et al. The LHC Olympics 2020: A Community Challenge for Anomaly
Detection in High Energy Physics. 1 2021.
[14] Gregor Kasieczka, Radha Mastandrea, Vinicius Mikuni, Benjamin Nachman, Mariel Pettee,
and David Shih. Anomaly Detection under Coordinate Transformations. 9 2022.
6

[15] Gregor Kasieczka, Ben Nachman, and David Shih. R&D Dataset for LHC Olympics 2020
Anomaly Detection Challenge, April 2019.
[16] Eric M. Metodiev, Benjamin Nachman, and Jesse Thaler. Classification without labels: Learningfrom mixed samples in high energy physics. JHEP, 10:174, 2017.
[17] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning.
MIT press, 2018.
[18] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan,
Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas
Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy,
Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style,high-performance deep learning library. In Advances in Neural Information Processing Systems
32, pages 8024–8035. Curran Associates, Inc., 2019.
[19] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher
Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the 44th
International Conference on Very Large Data Bases (VLDB), Rio de Janeiro, Brazil, 2018.
[20] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, andChristopher Ré. Training complex models with multi-task weak supervision. In Proceedings of
the AAAI Conference on Artificial Intelligence, Jul 2019.
[21] Torbjorn Sjostrand, Stephen Mrenna, and Peter Z. Skands. A Brief Introduction to PYTHIA
8.1. Comput. Phys. Commun., 178:852–867, 2008.

7

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes](c) Did you discuss any potential negative societal impacts of your work? [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes](b) Did you include complete proofs of all theoretical results? [Yes]3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes](c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] We report errors for M ULTI -CW O L A. For M ULTI SALAD, standard deviation on the signal efficiency to rejection rate can be extremely
large (as it is computed over the reciprocal of the FPR, which is expected to be noisy
across runs). Instead, we have included Figure 5, which shows several individual runsof our experiment.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes]4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes](b) Did you mention the license of the assets? [Yes](c) Did you include any new assets either in the supplemental material or as a URL? [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A] Data used is publicly available [13](e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]8

A

Potential Broader Impacts

Anomaly detection is used throughout science and industry and so our observations may be useful tosetups beyond collider physics. These tools are designed to find group anomalies and so even whenapplied to society at large, there is no way to single out individuals.
A.1

Code and Data

The code for this paper can be found at https://github.com/mayeechen/anomaly-detection-multi and the physics data sets are hosted on Zenodo at [15].

B

Appendix

We provide algorithmic details for M ULTI -CW O L A and M ULTI -SALAD in Section C. In section D,
we provide proofs for our theoretical results. In section E, we provide experimental details.

C

Additional Algorithmic Details

C.1

M ULTI -CW O L A Algorithm

M ULTI -CW O L A is described formally in Algorithm 1. Given a dataset D with k resonant featureswith corresponding thresholds Im for signal vs sideband region, we can construct a binary vectorper (x, m), M(m) = [M1 (m), . . . Mk (m)] ∈ {0, 1}k , where Mi (m) = 1 if x belongs to the ith
resonant feature’s signal region. The goal is to learn how to produce a label from k noisy “votes”M(m).
To aggregate M(m) into a label, we draw from weak supervision and aim to compute p(y|M(m)),
the distribution from which we produce an estimate ŷ. We assume p(y, M(m)) can be written as agraphical model as follows:


k
X
1
fi (m)e
θi M
y ,
(1)
p(y, M(m)) = exp θy ye +Z
i=1
where θy , θi for i ∈ [k] are the canonical parameters of the distribution, Z is for normalization, and yefi (m) are y and Mi (m) scaled from {0, 1} to {−1, 1}. Intuitively, θi represents the strength ofand M
the correlation between Mi (m) and y. This model also implies that Mi (m) ⊥ Mj (m)|y; that is, themixture membership components (i.e., the resonant features) are conditionally independent given y.
With the above construction, we learn ŷ as follows. First, we estimate p(Mi (m)|y), which corresponds
to the ith resonant feature’s accuracy. This can be done by adapting the triplet approach from [10].
First, we draw triplets of resonant features a, b, c ∈ [k]. If the distribution on y, M(m) follows thefa (m)]E[efb (m)] = E[Mfa (m)Mfb (m)]. Writing one suchgraphical model in (1), it holds that E[eyMyMequation for each pair in the triplet (a, b, c), we have thatfa (m)]E[efb (m)] = E[Mfa (m)Mfb (m)]
E[eyMyMfa (m)]E[efc (m)] = E[Mfa (m)Mfc (m)]
E[eyMyM
fb (m)]E[efc (m)] = E[Mfb (m)Mfc (m)],
E[eyMyM
since ye2 = 1. Solving this system, we obtain
v
u
u E[M
fa (m)Mfb (m)]E[Mfa (m)Mfc (m)]f|E[ey Ma (m)]| = t,
fb (m)Mfc (m)]
E[Mand similarly for b and c. We assume that each signal region is positively correlated with the truefa (m)]. Next, we can use E[efa (m)] to obtainsignal, which allows for us to uniquely recover E[eyMyMp(Ma (m)|y) due to the structure of the graphical model in (1). From these, we use Bayes’ rule9

Algorithm 1 M ULTI -CW O L A
1: Input: Dataset D = {(xi , mi )}ni=1 ; thresholds Imi that split D into signal and sideband regions,
DSRi and DSBi respectively, for each mi ; class balance probability of anomaly p(y = 1)
1 x ∈ DSRii
2: For each resonant feature m , define Mi (m) =.
0 x ∈ DSBi
3: for each triplet a, b, c ∈ [k] do
4:

αa :=
αb :=
αc :=

q
q
q

fa (m)Mfb (m)]Ê[Mfa (m)Mfc (m)]/Ê[Mfb (m)Mfc (m)]Ê[M(2)fa (m)Mfb (m)]Ê[Mfb (m)Mfc (m)]/Ê[Mfa (m)Mfc (m)]Ê[M(3)fa (m)Mfc (m)]Ê[Mfb (m)Mfc (m)]/Ê[Mfa (m)Mfb (m)] ,
Ê[M

(4)

where Ê is an empirical estimate of the expectation over D.
5: end for
6: Set p̂(Mi (m) = 1|y = 1) = p̂(Mi (m) = 0|y = 0) = p̂(Mi (m) = y) = αi2+1 .
Qm
7: Compute estimate p̂(y = 1|M(m)) ∝ i=1 p̂(Mi (m)|y = 1)p(y = 1).
8: Construct ŷ for each (x, m) ∈ D.
9: Output: Classifier fˆ for anomaly detection trained on {(xi , ŷi )}n
i=1 .
Qm

p(M (m)|y=1)p(y=1)

ito produce an estimate of p(y|M(m)) = i=1 p(M(m)), where we assume that the classbalance p(y = 1) is known; otherwise, it can be estimated [20]. In practice, all of these quantities arefa (xi )Mfb (xi ).
fa (m)Mfb (m)] = 1 Pn M
empirical estimates, starting with terms such as Ê[Mi=1nFinally, with labels ŷ for each (x, m) ∈ D, we train a classifier fˆ on points (x, ŷ). This procedure issummarized in Algorithm 1.
C.2

M ULTI -SALAD Algorithm

M ULTI -SALAD is described formally in Algorithm 2. We have simulation datasets D1sim , . . . Dksim ,simwhere Disim = {(xj , mj )}nj=1
and all points belong to the background (y = 0). As discussed in
Section 4, we propose using these simulation datasets by aggregating them into a single simulation
dataset Dsim (whether it be with uniform or stratified sampling, etc.) Then the rest of this section
proceeds as follows and is a review of the standard SALAD method.
Reweighting First, we learn weights to correct for the bias of the simulated background data. Wesim
sim
split the both simulation and true data along m to produce sets DSR, DSB
and DSR and DSB . We
simtrain a classifier over DSB and DSB to distinguish between simulation and real data in the sideband
region. That is, we train a binary classifier ĝ over points (x, m, z) in the sideband where x, m is either
from psim (·|y = 0) (z = 0) or p(·|y = 0) (z = 1), where we recall that simulation data only contains
y = 0, and no anomalies are present in the sideband. Denote q as the joint density of (x, m, z). We
define the weight as the estimated likelihood ratioĝ(x, m)q(z = 1|x, m)q(x, m|z = 1) q(z = 1)≈=·
1 − ĝ(x, m)q(z = 0|x, m)q(x, m|z = 0) q(z = 0)p(x, m|y = 0)q(x, m|z = 1)==.
q(x, m|z = 0)
psim (x, m|y = 0)ŵ(x, m) =

(5)
(6)

Here, we assume that q(z = 1) = q(z = 0) (i.e. balanced simulation and real dataset, which we canalways ensure by generating more or less simulation data). Equality is obtained in the expression
above when ĝ is Bayes-optimal.
Training The above ŵ(x, m) is defined on the sideband region. Next, we interpolate and correctsim
the bias of the simulation in the signal region. Let DSR
be the set of simulation data in the signal
SR
region of size nsim , and let DSR be the set of true data in the signal region of size nSR
data , for a total
10

Algorithm 2 M ULTI -SALAD
1: Input: Simulation datasets D1sim , . . . , Dksim and real dataset D.
Sk
2: Construct overall simulation dataset D sim = i=1 Disim .
3: Split each dataset into signal region and sideband region using resonant feature m to get
sim
sim
{DSR
, DSB
} and {DSR , DSB }.
ĝ(x,m)4: Learn weight ŵ(x, m) = 1−ĝ(x,m) , where ĝ is a classifier that distinguishes data DSB fromsim
simulation DSB
in the sideband region.
5: Train a new classifier ĥ on the signal region to distinguish between points in DSR and points insim
DSR
reweighted by ŵ, using the following loss:

 X
X
1
log h(x, m) +ŵ(x, m) log(1 − h(x, m)) .
(8)
L̂S (h, ŵ) = − SR
n
sim
x∈DSR

x∈DSR

6: Output: Classifier output ĥ(x, m) for anomaly detection.

of nSR points. We train a classifier h to distinguish between the reweighted simulated data, whichapproximates true background data, and the true data. In particular, the loss function used is
 X

X
1
L̂S (h, ŵ) = − SRlog h(x, m) +ŵ(x, m) log(1 − h(x, m)) .
(7)
n
sim
x∈DSR

x∈DSR

In expectation with an optimal w, we can see that minimizing this loss is equivalent to minimizing
the cross-entropy loss on a task that distinguishes between points drawn from p and points drawn
from p(·|y = 0) in the signal region. Therefore, h can be used for anomaly detection. The procedure
is summarized in Algorithm 2.

D

Theoretical Results

We first present our generalization error bound on M ULTI -CW O L A.
We define thePfollowing terms.  Define the Rademacher complexity of F as Rn (ℓ ◦ F) =n
E supf ∈F n1 i=1 εi ℓ(f (xi ), yi ) with Rademacher random variables Pr(ε = 1) = Pr(ε = −1) =1
2 . Define emin as the minimum eigenvalue of the covariance matrix on [y, M1 (x), . . . , Mk (x)], andfi (x)y] over all i.
let amin be the minimum value of E[MTheorem 1. Assume that p(y, M(x)) can be parametrized according to (1) and that ℓ is scaled
to be bounded in [0, 1]. Assume that the class balance p(y) is known (if not, there are ways toestimate it [20]), and that k ≥ 3. Then, with probability at least 1 − δ, the generalization error of
M ULTI -CW O L A on D is at most
r
r k
log
2/δ
c
c2 k 1⋆
LC (fˆ) − LC (f ) ≤ 4Rn (ℓ ◦ F) +
+
+√
5
2n
emin aminnn+ DKL (p(y|x)||p(y|M(m))),where c1 , c2 are positive constants.
We observe that there are four quantities controlling the above bound:• The Rademacher complexity of F: this term describes the model’s expressivity. Smaller
Rademacher complexity means that the model is easier to learn and that our fˆ will be closer
to the best model in F. This quantity can be readily computed for a variety of function
classes F, such as decision trees, linear models, and two-layer feedforward networks, whichmakes our bound in Theorem 1 tractable.
• Using n finite samples: as the amount of data increases, the error decreases in O(n−1/2 ).
11• Using noisy labels ŷ instead of y: for our weak supervision algorithm and graphical model,using ŷ rather than y contributes an additional O(n−1/2 ) error. Asymptotically, our approachthus does no worse than training with labeled data.
• The irreducible gap between p(y|m) and p(y|M(m)): we lose information about m by only
modeling mixture membership M(m).
bounded by the traditional ERM
Proof. From Theorem 3 of [10], we have that LC (fˆ) − LC (f ⋆ ) isPmgeneralization gap of LC (f¯) − LC (f ⋆ ), where f¯ = argminf ∈F n1 i=1 ℓ(f (xi ), yi ) is the classifier
q

c√2 k
k
learned on labeled data, plus the terms eminca1 5n + n + DKL (p(y|x)||p(y|M(m))).
min

We can apply standard learning theory bounds on LC (f¯) − LC (f ⋆ ). In particular, this quantity isequal toLC (f¯) − LC (f ⋆ ) = (LC (f¯) − L̂C (f¯)) + (L̂C (f¯) − L̂C (f ⋆ )) + (L̂C (f ⋆ ) − LC (f ⋆ ))≤ LC (f¯) − L̂C (f¯) + L̂C (f ⋆ ) − LC (f ⋆ )≤ 2 sup |LC (f ) − L̂C (f )|,f ∈Fwhere we have used the fact that L̂C (f¯) ≤ L̂C (f ⋆ ). Then, using uniform convergence bounds, suchas Theorem 3.3 of [17], we haver
log 2/δ ⋆¯.
LC (f ) − LC (f ) ≤ 2 2Rn (ℓ ◦ F) +2nThis gives us our desired result.

Next, we present our theoretical result on M ULTI -SALAD.
⋆⋆
We first set up some definitions. Let ĝ(x) ∈ [ĝmin , ĝmax ] and g ⋆ (x) ∈ [gmin, gmax], where g ⋆ isthe optimal classifier. Let RnSR (ℓS ◦ {H, G}) be the Rademacher complexity of the overall loss
LS (h, w) across function classes h ∈ H, g ∈ G. Define W = maxx,m w(x, m) as the maximum ratio
between the simulation and true background. Let B1 = max{− log h⋆ (x, m), − log(1 − h⋆ (x, m))}be based on the most extreme value of h⋆ (i.e. how far apart p and p(·|y = 0) can be). Letsim
η = max(− log(1 − h⋆ (x, m))) for x, m ∈ DSR. Let RnSB (ℓ ◦ G) is the Rademacher complexity of
the loss function class used for learning the reweighting, where ℓ is point-wise cross-entropy. Finally,⋆let B2 = − log(min{ĝmin , gmin}).

Theorem 2. With probability at least 1 − δ, there exists a constant c > 0 such that the generalizatione sim and D is at mosterror of M ULTI -SALAD on Dr
log 8/δ⋆LS (ĥ, ŵ) − LS (h , w) ≤ 2RnSR (ℓS ◦ {H, G}) + (1 + W B1 )
(9)
2nSR
s
r


ηnSR
log 4/δ
log 8/δsim+
4cRnSB (ℓ ◦ G) + 2c
+ B2.
⋆
(1 − ĝmax )(1 − gmax
)nSR
2nSB
2nSR
sim
We make several observations about this bound:−1/2• The bound scales in (nSB )−1/2 and (nSR
, where the former comes from the initialsim )reweighting step while the latter comes from the weighted classification step.

• The bound is also dependent on the Rademacher complexities of both classifiers g and h
used.
• The bound depends on the difference between the simulation and data distributions throughquantities W , B1 , B2 , η, ĝmax , gmax . If the distributions have very different densities, thesequantities will all be large, increasing the generalization error.
12

Proof. We define the true (cross-entropy) loss as
SR [w(x, m) log(1 − h(x, m))]LS (h, w) = − Pr(z ′ = 1)Ez′ =1 [log h(x, m)] − Pr(z ′ = 0)Ex,m∈Psim

where z ′ = 1 for x, m ∼ P and 0 for x, m ∼ P(·|y = 0). Next, define w(x, m) = q(x,m|z=1)q(x,m|z=0)
and let ŵ be the weight ratio learned by our model. Let ĥ = argminh∈H L̂S (h, ŵ), and letSRh⋆ = argminh∈H L(h, w⋆ ). Intuitively, h⋆ corresponds to the true difference between Pdata
andSR
Pdata (·|y = 0). We can first decompose the generalization error as
LS (ĥ, ŵ) − LS (h⋆ , w) = [LS (ĥ, ŵ) − L̂S (ĥ, ŵ)] + [L̂S (ĥ, ŵ) − L̂S (h⋆ , ŵ)]
⋆

⋆

⋆

⋆

+ [L̂S (h , ŵ) − L̂S (h , w)] + [L̂S (h , w) − LS (h , w)]

(10)
(11)

We know that L̂S (ĥ, ŵ) ≤ L̂S (h⋆ , ŵ), so
LS (ĥ, ŵ) − LS (h⋆ , w) ≤ |LS (ĥ, ŵ) − L̂S (ĥ, ŵ)| + |L̂S (h⋆ , w) − LS (h⋆ , w)|+ L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)
≤ sup |LS (h, w) − L̂S (h, w)| + |L̂S (h⋆ , w) − LS (h⋆ , w)| + L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)h,w

We first bound suph,w |LS (h, w) − L̂S (h, w)|. For notation, we rewrite LS (h, w) as LS (h, g),g(x,m)
where w(x, m) = 1−g(x,m)
and g belongs to some function class G. Then, using Theorem 3.3
q1/δ
from [17], we get that suph,w |LS (h, w) − L̂S (h, w)| ≤ 2RnSR (ℓS ◦ {H, G}) + log
with
2nSR
probability at least 1 − δ, where ℓS ◦ {H, G} is defined as satisfying ℓS (h(x, m), g(x, m), y) =g(x,m)−y log h(x, m) − (1 − y) 1−g(x,m)
log(1 − h(x, m)) for h ∈ H, g ∈ G.
Next, we bound |L̂S (h⋆ , w) − LS (h⋆ , w)|. Let W = max w(x, m) < ∞ be the maximum density
ratio, and let B1 = maxx,m {− log h⋆ (x, m), − log(1 − h⋆ (x, m))}. Assume that B1 < ∞. We canapplyqstandard concentration inequalities here (Hoeffding) to get that |L̂S (h⋆ , w) − LS (h⋆ , w)| ≤W B1

log 2/δ
with probability at least 1 − δ.
2nSR

Finally, we bound L̂S (h⋆ , ŵ) − L̂S (h⋆ , w). We can write L̂S (h⋆ , ŵ) − L̂S (h⋆ , w) as
X
1
L̂S (h⋆ , ŵ) − L̂S (h⋆ , w) = SR(ŵ(x, m) − w(x, m)) · (− log(1 − h⋆ (x, m)))
n
sim

(12)

x∈DSR

sim
Define η = max(− log(1 − h⋆ (x, m))) ≥ 0 for x, m ∈ DSR, which is small as long as h⋆ (x, m)
sufficiently classifies x and is hence a property of how separated the reweighted simulation and true
data is. Then,
Xη|L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)| ≤ SR|ŵ(x, m) − w(x, m)|
(13)
n
sim
x,m∈DSR⋆ĝ(x,m)g (x,m)⋆Recall that ŵ(x, m) = 1−ĝ(x,m)
and w(x, m) = 1−g⋆ (x,m) where g (x, m) = Pr(z = 1|x, m), so⋆|ĝ(x,m)−g (x,m)||ŵ(x, m) − w(x, m)| = (1−ĝ(x,m))(1−g⋆ (x,m)) . This denominator is greater than (1 − ĝmax )(1 −⋆gmax ). Then,
Xη|L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)| ≤|ĝ(x, m) − g ⋆ (x, m)| (14)⋆(1 − ĝmax )(1 − gmax
)nSR
sim
x,m∈DSR

We now look at the classifier for training g. The per-point cross entropy loss for (x, m, z)
is ℓ(g(x, m), z) = − log g(x, m) for z = 1 and − log(1 − g(x, m)) for z = 0. WLOG,
assume for some x and m, g ⋆ (x, m) > ĝ(x, m). Then |ℓ(g ⋆ (x, m), 1) − ℓ(ĝ(x, m), 1)| =
13


 ⋆

⋆
⋆
⋆
(x,m)
(x,m)
log gĝ(x,m)= log 1 + gĝ(x,m)−1≥ g g(x,m)/ĝ(x,m)−1= g (x,m)−ĝ(x,m)≥ |g ⋆ (x, m) −⋆ (x,m)/ĝ(x,m)⋆
g (x,m)1−ĝ(x,m)1−ĝ(x,m)
1 + 1−g≥ĝ(x, m)| and |ℓ(g ⋆ (x, m), 0) − ℓ(ĝ(x, m), 0)| = log 1−g⋆ (x,m) = log⋆ (x,m) − 1(1−ĝ(x,m))/(1−g ⋆ (x,m))−1g ⋆ (x,m)−ĝ(x,m)≥ |g ⋆ (x, m) − ĝ(x, m)|, where we use the inequality(1−ĝ(x,m))/(1−g ⋆ (x,m)) =1−ĝ(x,m)xlog(1 + x) ≥ 1+x for x > −1. Therefore, with probability 1 − δ,Xη|ℓ(ĝ(x, m), z) − ℓ(g ⋆ (x, m), z)|⋆SR(1 − ĝmax )(1 − gmax )nx,m∈SRsηnSR
log 2/δ⋆simE[|ℓ(ĝ(x,
m),
z)
−
ℓ(g
(x,
m),
z)|]
+
B
≤
2
⋆
(1 − ĝmax )(1 − gmax
)nSR
2nSR
sim

|L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)| ≤⋆where B2 = maxx,y {ℓ(ĝ(x, m), z), ℓ(g ⋆ (x, m), z)} = − log(min{ĝmin , gmin}). We assume that
B2 is finite, so there exists a constant c such that
s


ηnSR
log 2/δ⋆⋆⋆sim
c|L(ĝ) − L(g )| + B2
|L̂S (h , ŵ) − L̂S (h , w)| ≤⋆(1 − ĝmax )(1 − gmax
)nSR
2nSR
sim

where L(g) = Ex,m∈SR [ℓ(g(x, m), z)]. Since g ⋆ (x, m) is Bayes optimal, |L(ĝ) − L(g ⋆ )| =L(ĝ) − L(g ⋆ ) = L(ĝ) − L̂(ĝ) + L̂(ĝ) − L̂(g ⋆ ) + L̂(g ⋆ ) − L(g ⋆ ) ≤ 2 supg∈G |L(g) − L̂(g)|. Fromq
1/δTheorem 3.3 in [17], this is bounded by 2RnSB (ℓ ◦ G) + log
with probability at least 1 − δ.
2nSB
Then, applying a union bound, with probability 1 − δ, we have
|L̂S (h⋆ , ŵ) − L̂S (h⋆ , w)|
s
r


ηnSR
log 2/δ
log 4/δ
sim
SB
≤
+
B
4cR
(ℓ
◦
G)
+
2c
.
2
n
⋆
(1 − ĝmax )(1 − gmax
)nSR
2nSB
2nSR
sim
Putting everything together with another union bound, with probability 1 − δ, the generalization error
is at mostr
log 8/δ⋆LS (ĥ, ŵ) − LS (h , w) ≤ 2RnSR (ℓS ◦ {H, G}) + (1 + W B1 )
(15)
2nSR
s
r


ηnSR
log 4/δ
log 8/δsim
4cRnSB (ℓ ◦ G) + 2c
+
(16)
+ B2
⋆
(1 − ĝmax )(1 − gmax
)nSR
2nSB
2nSR
sim

E

Experiment Details

E.1

M ULTI -CW O L A Experiments

For the M ULTI -CW O L A experiment, we used the anomaly and simulation data from the Pythia
8 simulations in the LHC Olympics Dataset to create an unlabeled dataset we want to perform
anomaly detection on [13]. We have k = 3, and construct Mi (m) based on the thresholds[[3.3, 3.7], [0.09, 0.13], [0.3, 0.35]]. In standard CW O L A, only the first feature is regarded as theresonant feature, and it is thresholded with the interval [3.3, 3.7]. We measure the AUC of the
M ULTI -CW O L A versus CW O L A classifier and baselines in Figure 1 (Left). We constructed training
datasets of varying sizes from n = 59 to 6003. We used one test dataset with 65755 randomly
sampled anomaly points and 161658 randomly sampled background points.
All methods were trained using scikit-learn’s MLPClassifier with max_iter=5000.
For
M ULTI -CW O L A’s weak supervision step, we learn the parameters of the graphical model using SGD
and PyTorch [18] with class balance Pr(y = 1) = 0.25, 30000 epochs, and learning rate = 1e − 6.
14

Data (no anomaly)
Anomaly
Simulation 1
Simulation 2

counts

800
600
400
200
0

4

2

0
x

2

4

Figure 2: Synthetic data for evaluating M ULTI -SALAD.
E.2

M ULTI -SALAD Experiments

Setup For the synthetic, the true background is P(·|y = 0) = 12 N (−1, 0.2) + 12 N (1, 0.2), and1
the anomaly is P(·|y = 1) = 21 N (−2, 0.2) + 21 N (2, 0.2). Simulation 1 is Psim
= 21 N (1, 0.2) +11
12
2 N (0, 1), and simulation 2 is Psim = 2 N (−1, 0.2) + 2 N (0, 1). We generate 2000 points from1
the true background and 100 points that are anomalies to form D, and 2000 points each from Psim
2
sim
sim
and Psim to form D1 and D2 . We construct signal and sideband regions from these by splitting
datasets in half randomly, assuming they follow the same distribution over x (i.e., m is independentof x) except that there is no anomaly in the sideband regions.
We use MLPs from Keras [5], each with 3 hidden layers of dimension 32, ReLU activation, andtrained with cross-entropy loss and the Adam optimizer. We train for 50 epochs, batch size 200, anddefault parameters otherwise. Finally, we evaluate our approach on a new test set containing 200000
background points and 200000 anomaly points. This test set is used to produce the signal efficiency
to rejection rate. All experiments were run on a personal laptop.
Additional Results In Figure 2, we plot the synthetic example we evaluate on. In Figure 3, welook at the reweighting applied to the sideband region, and in Figure 4 we compare our reweighting
on simulation data to the true p(·|y = 0) in the signal region. In Figure 5, we show our results on
three individual runs. This is because computing the confidence intervals of these curves averaged
across the 10 random runs is too noisy due to the magnitude of the reciprocal 1/FPR.

15

50
40
30

40
countscounts50

Data (no anomaly)
Sim 1
Sim 1 SALAD

20
10
0

30

Data (no anomaly)
Sim 2
Sim 2 SALAD

20
10

1

0
x (SB)01

0x (SB)

1

Data (no anomaly)
Sim [1, 2]
Sim [1, 2] SALAD

50
40
counts

1

30
20
10
0

1

0
x (SB)

1

Figure 3: Top left: SALAD reweighting using simulation 1 on sideband region. Top right: reweighting using simulation 2. Bottom: reweighting using simulation 1 and 2 combined.

Data (with anomaly)
Sim 1 (SALAD)

150

200
countscounts

200
100
50
0

1

0
x (SR)

100
01Data (with anomaly)
Sim [1, 2] (SALAD-switch)

200
100

1

0
x (SR)

1

Data (with anomaly)
Sim [1, 2] (Multi-SALAD)

200
countscounts

150
50

300

0

Data (with anomaly)
Sim 2 (SALAD)

150
100
50

1

0
x (SR)011

0x (SR)

1

Figure 4: Top left: SALAD reweighting using simulation 1 on signal region. Top right: reweighting
using simulation 2. Bottom left: using both simulation 1 and 2 weights separately. Bottom right:
reweighting using simulation 1 and 2 combined.

16

1/FPR (Background Rejection)

104
103
102
101
100
0.4
0.6
TPR (Signal Efficiency)

1/FPR (Background Rejection)

0.8

102
101

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

104
103
102
101

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105

100

104
103
102
101
100

0.2

0.4
0.6
TPR (Signal Efficiency)

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
104
103
102
101

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

100

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
1/FPR (Background Rejection)

0.0

1/FPR (Background Rejection)

103

1.0

1/FPR (Background Rejection)

0.2

105

104
103
102
101
100

0.2

0.4
0.6
TPR (Signal Efficiency)

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
104
103
102
101

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

100

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
1/FPR (Background Rejection)

0.0

1/FPR (Background Rejection)

104

100
0.0

104
103
102
101
100

0.2

0.4
0.6
TPR (Signal Efficiency)

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
104
103
102
101

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

100

0.8

1.0

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
1/FPR (Background Rejection)

0.0

1/FPR (Background Rejection)

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105
1/FPR (Background Rejection)

Sim 1 SALADSim 2 SALAD
SALAD-Switch
Multi-SALAD

105

104
103
102
101
100

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

0.8

1.0

0.0

0.2

0.4
0.6
TPR (Signal Efficiency)

Figure 5: Results on individual runs.

17

0.8

1.0