dataesr/scientific-paragraphs-categorization huggingface dataset:
Learning the effect of nuisance parameters The correction term ∆ in eq. (  13 ) is the log-ratio between the likelihood of the Reference hypothesis evaluated with best-fit values of the nuisance parameters, and the one with centralvalue nuisance parameters. This object is of interest for any statistical analysis to be performed on the dataset D, as it provides a first indication of the data compatibility with the Reference hypothesis. In particular a sizable departure of the best-fit nuisance parameters from the central values should be monitored as an indication of a mis-modeling of the Reference hypothesis or possibly of a new physics effect. In order to introduce our strategy for the evaluation of ∆, it is convenient to first recall the standard approach, employed in most LHC analyses, based on a binned Poisson likelihood approximation of L(R ν |D).
In this approach, the dataset gets binned and the observed counting in each bin is compared with the corresponding ν-dependent cross section prediction. The predictions are obtained by computing each cross section for multiple values of the nuisance parameters and interpolating with a polynomial (or with the exponential of a polynomial, to enforce cross section positivity) around the central value ν = 0. A simple polynomial is sufficient to model the dependence of the cross section on the nuisances if their effect is small. The polynomial interpolation produces analytic expressions for the cross sections as a function of ν, which are fed into the Poisson likelihood. The maximization over ν in eq. (  13 ) is then performed with standard computer packages. In principle we could proceed to the evaluation of ∆ exactly as described above.
Evaluating N(H w,ν ) requires us to employ a Reference data set R = {x 1 ,. , x N R }. As described in Section 1, R consists of synthetic instances of the variable x that follow the Reference Model distribution. The R set plus the data D constitute the sample that we will employ for training the neural network f (x; w). Notice that the R dataset follows, by construction, the central-value distribution n(x|R 0 ). It might result from a first-principle Monte Carlo simulation, or have data-driven origin. In both cases it might take the form of a weighted event sample. 3 e choose the normalization of the weights such that e∈R w e = N(R 0 ). If the R sample is "unweighted", all the weights are equal, and equal to w e = N(R 0 )/N R , with N R the Reference sample size. The Reference sample plays conceptually the same role as the central-value in regular model-dependent LHC searches.
As already stressed in Section 1, however, asymptotic formulae such as the Wilks-Wald Theorem only hold in the limit of an infinitely large data set and therefore they offer no guarantee that P (t|R ν ) will resemble a χ 2 (and be independent of ν) in concrete analyses where the dataset has a finite size. At fixed dataset size, whether this is the case or not depends on the complexity (or expressivity) of the parameter-dependent hypothesis that is being compared with the data. When fitted by the likelihood maximization, an extremely flexible hypothesis will adapt its free parameters to reproduce (overfit) the observed data points individually. Therefore the value of t that results from the maximization can be driven by low-statistics portions of the dataset and thus violate the asymptotic condition even if the total size of the dataset is large.
The expressivity of our hypothesis is driven by the architecture (number of neurons and layers) of the neural network f (x; w), and by the other hyper-parameters (a weight clipping, in our implementation) that regularize the network preventing it from developing overly sharp features. We can thus enforce the validity of the asymptotic formula, i. e. ensure that P (t|R ν ) is close to a χ 2 and independent of ν, by properly selecting the neural network hyper-parameters. For the selection of the hyper-parameters according to the χ 2 compatibility criterion we proceed as in Ref. [2] , where this criterion had been already introduced on a more heuristic basis, unrelated with nuisance parameters. We generate toy datasets following the central-value hypothesis R 0 , we compute t and we compare its empirical distribution with a χ 2 with as many degrees of freedom as the number of parameters of the neural network.
Still it is interesting to discuss how the model-independent strategy that we are constructing would react to the presence of new physics effects in the auxiliary data. New (or mis-modeled) effects in auxiliary data could in general reduce the sensitivity of the test to new physics, however it is not obvious that this reduction will be significant. Consider the extreme case in which new physics is absent from the dataset of interest, and is present only in the auxiliary measurements. The new physics effects make the true auxiliary likelihood function different from the postulated one, L(ν|A). Therefore, in the likelihood maximization, the L(ν|A) term will push ν to values that are different from the true values of the nuisance parameters. This will occur both in the maximization of the L(R ν |D, A) and of the L(H w,ν |D, A) likelihoods.
For these incorrect values of the nuisance parameters, n(x|R ν ) does not provide a good description of the distribution of the data of interest D. Therefore the maximal likelihood of R ν will be small, due to the mismatch between the data and the Reference distribution estimated from the "signal-polluted" auxiliary dataset. The H w,ν hypothesis instead possesses enough flexibility to adapt n(x|H w,ν ) according to the data of interest, thanks to the flexibility of the neural network (4). The likelihood of H w,ν will thus possess a high maximum, in the configuration where ν maximizes the auxiliary likelihood and the neural network accounts for the discrepancy between the x distribution at that value of ν and the true x distribution at the true value of the nuisance parameters. This can enable our test to reveal a tension of the data with the Reference Model even in this limiting configuration, as we will see happening in Section 3.
5 in a simple setup. New physics effects in the auxiliary data might thus not spoil the potential to achieve a discovery. On the other hand, they would complicate its interpretation. Similar considerations hold for possible new physics contaminations in the Reference dataset R employed for training. These contaminations emerge if R has a data-driven origin, and if new physics affects the distribution of the data control region. Since the control region data are transferred to the region of interest by assuming the validity of the Reference Model, the net effect is a mismatch between the true distribution of x in the (central-value) Reference Model and the actual distribution of the instances of x in the Reference sample. As for auxiliary measurements, new physics in control regions does not necessarily spoil the sensitivity to new physics.
Indeed our test is sensitive to generic departures of the observed data distribution with respect to the distribution of the Reference dataset. Departures which are due to a mis-modeling of the Reference induced by new physics in the control region, rather than to new physics in the data of interest, could still be seen. Our strategy would instead loose completely sensitivity if new physics affects the control region and the data of interest in the exact same way, because in this case there would be strictly no difference between the distribution of the data and the one of the Reference dataset.
All the steps that are needed to set up our strategy, from the model selection to the evaluation of the distribution of the test statistic, are performed based exclusively on knowledge of the Reference Model and not on putative new physics signals, as appropriate for a model-independent search strategy. While presented in the context of a simple univariate problem that is rather far from a realistic LHC data analysis problem, the technical implementation of all the steps described in the present section is straightforwardly applicable to more complex situations. The application to a more realistic problem will be discussed in Section 4.
e. , σ n,s → 0). It is easy to see from the results of Section 2, or from Ref. s  [1, 2] , that the test statistic in this limit becomes t(D) = 2 max w log L(H w |D) L(R 0 |D) = -2 min w L [f (•; w); ]. (28) The minimization is performed by training the network f with the loss function L [f (•; w)] = - x∈D [f (x; w)] + e∈R w e e f (xe;w) -1. (29) The asymptotic distribution of t is a χ 2 with a number of degrees of freedom which is equal to the number of trainable parameters of the neural network. The χ 2 compatibility of a given neural network model will be monitored by generating toy instances of the dataset D in the R 0 hypothesis, running the training algorithm on each of them, computing the empirical probability distribution of t and comparing it with the χ 2. We first discuss how to select the weight clipping regularization parameter for a given architecture of the neural network.
We consider for illustration, in the simple univariate example at hand, a network with four nodes in the hidden layer (and one-dimensional input and output). We refer to this architecture as (1, 4, 1), for brevity. This network has a total of 13 trainable parameters, therefore the target t distribution is a χ 2 13 with 13 degrees of freedom. We generated a Reference sample R, with N R = 200 000 = 100 N(R 0 ) entries, following the R 0 distribution of the variable x as given by eq. (  26 ) for ν n,s = 0. The sample is unweighted, therefore the weights in the sample are all equal and w e = N(R 0 )/N R = 0. 01. We also generate 400 toy instances of the dataset D in the same hypothesis. The number of instances of x in D, N D , is thrown from a Poisson distribution with mean N(R 0 ) = 2 000 in accordance with the R 0 expectation.
For different values of the weight clipping parameter, ranging from 1 to 100, we train the neural network with the loss in eq. (  29 ) and we compute t(D) on the toy datasets using eq. (  28 ). The empirical P (t|R 0 ) distributions obtained in this way after 300 000 training epochs, and some of its percentiles as a function of the number of epochs, are reported in Figure  1. We see that for large values of the weight clipping parameter the distribution sits slightly to the right of the target χ 2 with 13 degrees of freedom. Furthermore the training is not stable and significant changes in the t percentiles (especially the 95% one) occur even after 150 000 epochs. Very small values of the weight clipping make the distribution stable with training, but push it lower than the χ 2 13 expectation.
A good compatibility is instead obtained for intermediate values of the weight clipping parameter. We see that a weight clipping equal to 9 reproduces the χ 2  13 formula quite accurately. The strategy to find the value of the weight clipping parameter that best complies with the χ 2 compatibility criterion can be refined and optimized. We can start from one small and one large value of the weight clipping, for which we expect that the distribution of t will, respectively, undershoot and overshoot the χ 2 expectation, and compute t by running the training algorithm on a limited number n of toy datasets. The average of t over the n toys will be below (above) the mean of the target χ 2 distribution (i. e. , 13, in the case at hand) for the small (large) value of the weight clipping. We thus obtain a window of values where the optimal weight clipping sits, which can be further narrowed by applying a standard root finding algorithm on the average t compared with the expected mean.
Clearly the average t will be affected by a relatively large error if n is small. Therefore after a few iterations of the root finding algorithm, it will become compatible with the expected mean, preventing us from further restricting the weight clipping compatibility window. Rather than looking at the compatibility of the average, a more powerful compatibility test should be employed at this stage in order to pick up the optimal weight clipping value inside the window. Furthermore this test should be sensitive to the entire shape of the distribution and not only to its central value. One can consider for instance a Kolmogorov-Smirnov (KS) test and maximize, in the window, the p-value for the compatibility with the target χ 2 of the empirical t distribution. 4 t is advantageous to implement the strategy described above using a rather small number n of toy datasets, because training could become computationally demanding in realistic applications of our strategy.
Computing the test statistic We finally have at our disposal all the ingredients to compute the test statistic t(D, A). This consists of the τ term, subtracted by the correction ∆. We now illustrate the evaluation of the two terms in turn, as implemented in the TensorFlow  [15]  package. The implementation is schematically represented in Figure  4 , and the corresponding code is available at  [16]. As described in Section 2. 4, computing τ requires the simultaneous optimization of the parameters w of the neural network model f (x; w) (dubbed "BSM network" in the figure) and of the nuisance parameters ν. The loss function is the one of eq. (25). It depends on ν through the distribution ratio r, or more precisely through its estimate r(x; ν) as in eq. (  16 ). The estimated r ratio is implemented as a TensorFlow "λ-layer" (denoted as "r layer" in the figure) that takes as input the output of the δ networks and builds the required polynomial function of ν.
Validation As previously emphasized, it is vital for the applicability of our strategy that the distribution P (t|R ν ) of the test statistic is nearly independent of ν. This is ensured in line of principle by the asymptotic formulae described in Section 2. 5. Verifying in practice the validity of the asymptotic formulae is thus the crucial validation step, which we will perform by computing the empirical P (t|R ν ) distribution on toy experiments. toy datasets are generated according to the R ν hypothesis, at different points ν = ν * = (ν * n , ν * s ) of the nuisances' parameter space. Each toy dataset D is accompanied by one instance of the nuisance parameters estimators ν = ( ν n , ν s ). As explained at the beginning of the present section, the estimators are thrown as Gaussians with standard deviations σ n,s centered at ν * n,s.
This potential issue should be kept in mind when dealing with poorly constrained nuisance parameters. Similar problems occur in traditional analyses, whenever the reference control sample statistics is insufficient. A typical mitigation of this effect is obtained binning the dataset with larger binwidths on distribution n , ν * s ) = (0, -0. 6). Increasingly accurate modelings of log r(x; ν) are employed in the three panels, namely the linear-and quadratic-order approximations and the analytic log-ratio in eq. (30). tails. For our method, which in its generic formulation does not make use of bins, possible solutions are either to restrict the variables to a region that is well-populated by the available Reference sample, or to produce a Reference sample that populates the tail of the features distribution more effectively.
Also this assumption is not crucial for the sensitivity since a modeling of the signal is not required in our method. Nuisance parameters related to the signal come at play whenever one wants to interpret the outcome of the method as a bound on the theoretical parameters of a specific scenario. We quantify the potential of our strategy to detect departures from the Reference Model, if one of the three NP 1,2,3 models is present in the data, in terms of the median Z-score Z obtained by running our algorithm on toy datasets generated according to the n(x|NP) distribution. For each NP-hypothesis toy we repeat the exact same operations we described in Section 3. 3 to obtain the test statistic t, in the exact same configuration (architecture, weight clipping, etc. ) we used in Section 3. 4 for validation on the Reference-hypothesis toy datasets.
The linear-order reconstruction of log r is employed for the modeling of the nuisance parameters effect. We saw in Section 3. 4 that this modeling is sufficiently accurate if we limit our analysis to the regime σ n,s ≤ 15%. The value of t on each NP toy is compared with the χ 2  13 distribution and converted to a p-value by exploiting the asymptotic formulae we verified Section 3. 4. For each NP 1,2,3 new physics scenario, the median p-value is computed using 100 NP toy datasets, obtaining Z = Φ -1 (1p), with Φ the cumulative of the Standard Gaussian. The results are reported in Figure  8  under multiple assumptions (σ n,s = 5, 10, 15%) for the nuisance parameters standard deviations and for different choices (ν * n,s = 0, ±σ n,s ) of the true values of the nuisance parameters that underly (through the R ν component of n(x|NP)) the generation of the NP toys.
[14]. Namely, we extend the NP hypothesis by a "signal strength" parameter µ ≥ 0 that rescales N i → µ N i (for i = 1, 2, 3) in eq. s  (32) (33) (34)   value of the signal strength parameter that maximizes the likelihood of the NP hypothesis, and ν the maximum in the nuisances' space, we define q 0 = -2 log max ν L(R ν |D, A) L(NP i; ν; µ |D, A) , (35) if µ > 0, and we set q 0 = 0 otherwise. In the equation, L denotes the extended likelihood constructed as in Section 2, exploiting the analytic knowledge of the new physics distributions provided by eq. s  (32) (33) (34). The "numerator" hypothesis R ν coincides by construction with the NP hypothesis at µ = 0. The distribution of q 0 under the Reference (numerator) hypothesis is known in the asymptotic limit. We can thus associate a p-value to the value of q 0 that is obtained on each NP toy data set.
Since we do not know about this bias, our auxiliary likelihood remains the one in eq. (  27 ), but ν s (A) in reality is not distributed around the true ν * s , but around ν * s + ∆ν s. In order to generate toy experiments that describe this scenario, one has to take ν * s + ∆ν s as the central-value for the generation of the toy ν s values while using the true ν * s for the generation of the x toy datasets. The mismatch, on average, between ν s and the value of ν s that truly underlies the x variable distribution can lead to the detection of new physics as explained in Section 2. 6. For σ s = 15% we find sensitivities In the previous section we described the practical implementation of our strategy and its validation in a very simple univariate toy problem. We now turn to a slightly more complex setup, which is inspired by the realistic problem of model-independent new physics searches in twobody final states at the LHC (see Ref.
We do not specify sharply the nature of the final state objects. In the typical cases we have in mind, these are either muons, electrons or τ leptons reconstructed by the detector. On the other hand, the same construction could be applied to objects with similar resolution, e. g. , trading electrons for photons or taus for jets. The kinematical distributions would be quite different in the different cases, however we do not expect these differences to impact the technical viability of our strategy, which we aim at demonstrating. The total cross-section of the process would be also different. However we can compensate this adjusting the assumed integrated luminosity of the dataset, making the total number of expected events N(R 0 ) roughly equal in the various cases. Therefore, for our purpose the only relevant difference between muons, electrons and τ final states resides in the increasingly large systematic uncertainties that affect the corresponding SM predictions.
Since larger uncertainties are more difficult to handle, as outlined in the previous section, it is instructive to investigate these three scenarios. Owing to the previous discussion, we ignore the difference in the distributions of the different final states and we model all of them as opposite-sign muons. Namely, the central-value Reference distribution n(x|R 0 ) is the same in all cases, and it corresponds to the SM simulation of pp → µ + µ -+ X at the 13 TeV LHC obtained with MadGraph5  [17]  at LO, with extra jets matching and using Pythia6  [18]  and Delphes3  [19]  for parton showering and detector simulation. The data samples we employ for the analysis are the ones described in Ref. [2]  and can be downloaded from Zenodo  [20]. We consider two Gaussian nuisance parameters ν n and ν s describing, as in the previous section, the uncertainty on the event yield normalization and on the scale factor in the measurement of the transverse momenta.
A similar analysis could also be repeated below the Z mass, as done by the CMS and the LHCb experiments, exploiting real-time analysis techniques  [25, 26]. We do not discuss this case here. In what follows we describe the implementation of our model-independent search strategy on a dataset whose integrated luminosity corresponds to N(R 0 ) = 8 700 expected events in the signal region defined by the acceptance and the 100 GeV invariant mass cut. In the case of opposite-sign muons, this number of events would correspond to an integrated luminosity of around 0. 35 fb -1. The expected event yield in the non-central Reference hypothesis, N(R ν ), is computed with the same integrated luminosity, duly taking into account the normalization nuisance factor e νn , and the effect of the scale nuisance ν s on the selection cuts efficiency.
A higher integrated luminosity, of 1. 1 fb -1 , is considered in the τ -like scenario in order to maintain N(R 0 ) as large as (specifically, N(R 0 ) = 8 400) in the other scenarios compensating for the higher invariant mass cut. Finally, in all scenarios we apply an upper cut p T,1(2) < 1 TeV. The phase space region excluded by this cut is populated, for the luminosity we are considering, with a probability as low as 10 -5 in the Reference model. Therefore it has essentially no impact on the analysis and on its sensitivity to new physics, also in light of the fact that the mere observation of a few events in the region excluded by the cut would constitute a discovery. On the other hand, it is technically important to set some upper cut (though extremely mild, as in this case) in order to strictly avoid the presence in toy datasets of high-p T outliers, falling in a region that is too rare to be populated even in the Reference sample.
This choice, which obviously enhances the statistical fluctuations of the Reference sample, was made in order to validate our strategy in a realistic context where an extremely abundant Reference sample might (possibly because of the resources needed to run the full detector simulation) not be available. 11 n the same spirit, the results of the present section are obtained (if not specified otherwise) using a single Monte Carlo sample of 3. 6 million unweighted events in total, generated with mild acceptance requirements. Each toy dataset was obtained by random sampling around (up to Poisson fluctuations) 200 000 events in the original sample. After the events are selected according to these requirements, the desired average number N(R 0 ) of toy events is found. The Reference dataset employed for the training of each toy experiment was obtained by sampling 1 million events from the original data, out of the remaining 3.