address: , , , ;  , , , ;  , , , ;  , , , ;  , , , ;  , , ,
author: ;  ;  ;  ;  ;  ;  ;  ;  ;  ;  ;  ;
title: BioPreDyn-bench: benchmark problems for kinetic modelling in systems biology

# Background

Systems biology aims at understanding the organization of complex biological systems with a combination of mathematical modelling, experiments, and advanced computational tools. To describe the behaviour of complex systems, models with sufficient level of detail to provide mechanistic explanations are needed. This leads to the use of large-scale dynamic models of cellular processes . By incorporating kinetic information, the range of applications of biological models can be widened. The importance of kinetic models is being increasingly acknowledged in fields such as bioprocess optimization , metabolic engineering , physiology, as well as cell and developmental biology .

Systems identification, or reverse engineering, plays an important part in the model building process. The difficult nature of reverse engineering was stressed in , where the different perspectives that coexist in the area of systems biology were reviewed. Specifically, large-scale dynamic biological models generally have many unknown, non-measurable parameters. For the models to encapsulate as accurately as possible our understanding of the system (i.e. reproducing the available data and, ideally, being capable of making predictions), these parameters have to be estimated. This task, known as parameter estimation, model calibration, or data fitting , consists of finding the parameter values that give the best fit between the model output and a set of experimental data. This is carried out by optimizing a cost function that measures the goodness of this fit. In systems biology models this problem is often multimodal (nonconvex), due to the nonlinear and constrained nature of the system dynamics. Hence, standard local methods usually fail to obtain the global optimum. As an alternative, one may choose a multistart strategy, where a local method is used repeatedly, starting from a number of different initial guesses for the parameters. However, this approach is usually not efficient for realistic applications, and global optimization techniques need to be used instead .

Many methods have been presented for this task, but less effort has been devoted to their critical evaluation. It is clear, however, that to make progress in this research area it is essential to assess performance of the different algorithms quantitatively, in order to understand their weaknesses and strengths. Furthermore, if a new algorithm is to be accepted as a valuable addition to the state of the art, it must be first rigorously compared with the existing plethora of methods. This systematic comparison requires adequate benchmark problems, that is, reference calibration case studies of realistic size and nature that can be easily used by the community. Several collections of benchmarks – and of methods for generating them – have already been published . An artificial gene network generator, which allows to choose from different topologies, was presented in . The system, known as A-BIOCHEM, generates pseudo-experimental noisy data in silico, simulating microarray experiments. An artificial gene network with ten genes generated in this way was later used to compare four reverse-engineering methods . More recently, a toolkit called GRENDEL was presented with the same purpose , including several refinements in order to increase the biological realism of the benchmark. A reverse-engineering benchmark of a small biochemical network was presented in . The model describes organism growth in a bioreactor and the focus was placed on model discrimination using measurements of some intracellular components. A proposal for minimum requirements of problem specifications, along with a collection of 44 small benchmarks for ODE model identification of cellular systems, was presented in . The collection includes parameter estimation problems as well as combined parameter and structure inference problems. Another method for generation of dynamical models of gene regulatory networks to be used as benchmarks is GeneNetWeaver , which was used to provide the international Dialogue for Reverse Engineering Assessments and Methods (DREAM) competition with three network inference challenges (DREAM3, DREAM4 and DREAM5) . Subsequent competitions (DREAM6, DREAM7) included also parameter estimation challenges of medium-scale models . Similar efforts have been carried out in related areas, such as in optimization, where BBOB workshops (Black-Box Optimization Benchmarking, ) have been organised since 2009. In this context it is also worth mentioning the collection of large-scale, nonlinearly constrained optimization problems from the physical sciences and engineering (COPS) .

Despite these contributions, there is still a lack of suitable benchmark problems in systems biology that are at the same time (i) dynamic, (ii) large-scale, (iii) ready-to-run, and (iv) available in several common formats. None of the above mentioned collections possesses all these features, although each one has a subset of them. Here we present a collection of medium and large-scale dynamic systems, with sizes of tens to hundreds of variables and hundreds to thousands of estimated parameters, which can be used as benchmarks for reverse-engineering techniques. The collection includes two *Escherichia coli* models , a genome-wide kinetic model of *Saccharomyces cerevisiae* , a metabolic model of Chinese Hamster Ovary (CHO) cells , a signal transduction model of human cells , and a developmental gene regulatory network of *Drosophila melanogaster* .

Ensuring standardisation allows systems biology models to be reused outside of their original context: in different simulators, under different conditions, or as parts of more complex models . Minimum requirements for published systems biology models are set out by the MIRIAM initiative : completeness of documentation, availability in standard formats, and semantic annotations connecting the model to web resources . To this end, we have made five of the six models (the exception is the spatial model of *D. melanogaster*) available in Systems Biology Markup Language (SBML ) format, allowing for their simulation in multiple software tools, including AMIGO and COPASI . Even when defined in a standard format such as SBML, large models such as the genome-wide kinetic model of *S. cerevisiae* may give different results when simulated in different software environments. The inherent size and stiffness of genome-scale systems biology models create new challenges to be addressed for their robust simulation by systems biology tools . To address this problem all the models have been consistently formatted, with their dynamics provided both in C and in Matlab. Additionally, a benchmark consisting of a parameter estimation problem has been defined for every model, for which ready-to-run implementations are provided in Matlab (optionally with the use of the AMIGO toolbox ) and, in some cases, also in COPASI . The availability of ready-to-run implementations is a highly desirable practice in computer science, since it ensures reproducibility of the results. Calibration results with state of the art optimization methods are reported, which can serve as a reference for comparison with new methodologies. Additionally, suggestions on how to compare the performance of several methods are also given in the Results section.

# Problem statement

Given a model of a nonlinear dynamic system and a set of experimental data, the parameter estimation problem consists of finding the optimal vector of decision variables $p$ (unknown model parameters). This vector consists of the set of parameter values that minimize a cost function that measures the goodness of the fit of the model predictions with respect to the data, subject to a number of constraints. The output state variables that are measured experimentally are called observables. The following elements need to be clearly stated in order to properly define the calibration problem:

-   cost function to optimize (i.e. metric which reflects the mismatch between experimental and predicted values)

-   dynamics of the systems (in our benchmark models they are given by systems of ordinary differential equations)

-   model parameters to be estimated

-   initial conditions for the dynamics (possibly unknown, in which case they are included among the parameters to be estimated)

-   upper and lower bounds for the parameters

-   state variables that can be measured (observed)

-   values of external stimuli, also known as control variables

-   measurements (over time and/or space) available for the calibration: number of experiments, stimuli (if any) for each experiment, data points per experiment, etc.

-   (optional) type and magnitude of errors considered for the experimental data

-   (optional) additional time series for cross-validation of the calibrated model

-   solver used to numerically simulate the systems, and the relative and absolute error tolerances used

Mathematically, it is formulated as a nonlinear programming problem (NLP) with differential-algebraic constraints (DAEs), where the goal is to find $p$ to minimize an objective function. The objective function, or cost function, is a scalar measure of the distance between data and model predictions. There are several common choices for the objective function. The generalized least squares cost function is given by: $$\label{eq:statement}
J_{lsq} = \sum^{n_{\epsilon}}_{\epsilon=1} \sum^{n^{\epsilon}_o}_{o=1} \sum^{n^{\epsilon,o}_s}_{s=1} w^{\epsilon,o}_s \left(ym^{\epsilon,o}_s - y^{\epsilon,o}_s(p) \right)^2$$ where $n_{\epsilon}$ is the number of experiments, $n^{\epsilon}_o$ is the number of observables per experiment, and $n^{\epsilon,o}_s$ is the number of samples per observable per experiment. The measured data will be denoted as $ym^{\epsilon,o}_s$ and the corresponding model predictions will be denoted as $y^{\epsilon,o}_s(p)$. Finally, $w^{\epsilon,o}_s$ are scaling factors used to balance the contributions of the observables, according to their magnitudes and/or the confidence in the measurements. When information about the experimental error is available, one may use the maximum (log-)likelihood function to look for the parameters with the highest probability of generating the measured data. Assuming independently identically distributed measurements with normally distributed noise, the likelihood is defined as:

$$\label{eq:llk}
J_{llk} = \sum^{n_{\epsilon}}_{\epsilon=1} \sum^{n^{\epsilon}_o}_{o=1} \sum^{n^{\epsilon,o}_s}_{s=1} \frac{ \left(ym^{\epsilon,o}_s - y^{\epsilon,o}_s(p) \right)^2 } {(\sigma^{\epsilon,o}_s)^2 }$$

For known constant variances the log-likelihood cost function is similar to the generalized least squares, with weights chosen as the inverse of the variance, $w^{\epsilon,o}_s = 1/(\sigma^{\epsilon,o}_s)^2$. This is the case of most of the benchmark problems presented here (B1, B2, B4, B5). The exceptions are: problem B3, in which no noise has been added to the data, and thus the scaling factors $w^{\epsilon,o}_s$ are taken as the squared inverse of the maximum experimental value for each observable; and problem B6, where the weights are inversely related to the level of expression. More details are given in the next section. Note that the cost functions used in Matlab/AMIGO are not exactly the same as the ones used in COPASI, since in COPASI the weights are scaled so that for each experiment the maximal occurring weight is 1.

The minimization of the objective function is subject to the following constraints: $$\label{eq:constraints1}
\dot{x} = f\left(x,p,t \right)$$ $$\label{eq:constraints2}
x(t_0) = x_0$$ $$\label{eq:constraints3}
y = g(x,p, t)$$ $$\label{eq:constraints4}
h_{eq}(x,y,p) = 0$$ $$\label{eq:constraints5}
h_{in}(x,y,p) \leq 0$$ $$\label{eq:constraints6}
p^L \leq p \leq p^U$$ where $g$ is the observation function, $x$ is the vector of state variables with initial conditions $x_0$, $f$ is the set of differential and algebraic equality constraints describing the system dynamics (that is, the nonlinear process model), $h_{eq}$ and $h_{in}$ are equality and inequality constraints that express additional requirements for the system performance, and $p^L$ and $p^U$ are lower and upper bounds for the parameter vector $p$. The problem defined above is the general formulation of a nonlinear least squares optimization subject to dynamic constraints and bounds in the parameters. The problems included in this collection of benchmarks do not make use of constraints (–).

## Remarks on parameter estimation methods

Fitting a large, nonlinear model to experimental (noisy) data is generally a multimodal problem. In these circumstances, the use of local optimization methods, which are usually gradient-based, entails the risk of converging to local minima. Hence it is needed to use global optimization methods that provide more guarantees of converging to the globally optimal solution . Global optimization strategies can be roughly classified as deterministic, stochastic and hybrid. Deterministic methods can guarantee the location of the global optimum solution; however, their computational cost makes them unfeasible for large-scale problems. Stochastic methods, which are based on probabilistic algorithms, do not provide those guarantees, but are frequently capable of finding optimal or near-optimal solutions in affordable computation times.

Some of the most efficient stochastic global optimization methods are the metaheuristic approaches. A heuristic is an algorithm originated not from formal analysis, but from an expert knowledge of the task to be solved. A metaheuristic can be seen as a general-purpose heuristic method designed to guide an underlying problem-specific heuristic. It is therefore a method that can be applied to different optimization problems with few modifications. Hybrid methods which combine metaheuristics for global optimization and local methods for accelerating convergence in the vicinity of local minima can be particularly efficient. One such method is the enhanced Scatter Search algorithm, eSS , and its parallel cooperative version, CeSS . Matlab and R implementations are publicly available as part of the MEIGO toolbox . The eSS method is available as a Matlab toolbox and is also included in AMIGO; this latter version is the one used in this work. It should be noted that AMIGO offers more than a dozen optimization solvers, including local and global methods, and the possibility of combining them to form user-defined sequential hybrid methods. In COPASI it is possible to choose among thirteen different optimization methods for parameter estimation, including deterministic and stochastic: Evolutionary Programming, Evolutionary Strategy (SRES), Genetic Algorithm, Hooke and Jeeves, Levenberg–Marquardt, Nelder–Mead, Particle Swarm, Praxis, Random Search, Simulated Annealing, Scatter Search, Steepest Descent, and Truncated Newton.

## Remarks on comparing optimization methods

Although the objective of this paper is to present a set of ready-to-run benchmarks, we list below several guidelines on how to compare different optimizers with these problems.

Many optimization methods require an initial point and/or bounds on the decision variables. For ensuring a fair comparison between different methods, the same bounds and initial points should be set. Obviously, the nominal solution can not be used as an initial point. Special emphasis should be laid on ensuring full reproducibility. This entails providing all source codes and binary files used in computations, as well as specifying all implementation details, such as software and hardware environment (including compiler versions and options, if any). If some aspects of a method can be tuned, these settings must be clearly indicated.

Many different criteria may be used for comparing the performance of optimization methods. It can be expressed as a function of CPU time, number of function evaluations, or iteration counts. When considering several problems, a solver's average or cumulative performance metric can be chosen. If an algorithm fails to converge a penalty can be used, in which case an additional decision is required to fix its value. An alternative is to use ranks instead of numerical values, although this option hides the magnitudes of the performance metric. All approaches have advantages and drawbacks, and their use requires making choices that are subjective to some extent. In an attempt to combine the best features of different criteria, Dolan and Moré proposed to compare algorithms based on their performance profiles, which are cumulative distribution functions for a performance metric. Performance profiles basically rely on calculations of the ratio of the solver resource time versus the best time of all the solvers. It should be noted that, for complex large-scale problems where identifiability is an issue, different methods often arrive at different solutions. In that case the use of performance profiles requires choosing a tolerance to define acceptable solutions. Performance profiles are a convenient way of summarizing results when there are many methods to be compared and many problems on which to test them. When this is not the case, however, more information can be provided by using convergence curves. Convergence curves plot the evolution of the objective function, as defined in Equation (), as a function of the number of evaluations or the computation time (since the overhead is different for each method). They provide information not only about the final value reached by an algorithm, but also about the speed of progression towards that value.

When comparing different optimization methods, the best result (cost) and the mean (or median) for N runs should be reported in a table. Similar statistics for computation and number of evaluations should apply to all the methods. However, since the final values can be greatly misleading, convergence curves should be provided in addition to this table.

Note that, in order to make a fair comparison of convergence curves obtained with different software tools and/or hardware environments, it is a good practice to report any speedup due to parallelism. This can happen in non-obvious situations. For example, COPASI can make use of several threads in multi-core PCs due to its use of the Intel MKL library. In summary, fair comparisons should be made taking into account the real overall computational effort used by each method/implementation.

## Remarks on identifiability

Parameter estimation is just one aspect of what is known as the inverse problem. This larger problem also includes identifiability analysis, which determines whether the unknown parameter values can be uniquely estimated . Lack of identifiability means that there are several possible parameter vectors that give the same agreement between experimental data and model predictions. We may distinguish between a priori structural identifiability and a posteriori or practical identifiability . The parameters are structurally identifiable if they can be uniquely estimated from the designed experiment under ideal conditions of noise-free observations and error-free model structure. Structural identifiability is a theoretical property of the model structure, which can be very difficult to determine for large and complex models . Even if a model is structurally identifiable, it may exhibit practical identifiability issues. Practical identifiability depends on the output sensitivity functions (partial derivatives of the measured states with respect to the parameters). If the sensitivity functions are linearly dependent the model is not identifiable, and sensitivity functions that are nearly linearly dependent result in parameter estimates that are highly correlated. Furthermore, even if they are linearly independent, low sensitivities may lead to an undesirable situation. Practical identifiability can be studied from sensitivity-based criteria like the Fisher information matrix (FIM). The practical identifiability of the models can be analyzed in this way with the AMIGO toolbox . The AMIGO_LRank method ranks the model parameters according to their influence on the model outputs, using several sensitivity measures. In large biological models identifiability issues are the norm rather than the exception . This may be partly due to inconsistent modelling practices, but even when a model has been carefully built and is structurally identifiable, the amount of data required for a perfect calibration (practical identifiability) is usually large. As an illustration, consider the general case of a model described by differential equations, and assume the ideal situation where the structure of the equations is perfectly known. Then, a well-known result states that identification of $r$ parameter values requires $2r+1$ measurements . However, it is frequently the case that a model with more than a thousand parameters has to be calibrated with only dozens or maybe hundreds of measurements. Finally, it should be noted that lack of identifiability does not preclude the use of model-based methods. Unique model predictions can in fact be obtained despite unidentifiability, as discussed by Cedersund .

# Benchmark problems

Here we present a collection of parameter estimation problems and their descriptions. The characteristics of the six dynamic models are summarized in Table . Four of the benchmark problems have been defined using in silico experiments, where pseudoexperimental data have been generated from simulations of the models and addition of artificial noise. The use of simulated data is usually considered the best way of assessing performance of parameter estimation methods, because the true solution is known. Additionally, we provide two benchmark problems that use real data. For each problem we provide the following information (see Supplementary Information online, <http://www.iim.csic.es/~gingproc/biopredynbench/>):

-   Dynamic model.

-   Experimental information: initial conditions, input functions, what is measured, measurement times, noise level and type.

-   Cost function to be used: its type (least squares, weighted least squares, maximum likelihood, etc), and why it should be chosen.

-   Parameters to estimate: lower and upper bounds, initial guesses, nominal values (the latter must not be used during estimations).

-   Implementations (Matlab with and without the AMIGO toolbox, C, COPASI): installation, requirements, and usage. Ready-to-run scripts are provided, with examples of how to execute them and their expected output.

```latex
\begin{table*}[t]\begin{flushleft}
  \caption{
  \textbf{Models}
  }
  \begin{tabular}{|l|l|l|l|l|l|l|}
  \hline
Model ID           & \textbf{B1}              & \textbf{B2}                  & \textbf{B3}             & \textbf{B4}  & \textbf{B5}  & \textbf{B6} \\
\hline
Model Ref          &\cite{smallbone2013large} &\cite{chassagnole2002dynamic} &\cite{kotte2010bacterial}&\cite{villaverde2013highconfidence}&\cite{macnamara2012state}&\cite{crombach2012efficient}\\
Cell               & \textit{S. cerevisiae}   & \textit{E. coli}             & \textit{E. coli}        & CHO          & Generic      & \textit{Drosophila}\\
                   &                          &                              &                         &              &              & \textit{melanogaster}\\
Description        & Metabolic:               & Metabolic:                   & Metabolic: CCM          & Metabolic    & Signal       & Developmental \\
level              & genome scale             & CCM                          & \& transcription        &              & transduction & GRN (spatial) \\
Parameters         & 1759                     & 116                          & 178                     & 117          & 86           & 37 \\
Dynamic states     & 276                      & 18                           & 47                      & 34           & 26           & 108 --212 \\
Observed states    & 44                       &  9                           & 47                      & 13           & 6            & 108 --212 \\
Experiments        &  1                       & 1                            &  1                      &   1          & 10           & 1 \\
Data points        & 5280                     & 110                          & 7614                    & 169          & 96           & 1804 \\
Data type          & simulated                & measured                     & simulated               & simulated    & simulated    & measured \\
Noise level        & $20\%$                   & real                         & no noise                & variable     & $5\%$        & real \\
\hline
\end{tabular}
\\
Main features of the benchmark models.
\end{flushleft}
\label{tab:models}
\end{table*}
```

## Problem B1: genome-wide kinetic model of *S. cerevisiae*

The biochemical structure of this model is taken from yeast.sf.net (version 6, ). In decompartmentalised form, this network has 1156 reactions and 762 variables. We fix some experimentally determined exchange fluxes, and use geometric FBA to choose a unique reference flux distribution consistent with the experimental data. We fix some initial concentrations to their experimentally determined levels and assign the remainder typical values. We define reaction kinetics using the common modular rate law, a generalised form of the reversible Michaelis-Menten kinetics that can be applied to any reaction stoichiometry . The final model contains 261 reactions with 262 variables and 1759 parameters. This model has been created according to the pipeline presented in , which ensures consistency with our sparse data set; whilst no data is required to produce the model, it can incorporate any known flux or concentration data or any kinetic constants. As an addition to the model developed in , this version has been alligned with previously unpublished experimental data. The new data consist of 44 steady-state measurements (38 concentrations and 6 fluxes), which are included in Tables 14 and 15 of the Supplementary Information online. The steady state is found to be stable. The number of measurements available at the present stage is not enough for carrying out a proper model calibration. Envisioning that dynamic (time-series) measurements of the 44 observed variables may be available in the near future, we show in this paper how they will be employed for re-estimating the parameter values. With this aim, we have generated pseudo-experimental noisy data corresponding to a pulse in the concentration of extracellular glucose, and have used this simulated data to re-calibrate the model. We generated 120 samples per observable and added artificial measurement noise (20%) to resemble realistic conditions.

## Problem B2: dynamic model of the Central Carbon Metabolism of *E. coli*

This model, originally published in and available at the BioModels database , reproduces the response to a pulse in extracellular glucose concentration. It includes 18 metabolites in two different compartments: the cytosol (17 internal metabolites), and the extracellular compartment (1 extracellular metabolite: glucose). These metabolites are involved in 48 reactions: 30 kinetic rate reactions, 9 degradation equations, 8 dilution equations, and 1 equation for extracellular glucose kinetics. Additionally, there are 7 analytical functions, thus the model is defined by a total of 55 mathematical expressions. We have reformulated the model to use it as a parameter estimation problem; the 116 parameters to be estimated consist of kinetic parameters and maximum reaction rates. As an addition to the model version available in the Biomodels database, we provide the experimental data that were used in the original publication but had not been published (Klaus Mauch, personal communication). The dataset is given in Table 16 of the Supplementary Information online, and consists of time-course concentration measurements of nine metabolites. The aim of the model calibration in this case is to find a better fit to the experimental data than the one obtained with the nominal parameter vector used in the original publication . Note that this is different from benchmarks 1 and 3–5, which use simulated data and where the aim is to recover a fit as good as the one obtained with the nominal parameter vector, with which the data were generated.

## Problem B3: enzymatic and transcriptional regulation of the Central Carbon Metabolism of *E. coli*

This model simulates the adaptation of *E. coli* to changing carbon sources. Complete information about this model is available as the supplementary information of . It is also included in the BioModels Database . It should be noted that there are some differences in parameter values between the original model and the BioModels version; however, these changes do not alter the simulation results, a fact that indicates unidentifiability. The model contains 47 ODEs and 193 parameters, of which 178 are considered unknown and need to be estimated. The other 15 parameters are constants known to the modeler (number of subunits of the multimers–enzymes–, scaling factors, universal protein degradation rate, and gene expression rate constant). The outputs of the system are the 47 state variables, which represent concentrations. Pseudo-experimental data were generated by simulation of the sixth scenario defined in the simulation files included as supplementary material in . This scenario simulates an extended diauxic shift which consists of three consecutive environments, where the carbon sources are first glucose, then acetate, and finally a mixture of both. Under these conditions, the 47 concentration profiles are sampled every 1000 seconds, for a total of 162 time points (45 hours). This model exhibits large differences in value among concentrations, which span five orders of magnitude. To equalize their contribution to the objective function, we scale each time-series dividing it by the maximum of the experimental value (scaled least squares).

## Problem B4: metabolic model of Chinese Hamster Ovary (CHO) cells

Chinese Hamster Ovary cells (CHO) are used for protein production in fermentation processes . This model simulates a batch process with resting cells: no metabolites are fed for a final time horizon of 300 hours. The fermenter medium contains glucose as main carbon source, and leucine and methionine are the main amino acids taken up. Lactate was modelled to be a by-product of the fermentation process. A generated protein serves as main product of the fermentation process. The model comprises 35 metabolites in three compartments (fermenter, cytosol, and mitochondria) and 32 reactions, including protein product formation, Embden-Meyerhof-Parnas pathway (EMP), TCA cycle, a reduced amino acid metabolism, lactate production, and the electron transport chain. The kinetics are modelled as in , and the resulting ODE model comprises 117 parameters in total. Some aspects of this model were partially discussed in . For optimization purposes pseudo-experimental data were generated, mimicking a typical cell behavior. The following 13 metabolites are assumed to be measured: in fermenter, glucose, lactate, product protein, leucine, and methionine; in cytosol, aspartate, malate, pyruvate, oxaloacetate, ATP, and ADP; and in mitochondria, ATP and ADP. Samples were assumed to be daily taken over the whole fermentation time.

## Problem B5: signal transduction logic model

To illustrate the advantages and disadvantages of different formalisms related to logic models, MacNamara and colleagues constructed a plausible network of interactions consisting of signaling factors known to be activated downstream of $EGF$ and $TNF\text{-}\alpha$ . The model consists of 26 ODEs that use a logic-based formalism, which is explained in detail in . In this formalism, state values can vary between 0 and 1 and represent the normalized activity of a given protein, which is typically measured as the level of phosphorylation. In total the model includes 86 continuous parameters, corresponding to the half maximal activations ($k)$, the Hill coefficients ($n$) and a set of parameters controlling the rate of activation/deactivation of a given protein ($\tau$). The model incorporates $EGF$ and $TNF\text{-}\alpha$ which are treated as stimuli that trigger the pathway response. In addition to these two stimuli, the model includes two kinase inhibitors for $RAF1$ and $PI3K$, which can block the activity of both species. In total the model can be perturbed by these 4 cues, allowing a rich variation in the dynamic profiles of the model signaling components, an essential requirement for parameter estimation. In order to generate a data-set for reverse engineering the model structure, the authors generated data, corresponding to 10 in-silico experiments, where the different cues (stimuli and inhibitors) are added in different combinations. For each experiment 6 strategically located states are observed. Each observable was measured at 16 equidistant time points per experiment. In addition to this Gaussian noise was added to the data in order to mimic a reasonable amount of experimental error. Note that the SBML implementation of this model uses the SBML qual format , an extension of SBML developed for qualitative models of biological networks.

## Problem B6: the gap gene network of the vinegar fly, *Drosophila melanogaster*

Our last benchmark model is slightly different from those previously described, in that it represents a spatial model of pattern formation in multi-cellular animal development, and the data for fitting are based on microscopy, rather than metabolomics or transcriptomics. The gap genes form part of the segmentation gene network, which patterns the anterior–posterior (AP) axis of the *Drosophila melanogaster* embryo. They are the primary regulatory targets of maternal morphogen gradients, and are active during the blastoderm stage in early development. In the model, the embryo is a single row of dividing nuclei along the AP axis, with each nucleus containing the four gap genes and receiving input from four external factors. The gap genes included in the model are *hunchback* (*hb*), *Krüppel* (*Kr*), *giant* (*gt*), and *knirps* (*kni*), and the external inputs Bicoid (Bcd), Caudal (Cad), Tailless (Tll), and Huckebein (Hkb). Three processes occur within and between nuclei: (1) regulated gene product synthesis, (2) Fickian gene product diffusion, and (3) linear gene product decay. These processes are formalised with ODEs, and result in the model having 37 unknown parameters. This model implements the gene circuit approach used to reverse-engineer the regulatory interactions of the gap genes by fitting to quantitative spatio-temporal gene expression data, which can be mRNA or protein . The data consist of 9 time points spanning 71 minutes of *Drosophila* development, and at each time point maximally 53 nuclei with data points for the four gap genes, and the four external inputs. The fit is measured with a weighted least squares scheme (WLS) with variable weights, which, in the case of the mRNA data used here , are inversely related to the level of expression. The weights were created from normalized, integrated mRNA expression data according to the formula: $w = 1.0 - 0.9y$, with $y\in[0,1]$ being the normalized staining intensity. This proportionality of variation with expression level reflects the fact that gap domains (showing high levels of expression) show more variation than those regions of the embryo in which a gene is not expressed .

# Results

We show how our collection of benchmark problems can be used by reporting selected results using several parameter estimation methods. We emphasize that the purpose of this work is not to provide a comprehensive comparison of all existing approaches, but to provide a useful, versatile, and practical test set and illustrate its use. For simplicity, and to enable direct comparisons among benchmarks, all the computations reported in this section have been carried out in Matlab, using the algorithms available in the AMIGO toolbox . This includes both global and local optimization methods; the latter have been used in a multistart procedure, where multiple instances are launched from different initial points selected randomly within the parameter bounds.

Before estimating the parameter values we assessed the identifiability of the models. Model parameters were ranked according to their influence on the system output (sensitivity), using the local rank routine (AMIGO_LRank) from the AMIGO toolbox as described in the previous section. As is typical of models of this size, it was found that all benchmarks have identifiability issues, with a portion of their parameters exerting very little influence on the model outputs. Therefore, the goal of these benchmarks is not to obtain accurate estimates of all the parameters, but rather to obtain a good fit to the data: when tested on this collection of benchmarks, optimization methods should be evaluated by their ability to minimize the objective function. As an illustration of the typical outcome that can be obtained from the local rank method, we show in Figure the results of the practical identifiability analysis for problem B2. Figure ranks the parameters in decreasing order of their influence on the system's behaviour, which is quantified by means of the importance factors $\delta_p^{msqr}$: $$\delta_p^{msqr} = \frac{1}{n_{lhs}n_d}\sqrt{\sum_{mc=1}^{n_{lhs}}\sum_{d=1}^{n_{d}}([s_d]_{mc})^2}$$ where $n_{lhs}$ are the different values for each of the parameters selected by Latin Hypercube Sampling, $n_d$ is the number of experiments, and $[s]$ are the relative sensitivities. Figure shows the sensitivity of the state variables with respect to the parameters. From this figure it becomes clear that many parameters such as 8-10, 32-38, 56-64, are not influencing observables. Therefore those parameters are expected to be poorly identifiable.

In the remainder of this section we show selected results of the best performing optimization methods in every parameter estimation problem. Complete results for every benchmark are reported in the Supplementary Information online.

To evaluate the performance of local methods we launched repeated local searches in a multistart procedure, starting from initial parameter vectors with values chosen randomly from within the parameter bounds. It should be noted that, while multistarts of local searches are a popular option for parameter estimation, they are usually not the most efficient solution when dealing with large-scale nonlinear models. Due to the multimodal nature of these problems, local methods tend to be stuck in local minima, which can sometimes be very far from the global optimum. Launching local methods from random points leads to spending a large fraction of the computational time in unsuccessful searches. Hence, global optimization methods usually perform better in these cases, especially if–as happens with eSS–they are used in combination with local searches. As an example, Figure shows histograms of the results (i.e., objective function values reached and the frequency with which they were found) obtained with the DHC local method for benchmark B3. Similar outcomes were obtained with the other benchmarks and methods. Complete results for all the benchmarks and with different methods are included in the Supplementary Information online. In all cases, the number of local searches was fixed so that their overall CPU time was comparable to that consumed in optimizations where the global method eSS was used. While there was great variability in the results obtained for the different benchmarks, a conclusion was common to all of them: in all cases, the local methods were outperformed by the global optimization method eSS.

The convergence curves of the six benchmarks are shown in Figure . Results were obtained with the eSS method on a computer with Intel Xeon Quadcore processor, 2.50 GHz. It can be clearly noticed that, due to the differences in size and complexity, the computational cost of estimating the parameters varies among benchmarks. Results show that they can be naturally classified in three different levels:

-   B1 and B3 are the most expensive: in our computers, obtaining a reasonably good fit took at least one week.

-   B5 and B6 are intermediate in terms of cost; a good fit could be obtained in one day.

-   B2 and B4 are the least expensive, with good fits obtained in one or a few hours.

These computation times can be used as a reference to select the appropriate benchmarks to test a particular optimization method, depending on its focus and the available time. Due to the stochastic nature of the eSS algorithm, results may vary among optimization runs. Figure shows the dispersion of 20 different optimization results for benchmark B4.

Table summarizes the settings and outcomes of the parameter estimations with eSS, including the local method used for each problem. Note that, while DN2FB is generally recommended , we have realized that for large-scale problems it may not be the most efficient local method, due to the large number of evaluations needed to calculate the derivatives. Hence, for the problems considered here it is outperformed by other methods like DHC, SOLNP, or FMINCON.

One of the outcomes reported in Table is the cumulative normalized root-mean-square error, $\sum{}$NRMSE. The root-mean-square error is a standard measure of the goodness of fit obtained for an observable which is defined as $$\mathrm{RMSE} = \sqrt{ \frac{  \sum^{n_{\epsilon}}_{\epsilon=1} \sum^{n^{\epsilon,o}_s}_{s=1} \left(ym^{\epsilon,o}_s - y^{\epsilon,o}_s(p) \right)^2 } {n_{\epsilon} \cdot n^{\epsilon,o}_s } }$$ with the same notation as in equation (). To account for the different magnitudes of the observables it is useful to report the normalized root-mean-square error, NRMSE, which scales the NRMSE by dividing it by the range of values of the observable: $$\label{eq:NRMSE}
\mathrm{NRMSE} = \frac{\mathrm{RMSE}}{max( ym^{\epsilon,o}  )-min( ym^{\epsilon,o} )}$$ The cumulative normalized root-mean-square error, $\sum{}$NRMSE, is simply the sum of the NRMSE for all observables.

```latex
\begin{table*}[t]\begin{flushleft}
  \caption{
  \textbf{Parameter estimation with eSS (AMIGO implementation): settings and results}
  }
  \begin{tabular}{|l|l|l|l|l|l|l|}
  \hline
Model ID            & \textbf{B1}          & \textbf{B2}             & \textbf{B3}              & \textbf{B4}             & \textbf{B5}           & \textbf{B6}  \\
\hline
$p^U$               & $5\cdot p_{nom}$     & $10\cdot p_{nom}^{(ex)}$  & $10\cdot p_{nom}^{(ex)}$  & $5\cdot p_{nom}$     & varying               & varying      \\
$p^L$               & $0.2\cdot p_{nom}$   & $0.1\cdot p_{nom}^{(ex)}$ & $0.1\cdot p_{nom}^{(ex)}$ & $0.2\cdot p_{nom}$   & varying               & varying      \\
Local method        & DHC                  & FMINCON                 & none                     & FMINCON                 & DHC                   & FMINCON      \\
CPU time            & $\approx$170 hours   & $\approx$3 hours        & $\approx$336 hours       & $\approx$1 hour         & $\approx$16 hours     & $\approx$24 hours  \\
Evaluations         & $6.9678\cdot10^5$    & $9.0728\cdot10^4$       & $7.2193\cdot10^6$        & $1.6193\cdot10^5$       & $8.8393\cdot10^4$     & $2.0751\cdot10^6$ \\
$J_0$               & $5.8819\cdot10^9$    & $3.1136\cdot10^4$       & $4.6930\cdot10^{16}$     & $6.6034\cdot10^8$       & $3.1485\cdot10^4$     & $8.5769\cdot10^5$ \\
$J_f$                 & $1.3753\cdot10^4$    & $2.3390\cdot10^2$       & $3.7029\cdot10^{-1}$     & $4.5718\cdot10^1$       & $3.0725\cdot10^3$     & $1.0833\cdot10^5$ \\
$J_{nom}$           & $1.0846\cdot10^6$    & $-$                     & $0$                      & $3.9068\cdot10^1$       & $4.2737\cdot10^3$     & $ -$ \\
$\sum$NRMSE$_0$     & $3.5834\cdot10$      & $8.5995\cdot10^{-2}$    & $3.5457\cdot10$          & $4.8005\cdot10$         & $4.0434\cdot10^1$     & $2.3808\cdot10^2$ \\
$\sum$NRMSE$_f$     & $5.7558$             & $2.4921$                & $2.9298\cdot10^{-1}$     & $2.8010$                & $2.7430\cdot10^1$     & $1.6212\cdot10^2$ \\
$\sum$NRMSE$_{nom}$ & $3.8203$             & $-$                     & $0$                      & $2.8273$                & $3.0114\cdot10^1$     & $ -$ \\
\hline
\end{tabular}
\\
Optimization settings and results obtained for each of the benchmarks with the eSS method, using the implementation provided in the AMIGO toolbox. In some cases the lower ($p^L$) and upper ($p^U$) bounds in the parameters are specified as a function of the nominal parameter vector, $p_{nom}$. There may be exceptions to these bounds, in cases where it makes sense biologically to have a different range of values (e.g. Hill coefficients in the range of 1--12). Cases with exceptions are marked by $^{(ex)}$. In other cases all the parameters have specific bounds; this is marked as ``varying''. The initial objective function value, $J_0$, corresponds to the parameter vector $p_0$ used as initial guess in the optimizations, which is randomly selected between the bounds $p^L$ and $p^U$. The only exception is benchmark B2, where $p_0$ is the parameter vector reported in the original publication.
The final value achieved in the optimizations is $J_f$. $\sum$NRMSE is the cumulative normalized root-mean-square error as defined in eq. (\ref{eq:NRMSE}).
Results obtained on a computer with Intel Xeon Quadcore processor, 2.50 GHz, using Matlab 22.214.171.1249 (R2009b) 32-bit.
\end{flushleft}
\label{tab:pe}
\end{table*}
```

Note that, due to the realistic nature of most of these problems, there may be lack of identifiability and optimization may result in overfitting: that is, an optimal solution may be found that gives a better fit to the pseudoexperimental data than the one obtained with the nominal parameter vector used to generate the data. This is explained because, in the presence of measurement noise, the optimal solution manages to fit partially not only the system dynamics, but also the noise itself–which of course cannot be achieved by the nominal solution. Hence in the results reported in Table the optimal objective function value ($J_f$) is sometimes smaller (i.e. better) than the nominal one ($J_{nom}$). This may also happen with the NRMSE values. Note however that, since the objective functions used in the calibration ($J$) and the NRMSE are different metrics, their behavior may be different. For example, for B1 $J_f<J_{nom}$ and NRMSE$_f>$NRMSE$_{nom}$, while for B4 the opposite is true: $J_f>J_{nom}$ and NRMSE$_f<$NRMSE$_{nom}$.

As an example of the fit between data and model output that is obtained after calibration, let us consider benchmark B5, which uses pseudoexperimental data corresponding to ten different experiments. Figure reports a good match between data and model output; notably, the algorithm manages to reproduce the oscillations in NF$\kappa$B.

The fit can also be represented with histograms of the residuals, which show the distribution of the errors in the state variables. This kind of plot can also be used for showing the errors in the recovered parameters when compared to the nominal (true) values. An alternative way of visualizing this relation is by plotting the predicted states (or parameter) values as a function of the true values. This results in a diagonal–like plot; the larger the deviations from the diagonal, the larger the prediction errors. When there are identifiability issues, the fit is typically better for the states than for the parameters, because a good fit to the data does not necessarily ensure that the correct parameters have been recovered. Examples of these plots are shown in Figure , which shows the fits obtained for benchmark B4.

# Conclusions

To address the current lack of ready-to-run benchmarks for large-scale dynamic models in systems biology, we have presented here a collection of six parameter estimation problems. They cover the most common types, including metabolism, transcription, signal transduction, and development. The benchmarks are made available in a number of formats. As a common denominator, all of the models have been implemented in Matlab and C. When possible (i.e. for benchmarks B1–B5), model descriptions are also given in SBML. Ready-to-run implementations of all the benchmarks are provided in Matlab format (both with and without the AMIGO toolbox) and in COPASI (for benchmarks B1–B4). With these files it is straightforward to reproduce the results reported here.

More importantly, the benchmark files can be easily adapted to test new parameter estimation methods for which a Matlab, C, or COPASI implementation is available. The performance of an existing or newly developed method can be evaluated by comparing its results with those reported here, as well as with those obtained by other methods. To this end, we have provided guidelines for comparing the performance of different optimizers. The problems defined here may also be used for educational purposes, running them as examples in classes or using them as assignments.

Finally, it should be noted that the utility of this collection goes beyond parameter estimation: the models provided here can also be used for benchmarking methods for optimal experimental design, identifiability analysis, sensitivity analysis, model reduction, and in the case of metabolic models also for metabolic engineering purposes.

# Competing interests

The authors declare that they have no competing interests.

# Author's contributions

JRB, PM, and JJ conceived of the study. JRB and AFV coordinated the study. KS, SB, JS, DCS, AC, JSR, KM, PM, and JJ contributed with models. AFV, DH, KS, and DCS formatted the models and tested the implementations. AFV and DH carried out the computations. AFV, EBC, and JRB analyzed the results. AFV and JRB drafted the manuscript. KS, SB, AC, JSR, KM, EBC, and JJ helped to draft the manuscript. All authors read and approved the final manuscript.

# Supplementary Information

<http://www.iim.csic.es/~gingproc/biopredynbench/>

# Acknowledgments

This work was supported by the EU project "BioPreDyn" (EC FP7-KBBE-2011-5, grant number 289434). We would like to thank Attila Gábor for reading the manuscript, providing critical comments, and finding bugs in the codes; David Rodríguez Penas for helping in debugging the codes; and Thomas Cokelaer for providing the SBML qual file of model B5.


abstract: Quantum entanglement is one of the core features of quantum theory. While it is typically revealed by measurements along carefully chosen directions, here we review different methods based on so-called *random* or *randomized measurements*. Although this approach might seem inefficient at first, sampling correlations in various random directions is a powerful tool to study properties which are invariant under local-unitary transformations. Based on random measurements, entanglement can be detected and characterized without a shared reference frame between the observers or even if local reference frames cannot be defined.
  .
  This overview article discusses different methods using random measurements to detect genuine multipartite entanglement and to distinguish SLOCC classes. Furthermore, it reviews how measurement directions can efficiently be obtained based on spherical designs.
author: Lukas Knips
title: A Moment for Random Measurements

# Introduction

*Entangled quantum systems are more strongly correlated than classical systems can be.* This simplified statement proves to be remarkably rich by raising several questions, from practical ones - "How do we measure those correlations?" - to more subtle ones - "What does *stronger* mean here and *how* much stronger are the correlations? Are there exceptions to this rule? Can we use this relation to characterize entanglement?"

While entanglement of bipartite or multipartite systems is typically detected using a set of carefully chosen measurements on all subsystems, here, we discuss a simpler and, as it turns out, yet powerful method to analyze correlations and entanglement. There, one samples from the entire set of correlations by measuring in random directions, instead of considering a specific set of correlations. This type of measurement is often called a *random* or *randomized measurement*. Depending on the context, these can be either controlled measurements without a shared reference frame, measurements without active control over (but knowledge of) the measurement direction, or even measurements with neither control nor knowledge about the measurement direction.

Previous works showed the violation of Bell-type inequalities without the need for a shared reference frame , but with the ability to repeat previously conducted measurements. Random measurements in the sense of measurements without control have been used in the context of many-body systems , for the verification of quantum devices , for the detection of entanglement , for the prediction of fidelities, entanglement entropies and various other properties  as well as for the characterization and classification of genuine multipartite entanglement . Recently, it was shown that even bound entangled states, i.e., states so weakly entangled that their entanglement is not recognized by the PPT (positive partial transpose) criterion , can be characterized in a reference-frame independent manner .

In this perspective article, we will discuss a work of Ketterer et al., which recently appeared in *Quantum* . Before we will review their means of entanglement detection and classification, we will introduce the general concept of random measurements, provide an intuitive understanding for them and give context by discussing other methods for detection and classification of entanglement in this scenario. Finally, we will discuss their approach for selecting local measurement directions based on spherical $t$-designs.

# Scenario

To illustrate the scenario, we can first think of a two-qubit state whose subsystems are sent to two observers via unitary but unknown quantum channels as shown in Fig. . Hence, the goal is to characterize the quantum state as well as possible despite the lack of a shared reference frame. Interestingly, although the correlation value of the outcomes of both observers (which quantifies the probability for both results being equal or opposite) is random as it depends on the respective measurement directions, the *distribution* of correlation values turns out to be a useful resource for describing the state.

If the unknown unitary transformations of the quantum channels are furthermore time-dependent, i.e., the channels are *fluctuating* or *noisy*, one could naively perform measurements in any fixed direction. However, the type of fluctuations strongly influences the measurement results: are the fluctuations distributed uniformly (according to the Haar measure) or are we oversampling some and undersampling other directions? To mitigate this problem and avoid any dependence on the type of noise of the quantum channels, the measurement directions can themselves be chosen Haar-randomly. In this way, the concatenation of the quantum channel of the noisy environment featuring possibly biased noise with the channel corresponding to a Haar random distribution of unitary transformations due to intentional rotation removes any bias.

# Intuitive Picture

To establish an intuitive understanding how entangled and separable quantum states behave in this scenario and how we can make use of the stronger correlations of an entangled state, let us do a short numerical experiment. Take the pure product state $|\psi^{\mathrm{prod}}\rangle=|00\rangle$ and a Bell state, e.g., $|\psi^-\rangle=(|01\rangle-|10\rangle)/\sqrt{2}$. Now repeatedly draw two $2\times2$ unitary matrices according to the Haar measure (see, e.g., for a practical recipe). Apply the first (second) of those unitaries to the first (second) qubit of both states and evaluate, e.g., $\langle\sigma_z\otimes\sigma_z\rangle$, i.e., the correlation value in the $zz$-direction. A histogram of those two distributions shows the very distinct behavior of the product and the maximally entangled state, as shown in figure . The maximally entangled state results in a uniform distribution of the expectation values of the correlations, whereas small absolute values are much more probable than large ones for the product state.

For those two states, it is still easy to understand the origin of the corresponding distributions. Consider the schematic arrow diagrams in Fig. , where in a) the two red arrows represent the spins of a product state after applying arbitrary local unitary (LU) transformations each parametrized here by a single angle. A measurement of $\sigma_z\otimes\sigma_z$ is just given by the product of the results, which are obtained by the projection onto the $\sigma_z$ directions. Both angles, $\alpha$ and $\beta$, have to be close to $0$ or to $\pi$ to give a large absolute correlation value. In the example, we show the case of $\alpha=75^\circ$ and $\beta=60^\circ$ leading to a $zz$-correlation of about $0.13$.

For the maximally entangled state in Fig.  b), the situation is very different. We illustrate the state as a superposition of both red arrows with both yellow arrows. By applying an LU transformation of the form $U\otimes U$ on both qubits such that, say, the first qubit is aligned with its measurement direction (i.e., such that $U$ is a rotation by an angle $-\alpha$), the state does not change (up to a global phase factor). The expectation value of $\sigma_z\otimes\sigma_z$ therefore now obviously only depends on the single relative angle $\beta-\alpha$ instead of the two angles $\alpha$ and $\beta$. For the same angles as in the product-state case, we now obtain a $zz$-correlation of about $-0.71$.

Formally, we can obtain the distributions of correlations $E$ by integration over the Bloch spheres $S^{2}\times S^{2}$ as $$\begin{aligned}
&p_{\mathrm{prod}}(E) = \frac{1}{(4\pi)^2} \int_{S^2} {\mathrm{d}U_1}\int_{S^2} {\mathrm{d}U_2} \delta(E-E_{\mathrm{prod}}(U_1,U_2)) \nonumber \\
&= \frac{1}{4} \int_0^{\pi} \sin(\theta_1) {\mathrm{d}\theta_1}\int_0^{\pi} \sin(\theta_2){\mathrm{d}\theta_2} \delta(E - \cos(\theta_1) \cos(\theta_2)  ) \nonumber \\
&= -\frac{1}{2} \log(|E|),\\
&p_{\mathrm{Bell}}(E) = \frac{1}{(4\pi)^2} \int_{S^2} {\mathrm{d}U_1}\int_{S^2} {\mathrm{d}U_2} \delta(E-E_{\mathrm{Bell}}(U_1,U_2)) \nonumber \\
&= \frac{1}{4} \int_0^{\pi} \sin(\theta_1) {\mathrm{d}\theta_1}\int_0^{\pi} \sin(\theta_2){\mathrm{d}\theta_2} \delta(E - \cos(\theta_1-\theta_2)  ) \nonumber \\
&= \frac{1}{2}.
\end{aligned}$$ In the histograms of Fig. , we have also shown the distributions of two other states. The Werner state, as the mixture of the Bell state (corresponding to a uniform distribution) and the maximally mixed state (corresponding to a Dirac delta peak at $0$ as the maximally mixed state $\mathbbm{1}/4$ always results in $\langle \sigma_z\otimes\sigma_z\rangle=0$, independently of the measurement directions) with mixing parameter $p$, results in a uniform distribution in the range $[-p,p]$ (shown above for $p=1/\sqrt{3}$), i.e., mixing white noise to a Bell state bounds the possible correlations in this sense. The two-qubit marginal of a tripartite $W$ state, however, gives yet another distinct distribution.

Due to the nature of the measurements, two quantum states which are equivalent up to local unitary transformations will show the same distribution of correlations. Therefore, all pure two-qubit product states result in a logarithmic distribution, while, for example, all maximally entangled two-qubit states result in a uniform distribution.

# Statistical Moments

To characterize such distributions of correlations, statistical moments have proven to be powerful. The $t$-th moment of a probability distribution $p(x)$ is given by $$m^{(t)} = \int x^t p(x) {\mathrm{d}x}.$$ The first moment ($t=1$) is the mean value. The next lower *centralized* moments (i.e., the moments after shifting the distribution around its mean) are the variance, the skewness and the kurtosis. As we are dealing with a symmetric distribution here, the mean value vanishes and, hence, the centralized moments are identical to the moments themselves. In our scenario, where all moments are finite and the moment-generating function has a positive radius of convergence, knowing all moments allows one to uniquely determine the distribution . Random measurements can be used to obtain the statistical moments of the distribution of expectation values. The scheme naturally generalizes to the case of more than two parties  and is not limited to qubits .

By considering the measurement results on a subset of parties, also distributions of correlations of marginal states can be retrieved. As the second moment is the first non-trivial one here, we will from now on denote it just as $m:= m^{(2)}$ and specify the respective set of parties it pertains to using a subscript. The combined information of the second moments of the *full* distribution ($m_{1,2,\dots,n}$) involving all $n$ observers together with those of all marginal states ($m_{1}$, $m_{1,2}$, $m_{1,2,3}$, $\dots$, $m_{(n-1),n}$, $\dots$, $m_{n}$) allows us to determine the purity of an $n$-qubit state  as $$\begin{aligned}
\operatorname{tr}{\varrho^2}=\frac{1}{2^n} \sum_{\mathcal{A} \in \mathbbm{P}(\mathcal{S})}{ 3^{|\mathcal{A}|} \, m_\mathcal{A}},
\end{aligned}$$ where $\mathbbm{P}(\mathcal{S})$ is the set of all subsets of $\mathcal{S}=\{1,\ldots,n\}$ and $|\mathcal{A}|$ denotes the cardinality (number of elements) of the set $\mathcal{A}$. Here, $m_\mathcal{A}$ is the second moment of the distribution involving the observers given by $\mathcal{A}$. In other words, the purity is given by the weighted sum of second moments of the full distribution and all marginal distributions. In the remainder of this perspective, we will discuss how to detect and to characterize entanglement based on statistical moments.

# Detecting Entanglement

A remarkably simple method to detect entanglement was presented in . For any pure, fully separable $n$-qubit state $|\psi\rangle=|\psi_1\rangle\otimes|\psi_2\rangle\otimes\dots|\psi_n\rangle$, the length of correlations (sum of the correlations of all basis directions squared) is $1$. If *each* observer $j$ aligns their measurement apparatus along $|\psi_j\rangle$, they jointly observe a correlation of $1$, whereas any orthogonal measurement direction will result in the loss of correlated outcomes. This relation holds independently of the number of parties and can be adapted for arbitrary dimensions. Choosing $M$ measurement directions completely randomly with $K$ repetitions each allows to estimate the length of correlations. If one significantly overcomes the threshold of a product state, one can, after proper statistical analysis, conclude that the state must carry *some* entanglement.

Subsequent work  studied the length of correlations of various genuinely multipartite entangled states and extended the results of  to mixed states. There, entanglement detection based on a single measurement setting is derived explicitly. However, it was found that the length of correlations (alone) is not an entanglement measure as it can increase under local operations and classical communication (LOCC). Nevertheless, random measurements can be used to witness genuine multipartite entanglement, i.e., entanglement truly involving all parties, as we will see below. Moreover, classes of states inequivalent under stochastic local operations and classical communication (SLOCC)  can be distinguished in this way, as we will discuss.

Yet another approach for entanglement detection is used in . There, a measurement direction is randomly drawn from a specific set. In addition, a framework for the probabilistic use of entanglement witnesses is provided. With this, entanglement can in some cases be detected with a single copy with high confidence  or different classes of entanglement can be discriminated using a few copies of the state in a more general approach .

Also note that it is common that entanglement criteria are sufficient, but not necessary. For example, any method, whether based on random measurements or on fully controlled ones, that is only using *full* correlations, i.e., correlations between all $n$ parties without inspecting correlations between subsets of parties, will miss some entangled states: There are genuinely $n$-partite entangled states without any correlations between all $n$ parties , which therefore will result in a Dirac-delta peak for the full distribution and are on that level indistinguishable from white noise.

# Detecting Genuine Multipartite Entanglement

## Using Second Moments of Marginals

In , the second moments of the distributions are used for the detection of genuine multipartite entanglement (entanglement which truly involves all parties and cannot be broken down into pure biseparable states or even mixtures of states biseparable with respect to different bipartitions). The key ingredient of that strategy is to relate the second moment of the distribution of correlations to the second moments of marginal distributions and test that quantifier with a purity-dependent threshold. For a pure product state $|\psi_{\mathcal S}\rangle=|\psi_{{\mathcal A}_1}\rangle\otimes|\psi_{{\mathcal A}_2}\rangle$, the second moment of the distribution of correlations when considering the set of parties $\mathcal S$ factorizes into the second moments of the corresponding marginal distributions, i.e., $m_{{\mathcal S}}=m_{{\mathcal A}_1}m_{{\mathcal A}_2}$. For a pure state, genuine multipartite entanglement can therefore be detected by verifying $m_{{\mathcal S}}>m_{{\mathcal A}_1}m_{{\mathcal A}_2}$ for every possible bipartition ${\mathcal A}_1|{\mathcal A}_2$ such that ${\mathcal S}={\mathcal A}_1\cup{\mathcal A}_2$.

Unfortunately, for non-pure biseparable states, the second moment of the full distribution might be larger than the product of the marginals' second moments. To mitigate this problem, purity-dependent bounds for detecting genuine multipartite entanglement can be used. For an $n$-qubit state we can consider $${\mathcal M}_{n} := m_{{\mathcal S}} - \frac{1}{2} \sum_{{\mathcal A} \in \{\mathbbm{P}({\mathcal S}) \setminus ({\mathcal S} \cup \varnothing)\}}{ m_{\mathcal A} m_{{\mathcal S} \setminus {\mathcal A}}}$$ to capture by how much the full distribution's second moment can be expressed in terms of the marginals' ones. For example, it has been found  that in the case of $n=4$ qubits, all biseparable states fulfill the relation $$\begin{aligned}
{\mathcal M}_{4}
\le \tfrac{8}{81}(1-\operatorname{tr}\varrho^2).
\label{eq:M4}
\end{aligned}$$ A violation of the latter inequality therefore indicates genuine fourpartite entanglement.

In Fig. , the values of ${\mathcal M}_{4}$ based on the distribution of fourpartite correlations for experimental photonic states (four qubits encoded in polarization and path degrees of freedom of two photons from type-I spontaneous parametric down-conversion, see  for details on the setup) are compared with the purity-dependent threshold given by Eq. () . As the red (four-qubit GHZ state) and blue (linear four-qubit Cluster state) data points are above the threshold, those states are shown to be genuinely fourpartite entangled. The distributions of a prepared triseparable state \[$\propto \left(|00\rangle+|11\rangle\right)\otimes|0\rangle\otimes|0\rangle$\] and of a biseparable state \[$\propto \left(|00\rangle+|11\rangle\right)\otimes\left(\sin\varphi\,|00\rangle+\cos\varphi\,|11\rangle\right)$ with $\varphi\approx0.2$\], however, contain only one and two bipartite entangled marginals, respectively, as one finds by considering ${\mathcal M}_{3}$ for all tripartite and ${\mathcal M}_{2}$ for all bipartite marginals of those states. Thus, this approach not only allows to detect genuine multipartite entanglement for mixed states, but generally provides insights into the entanglement structure.

## Using Higher Order Moments

The combination of moments of various orders may in general capture more information about a distribution of correlations and, hence, about the underlying quantum state than restricting the analysis to moments of second order, only. In two recent works , the latest of which was published in *Quantum* , Ketterer et al. use a combination of the second and the fourth moment, denoted there by $\mathcal{R}^{(2)}$ and $\mathcal{R}^{(4)}$ (with $\mathcal{R}^{(t)}\equiv m^{(t)}$), respectively. Obviously, those moments are not entirely independent of each other. For example, a vanishing second moment $\mathcal{R}^{(2)}$ indicates a Dirac delta distribution, which in turn requires $\mathcal{R}^{(4)}$ to vanish. They discuss possible combinations of those two moments for two, three and four qubits. In a combined analytical and numerical study, the authors identify regions in an $\mathcal{R}^{(2)}$-$\mathcal{R}^{(4)}$ plane which allow them to directly indicate that a state is, e.g., biseparable or that it cannot belong to a specific SLOCC class.

In Fig. , three-qubit states are sampled and represented in the $\mathcal{R}^{(2)}$-$\mathcal{R}^{(4)}$ plane. The blue-shaded area is outlined by different LU-inequivalent types of biseparable states. Ketterer et al. propose the inequality $$\mathcal{R}^{(4)} \geq \frac{1}{425}\left[972 \left(\mathcal{R}^{(2)}\right)^2+90\mathcal{R}^{(2)}-5\right],\label{eq:threequbit_bisep}
%\mathcal{R}^{(4)}_{\varrho_{\mathrm{bisep}}} \geq \frac{1}{425}\left[972 \left(\mathcal{R}^{(2)}_{\varrho_{\mathrm{bisep}}}\right)^2+90\mathcal{R}^{(2)}_{\varrho_{\mathrm{bisep}}}-5\right],\label{eq:threequbit_bisep}$$ which is shown as the green dashed line, as demarcation between biseparable and genuine multipartite entangled states. Therefore, states for which the fourth moment $\mathcal{R}^{(4)}$ is below this threshold are not biseparable and are hence shown to be genuinely multipartite entangled.

# Witnessing SLOCC classes

Furthermore, in the authors discuss moments of distributions of correlations for witnessing SLOCC classes, which allows to decide if a state is reversibly convertible into, say, a $W$ state. Figure shows sampled four-qubit states in the $\mathcal{R}^{(2)}$-$\mathcal{R}^{(4)}$ plane. The region with the solid border contains states of the $\mathcal{W}^{(4)}$ class, i.e., the SLOCC class of $W$ states, whereas the region surrounded by the dashed line encompasses its convex hull $\operatorname{Conv}(\mathcal{W}^{(4)})$. States whose moments lie outside of the regions enclosed by the solid and dashed lines are shown not to belong to the SLOCC classes $\mathcal{W}^{(4)}$ and $\operatorname{Conv}(\mathcal{W}^{(4)})$, respectively.

We already see from Fig.  that $\mathcal{R}^{(2)}$ might be very helpful for witnessing that a state is not a member of the mixed $W$ class $\operatorname{Conv}(\mathcal{W}^{(4)})$. In contrast, additional consideration of $\mathcal{R}^{(4)}$ does not improve the ability for detection significantly. More generally, the authors of  derive a witness for $\operatorname{Conv}(\mathcal{W}^{(n)})$ for an arbitrary number of $n$ qubits. If the second moment $\mathcal{R}^{(2)}$ of a distribution of correlations is larger than $$\chi^{(n)}:=\frac{5-\frac{4}{n}}{3^n},$$ the $n$-qubit state is not a member of the mixed $W$ class $\operatorname{Conv}(\mathcal{W}^{(n)})$ . For $4$ qubits, the threshold is $\chi^{(4)}=4/81\approx0.049$. Hence, all states with $\mathcal{R}^{(2)}>\chi^{(4)}=4/81$, i.e., on the right-hand side of $|\mathrm{W}_4\rangle$ and $|\phi\rangle\otimes|\mathrm{GHZ}_3\rangle$ in Fig. , are shown not to belong to the mixed $W$ class.

Please note that this is not a statement about genuine multipartite entanglement. For example, both the biseparable state $|\mathrm{Bell}\rangle\otimes|\mathrm{Bell}\rangle$ and the genuinely fourpartite entangled state $|\mathrm{GHZ}_4\rangle$ are outside of this region as both are not members of $\operatorname{Conv}(\mathcal{W}^{(4)})$.

# Spherical Designs

Ketterer et al.  not only use higher-order moments to detect and characterize entanglement, but they also follow a different approach for selecting measurement directions. Up to this point of the discussion it has been assumed that the distributions of correlations are obtained by sampling over a large set of random directions, where the distribution of directions should follow a Haar random distribution. Whereas in, e.g., the sampling was done over a large set of Haar randomly distributed measurement directions to describe the distributions of correlations, the approach of Ketterer et al. allows one to fix the number of measurement directions if a specific moment is to be calculated. This significantly reduces the measurement effort at the cost of requiring active control over the local measurement directions.

Using the notation of Ref. , the $t$-th moment $\mathcal{R}^{(t)}$ of the distribution of correlations of an $n$-qubit state can be obtained from $$\begin{aligned}
\mathcal{R}^{(t)} &= \!\!\int\limits_{\mathcal{U}(2)} \!\!\!\mathrm{d}\eta (U_1) \cdots\!\! \int\limits_{\mathcal{U}(2)}\!\!\! \mathrm{d}\eta(U_n)\langle U_1\sigma_z U_1^\dagger\otimes \dots \otimes U_n\sigma_z U_n^\dagger \rangle^t \nonumber \\
&= \frac{1}{\left(4\pi\right)^n} \int_{S^2} \mathrm{d}\vec{u}_1 \cdots \int_{S^2} \mathrm{d}\vec{u}_n E(\vec{u}_1,\dots,\vec{u}_n)^{t}, \label{eq:tth_moment_integration}
\end{aligned}$$ where $E(\vec{u}_1,\dots,\vec{u}_n):=\langle \sigma_{\vec{u}_1}\otimes\dots\otimes\sigma_{\vec{u}_n} \rangle$ denotes the correlation along specific local measurement directions $\vec{u}_i$ with $\sigma_{\vec{u}_i}=\vec{u}_i\cdot\vec{\sigma}$, where $\vec{\sigma}=\left(\sigma_x,\sigma_y,\sigma_z\right)^{T}$ is a vector of the Pauli matrices $\sigma_x$, $\sigma_y$ and $\sigma_z$, while $\eta$ and $\mathrm{d}\vec{u}_i=\sin\theta_i\mathrm{d}\theta_i\mathrm{d}\varphi_i$ are the Haar measure on the unitary group $\mathcal{U}(2)$ and the uniform measure on the Bloch sphere $S^2$, respectively.

To determine the average of a homogeneous polynomial $P_{t^\prime}:S^2\rightarrow\mathbb{R}$ of order $t^\prime$ over the Bloch sphere $S^2$, it is sufficient to sample a finite set of points as shown in . For that, they use a so-called spherical $t$-design in dimension three which is defined by the finite set of points $\{\vec{u}_i|i=1,\dots,L^{(t)}\}\subset S^2$ such that $$\int_{S^2} \mathrm{d}\vec{u} P_{t^\prime}(\vec{u}) = \frac{1}{L^{(t)}}\sum_{k=1}^{L^{(t)}}P_{t^\prime}(\vec{e}_k)$$ holds for all homogeneous polynomials of order $t^\prime$ with $t^\prime\leq t$. Hence, for the respective spherical $t$-design, $L^{(t)}$ determines the number of measurement directions to consider. Using this framework, Ketterer et al. evaluate the $t$-th moment of the correlations of an $n$-qubit state as $$\mathcal{R}^{(t)} = \frac{1}{\left(L^{(t)}\right)^n}\sum_{k_1,\dots,k_n=1}^{L^{(t)}} \langle \sigma_{\vec{u}_1}\otimes\dots\otimes\sigma_{\vec{u}_n} \rangle^{t},$$ instead of using the integration as in Eq. (). Although they also show a similar derivation for qudit states employing *unitary* $t$-designs, we here restrict our discussion to the qubit case using *spherical* $t$-designs.

In Fig. , the $L^{(3)}=6$ directions $\{\pm \vec{e}_i|i=x,y,z\}$ of a spherical $3$-design as well as the $L^{(5)}=12$ directions of a $5$-design are shown. If one is only interested in the second moment (polynomial of order $2$), the $6$ local measurement directions of $L^{(3)}$ are sufficient. Moreover, as even-order moments are invariant under a parity transformation of the measurement direction, skipping $\{-\vec{e}_i|i=x,y,z\}$ does no harm. For obtaining the fourth moment, $L^{(5)}/2=6$ local measurement directions are suitable. With this method, the selection of measurement directions from the pseudo-random process of a spherical design allows one to mimic uniform averages over the sphere .

# Conclusion and Future Research

In this perspective, we have discussed what we can learn from random measurements despite the lack of shared or even local reference frames. Distributions of correlations can reveal entanglement and exclude any type of separability. Although considering only the second moment of these distributions is not sufficient for constructing an entanglement measure, it can be used for witnessing SLOCC classes as recently shown in . As random measurements are inherently not sensitive to local unitary transformations, they do not divert one's gaze from LU-invariant properties.

Random measurements turn out to be a powerful tool for entanglement detection and classification. Here, we did not elaborate on statistical errors involved in those measurements, which require some further research. Also, we focused on quantum *states*. Of course, random measurements are also of interest for characterizing quantum *processes*. Another open question is the tomographic reconstruction using random measurements: Which states can be discriminated and what information will stay hidden? Also, it is worth to discuss how and to what degree random measurements can be employed for applications such as quantum metrology.

# Acknowledgments

I am grateful to Tomasz Paterek, Jasmin D. A. Meinecke, Karen Wintersperger, and Nicolai Friis for their helpful comments and suggestions for improvements of the manuscript.