# Decision support for the comparative evaluation and selection of analytical methods: detection of genetically modified organisms as an example

David Dobnik, Kristina Gruden, Jana Zel, Yves Bertheau, Arne Holst-Jensen, Marko Bohanec

<figure><img src="image_1.png" /></figure>

## To cite this version:

David Dobnik, Kristina Gruden, Jana Zel, Yves Bertheau, Arne Holst-Jensen, et al.. Decision support for the comparative evaluation and selection of analytical methods: detection of genetically modified organisms as an example. Food Analytical Methods, 2018, pp.1-18. 10.1007/s12161-018-1194-1. hal-02623338

HAL Id: hal-02623338

https://hal.inrae.fr/hal-02623338

Submitted on 26 May 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Decision Support for the Comparative Evaluation and Selection of Analytical Methods: Detection of Genetically Modified Organisms as an Example

David Dobnik<fnref n="1" /> Ⓒ Kristina Gruden<fnref n="1" /> · Jana Žel<fnref n="1" /> · Yves Bertheau<fnref n="2" />,<fnref n="3" /> · Arne Holst-Jensen<fnref n="4" /> · Marko Bohanec<fnref n="5" />

Received: 8 August 2017 / Accepted: 5 February 2018
© The Author(s) 2018. This article is an open access publication

## Abstract

The selection of the best-fit-for-purpose analytical method to be implemented in the laboratory is difficult due to availability of multiple methods, targets, aims of detection, and different kinds and sources of more or less reliable information. Several factors, such as method performance, practicability, cost of setup, and running costs need to be considered together with personnel training when selecting the most appropriate method. The aim of our work was to prepare a flexible multicriteria decision analysis model suitable for evaluation and comparison of analytical methods used for the purpose of detecting and/or quantifying genetically modified organisms, and to use this model to evaluate a variety of changing analytical methods. Our study included selection of PCR-, isothermal-, protein-, microarray-, and next-generation sequencing-based methods in simplex and/or multiplex formats. We show that the overall result of their fitness for purpose is relatively similar; however, individual criteria or a group of related criteria exposed more substantial differences between the methods. The proposed model of this decision support system enables easy modifications and is thus suitable for any other application of complex analytical methods.

**Keywords** Multicriteria decision analysis · Genetically modified organisms · Method evaluation · DEXi · Decision support system · DSS

## Introduction

Analytical laboratories have to select and implement new methods from time to time. Alternatively, based on received real-life samples, they may have to select among a panoply of methods for multiple target analyses. This selection can be rather arbitrary, primarily directed by tradition or the expertise

of trained persons, but it can also be based on systematic objective and reproducible comparative analysis. Options of choice often include the use of different technological platforms and the choice between simplex and multiplex methods. With increasing numbers of optional methods, the complexity of selecting analyses will either require a long reflection period of trained analysts or some sort of decision support system, selection criteria, and rules for comparison of attributes as a way to reduce uncertainties and ensure objective and reproducible choices independently of the presence of trained experts. In many fields where analytical methods are applied, the situation is quite complex. In this manuscript, we take detection (including identification and quantification) of genetically modified (GM) organisms (GMOs) as an example to illustrate how the decision process can be facilitated in a complex situation through the use of decision support system (DSS).

Many countries around the world regulate cultivation and trade with GMOs and have implemented an authorization system and mandatory labeling above a certain threshold (Gruère and Rao 2007; Vigani et al. 2012; Milavec et al. 2014). For example, in the European Union (EU), the labeling threshold

**Electronic supplementary material** The online version of this article (https://doi.org/10.1007/s12161-018-1194-1) contains supplementary material, which is available to authorized users.

<figure><img src="image_4.png" /></figure>

David Dobnik
david.dobnik@nib.si

<footnote n="1">¹ Department of Biotechnology and Systems Biology, National Institute of Biology, Večna pot 111, Ljubljana, Slovenia</footnote>

<footnote n="2">² Institut national de la recherche agronomique, Versailles, France</footnote>

<footnote n="3">³ Present address: Muséum national d'histoire naturelle, Paris, France</footnote>

<footnote n="4">⁴ Norwegian Veterinary Institute, Oslo, Norway</footnote>

<footnote n="5">⁵ Jožef Stefan Institute, Ljubljana, Slovenia</footnote>
for food products containing, consisting of, or produced from authorized GMOs is set at 0.9% by Regulation (EC) 1829/2003 (European Commission 2003). After more than two decades of commercialization of GM plants, the number of new GM events and the diversity of GM traits and constructs, as well as the mixture of GMOs and of their derived products and the purpose of analyses (labeling vs. market withdrawal), is still steadily increasing (James 2015). The complexity of the products that need to be tested for the presence of GMOs is therefore also increasing. This complexity will increase even more, if products derived from new breeding techniques would also have to be labeled. International trade and differences in national regulations combined with the need for harmonized testing and results interpretation add to the complexity of the method selection in the analytical laboratories.

Currently, detection, identification, and quantification of GMOs (Holst-Jensen et al. 2012) are mostly done using quantitative real-time polymerase chain reaction (qPCR). The individual methods either target a DNA motif present inside the inserted GM construct or a junction between the hosts' native DNA and the inserted sequence (see, e.g., Holst-Jensen et al. (2012) for a detailed review). Screening for the presence of genetic elements commonly found in GMOs can be an effective approach to reduce the number of needed analyses (Holst-Jensen et al. 2012). It is, however, prone to misinterpretation and normally will have to be complemented with the use of methods targeting specific GMOs (transformation event-specific methods) (Holst-Jensen et al. 2012). Multiplex screening approaches can improve the cost efficiency (Huber et al. 2013) but also have important limitations. Several more or less specific PCR (Randhawa et al. 2009, 2010; Holck et al. 2010; Luan et al. 2012) and qPCR (Germini et al. 2004; Waiblinger et al. 2007; Bahrdt et al. 2010) multiplex methods have already been developed. Additionally, alternative testing methods and targets are available (reviewed in Milavec et al. (2014)), and new methods and technologies/platforms are constantly developed.

Altogether, the GMO detection and identification processes should be reproducible, understandable, transparent, and user-friendly for both the analyst and the decision maker. Selecting among analytical strategies to meet these objectives is challenged by the complexity of the methods. The integrated use of additional sources of information would be beneficial, as would the possibility to weigh issues like personnel training costs and restricted budgets.

Factors considered for the selection of each method are as follows: properties of the product (product type, ingredients, diversity of targets that may have to be detected, etc.), purpose of the analysis (screening, identification, quantification, as well as unapproved product, detection for market withdrawal, etc.), performance parameters of the methods (limit of detection, applicability to the situation, compatibility with other methods in the sequence, etc.), capabilities of the laboratory

(available equipment, skilled personnel and other resources, reference materials, etc.), duration of analysis in case of urgency, practicability, cost of setup, and running costs. Consequently, this proves to be a complex decision problem, not restricted to analytical methods, which requires extensive knowledge and experience.

The assessment and selection of analytical methods involves a multitude of criteria, which are generally conflicting and affect the decision in different ways. Until now, there are some reports on comparisons of DNA extraction methods, where performance of individual parameters was compared and statistically evaluated (Jasbeer et al. 2009; Volk et al. 2014). However, no overall assessment of all parameters together was performed. The problem of including many parameters and criteria in the evaluation can be defined as a multicriteria decision problem. It is unlikely that one method will be the best for all of the considered criteria. Thus, approaches implementing multicriteria decision analysis (MCDA) (Ishizaka and Nemery 2013; Greco et al. 2016) may prove useful. In MCDA, each alternative (method) is first assessed according to each criterion. These individual assessments are then aggregated into an overall evaluation of each alternative method. On this basis, the alternatives are compared and/or ranked, and the best one can be chosen. Various analyses, such as sensitivity analysis and “what-if” analysis, are also possible to further support and justify the assessment. Pakpour (2012) reported the use of MCDA with weighted sum method for evaluation of sample preservation coupled with DNA extraction methods. The reported system, however, included a rather small amount of attributes in the study. Another tool named Analytical Method Performance Evaluation (AMPE), implementing more computational analysis, was developed to help with analytical method validation, including data handling capabilities and series of statistical calculations (Acutis et al. 2007). This tool is thus suitable only for comparisons of different parameters (e.g., limits of quantification and detection, variability, accuracy, trueness, precision) of one analytical method, produced by different users, which is usually the case in collaborative method validations. Currently, there are no tools available that would help in a thorough, objective (human independent), and reproducible evaluation and comparison of several analytical methods from different platforms (e.g., PCR, next-generation sequencing [NGS], and protein-based methods), taking into account a multitude of criteria that affect the methods’ performance and applicability.

This study addresses a common situation faced by enforcement or private analytical laboratories and other organizations involved in the analytical traceability and controls in the food/ feed production chains (Holst-Jensen 2009). Numerous traceability data (country of origin, species, industrial process, etc.) are available and processed, before a sample is submitted to the analytical laboratory. These data can obviously be used to
direct the analyses, i.e., choosing the appropriate sampling strategy and the required detection process (routine or more targeted analysis). All these data can be integrated in MCDA by the analyst for detection, results interpretation, or decision-making (Bohanec et al. 2013a).

Although MCDA could also be applied to additional steps, such as sample processing and DNA extraction, we have here limited our focus to the analysis of purified DNA from the sample(s). DNA analysis is usually carried out as a sequence of methods, most commonly involving a combination of screening, identification, and quantification methods.

The aim of our work, as part of a more general work on the development of GMO detection and identification strategies, was twofold: (1) to prepare an MCDA model suitable for evaluation and comparison of analytical methods and (2) use the model to evaluate a variety of analytical methods taking the field of GMO detection and/or quantification as an example case. Methodologically, we built upon the results of the European FP6 project Co-Extra (Bertheau 2013). There, as part of the Co-Extra decision support system (Bohanec et al. 2013a), a number of MCDA models were developed, including two models for the assessment of analytical methods. For the purpose of this study, we have, within European FP7 Decathlon project (www.decathlon-project.eu), adapted one of the models, called AM_DetQuant (Bohanec et al. 2013a). We have narrowed the original model to only assess whether a method is fit for purpose (FitForPurpose) in the situation being evaluated and to determine, which method is best for purpose (BestForPurpose). Rather than focusing on the comparison of several different methods for qPCR, our aim was to prepare and use the multicriteria model for comparing and evaluating a variety of different platforms used for running analytical methods. Altogether 15 different methods were lined up and evaluated from different aspects.

For clearer presentation, we prepared decision rules in a way that would allow all of these methods to be compared side by side, independently on their position in the whole pipeline of GMO analysis. In current GMO testing, several methods are usually combined and performed sequentially. The idea behind the selected decision rules was that they enable comparison and selection of methods for the same purpose (e.g., quantification), rather than implementing rules for more complex scenarios where methods for different purposes, such as screening and event-specific quantification, are combined. This also means that the exercise must be repeated for each desired purpose, with purpose-based decision rules.

In this manuscript, we show that the MCDA model makes possible the direct comparisons of several unrelated technologies. Notably, even when their overall fitness for purpose is relatively similar, comparing and evaluating individual criteria or a group of related criteria can uncover substantial differences between methods and technologies. The newly

developed model enables easy modifications of the criteria and of their influence on final evaluation. Thus, it can be easily adapted to any other complex analytical situation for selecting the most suitable analytical method(s). Particularly, it could be of great help in highly complex situations, where results of different identification techniques and approximate data, such as NGS sequences, would have to be combined with traceability elements.

# Materials and Methods

## Analytical Methods Assessed in This Study

Methods, selected for evaluation and comparison (assessment) in this study, are listed in Table 1. Four of the methods represent the qPCR system currently used in routine GMO diagnostics covering different applications: simplex and multiplex screening/identification (Alary et al. 2002; Kuribara et al. 2002; Pla et al. 2013) and simplex quantification (Holck et al. 2002). Two additional qPCR applications (SIMQUANT; Berdal et al. 2008) use qPCR chemistry together with the limiting dilutions principle, which is near to the idea of the ddPCR-based methods, of which two were included (Morisset et al. 2013; Dobnik et al. 2015). Other selected methods include LAMP with end-point fluorescent (Chen et al. 2011; Wang et al. 2015) or bioluminescence real-time detection (Kiddle et al. 2012), multiplex PCR with hybridization on microarrays (Leimanis et al. 2008; Hamels et al. 2009) or detection with capillary gel electrophoresis (Nadal et al. 2006), a protein-based method (Van Den Bulcke et al. 2007), and two NGS methods (unpublished, developed within the EU FP7 Decathlon project), one for enriched samples and another for whole genome sequencing (see, e.g., Arulandhu et al. 2016; Holst-Jensen et al. 2016). The majority of the selected methods are validated in-house or within international collaborative trials and their fitness for purpose demonstrated elsewhere (see Table 1 for references).

## Qualitative MCDA Method DEX

The original AM_DetQuant (Bohanec et al. 2013a) model, which was adapted for the purpose of this research, was developed using an MCDA method DEX (Decision EXPert) (Bohanec et al. 2013b; Bohanec 2015). DEX is a qualitative MCDA method, specifically designed to support expert modeling, that is, acquisition of decision knowledge from experts and decision makers in the form of decision rules. As all other MCDA methods, DEX assesses decision alternatives using multiple criteria. Alternatives are described with variables, called attributes, which represent observable properties of alternatives, such as *Price*, *Availability*, or *Accuracy*. DEX has the following distinctive characteristics:
<table><caption>**Table 1** Methods selected for evaluation and comparison within the developed DSS</caption><thead><tr><td>Method</td><td>Detection principle</td><td>Reference</td></tr></thead><tbody><tr><td>qPCR event</td><td>Simplex real-time PCR</td><td>Holck et al. (2002)</td></tr><tr><td>qPCR screen</td><td>Simplex real-time PCR</td><td>Alary et al. (2002), Kuribara et al. (2002)</td></tr><tr><td>qPCR triplex 35S-lec1-IPC</td><td>Multiplex real-time PCR</td><td>Pla et al. (2013)</td></tr><tr><td>qPCR pentaplex</td><td>Multiplex real-time PCR</td><td>Huber et al. (2013)</td></tr><tr><td>SIMQUANT simplex</td><td>Simplex real-time PCR with limiting dilutions</td><td>Berdal et al. (2008)</td></tr><tr><td>SIMQUANT multiplex</td><td>Multiplex real-time PCR with limiting dilutions</td><td>Berdal et al. (2008)</td></tr><tr><td>ddPCR MON810/hmgA</td><td>End-point multiplex droplet digital PCR</td><td>Morisset et al. (2013)</td></tr><tr><td>ddPCR multiplex per ingredient</td><td>End-point multiplex droplet digital PCR</td><td>Dobnik et al. (2015)</td></tr><tr><td>EAT DualChip</td><td>Multiplex PCR coupled with microarray detection</td><td>Leimanis et al. (2008), Hamels et al. (2009)</td></tr><tr><td>Pentaplex-CGE</td><td>Multiplex PCR coupled with capillary gel electrophoresis detection</td><td>Nadal et al. (2006)</td></tr><tr><td>LAMP screening/detection</td><td>Loop-mediated isothermal amplification with end-point visual detection</td><td>Chen et al. (2011), Wang et al. (2015)</td></tr><tr><td>LAMP-BART</td><td>Loop-mediated isothermal amplification with real-time bioluminescence detection</td><td>Kiddle et al. (2012)</td></tr><tr><td>LFD</td><td>Lateral flow device for antibody-based visual detection</td><td>Van Den Bulcke et al. (2007)</td></tr><tr><td>NGS-flanking regions</td><td>Enrichment PCR coupled with next-generation sequencing</td><td>Arulandhu et al. (2016)*</td></tr><tr><td>NGS-wgs</td><td>Whole genome sequencing</td><td>Holst-Jensen et al. (2016)*, Willems et al. (2016)</td></tr></tbody></table>

* DEX models are *hierarchical*. Attributes are hierarchically organized, so that the attributes at higher levels of the hierarchy depend on (and are determined on the basis of) lower-level attributes. This effectively splits attributes into basic attributes (terminal nodes), which represent inputs to the model, and aggregated attributes (internal nodes), which represent evaluations of alternatives. The topmost node(s) represent the final evaluations.

* DEX is a *qualitative* MCDA method. Unlike the majority of MCDA methods, which use numerical attributes, DEX uses symbolic attributes. Each attribute in DEX has a value scale consisting of words, such as (no, yes) or (low, medium, high). Scales are usually small (up to five values) and, whenever possible, preferentially ordered from “bad” to increasingly “good” values, according to the purpose of the choice.

* DEX is a *rule-based* method. The evaluation of alternatives is defined in terms of decision rules, which are defined by the model developer (expert of decision maker) and represented in the form of decision table. Each aggregated attribute in the model has a decision table that determines its value for all possible combinations of values of descendant attributes in the hierarchy.

The method DEX is currently implemented in the software called DEXi (Bohanec 2015) (downloadable from http://kt.ijs.si/MarkoBohanec/dexi.html). DEXi supports the development of DEX models and their application for the evaluation and analysis of decision alternatives. In the model

development stage, DEXi checks the quality of decision rules in terms of completeness (providing evaluation results for all possible inputs) and consistency (defined decision rules obey the principle of dominance, i.e., they monotonically increase with increasing input values).

DEX and DEXi have so far been used to support real-world decisions in health care, public administration, agriculture, food production, ecology, land use planning, tourism, housing, traffic control, sports, and finance (Bohanec et al. 2013b). Some recent large-scale applications include assessment of food and feed for the presence of genetically modified organisms (Bohanec et al. 2017) and assessment of energy production technologies (Kontić et al. 2016). Overall, DEX is particularly suitable for helping to solve complex decision problems that require judgment and qualitative knowledge-based reasoning, dealing with inaccurate and/or missing data, and analyzing and justifying the results of evaluation (Bohanec et al. 2013b).

## MCDA Model for the Assessment of Analytical Methods

The developed AnalyticalMeth model (available as .dxi file in Online Resource 1) is suitable for evaluation of analytical detection/quantification methods. Its overall architecture is shown in Fig. 1. As input, the model takes data describing analytical methods of the corresponding type. As output, the model provides two assessments: *FitForPurpose*, which tells whether a method is appropriate for a given analytical purpose

<footnote>*Review publications with limited experimental data reported</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 1** Schematic structure of the *AnalyticalMeth* model. The two main output attributes (*FitForPurpose* and *BestForPurpose*) appear on the upper side, and the input (basic) attributes that describe the detection and quantification analytical methods appear as terminal nodes at the bottom. Internal (aggregate) attributes serve for the aggregation of basic attributes into the three overall assessments. The aggregation is governed by expert-defined decision rules</figcaption></figure>

(using the value scale (no, partly, yes)), and *BestForPurpose*, which assesses the method's quality, depending on its fitness for purpose, level of development, performance, and overall applicability (using the value scale (unacc, acc, good, v-good, exc)).

The model has a complex internal structure and contains 77 attributes (34 basic, 10 linked, and 33 aggregated). The 34 basic attributes, represented with scales and descriptions, are listed in Table 2. Figure 2 shows the hierarchical structure of attributes in the *AnalyticalMeth* model. Fitness for purpose is assessed through the *FitForPurpose* submodel, which includes two branches: *PurposeFitness* and *SiteFitness*. *PurposeFitness* determines quantitative and screening performance of methods based on the following properties: linearity, accuracy, absolute and relative limit of quantification (LOQ), specificity, robustness, and limit of detection (LOD). The second branch, *SiteFitness*, determines whether the method is fit for on-site applications, meaning that it uses portable and less expensive equipment and that the actual analysis can be performed on the site of sampling (e.g., on the field). The most important part of the model is the *BestForPurpose* branch, which assesses the overall quality of the method in terms of *Constraints* and *Method Evaluation*. The *Constraints* submodel requires that the method is fit for purpose (*FitForPurpose*) and sufficiently developed (*MethDeveloped*) in terms of availability of reference materials (*SuggestedSamples*), defined standard operating procedure (*SOP*), current stage of development (*DevelopmentStage*), known specificity, and proficiency test outcomes. The second submodel, *MethEvaluation*, includes *Costs* (fixed and running costs), *Method Performance* in relation to its primary purpose (detection and/or quantification), and *Method Applicability* (the set of different functionalities of the method, see Fig. 2 and Table 2).

The hierarchical structure of the model provides a framework for decision tables, which define the bottom-up aggregation of attributes in the model. For each of the 33 aggregated attributes, a corresponding decision table was defined. We illustrate here the concept of decision tables with only two examples (Tables 3 and 4); for all decision tables, please see the general model description in Online Resource 2. The decision rules are indeed critical to the final outcome of the evaluation and selection process. If, for example, the laboratory must be able to identify and/or quantify a particular group of targets, then the rules must be designed so that only methods compliant with such a requirement are accepted.

As a first simple example, let us consider the decision table that assesses whether or not an analytic method is suitable for quantification. In the model (Fig. 2), the corresponding aggregated attribute is *FitForQuantification*, which depends on two descendant attributes: *FitForScreening* and *QuantitativePerformance*. The corresponding decision table (Table 3) thus defines the value of *FitForQuantification* for all possible value combinations of the latter attributes. There are nine possible combinations, from

$$ \text{Rule 1: if FitForScreening = no and QuantitativePerformance} \\ \qquad = \text{unacc then FitForQuantification = no} $$

to

$$ \text{Rule 9: if FitForScreening = yes and QuantitativePerformance} \\ \qquad = \text{good then FitForQuantification = yes} $$

This decision table is complete (defined for all nine possible combinations) and monotonically increasing.

For a more complex example of an evaluative decision table, let us consider the root attribute *AnalyticalMeth*, which makes an overall assessment of the method combining the attributes *FitForPurpose* (no, partly, yes) and *BestForPurpose* (unacc, acc, good, v-good, exc). Thus, the corresponding decision table contains 3 × 5 = 15 combinations. To save space, and in contrast with Table 3, Table 4 shows the rules in a more compact way, employing the symbols “:” (interval), “*” (any value), and “>=” (at least as good as). For example, two of the rules are interpreted as:

Rule 1: if the method is not *FitForPurpose*, then it is unacceptable (regardless of *BestForPurpose*).

Rule 4: if the method is at least partly *FitForPurpose* and is acceptable with respect to *BestForPurpose*, then it is acceptable.

In standard DEX notation for input data that is unknown or so uncertain that it cannot be represented by a single scale value, “*” is used. In our case, the “*” is most often used in connection with *LOQ_Abs*, *LOQ_Rel*, and *Linearity* because these parameters were not relevant for specific methods (i.e., quantification
<figure><img src="image_6.png" /><figcaption>**Table 2** List of all basic attributes and their scales and descriptions</figcaption></figure>

<table><thead><tr><th>Basic attribute</th><th>Scale</th><th>Description of scale values</th><th>Description of attribute</th></tr></thead><tbody><tr><td>Purpose</td><td>Screening, quantification</td><td></td><td>Primary purpose of analysis: screening or quantification</td></tr><tr><td>Site</td><td>Lab, on-site</td><td></td><td>Site of performing analysis: lab or on-site</td></tr><tr><td>DevelopmentStage</td><td>Proof, opt. assay, tested, prevalidation, validation</td><td></td><td>Current state of method development</td></tr><tr><td>SOP</td><td>No, yes</td><td></td><td>Is standard operating procedure available?</td></tr><tr><td>SuggestedSamples</td><td>No, yes</td><td></td><td>Are samples for method development available? At least reference material and one independent positive sample are in general needed to fully assess the method</td></tr><tr><td>SpecificityAssessed</td><td>No, partly, yes, not required</td><td>No, only for targets available, for all relevant targets, not applicable for the method</td><td>To which extent was method specificity assessed? Definition of specificity: property of a method to respond exclusively to the characteristic or analyte of interest</td></tr><tr><td>Proficiency</td><td>No, yes</td><td></td><td>Was the method tested on proficiency samples?</td></tr><tr><td>MethodType</td><td>Detection, quantification</td><td></td><td>Is the method specific?</td></tr><tr><td>Specificity</td><td>No, yes</td><td></td><td>Absolute limit of detection: target copy numbers detected in single reactions</td></tr><tr><td>LOD_Abs</td><td>Insensitive, med, good</td><td>Above 100, 25–100, 1–25 target copies</td><td>Relative limit of detection: lowest detected percentage of GMO in the sample</td></tr><tr><td>LOD_Rel</td><td>Insensitive, med, good</td><td>Above 0.5% GMO, 0.1–0.5% GMO, below 0.1% GMO</td><td>Is the method robust? Definition: the robustness of a method is a measure of its capacity to remain unaffected by small, but deliberate deviations from the experimental conditions described in the procedure. Examples of factors that a robustness test could address are use of different instrument types, operator, brand of reagents, concentration of reagents, and temperature of reaction</td></tr><tr><td>Robustness</td><td>Low, med, high</td><td>The method is running on one machine type only, needs skilled operators, difficult to transfer to other system; the method is well transferable but is sensitive to deviations in experimental conditions; the method is well transferable and not sensitive to experimental conditions</td><td>How is method handling inhibitor-rich samples?</td></tr><tr><td>InhibitorHandling</td><td>None, med, good</td><td>Method is sensitive to inhibitors, the method is partially sensitive to inhibitors, the method is not sensitive to inhibitors<br/>&gt; 50% discrepancy, 25–50%, &lt; 25%</td><td>How accurate is the method? Definition of accuracy: the closeness of agreement between a test result and the accepted reference value</td></tr><tr><td>Accuracy</td><td>Low, med, high</td><td>Above 1% GMO, 0.1–1% GMO, below 0.1% GMO<br/>Above 200, 50–200, below 50 target copies<br/>R<sup>2</sup> &lt; 0.95 and dynamic range (DR) spanning 2 logs, R<sup>2</sup> &gt; 0.96 and DR spanning 2–4 logs, R<sup>2</sup> &gt; 0.96 and DR spanning more than 4 logs</td><td>Relative limit of quantification: lowest quantifiable percentage of GMO in sample<br/>Absolute limit of quantification: target copy numbers quantified in single reactions<br/>How linear is the method within range of quantification (= coefficient of determination, R<sup>2</sup> of linear regression)? What is the DR of the method?<br/>Definition of dynamic range: The range of concentrations over which the method performs in a linear manner with an acceptable level of accuracy and precision.</td></tr><tr><td>MixedSamples</td><td>No, yes</td><td></td><td>Is the method applicable for mixed samples?</td></tr><tr><td>ProcessedSamples</td><td>No, yes</td><td></td><td>Is the method applicable for processed samples?</td></tr><tr><td>RequiredAmount</td><td>High, med, low</td><td>&gt; 5, 1–5, &lt; 1 µg</td><td>Required amount of DNA</td></tr><tr><td>Screening</td><td>No, yes</td><td></td><td>Is the method useful for screening?</td></tr><tr><td>Identification</td><td>No, yes</td><td></td><td>Is the method useful for identification of specific GM line?</td></tr><tr><td>SpeciesDetection</td><td>No, yes</td><td></td><td>Is the method useful for species detection?</td></tr><tr><td>Quantification</td><td>No, yes</td><td></td><td>Is the method useful for quantification?</td></tr></tbody></table>
<table><caption>**Table 2 (continued)**</caption><thead><tr><th>Basic attribute</th><th>Scale</th><th>Description of scale values</th><th>Description of attribute</th></tr></thead><tbody><tr><td>UnauthorisedGM</td><td>No, yes</td><td rowspan="2">Higher than PCR, as in PCR, lower than PCR</td><td>Can the method detect unauthorized GMOs?</td></tr><tr><td>OnSiteDetection</td><td>No, yes</td><td>Is the method suitable for on-site detection?</td></tr><tr><td>Consumables/Sample</td><td>High, med, low</td><td rowspan="2">Under 10, 10–50, more than 50</td><td>Costs of consumables per sample compared to simplex PCR</td></tr><tr><td>Samples/Day</td><td>Low, med, high</td><td>Number of samples that can be analyzed in one working day with one detection instrument</td></tr><tr><td>Targets/Method</td><td>Low, med, high, v-high</td><td rowspan="2">1, 2 to 4, 5 to 10, more than 10<br>More than 3 h, 30 min to 3 h, less than 30 min</td><td>Number of targets that can be detected simultaneously</td></tr><tr><td>PersonalCosts</td><td>High, med, low</td><td>Costs of labor per sample [working hours needed per sample] (not including sampling, sample processing, and DNA extraction)</td></tr><tr><td>EquipmentCosts</td><td>V-high, high, med, low</td><td>Range of microarray or ddPCR, range of qPCR, range of conv. PCR, basic lab or even on-site</td><td>Costs of required equipment, including maintenance</td></tr><tr><td>InfrastructureCosts</td><td>High, med, low</td><td>Mol. lab, basic lab, no lab needed</td><td>Cost of infrastructure</td></tr><tr><td>CompetenceCosts</td><td>High, med, low</td><td>Experienced techn., any techn. or student, not qualified personnel</td><td>Personal competence required for method</td></tr><tr><td>ValidationCosts</td><td>High, med, low</td><td>Range of multiplex methods, range of simplex qPCR, lower</td><td>Cost of in-house validation (implementation in the lab)</td></tr></tbody></table>

parameters are not relevant for qualitative methods). Also, the values of *Robustness*, *InhibitorHandling*, and *Accuracy* for the method next-generation sequencing-whole genome sequencing (NGS-wgs) are missing because they were not assessed yet. When evaluating analytical methods with unknown input values, DEX treats the symbol "*" as a set of all possible values of the corresponding attribute and repeats the evaluation for all of them. Consequently, any DEX evaluation may yield a set of values rather than a single value (Bohanec 2015).

The decision tables in the *AnalyticalMeth* model are complete and consistent. All decision tables are presented in the model description (see Online Resource 2).

## Results

### Filling Up the Model with Methods' Information

The information about the individual methods and their performance must be manually entered into the model. Different variables or observable properties of the methods, called attributes, are taken into account for final evaluation. For each basic attribute, a corresponding scale value must be chosen. For the GMO-related examples presented in this study, the 34 basic attributes and their scale values are listed in Table 2. We determined a specific scale value of each basic attribute for each individual method listed in Table 1. We have performed this task with the help of published data and with the data from our experiments for yet unpublished methods. The constructed methods/attributes matrix (presented in Table 5) is the base for the calculations of the model.

### Overall Evaluation of Compared Methods

The developed model was tested using 15 analytical methods. The overall assessment of these 15 methods with the *AnalyticalMeth* model on the basis of the applied decision rules is shown in Fig. 3. Other decision rules might or would have yielded other results. The gold standard technologies, qPCR event and qPCR screen, were assessed as very good and good, respectively. The majority of other methods (qPCR triplex, qPCR pentaplex, SIMQUANT multiplex, both ddPCR methods, EAT Dual Chip, pentaplex-CGE, and LAMP-BART) were assessed as very good and one (SIMQUANT simplex) as good. Only one technology was assessed as excellent, i.e., simplex LAMP for GMO screening or detection. In the current context, the LFD technology was assessed as acceptable, due to weaker performance in terms of sensitivity and accuracy, despite that it can be improved by subsampling strategies (Remund et al. 2001; Kobilinsky and Bertheau 2005). Both NGS methods were assessed as unacceptable, mostly because their sensitivity and throughput are not as good as with other methods and also the price of analyses is relatively
<figure><img src="image_7.png" /><figcaption>Fig. 2 Hierarchical structure of attributes in the AnalyticalMeth model</figcaption></figure>

high. In case of different rules (one example in Table 6), for instance when there would be a need to perform the detection and identification of unauthorized GMOs, the NGS methods would get a better (or even the best) score.

## Detailed Method Assessment

To explain the overall assessment and to pinpoint the differences that contribute the most to the outcome of method assessments, a more detailed analysis can be performed when assessing the methods at a lower level. To illustrate this, we have selected three cases to evaluate the effectiveness of increasing the number of targets per analysis (Fig. 4), to compare different detection platforms (Fig. 5) and to compare the methods with the same

purpose (e.g., quantification, Fig. 6). For these cases, we have selected five sublevels, which are, in our opinion, the most informative for the given situation (BestForPurpose, Costs, MethPerformance, MethApplicability, and Constraints for the first two cases and QuantitativePerformance, Costs, MethFunctions, LOD, and Targets/Method for quantitative methods comparison). For their position within the attribute tree, see section A in Online Resource 2.

## Increasing the Number of Targets per Analysis

In order to compare the performance of qPCR-based detection methods when increasing the number of targets detected in a single analysis, we have included simplex qPCR (event

<table><caption>**Table 3** Decision table defining FitForQuantification output depending on FitForScreening and QuantitativePerformance inputs</caption><thead><tr><th rowspan="2">Rule no.</th><th colspan="2">Input</th><th>Output</th></tr><tr><th>FitForScreening</th><th>QuantitativePerformance</th><th>FitForQuantification</th></tr></thead><tbody><tr><td>1</td><td>No</td><td>Unacc</td><td>No</td></tr><tr><td>2</td><td>No</td><td>Acc</td><td>No</td></tr><tr><td>3</td><td>No</td><td>Good</td><td>No</td></tr><tr><td>4</td><td>Partly</td><td>Unacc</td><td>No</td></tr><tr><td>5</td><td>Partly</td><td>Acc</td><td>Partly</td></tr><tr><td>6</td><td>Partly</td><td>Good</td><td>Yes</td></tr><tr><td>7</td><td>Yes</td><td>Unacc</td><td>No</td></tr><tr><td>8</td><td>Yes</td><td>Acc</td><td>Partly</td></tr><tr><td>9</td><td>Yes</td><td>Good</td><td>Yes</td></tr></tbody></table>

unacc unacceptable, acc acceptable
<table><caption>**Table 4** Decision table for the assessment of *AnalyticalMeth* from *FitForPurpose* and *BestForPurpose* inputs</caption><thead><tr><th rowspan="2">Rule no.</th><th colspan="2">Input</th><th>Output</th></tr><tr><th>FitForPurpose</th><th>BestForPurpose</th><th>AnalyticalMeth</th></tr></thead><tbody><tr><td>1</td><td>No</td><td>*</td><td>Unacc</td></tr><tr><td>2</td><td>*</td><td>Unacc</td><td>Unacc</td></tr><tr><td>3</td><td>Partly</td><td>Acc:good</td><td>Acc</td></tr><tr><td>4</td><td>&gt;=partly</td><td>Acc</td><td>Acc</td></tr><tr><td>5</td><td>Partly</td><td>&gt;=v-good</td><td>Good</td></tr><tr><td>6</td><td>Yes</td><td>Good</td><td>Good</td></tr><tr><td>7</td><td>Yes</td><td>V-good</td><td>V-good</td></tr><tr><td>8</td><td>Yes</td><td>Exc</td><td>Exc</td></tr></tbody></table>

unacc unacceptable, acc acceptable, v-good very good, exc excellent, ":" interval, "*" any value, ">=" at least as good as

specific and screening), multiplex qPCR (triplex and pentaplex), and SIMQUANT methods. In the overall evaluation, triplex qPCR screen and SIMQUANT simplex were assessed as good, while all the others were assessed as very good (Fig. 3). As can be seen from Fig. 4, lower overall score partially comes from lower scores for the aggregate attributes *BestForPurpose*, *MethApplicability*, and/or *Costs*. When multiplexing and comparing simplex and multiplex of the same platform, more targets are detected in one reaction. Thus, the cost for analysis per target is generally lower. Based on the set decision rules, triplex qPCR was still in the same cost level as both simplex qPCR methods, but pentaplex qPCR already showed cost benefit. The SIMQUANT method not only improved the cost factor with multiplexing, but also the *BestForPurpose* score was better (Fig. 4).

## Different Detection Platforms

In order to compare different detection platforms, considering their complete application potential in terms of their purpose, we assessed nine different methods (Fig. 5). We observed that the majority of detection methods alternative to qPCR (droplet digital PCR, microarray detection, capillary gel electrophoresis, and LAMP coupled with bioluminescence detection in real-time) are comparable between each other, i.e., having the same scores, when performing evaluation with the given decision rules. A relatively simple and inexpensive method for on-site detection, LFD, was assessed as less advantageous in our conditions of use, mostly because in general, it targets proteins and thus cannot be applied to the analysis of processed samples (Fig. 5). The NGS method is currently, based on the scores of the *AnalyticalMeth* model, still not a good option for GMO analysis. Nevertheless, NGS is the newest fashionable method, and with more extensive use driven by a predicted drop in overall costs it might find a place in GMO testing in the future, when sequencing methods will have been improved for in-

depth sequencing, sequences' assembly, and comparison with, currently missing, gold reference genomes. Already now, it is the best option for very specific applications when no other methods are applicable (e.g., for identification of completely unknown GMO), and with new platforms it is getting less expensive (Pennisi 2017). Another thing is that while new alternative detection platforms often have very good applicability, the fixed costs (equipment, training, skilled personnel, etc.) for implementation in the laboratory can be prohibitively high, resulting in a low overall score in this *AnalyticalMeth* model (e.g., NGS in Fig. 5).

## Methods for Target Quantification

The comparison of the methods that enable quantification of the targets, including the current golden standard detection method (qPCR on transformation event's sequences), is shown in Fig. 6. For this comparison, we selected the attributes that are important for quantification methods and show most pronounced differences. For example, the attribute *FitForQuantification* would not provide any new relevant information, as all of the methods except NGS-wgs have reported quantification purposes. However, *MethFunctions*, and *Costs*, with sublevel of *Targets/Method*, produce a lot of information, if one is planning to implement the method into routine diagnostics. Additionally, *QuantitativePerformance* and *LOD* (for aspect of sensitivity) attributes were added to the comparison. With such a selection of attributes, the output results of the *AnalyticalMeth* model were the most relevant for the purpose of GMO quantification. The comparison of SIMQUANT simplex and multiplex shows improvement in terms of costs (with more targets analyzed per method). However, when moving to ddPCR (considering it as a technical improvement of the SIMQUANT method), we also gain additional information within *MethFunctions* due to simultaneous species reference sequence and event detection (Fig. 6). When finally moving to ddPCR multiplex per ingredient, we got the best scores in this comparison with the highest number of targets. When comparing ddPCR multiplex to a qPCR event method, ddPCR multiplex outperforms qPCR event due to the possibility of endogenous sequence quantification (and species identification) and due to numerous targets quantified simultaneously in one reaction (Fig. 6). NGS-wgs that theoretically also enables quantification, has its strong points in detecting/quantifying numerous targets (events, species, and also unauthorized GMOs). However, for the current case study of a routine laboratory detecting approved GMOs, other attributes of NGS-wgs presented low scores; therefore, the overall score for this method is relatively poor in comparison to other methods (Fig. 6). In case that a model would be designed to evaluate the best method for the purpose of unauthorized GM detection, the overall score of NGS-wgs would probably be the highest.
<figure><img src="image_8.png" /><figcaption>**Table 5** Methods and attributes matrix in the model. For individual method a value is selected for each of the attributes</figcaption></figure>

<table><thead><tr><th>Attribute</th><th>qPCR event</th><th>qPCR screen</th><th>qPCR triplex<br>35S-LEC1-IPC</th><th>qPCR pentaplex<br>screening</th><th>SIMQUANT simplex</th><th>SIMQUANT<br>multiplex</th><th>ddPCR MON<br>810/hmgA</th></tr></thead><tbody><tr><td>Reference</td><td>Holck et al. (2002)</td><td>Alary et al. (2002),<br>Kuribara et al. (2002)</td><td>Pla et al. (2013)</td><td>Huber et al. (2013)</td><td>Berdal et al. (2008)</td><td>Berdal et al. (2008)</td><td>Morisset et al.<br>(2013)</td></tr><tr><td>Purpose</td><td>Quantification</td><td>Screening</td><td>Screening</td><td>Screening</td><td>Quantification</td><td>Quantification</td><td>Quantification</td></tr><tr><td>Site</td><td>Lab</td><td>Lab</td><td>Lab</td><td>Lab</td><td>Lab</td><td>Lab</td><td>Lab</td></tr><tr><td>DevelopmentStage</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td></tr><tr><td>SOP</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>SuggestedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>SpecificityAssessed</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Proficiency</td><td>Quantification</td><td>Detection</td><td>Detection</td><td>Detection</td><td>Quantification</td><td>Quantification</td><td>Quantification</td></tr><tr><td>MethodType</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>Specificity</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td></tr><tr><td>LOD_Abs</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td></tr><tr><td>LOD_Rel</td><td>Good</td><td>Good</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td></tr><tr><td>Robustness</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td></tr><tr><td>InhibitorHandling</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td></tr><tr><td>Accuracy</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td></tr><tr><td>LOQ_Rel</td><td>Good</td><td>*</td><td>*</td><td>*</td><td>Good</td><td>Good</td><td>Good</td></tr><tr><td>LOQ_Abs</td><td>Good</td><td>*</td><td>*</td><td>*</td><td>Good</td><td>Good</td><td>Good</td></tr><tr><td>Linearity</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td></tr><tr><td>MixedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>ProcessedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>RequiredAmount</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td></tr><tr><td>Screening</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>Identification</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>SpeciesDetection</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>Quantification</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>UnauthorisedGM</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td></tr><tr><td>OnSiteDetection</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Med</td></tr><tr><td>Consumables/Sample</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>High</td><td>High</td><td>Med</td></tr><tr><td>Samples/Day</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>Med</td></tr><tr><td>Targets/Method</td><td>Low</td><td>Low</td><td>Med</td><td>Med</td><td>Low</td><td>Low</td><td>Med</td></tr><tr><td>PersonalCosts</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td></tr><tr><td>EquipmentCosts</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>V-high</td></tr><tr><td>InfrastructureCosts</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td></tr><tr><td>CompetenceCosts</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>High</td></tr><tr><td>ValidationCosts</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>Med</td><td>High</td></tr></tbody></table><table><thead><tr><th>Attribute</th><th>ddPCR multiplex<br>per ingredient</th><th>EAT DualChip</th><th>Pentaplex-CGE</th><th>LAMP screening/detection</th><th>LAMP-BART</th><th>LFD</th><th>NGS-flanking regions</th><th>NGS-wgs</th></tr></thead><tbody><tr><td>Reference</td><td>Dobnik et al. (2015)</td><td>Leimanis et al. (2008),<br>Hamels et al. (2009)</td><td>Nadal et al. (2006)</td><td>Chen et al. (2011),<br>Wang et al. (2015)</td><td>Kiddle et al. (2012)</td><td>Van Den Bulcke<br>et al. (2007)</td><td>NA</td><td>NA</td></tr><tr><td>Purpose</td><td>Quantification</td><td>Screening</td><td>Screening</td><td>Screening</td><td>Screening</td><td>Screening</td><td>Screening</td><td>Quantification</td></tr></tbody></table>
<table><caption>**Table 5 (continued)**</caption><thead><tr><th>Attribute</th><th>ddPCR multiplex per ingredient</th><th>EAT DualChip</th><th>Pentaplex-CGE</th><th>LAMP screening/detection</th><th>LAMP-BART</th><th>LFD</th><th>NGS-flanking regions</th><th>NGS-wgs</th></tr></thead><tbody><tr><td>Site</td><td>Lab</td><td>Lab</td><td>Lab</td><td>On-site</td><td>On-site</td><td>On-site</td><td>Lab</td><td>Lab</td></tr><tr><td>DevelopmentStage</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Validation</td><td>Proof</td><td>Proof</td></tr><tr><td>SOP</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td></tr><tr><td>SuggestedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>SpecificityAssessed</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Partly</td><td>Partly</td></tr><tr><td>Proficiency</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td></tr><tr><td>MethodType</td><td>Quantification</td><td>Detection</td><td>Detection</td><td>Detection</td><td>Detection</td><td>Detection</td><td>Detection</td><td>Quantification</td></tr><tr><td>Specificity</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>LOD_Abs</td><td>Good</td><td>Med</td><td>Good</td><td>Good</td><td>Good</td><td>Med</td><td>Med</td><td>Med</td></tr><tr><td>LOD_Rel</td><td>Good</td><td>Med</td><td>Good</td><td>*</td><td>High</td><td>Med</td><td>Insensitive</td><td>Insensitive</td></tr><tr><td>Robustness</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>Med</td><td>Med</td><td>*</td></tr><tr><td>InhibitorHandling</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Good</td><td>Med</td><td>Med</td><td>*</td></tr><tr><td>Accuracy</td><td>High</td><td>High</td><td>High</td><td>High</td><td>High</td><td>Med</td><td>Med</td><td>*</td></tr><tr><td>LOQ_Rel</td><td>Good</td><td>*</td><td>*</td><td>*</td><td>*</td><td>*</td><td>*</td><td>Bad</td></tr><tr><td>LOQ_Abs</td><td>Good</td><td>*</td><td>*</td><td>*</td><td>*</td><td>*</td><td>*</td><td>Bad</td></tr><tr><td>Linearity</td><td>High</td><td>Med</td><td>High</td><td>High</td><td>*</td><td>*</td><td>*</td><td>*</td></tr><tr><td>MixedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>ProcessedSamples</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>RequiredAmount</td><td>Low</td><td>Med</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td><td>Low</td><td>Med</td></tr><tr><td>Screening</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>Identification</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td></tr><tr><td>SpeciesDetection</td><td>Yes</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>Quantification</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td></tr><tr><td>UnauthorisedGM</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Yes</td><td>Yes</td></tr><tr><td>OnSiteDetection</td><td>No</td><td>No</td><td>No</td><td>Yes</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td></tr><tr><td>Consumables/Sample</td><td>Med</td><td>Low</td><td>Low</td><td>Med</td><td>Med</td><td>Low</td><td>High</td><td>High</td></tr><tr><td>Samples/Day</td><td>Med</td><td>Low</td><td>Med</td><td>High</td><td>High</td><td>High</td><td>Low</td><td>Low</td></tr><tr><td>Targets/Method</td><td>V-high</td><td>V-high</td><td>High</td><td>High</td><td>Low</td><td>Low</td><td>V-high</td><td>V-high</td></tr><tr><td>PersonalCosts</td><td>Med</td><td>High</td><td>Med</td><td>Low</td><td>Med</td><td>Low</td><td>High</td><td>High</td></tr><tr><td>EquipmentCosts</td><td>V-high</td><td>V-high</td><td>V-high</td><td>V-high</td><td>Low</td><td>Med</td><td>V-high</td><td>V-high</td></tr><tr><td>InfrastructureCosts</td><td>High</td><td>High</td><td>High</td><td>High</td><td>Low</td><td>Low</td><td>High</td><td>High</td></tr><tr><td>CompetenceCosts</td><td>High</td><td>High</td><td>High</td><td>High</td><td>Low</td><td>Med</td><td>High</td><td>High</td></tr><tr><td>ValidationCosts</td><td>High</td><td>Med</td><td>High</td><td>High</td><td>Med</td><td>Med</td><td>High</td><td>High</td></tr></tbody></table>

NA not available, because the method was not published yet; unacceptable; acc acceptable; v-high very high; "*" any value

<figure><img src="image_9.png" /></figure>
<figure><img src="image_10.png" /><figcaption>**Fig. 3** Overall evaluation of the methods by the *AnalyticalMeth* model given the specific set of decision rules defined for the purpose of GMO detection or quantification in Online Resource 2 part B. Colors of the chart are for easier visualization of the results, as they visually stress the appropriateness of the methods (green—excellent result; blue—acceptable to very good result; red—unacceptable result)</figcaption></figure>

Discussion

The presented results from the *AnalyticalMeth* model showed its usefulness for evaluating a set of GMO testing methods of diverse detection platforms. The results also showed that new

methods, developed for the purpose of GMO detection, are as good, or even better, when compared to the golden standard method of qPCR. Most of the compared methods showed a very good final evaluation score. However, it should be noted that the comparison of scores at sublevels varied between the

<table><caption>**Table 6** Decision table for the assessment of *MethFunctions* with theoretical example of assessment output, when the laboratory would require detection and identification of unauthorized GMO</caption><thead><tr><th rowspan="2">Rule no.</th><th colspan="6">Input</th><th colspan="2">Output</th></tr><tr><th>Screening</th><th>Identification</th><th>SpeciesDetection</th><th>Quantification</th><th>UnauthorisedGM</th><th>OnSiteDetection</th><th>MethFunctions</th><th>MethFunctions—alternative<sup>a</sup></th></tr></thead><tbody><tr><td>1</td><td>No</td><td>No</td><td>No</td><td>*</td><td>*</td><td>*</td><td>Bad</td><td>Bad</td></tr><tr><td>2</td><td>No</td><td>No</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>Acc</td><td>Bad</td></tr><tr><td>3</td><td>No</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Acc</td><td>Acc</td></tr><tr><td>4</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>No</td><td>No</td><td>Acc</td><td>Bad</td></tr><tr><td>5</td><td>No</td><td>No</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>Good</td><td>Bad</td></tr><tr><td>6</td><td>*</td><td>Yes</td><td>No</td><td>Yes</td><td>No</td><td>No</td><td>Good</td><td>Good</td></tr><tr><td>7</td><td>*</td><td>Yes</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>Good</td><td>Good</td></tr><tr><td>8</td><td>Yes</td><td>*</td><td>No</td><td>Yes</td><td>No</td><td>No</td><td>Good</td><td>Bad</td></tr><tr><td>9</td><td>Yes</td><td>*</td><td>Yes</td><td>No</td><td>No</td><td>No</td><td>Good</td><td>Bad</td></tr><tr><td>10</td><td>Yes</td><td>Yes</td><td>No</td><td>*</td><td>No</td><td>No</td><td>Good</td><td>Good</td></tr><tr><td>11</td><td>Yes</td><td>Yes</td><td>*</td><td>No</td><td>No</td><td>No</td><td>Good</td><td>Good</td></tr><tr><td>12</td><td>*</td><td>*</td><td>Yes</td><td>*</td><td>*</td><td>Yes</td><td>Exc</td><td>Bad</td></tr><tr><td>13</td><td>*</td><td>*</td><td>Yes</td><td>*</td><td>Yes</td><td>*</td><td>Exc</td><td>Exc</td></tr><tr><td>14</td><td>*</td><td>Yes</td><td>*</td><td>*</td><td>*</td><td>Yes</td><td>Exc</td><td>Good</td></tr><tr><td>15</td><td>*</td><td>Yes</td><td>*</td><td>*</td><td>Yes</td><td>*</td><td>Exc</td><td>Exc</td></tr><tr><td>16</td><td>*</td><td>Yes</td><td>Yes</td><td>Yes</td><td>*</td><td>*</td><td>Exc</td><td>Good</td></tr><tr><td>17</td><td>Yes</td><td>*</td><td>*</td><td>*</td><td>*</td><td>Yes</td><td>Exc</td><td>Bad</td></tr><tr><td>18</td><td>Yes</td><td>*</td><td>*</td><td>*</td><td>Yes</td><td>*</td><td>Exc</td><td>Exc</td></tr><tr><td>19</td><td>Yes</td><td>*</td><td>Yes</td><td>Yes</td><td>*</td><td>*</td><td>Exc</td><td>Bad</td></tr></tbody></table>

<footnote><sup>a</sup> The suggested rule would favor the detection and identification of unauthorized GMOs</footnote>
SIMPLEX

<figure><img src="image_11.png" /><figcaption>qPCR screen</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>SIMQUANT simplex</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 4** Comparison of qPCR-based methods to evaluate the effectiveness of increasing the number of targets per analysis. The left side of the figure shows evaluation of simplex methods with qPCR event, qPCR screen, and SIMQUANT simplex as examples. On the right side, a multiplex method evaluation is given for qPCR triplex, qPCR pentaplex, and</figcaption></figure>

MULTIPLEX

<figure><img src="image_14.png" /><figcaption>qPCR pentaplex screening</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>SIMQUANT multiplex</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>SIMQUANT multiplex. Both qPCR pentaplex and SIMQUANT multiplex showed improvement in costs, whereas triplex qPCR was still in the range of simplex qPCR. Blue line—overall score for evaluation of the method was “acceptable,” good, or very good</figcaption></figure>

methods. As the compared methods are meant for different purposes, the comparison at these sublevels is indeed more informative for the laboratories. Importantly, the final evaluation score depends on the decision rules set by the user. Therefore, the current model can only serve as an example

for the decision rules set by ourselves. Indeed, the decision rules, giving more importance to other attributes, set by other users, might result in different evaluation scores in the end.

For our model, one of the important attributes was the cost effectiveness of a method. Since the costs in this model are
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /><figcaption>**Fig. 5** Comparison of different detection platforms. Majority of platforms have equal score in terms of selected attributes (top left) and three other platforms have specific scores that are better (LAMP platform) or worse than majority (LFD and NGS). Green line—overall score for</figcaption></figure>

compound of running and fixed costs, each one with addition-
al sublevels, the final score of different methods could be
comparable. In such cases, one should carefully evaluate also
other levels to take a decision, which method should be im-
plemented in the laboratory. If fixed costs are low, but running
costs are high, then laboratories with lots of samples might
rather make a choice of methods with higher fixed costs and
lower running costs as it would be more cost-effective in the
long run.

New multiplexing methods (e.g., multiplex ddPCR) do
show some increased performance in comparison to sim-
plex qPCR methods, as the number of targets detected in
one analysis far exceeds the single target from simplex
methods (Morisset et al. 2013; Dobnik et al. 2015, 2016;
Košir et al. 2017). However, their main drawback is the
investment in new equipment, additional personnel train-
ing costs, and the longer time for analysis of one sample.
This is generally a problem of new technologies, for which
the costs are quite high compared to already established
technologies. On the other hand, they can offer more infor-
mation from one run, outweighing some additional costs.
Pentaplex qPCR (Huber et al. 2013) exemplifies this as it
incorporates both qPCR and multiplexing, but the purpose

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>evaluation of the platform was “excellent.” Blue line—overall score for
evaluation of the platform was “acceptable,” good, or very good. Red
line—overall score for evaluation of the platform was “unacceptable”</figcaption></figure>

of this method is limited to screening. With the increasing
number of GMOs on the market (James 2015), the situa-
tion will most probably go in the direction of bringing
multiplex methods to the position as *BestForPurpose* tech-
nology on the market. Since the quantitative aspect of
qPCR multiplex is rather limited, with only two available
interlaboratory-validated duplex methods (Waiblinger
et al. 2007; Takabatake et al. 2011), new technologies such
as ddPCR can become the leading technology on the mar-
ket for routine GMO diagnostics. When new technologies
emerge, they are often more costly at the beginning, but
with gradually broader adoption, the prices usually de-
crease. In the long run, the accumulated costs of delayed
implementation can sometimes exceed the accumulated
savings perceived in a short-term perspective. This per-
spective is not included in the cost calculation in the cur-
rent model, in part because it would add a factor of spec-
ulation, as future cost fluctuations cannot be reliably pre-
dicted. Nevertheless, up to now in some cases, when con-
sidering the need of accurate results for taking some deci-
sions, such as removing products from the market, the cur-
rent need for parallel use of different NGS platforms
and software, to discard tool-linked specific errors,
<figure><img src="image_21.png" /><figcaption>SIMQUANT multiplex</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>ddPCR multiplex per ingedient</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Fig. 6** Comparison of methods that enable GMO quantification. The selected attributes were the ones where the distinction between the methods was the greatest. In comparison to qPCR, only ddPCR showed</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>ddPCR MON 810/hmgA</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>NGS-wgs</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>improvement in more than two out of five attributes. Blue line—overall score for evaluation of the method was “acceptable,” good, or very good. Red line—overall score for evaluation of the method was “unacceptable”</figcaption></figure>

increases drastically the associated costs, although the costs of NGS are reducing (Liu et al. 2012; Goldfeder et al. 2016; Potapov and Ong 2017).

The GMO analysis testing pipeline generally involves several steps, selected based on a classification by the matrix approach (Chaouachi et al. 2008; Van den Bulcke et al. 2010; Block et al. 2013) and dependent on the sample type. The sample can be analyzed with a multiplex screening method and then further analyzed with a quantitative method. Thus, the comparison of these two methods would not give any relevant information as both are needed in the analysis process. Therefore, we suggest that methods with similar purpose, suitable for each of the steps, should be evaluated independently. Until now, unless a specific GMO is targeted (e.g.,

during an emergency period linked to a specific unauthorized
GMO, such as FP967 flax (EURL-GMFF 2009) or BT10
maize (EURL-GMFF 2005)), initial screening is the predom-
inant first step in GMO testing. Based on screening results, it
is possible to predict which GM events the sample contains.
To eliminate the need for screening steps, a ddPCR multiplex
method that quantifies a whole group of GMOs (Dobnik et al.
2015) can for instance be used when only one species is
targeted. With the recent emergence and growing number of
GMOs lacking the most common screening elements, the cost
efficiency of element screening is reduced, since additional
complementary identification methods must always be run.
In such cases, new methods and more universal approaches
could be more suitable for the analysis (e.g., performing
specific event detections with the usual screenings, or using multiplex ddPCR or NGS). Modifications, i.e., setting new decision rules, in the developed MCDA *AnalyticalMethod* model, could help in direct comparison of selected methods.

Again, it is important to note that to get the best possible evaluation and avoid cases where methods would have the same result, the laboratories should define their needs first, and then set up the decision rules accordingly. As two different sets of decision rules might give two different scores for any individual method, the setting of decision rules to fit a laboratory’s needs is critical. The developed *AnalyticalMethod* model is therefore not fixed, but fully flexible, to allow each user to select decision rules according to own needs. Additional methods can also be added to the model as they become available. With the emergence of new methods and other relevant parameters, new attributes can easily be added and/or deleted from the model. For instance, seed quality control during production would probably benefit from emphasis on fast on-site applicability. This could improve the ranking of, e.g., LFD, and lower that of PCR. On the other hand, perceived risk of presence of multiple events including unauthorized and possibly even unknown events would suggest to put emphasis on criteria that could favor NGS. One of the possible additions to the model could be a module on DNA extraction, because different methods may require different quantities and purities of DNA. As at least simplified DNA extraction must be performed for methods such as LAMP and by implementing this in the model, its overall on-site applicability might be a bit lower. On the other hand, with available small portable qPCR machines, simplified DNA extraction protocols, and inhibitor-tolerant enzymes for qPCR, a score for qPCR might be a bit higher for on-site detection.

There is actually no limit for the complexity of attributes and rules, which could also include some more laboratory-based observations, such as trust in the reagents (e.g., variability between batches), number and reliability of available reference methods, and amount of costly training needed. Individual laboratories might even put more weight to specific attributes when selecting a series of methods. Such additions to the model could thus provide a substantial contribution to the final evaluation. Since this manuscript compared only the methods that are already publicly available, it is really important that the model offers the possibility of modifying the attributes with emergence of new methods and requirements.

The *AnalyticalMeth* model as presented here has some clear limitations. It can inform but not conclude on which methods to combine for specific aims of GMO testing (e.g., detection only, identification, quantification or detection of unauthorized GMOs). It provides information on a general comparison of individual methods based on their purpose in separate steps of GMO analysis. But it remains open and flexible for future changes that could also set the premises for such even more complex evaluations. The *AnalyticalMeth* model

file is available as Online Resource 1 and can be opened, viewed, and changed using DEXi (downloadable from http://kt.ijs.si/MarkoBohanec/dexi.html).

To take one step further, beyond current GMO detection, MCDA could, for instance, consider including epigenomic as well as epitranscriptomic detection by, e.g., sequencing, to be combined with genetic data for, e.g., detecting plants produced with new breeding techniques. Values could be defined for stakeholders who are looking only to detect the products and identification sets for enforcement laboratories, which could be interested in identifying the patent owner of a product or the genome or epigenome modification (i.e., modifications of DNA, associated proteins, and/or RNA) technique used.

## Conclusions

The idea of the developed MCDA model was to integrate the evaluation of different GMO detection methods in a decision support system that is operational and easily accessible for various categories of users and that provides data and advice for decision problems that occur in supply chains involving GMOs. In principle, the models’ objective was to provide a tool to assess “decision alternatives,” to change decision-related parameters and investigate their effects, to visualize the results of evaluations and analyses, and to maintain data related to the decisions involved. We have shown that use of the model can objectively evaluate different kinds of methods that can help in selecting the best for the purpose of interest. Due to the adaptability of the models’ generic structure, it can be easily modified for evaluation of methods in other fields.

**Acknowledgements** We thank Jeroen van Dijk and Bjørn Spilsberg for sharing information on unpublished NGS-based methods.

**Funding** The research leading to these results has received funding from the European Union under FP7 grant agreement no. 613908 (project DECATHLON) and FP6 contract no. 007158 (Project Co-Extra). Complementary financial support was given by the Slovenian Research Agency (contract number P4-0165) and the Research Council of Norway.

## Compliance with Ethical Standards

**Conflict of Interest** David Dobnik declares that he has no conflict of interest. Kristina Gruden declares that she has no conflict of interest. Jana Žel declares that she has no conflict of interest. Yves Bertheau declares that he has no conflict of interest. Arne Holst-Jensen declares that he has no conflict of interest. Marko Bohanec declares that he has no conflict of interest.

**Ethical Approval** This article does not contain any studies with human participants or animals performed by any of the authors.

**Informed Consent** Not applicable.
**Open Access** This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## References

Acutis M, Trevisiol P, Confalonieri R, Bellocchi G, Grazioli E, van den Eede G, Paoletti C (2007) Analytical method performance evaluation (AMPE)—a software tool for analytical method validation. J AOAC Int 90:1432–1438

Alary R, Serin A, Maury D, Ben Jouira H, Sirven JP, Gautier MF, Joudrier P (2002) Comparison of simplex and duplex real-time PCR for the quantification of GMO in maize and soybean. Food Control 13: 235–244. https://doi.org/10.1016/S0956-7135(02)00015-4

Arulandhu AJ, van Dijk JP, Dobnik D, Holst-Jensen A, Shi J, Zel J, Kok EJ (2016) DNA enrichment approaches to identify unauthorized genetically modified organisms (GMOs). Anal Bioanal Chem 408: 4575–4593. https://doi.org/10.1007/s00216-016-9513-0

Bahrdt C, Krech AB, Wurz A, Wulff D (2010) Validation of a newly developed hexaplex real-time PCR assay for screening for presence of GMOs in food, feed and seed. Anal Bioanal Chem 396:2103–2112. https://doi.org/10.1007/s00216-009-3380-x

Berdal KG, Bøydler C, Tengs T, Holst-Jensen A (2008) A statistical approach for evaluation of PCR results to improve the practical limit of quantification (LOQ) of GMO analyses (SIMQUANT). Eur Food Res Technol 227:1149–1157. https://doi.org/10.1007/s00217-008-0830-1

Bertheau Y (2013) Genetically modified and non-genetically modified food supply chains: co-existence and traceability. Wiley-Blackwell, Oxford

Block A, Debode F, Grohmann L, Hulin J, Taverniers I, Kluga L, Barbau-

Piednoir E, Broeders S, Huber I, van den Bulcke M, Heinze P, Berben G, Busch U, Roosens N, Janssen E, Žel J, Gruden K, Morisset D (2013) The GMOseek matrix: a decision support tool for optimizing the detection of genetically modified plants. BMC Bioinformatics 14:256. https://doi.org/10.1186/1471-2105-14-256

Bohanec M (2015) DEXi: program for multi-attribute decision making, user's manual, version 5.00. IJS report DP-11897. Ljubljana

Bohanec M, Bertheau Y, Brera C, Gruden K, Holst-Jensen A, Kok EJ, Lécroart B, Messéan A, Miraglia M, Onori R, Prins TW, Soler LG, Žnidaršič M (2013a) The co-extra decision support system: a model-based integration of project results. In: Genetically modified and non-genetically modified food supply chains: co-existence and traceability. Wiley-Blackwell, Oxford, pp 459–489. https://doi.org/10.1002/9781118373781.ch25

Bohanec M, Žnidaršič M, Rajkovič V et al (2013b) DEX methodology: three decades of qualitative multi-attribute modeling. Inform 37:49–54

Bohanec M, Boshkoska BM, Prins TW, Kok EJ (2017) SIGMO: a decision support system for identification of genetically modified food or feed products. Food Control 71:168–177. https://doi.org/10.1016/j.foodcont.2016.06.032

Chaouachi M, Chupeau G, Berard A et al (2008) A high-throughput multiplex method adapted for GMO detection. J Agric Food Chem 56:11596–11606. https://doi.org/10.1021/jf801482r

Chen L, Guo J, Wang Q, Kai G, Yang L (2011) Development of the visual loop-mediated isothermal amplification assays for seven genetically modified maize events and their application in practical samples analysis. J Agric Food Chem 59:5914–5918. https://doi.org/10.1021/jf200459s

Dobnik D, Spilsberg B, Bogožalec Košir A, Holst-Jensen A, Žel J (2015) Multiplex quantification of 12 European Union authorized genetically modified maize lines with droplet digital polymerase chain

reaction. Anal Chem 87:8218–8226. https://doi.org/10.1021/ac501208

Dobnik D, Štebih D, Blejec A, Morisset D, Žel J (2016) Multiplex quantification of four DNA targets in one reaction with Bio-Rad droplet digital PCR system for GMO detection. Sci Rep 6. https://doi.org/10.1038/srep35451

EURL-GMFF (2005) Detection method for event Bt 10 using a qualitative PCR assay. http://gmo-crl.jrc.ec.europa.eu/summaries/Bt10. Detection Protocol.pdf. Accessed 24 May 2017

EURL-GMFF (2009) NOST-Spec construct-specific method for the detection of CDC Trifid Flax (Event FP967) using real-time PCR. http://gmo-crl.jrc.ec.europa.eu/doc/Flax-CDCTriffi. Accessed 24 May 2017

European Commission (2003) Regulation (EC) No 1829/2003 of the European Parliament and of the Council of 22 September 2003 on genetically modified food and feed. Off J Eur Union L 1–23

Germini A, Zanetti A, Salati C, Rossi S, Forré C, Schmid S, Marchelli R, Fogher C (2004) Development of a seven-target multiplex PCR for the simultaneous detection of transgenic soybean and maize in feeds and foods. J Agric Food Chem 52:3275–3280. https://doi.org/10.1021/jf035052x

Goldfeder RL, Priest JR, Zook JM, Grove ME, Waggott D, Wheeler MT, Salit M, Ashley EA (2016) Medical implications of technical accuracy in genome sequencing. Genome Med 8:24. https://doi.org/10.1186/s13073-016-0269-0

Greco S, Ehrogott M, Figueira J (2016) Multiple criteria decision analysis. Springer, New York. https://doi.org/10.1007/978-1-4939-3094-4

Gruère GP, Rao SR (2007) A review of international labeling policies of genetically modified food to evaluate India's proposed rule 10:51–64

Hamels S, Glouden T, Gillard K, Mazzara M, Debode F, Foti N, Sneyers M, Esteve Nuez T, Pla M, Berben G, Moens W, Bertheau Y, Audéon C, van den Eede G, Remacle J (2009) A PCR-microarray method for the screening of genetically modified organisms. Eur Food Res Technol 228:531–541. https://doi.org/10.1007/s00217-008-0960-5

Holck A, Vaïtilingom M, Didierjean L, Rudi K (2002) 5'-Nuclease PCR for quantitative event-specific detection of the genetically modified Mon810 MaisGard maize. Eur Food Res Technol 214:449–454. https://doi.org/10.1007/s00217-001-0473-y

Holck A, Pedersen BO, Heir E (2010) Detection of five novel GMO maize events by qualitative, multiplex PCR and fluorescence capillary gel electrophoresis. Eur Food Res Technol 231:475–483. https://doi.org/10.1007/s00217-010-1302-y

Holst-Jensen A (2009) Testing for genetically modified organisms (GMOs): past, present and future perspectives. Biotechnol Adv 27: 1071–1082. https://doi.org/10.1016/j.biotechadv.2009.05.025

Holst-Jensen A, Bertheau Y, de Loose M, Grohmann L, Hamels S, Hougs L, Morisset D, Pecoraro S, Pla M, den Bulcke MV, Wulff D (2012) Detecting un-authorized genetically modified organisms (GMOs) and derived materials. Biotechnol Adv 30:1318–1335. https://doi.org/10.1016/j.biotechadv.2012.01.024

Holst-Jensen A, Spilsberg B, Arulandhu AJ, Kok E, Shi J, Zel J (2016) Application of whole genome shotgun sequencing for detection and characterization of genetically modified organisms and derived products. Anal Bioanal Chem 408:4595–4614. https://doi.org/10.1007/s00216-016-9549-1

Huber I, Block A, Sebah D, Debode F, Morisset D, Grohmann L, Berben G, Štebih D, Milavec M, Žel J, Busch U (2013) Development and validation of duplex, triplex, and pentaplex real-time PCR screening assays for the detection of genetically modified organisms in food and feed. J Agric Food Chem 61:10293–10301. https://doi.org/10.1021/jf402448y

Ishizaka A, Nemery P (2013) Multi-criteria decision analysis: methods and software. Wiley, Somerset

James C (2015) Global status of commercialized biotech/GM crops: 2015. ISAAA brief no. 51. Ithaca, NY
Jasbeer K, Son R, Mohamad Ghazali F, Cheah YK (2009) Real-time PCR evaluation of seven DNA extraction methods for the purpose of GMO analysis. Int Food Res J 16:329–341

Kiddle G, Hardinge P, Buttigieg N, Gandelman O, Pereira C, McElgunn
CJ, Rizzoli M, Jackson R, Appleton N, Moore C, Tisi LC, Murray
JAH (2012) GMO detection using a bioluminescent real time report-
er (BART) of loop mediated isothermal amplification (LAMP) suit-
able for field use. BMC Biotechnol 12:15. https://doi.org/10.1186/
1472-6750-12-15

Kobilinsky A, Bertheau Y (2005) Minimum cost acceptance sampling plans for grain control, with application to GMO detection. Chemom Intell Lab Syst 75:189–200. https://doi.org/10.1016/j.chemolab.2004.07.005

Kontić B, Bohanec M, Kontić D, Trdin N, Matko M (2016) Improving appraisal of sustainability of energy options—a view from Slovenia. Energy Policy 90:154–171. https://doi.org/10.1016/j.enpol.2015.12.022

Košir AB, Spilsberg B, Holst-Jensen A, Žel J, Dobnik D (2017) Development and inter-laboratory assessment of droplet digital PCR assays for multiplex quantification of 15 genetically modified soybean lines. Sci Rep 7:8601. https://doi.org/10.1038/s41598-017-09377-w

Kuribara H, Shindo Y, Matsuoka T, Takubo K, Futo S, Aoki N, Hirao T, Akiyama H, Goda Y, Toyoda M, Hino A (2002) Novel reference molecules for quantitation of genetically modified maize and soybean. J AOAC Int 85:1077–1089

Leimanis S, Hamels S, Nazé F, Mbongolo Mbella G, Sneyers M, Hochegger R, Broll H, Roth L, Dallmann K, Micsinai A, la Paz JL, Pla M, Brünen-Nieweler C, Papazova N, Taverniers I, Hess N, Kirschneit B, Bertheau Y, Audeon C, Laval V, Busch U, Pecoraro S, Neumann K, Rösel S, van Dijk J, Kok E, Bellocchi G, Foti N, Mazzara M, Moens W, Remacle J, van den Eede G (2008) Validation of the performance of a GMO multiplex screening assay based on microarray detection. Eur Food Res Technol 227:1621–1632. https://doi.org/10.1007/s00217-008-0886-y

Liu L, Li Y, Li S, Hu N, He Y, Pong R, Lin D, Lu L, Law M (2012) Comparison of next-generation sequencing systems. J Biomed Biotechnol 2012:1–11. https://doi.org/10.1155/2012/251364

Luan FX, Tao R, Xu YG, Wu J, Guan XJ (2012) High-throughput detection of genetically modified rice ingredients in foods using multiplex polymerase chain reaction coupled with high-performance liquid chromatography method. Eur Food Res Technol 234:649–654. https://doi.org/10.1007/s00217-012-1671-5

Milavec M, Dobnik D, Yang L, Zhang D, Gruden K, Žel J (2014) GMO quantification: valuable experience and insights for the future. Anal Bioanal Chem 406:6485–6497. https://doi.org/10.1007/s00216-014-8077-0

Morisset D, Štebih D, Milavec M, Gruden K, Žel J (2013) Quantitative analysis of food and feed samples with droplet digital PCR. PLoS One 8:e62583. https://doi.org/10.1371/journal.pone.0062583

Nadal A, Coll A, La Paz JL et al (2006) A new PCR-CGE (size and color) method for simultaneous detection of genetically modified maize events. Electrophoresis 27:3879–3888. https://doi.org/10.1002/elpss.200600124

Pakpour S (2012) A multi-criteria decision-making approach for compar-
ing sample preservation and DNA extraction methods from swine
feces. Am J Mol Biol 2:159–169. https://doi.org/10.4236/ajmb.
2012.22018

Pennisi E (2017) Pocket-sized sequencers start to pay off big. Science (80-). https://doi.org/10.1126/science.356.6338.572

Pla M, Nadal A, Baeten V, Bahrdt C, Berben G, Bertheau Y, Coll A, van Dijk JP, Dobnik D, Fernandez Pierna JA, Gruden K, Hamels S, Holck A, Holst-Jensen A, Janssen E, Kok EJ, la Paz JL, Laval V, Leimanis S, Malcevschi A, Marmiroli N, Morisset D, Prins TW, Remacle J, Ujhelyi G, Wulff D (2013) New multiplexing tools for reliable GMO detection. In: Bertheau Y (ed) Genetically modified and non-genetically modified food supply chains: co-existence and traceability. Wiley-Blackwell, Oxford, pp 333–366. https://doi.org/10.1002/9781118373781.ch19

Potapov V, Ong JL (2017) Examining sources of error in PCR by single-molecule sequencing. PLoS One 12:e0169774. https://doi.org/10.1371/journal.pone.0169774

Randhawa GJ, Chhabra R, Singh M (2009) Multiplex PCR-based simul-
taneous amplification of selectable marker and reporter genes for the
screening of genetically modified crops. J Agric Food Chem 57:
5167-5172. https://doi.org/10.1021/jf900604h

Randhawa GJ, Chhabra R, Singh M (2010) Decaplex and real-time PCR based detection of MON531 and MON15985 Bt cotton events. J Agric Food Chem 58:9875-9881. https://doi.org/10.1021/jf100466n

Remund KM, Dixon DA, Wright DL, Holden LR (2001) Statistical considerations in seed purity testing for transgenic traits. Seed Sci Res 11:101–119. https://doi.org/10.1079/SSR200166

Takabatake R, Koiwa T, Kasahara M et al (2011) Interlaboratory validation of quantitative duplex real-time PCR method for screening analysis of genetically modified maize. Food Hyg Saf Sci (Shokuhin Eiseigaku Zasshi) 52:265–269. https://doi.org/10.3358/shokueishi.52.265

Van Den Bulcke M, De Schrijver A, De Bernardi D et al (2007) Detection of genetically modified plant products by protein strip testing: an evaluation of real-life samples. Eur Food Res Technol 225:49–57. https://doi.org/10.1007/s00217-006-0381-2

Van den Bulcke M, Lievens A, Barbau-Piednoir E et al (2010) A theoretical introduction to “Combinatory SYBR@Green qPCR Screening”, a matrix-based approach for the detection of materials derived from genetically modified plants. Anal Bioanal Chem 396: 2113–2123. https://doi.org/10.1007/s00216-009-3286-7

Vigani M, Raimondi V, Olper A (2012) International trade and endoge-
nous standards: the case of GMO regulation. World Trade Rev 11:
415-437

Volk H, Piskernik S, Kurinčič M et al (2014) Evaluation of different methods for DNA extraction from milk. J Food Nutr Res 53:97–105

Waiblinger H-U, Ernst B, Anderson A, Pietsch K (2007) Validation and collaborative study of a P35S and T-nos duplex real-time PCR screening method to detect genetically modified organisms in food products. Eur Food Res Technol 226:1221–1228. https://doi.org/10.1007/s00217-007-0748-z

Wang C, Li R, Quan S, Shen P, Zhang D, Shi J, Yang L (2015) GMO detection in food and feed through screening by visual loop-mediated isothermal amplification assays. Anal Bioanal Chem 407:4829–4834. https://doi.org/10.1007/s00216-015-8652-z

Willems S, Fraiture M-A, Deforce D, de Keersmaecker SCJ, de Loose M, Ruttink T, Herman P, van Nieuwerburgh F, Roosens N (2016) Statistical framework for detection of genetically modified organisms based on next generation sequencing. Food Chem 192:788–798. https://doi.org/10.1016/j.foodchem.2015.07.074