# Revised annual post-market environmental monitoring (PMEM) report on the cultivation of genetically modified maize MON 810 in 2013 from Monsanto Europe S.A

Andrew Nicholas Birch, Josep Casacuberta, Adinda de Schrijver, Achim Gathmann, Mikolaj Antoni Gralak, Philippe Guerche, Huw Jones, Barbara Manachini, Antoine Messéan, Hanspeter Naegeli, et al.

► To cite this version:

Andrew Nicholas Birch, Josep Casacuberta, Adinda de Schrijver, Achim Gathmann, Mikolaj Antoni Gralak, et al.. Revised annual post-market environmental monitoring (PMEM) report on the cultivation of genetically modified maize MON 810 in 2013 from Monsanto Europe S.A. EFSA Journal, 2015, 13 (11), 10.2903/j.efsa.2015.4295. hal-01602124

HAL Id: hal-01602124

https://hal.science/hal-01602124

Submitted on 27 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
ADOPTED: 28 October 2015

doi:10.2903/j.efsa.2015.4295

PUBLISHED: 17 November 2015

# Revised annual post-market environmental monitoring (PMEM) report on the cultivation of genetically modified maize MON 810 in 2013 from Monsanto Europe S.A.

## EFSA Panel on Genetically Modified Organisms (GMO)

### Abstract

Following a request from the European Commission, the Panel on Genetically Modified Organisms of the European Food Safety Authority (EFSA GMO Panel) assessed the results of the general surveillance activities contained in the revised annual post-market environmental monitoring (PMEM) report for the 2013 growing season of maize MON 810 provided by Monsanto Europe S.A. The supplied data do not indicate any unanticipated adverse effects on human and animal health or the environment arising from the cultivation of maize MON 810 cultivation in 2013. Similar methodological shortcomings to those observed in previous annual PMEM reports were identified in the analysis of farmer questionnaires and the conduct of the literature review. The EFSA GMO Panel therefore strongly reiterates its previous recommendations to improve the methodology of future annual PMEM reports on maize MON 810. The EFSA GMO Panel urges the applicant to consider how to make best use of the information recorded in national registers in order to optimise sampling for farmer questionnaires, reiterates its previous recommendations on insect resistance monitoring and continued screening, and requests to continue reviewing and discussing relevant scientific publications on possible adverse effects of maize MON 810 on rove beetles. Also, the EFSA GMO Panel encourages relevant parties to continue developing a methodological framework to use existing networks in the broader context of environmental monitoring.

© European Food Safety Authority, 2015

**Keywords:** Cry1Ab, farmer questionnaires, general surveillance, literature searches, risk assessment, Zea mays

**Requestor:** European Commission

**Question number:** EFSA-Q-2015-00432

**Correspondence:** gmo@efsa.europa.eu
**Panel members:** Andrew Nicholas Birch, Josep Casacuberta, Adinda De Schrijver, Achim Gathmann, Mikolaj Gralak, Philippe Guerche, Huw Jones, Barbara Manachini, Antoine Messéan, Hanspeter Naegeli, Elsa Ebbesen Nielsen, Fabien Nogué, Christophe Robaglia, Nils Rostoks, Jeremy Sweet, Christoph Tebbe, Francesco Visioli, Jean-Michel Wal.

**Competing interests:** In line with EFSA's policy on declarations of interest, Panel member Achim Gathmann did not participate in the development and adoption of this scientific output.

**Acknowledgements:** The Panel wishes to thank the members of the Working Group on annual PMEM reports, Salvatore Arpaia, Barbara Manachini, Antoine Messéan and Jeremy Sweet, for the preparatory work on this scientific and EFSA staff members, Fernando Álvarez, Michele Ardizzone, Yann Devos, Marios Georgiadis, Ana Gomes, Anna Lanzoni and Jane Richardson, for the support provided to this scientific output.

**Suggested citation:** EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2015. Scientific opinion on the revised annual post-market environmental monitoring (PMEM) report on the cultivation of genetically modified maize MON 810 in 2013 from Monsanto Europe S.A. EFSA Journal 2015;13(11):4295, 37 pp. doi:10.2903/j.efsa.2015.4295

**ISSN:** 1831-4732

© European Food Safety Authority, 2015

Reproduction is authorised provided the source is acknowledged.
## Summary

Following a request from the European Commission, the Panel on Genetically Modified Organisms of the European Food Safety Authority (EFSA GMO Panel) assessed the results of the general surveillance (GS) activities contained in the revised annual post-market environmental monitoring report on the cultivation of maize MON 810 during the 2013 growing season provided by Monsanto Europe S.A.

The EFSA GMO Panel assessed the 2013 GS dataset on maize MON 810, which consists of a survey based on 256 questionnaires received from farmers in four European countries, peer-reviewed publications relevant to the risk assessment and/or management of maize MON 810 (published between June 2013 and the beginning of June 2014), and alerts on environmental issues issued by regulatory authorities and existing surveillance networks. To identify relevant publications not reported by the applicant, the EFSA GMO Panel performed a literature search and assessed the relevance of retrieved scientific publications for the safety of maize MON 810. The available data do not indicate any unanticipated adverse effects on human and animal health or the environment arising from the cultivation of maize MON 810 during the 2013 growing season. Therefore, the EFSA GMO Panel considers that its previous conclusions on the safety of maize MON 810 remain valid and applicable.

Similar methodological shortcomings to those found in previous annual PMEM reports on maize MON 810 were identified by the EFSA GMO Panel in the analysis of farmer questionnaires and conduct of the literature review. The EFSA GMO Panel therefore strongly reiterates its recommendations to provide more detailed information on the sampling methodology and to reduce the possibility of selection bias in farmer questionnaires, and to ensure that all relevant scientific publications are identified. In order to improve the sampling frame of the farmer survey, the EFSA GMO Panel reiterates the importance of national GMO cultivation registers; and its recommendations to applicants to consider how they may make best use of the information recorded in national registers and foster dialogue with those responsible for the administration of these registers where maize MON 810 is cultivated.

No information collected from existing monitoring networks in the EU was provided by the applicant. However, the EFSA GMO Panel notes that initiatives have been taken to develop a methodological framework to use existing networks in the broader context of environmental monitoring, and encourages relevant parties to continue to use these.

The outcome of the literature review reinforces previous recommendations on insect resistance monitoring as part of the insect resistance management plan, and to use farmer questionnaires as a tool to capture early warning signs indicating increases in tolerance in populations of non-target lepidopteran pests in the field.

With regard to rove beetles (Coleoptera: Staphylinidae), the applicant is requested to continue screening, reviewing and discussing relevant scientific publications on possible adverse effects of maize MON 810 on rove beetles.
Table of contents

<table><tr><td>Abstract</td><td>1</td></tr><tr><td>Summary</td><td>3</td></tr><tr><td>1. Introduction</td><td>5</td></tr><tr><td>1.1. Background and Terms of Reference provided by the requestor</td><td>5</td></tr><tr><td>2. Data and Methodologies</td><td>6</td></tr><tr><td>2.1. Data</td><td>6</td></tr><tr><td>2.2. Methodologies</td><td>7</td></tr><tr><td>3. Assessment</td><td>7</td></tr><tr><td>3.1. Farmer questionnaires</td><td>7</td></tr><tr><td>3.2. Existing monitoring networks</td><td>8</td></tr><tr><td>3.3. Literature review</td><td>8</td></tr><tr><td>3.3.1. Relevant publications reported by the applicant</td><td>8</td></tr><tr><td>3.3.2. Additional publications identified by the EFSA GMO Panel</td><td>9</td></tr><tr><td>3.3.3. Conclusions of the literature review</td><td>11</td></tr><tr><td>4. Conclusions</td><td>12</td></tr><tr><td>5. Recommendations</td><td>12</td></tr><tr><td>Documentation provided to EFSA</td><td>12</td></tr><tr><td>References</td><td>13</td></tr><tr><td>Abbreviations</td><td>17</td></tr><tr><td>Appendix A – Peer-reviewed scientific publications relevant to the risk assessment and/or management of maize MON 810 assessed by the EFSA GMO Panel as part of the revised annual 2013 PMEM report on the cultivation of maize MON 810</td><td>18</td></tr><tr><td>Annex A – EFSA AMU Unit technical report on the evaluation of farmer questionnaires</td><td>21</td></tr></table>
1. Introduction

The event MON 810 has been introduced into a wide range of maize varieties that have been
cultivated in the European Union (EU) since 2003. Maize MON 810 produces the insecticidal protein
Cry1Ab from *Bacillus thuringiensis* (Bt), which confers resistance to certain lepidopteran pests, such as
the European corn borer (ECB), *Ostrinia nubilalis* (Hübner) (Lepidoptera: Crambidae), and the
Mediterranean corn borer (MCB), *Sesamia nonagrioides* (Lefebvre) (Lepidoptera: Noctuidae). In 2013,
maize MON 810 was grown in Spain (136 962 ha), Portugal (8 202 ha), the Czech Republic
(2 560 ha), Romania (835 ha) and Slovakia (100 ha) over a total area of approximately 148 659 ha.

According to Articles 13 and 20 of Directive 2001/18/EC (EC, 2001)<fnref n="1" />, each notification for placing on the market of a genetically modified organism (GMO) shall contain a plan for monitoring in accordance with Annex VII of the Directive. Similarly, according to Articles 5.5(b) and 17.5(b) of Regulation (EC) No 1829/2003 (EC, 2003)<fnref n="2" />, each application for the placing on the market of a GMO or food/feed containing or consisting of that GMO shall be accompanied by a monitoring plan for environmental effects conforming with Annex VII to Directive 2001/18/EC. Annex VII was supplemented by notes providing guidance on the objectives, general principles and design of the monitoring plan (EC, 2002)<fnref n="3" />.

Results of post-market environmental monitoring (PMEM) activities on the cultivation of maize MON 810 in the EU are reported to the European Commission (EC) and Member States on an annual basis by the applicant (Monsanto Europe S.A.). Since 2010, the Scientific Panel on Genetically Modified Organisms of the European Food Safety Authority (hereafter referred to as EFSA GMO Panel) assesses these annual PMEM reports on the cultivation of maize MON 810 (EFSA GMO Panel, 2011a, 2012a, 2013, 2014a, 2015).

1.1. Background and Terms of Reference provided by the requestor

The marketing of maize MON 810 (notification C/F/95/12-02) was authorised under Directive 90/220/EEC in the EU for all, other than food, uses by the Commission Decision 98/294/EC of 22 April 1998 (EC, 1998)<fnref n="4" />. Consent was granted to the applicant (Monsanto Europe S.A.) on 3 August 1998 by the Competent Authority of France. Food uses of maize derivatives were notified according to Article 5 of the Novel Food Regulation (EC) No 258/97 on 6 February 1998.

Following the request by the applicant for the renewal of the authorisation for placing maize MON 810 on the market, the EFSA GMO Panel adopted a scientific opinion on the renewal under Regulation (EC) No 1829/2003 of maize MON 810 for: existing food and food ingredients produced from maize MON 810; feed consisting of and/or containing maize MON 810, including the use of seed for cultivation; and food and feed additives, and feed materials produced from maize MON 810 (EFSA, 2009). The EFSA GMO Panel concluded that 'maize MON 810 is as safe as its conventional counterpart with respect to potential effects on human and animal health', and that 'maize MON 810 is unlikely to have any adverse effect on the environment in the context of its intended uses, especially if appropriate management measures are put in place in order to mitigate possible exposure of non-target (NT) Lepidoptera'. The EFSA GMO Panel recommended that 'especially in areas of abundance of NT Lepidoptera populations, the adoption of the cultivation of maize MON 810 be accompanied by management measures in order to mitigate the possible exposure of these species to maize MON 810 pollen'. In addition, the EFSA GMO Panel advised that 'resistance management strategies continue to be employed and that the evolution of resistance in lepidopteran target pests continues to be monitored, in order to detect potential changes in resistance levels in pest populations' (EFSA, 2009).

<footnote n="1">¹ EC (European Commission), 2001. Directive 2001/18/EC of the European Parliament and of the Council of 12 March 2001 on the deliberate release into the environment of genetically modified organisms and repealing Council Directive 90/220/EEC. OJ L 106, 17.04.2001, 1–39.</footnote>

<footnote n="2">² EC (European Commission), 2003. Regulation (EC) No 1829/2003 of the European Parliament and of the Council of 22 September 2003 on genetically modified food and feed. OJ L 268, 18.10.2003, 1–23.</footnote>

<footnote n="3">³ EC (European Commission), 2002. Council Decision 2002/811 of 3 October 2002 establishing guidance notes supplementing Annex VII to Directive 2001/18/EC of the European Parliament and of the Council on the deliberate release into the environment of genetically modified organisms and repealing Council Directive 90/220/EEC. OJ L 280, 18.10.2002, 27–36.</footnote>

<footnote n="4">⁴ EC (European Commission), 1998. Commission Decision of 22 April 1998 concerning the placing on the market of genetically modified maize (Zea mays L. line MON 810), pursuant to Council Directive 90/220/EEC (98/294/EC). OJ L 131, 05.05.1998, 32–33.</footnote>
From 2005 onwards, the applicant submitted to the EC its PMEM reports on the cultivation of maize MON 810 according to the provisions of Directive 2001/18/EC (EC, 2001). These annual PMEM reports are composed of case-specific monitoring (CSM), to assess the efficacy of the 'high dose/refuge' strategy, and general surveillance (GS), to detect unanticipated adverse effects caused by the cultivation of maize MON 810.

Since 2010, the EC requested the EFSA GMO Panel to assess the annual PMEM reports on the cultivation of maize MON 810 submitted by Monsanto Europe S.A. The EFSA GMO Panel therefore adopted scientific opinions on the 2009, 2010, 2011 and 2012 annual PMEM reports (EFSA GMO Panel, 2011a, 2012a, 2013, 2014a). From the data provided in these reports, the EFSA GMO Panel did not identify adverse effects on the environment, human and animal health resulting from the cultivation of maize MON 810. However, the EFSA GMO Panel noted shortcomings in the methodology for CSM and GS, and made recommendations to improve future annual PMEM reports on maize MON 810.

On 22 May 2012, the EC requested EFSA to compile an inventory of existing environmental surveillance networks at European and national level, and to develop a set of assessment criteria to support the selection of such networks for PMEM of GM plants. Following this request, an external open call was launched by the EFSA Unit for Assessment and Methodological Support (EFSA AMU Unit). The external report reviewed statistical methods used in the analysis of ecological and environmental datasets; provided an inventory of statistical approaches in ecological and environmental monitoring and identification of data requirements for the items in the inventory; provided an inventory of European, National and Regional existing surveillance networks/programmes; and gave recommendations of the most appropriate analysis methodologies for PMEM of agro-ecosystems (Centre for Ecology and Hydrology et al., 2014).

On 14 November 2014, the EC asked the EFSA GMO Panel to assess the PMEM report on the cultivation of maize MON 810 during the 2013 growing season submitted by Monsanto Europe S.A. The report contained information on CSM, but not on GS activities. In its scientific opinion, the EFSA GMO Panel could therefore not conclude on potential unanticipated adverse effects arising from the cultivation of maize MON 810 in 2013, or on possible changes of the methodology on the GS as compared to previous growing seasons (EFSA GMO Panel, 2015).

On 24 March 2015, the EC requested EFSA to assess the concerns raised by Monsanto about the previous EFSA GMO Panel recommendations on the insect resistance management (IRM) strategy of maize MON 810. EFSA concluded that the previous conclusions and recommendations by the EFSA GMO Panel remain valid (EFSA, 2015a).

On 24 April 2015, the EC received from the applicant a revised monitoring report for the 2013 cultivation season of maize MON 810, which includes information on GS activities.

On 18 June 2015, the EC requested the EFSA GMO Panel 'to evaluate the findings of the general surveillance activities, taking into consideration the comments received from Member States and to assess the appropriateness of the methodology if this is found to differ compared to the previous season'.

## 2. Data and Methodologies

### 2.1. Data

In delivering this scientific opinion, the EFSA GMO Panel took into account the information on GS activities provided by the applicant:<fnref n="5" />

* a survey based on 256 questionnaires received from farmers in four European countries:<fnref n="6" /> 190 in Spain, 46 in Portugal, 18 in the Czech Republic and 2 in Romania;

<footnote n="5">⁵ The literature review on possible adverse effects of maize MON 810 on rove beetles (Coleoptera: Staphylinidae) was submitted by the applicant in its first annual 2013 PMEM report and already assessed by the EFSA GMO Panel (EFSA GMO Panel, 2015).</footnote>

<footnote n="6">⁶ Revised annual 2013 PMEM report, Appendix 1.</footnote>
* a list of peer-reviewed publications relevant to the risk assessment and/or management of maize MON 810, which were published between June 2013 and the beginning of June 2014;<fnref n="7" />

* company stewardship activities;<fnref n="8" /> and

* alerts on environmental issues by regulatory authorities and existing surveillance networks.

In addition, the EFSA GMO Panel assessed additional relevant peer-reviewed publications between June 2013 and the beginning of June 2014 that were not included in the revised annual 2013 PMEM report supplied by the applicant.

## 2.2. Methodologies

Following the terms of reference of the EC mandate, the EFSA GMO Panel considered whether the GS methodology applied during the 2013 growing season of maize MON 810 differs from that followed in the previous PMEM reports on maize MON 810 (EFSA GMO Panel, 2011a, 2012a, 2013, 2014a).

The EFSA GMO Panel assessed the new 2013 GS dataset on maize MON 810 (see section 3).

In addition, the EFSA GMO Panel performed a literature search to identify relevant scientific publications not reported by the applicant, and subsequently assessed their relevance for the risk assessment and/or risk management of maize MON 810.

In its assessment, the EFSA GMO Panel also considered the comments from Member States on the revised annual 2013 PMEM report.<fnref n="9" />

## 3. Assessment

The EFSA GMO Panel assessed the methodology and results of the GS activities on maize MON 810 reported in the revised annual 2013 PMEM report,<fnref n="10" /> taking into consideration Member States' comments on this report.

The EFSA GMO Panel assessed the reported GS data originating from farmer questionnaires, existing monitoring networks and scientific literature.

### 3.1. Farmer questionnaires

In its revised annual 2013 PMEM report, the applicant submitted a survey completed between December 2013 and March 2014 based on 256 questionnaires received from farmers in four European countries: 190 in Spain, 46 in Portugal, 18 in the Czech Republic and 2 in Romania. The applicant concluded that the analysis of the questionnaires 'did not reveal any unanticipated adverse effects that could be associated with the genetic modification in MON 810'.

The EFSA GMO Panel, in close collaboration with the EFSA AMU Unit, assessed the methodology followed by the applicant to analyse the farmer questionnaires. Alongside the methodological guidance for a systematic evaluation of farmer questionnaires, the evaluation of the overall 2013 farmer's survey (including, for example, sampling of farmers, types of questions, method of conduct interviews, data validation, method used for the design of the statistical analysis) is given in Annex A.

The methodology followed by the applicant to identify unanticipated adverse effects caused by cultivation of maize MON 810 through the use of farmer questionnaires did not differ from previous annual PMEM reports (EFSA GMO Panel, 2011a, 2012a, 2013, 2014a). Similar weaknesses in the methodology as in previous annual PMEM reports were observed, and recommendations to the applicant for the improvement of the methodology are listed in Annex A.

The EFSA GMO Panel examined the results of the analysis of the 2013 farmer questionnaires on maize MON 810, and concludes that there is no indication that unanticipated adverse effects have occurred.

<footnote n="7">⁷ Revised annual 2013 PMEM report, Appendices 5.1 and 5.2.</footnote>

<footnote n="8">⁸ Revised annual 2013 PMEM report, Appendices 3.1 to 3.5.</footnote>

<footnote n="9">⁹ Comments were received from Austria, Finland, Germany, the United Kingdom and the Netherlands.</footnote>

<footnote n="10">¹⁰ The revised annual 2013 PMEM report on maize MON 810 is published online at: http://ec.europa.eu/food/plant/docs/plant_gmo_report_studies_report_2013_mon_810_revised_en.pdf</footnote>
## 3.2. Existing monitoring networks

Directive 2001/18/EC and Council Decision 2002/811/EC propose to make use of existing monitoring networks, as it complements farmer questionnaires and provides an additional tool for the GS of GM plants. Member States have various networks in place—some of which have a long history of data collection—that may be helpful in the context of GS of GM plants. The networks involved in routine monitoring offer recognised expertise in a specific domain and have the tools to capture information on important environmental aspects over a large geographical area.

The applicant referred to an on-going Europabio project which aims to map existing European monitoring networks, but which has not yet delivered information on possible networks that could be involved in the GS of maize MON 810. Therefore, as in previous years, the applicant provided no information collected from existing monitoring networks in the EU. However, the EFSA GMO Panel notes that initiatives have been taken to develop a methodological framework to use existing networks in the broader context of environmental monitoring (EFSA GMO Panel, 2014b; Centre for Ecology and Hydrology et al., 2014; Smets et al., 2014), and encourages that these are continued by relevant parties (EFSA GMO Panel, 2011b).

## 3.3. Literature review

### 3.3.1. Relevant publications reported by the applicant

The applicant performed a literature search to identify potentially relevant publications published in the peer-reviewed scientific literature between June 2013 and the beginning of June 2014.

The search terms<fnref n="11" /> used in the literature search differed slightly from those used in previous annual PMEM reports; the main difference is that, in previous reports, terms included crops other than maize (e.g. cotton, soybean, rape, potato, brinjal, rice), GM maize events other than MON 810 (e.g. TC1507, 59122) and Bt-proteins other than Cry1Ab (e.g. Cry1F, Cry1Ac).

The EFSA GMO Panel considered that the search terms used by the applicant in its revised annual 2013 PMEM report are adequate to retrieve the relevant scientific publications. The search terms are broad and include synonyms ('tolerant' and 'resistant'), scientific and common names ('maize' and 'Zea mays'), brand and generic names ('Yieldgard' or 'Bt maize'), British and US variants ('maize' and 'corn'), etc. Boolean operators (i.e. OR, AND) were appropriately used to combine terms, while wild card symbols allowed to retrieve variant spellings (e.g. toleran*, protec*).

The applicant used Web of Science<sup>SM12</sup> as the only scientific literature database to identify relevant publications.

The applicant identified 21 scientific publications related to maize MON 810 and/or the Cry1Ab protein published between June 2013 and the beginning of June 2014 (Appendix A). One publication was relevant to the molecular characterisation of maize MON 810, 8 publications were relevant for the food and feed (FF) safety assessment (specifically, animal feeding studies and human *in vitro* tests), and 12 publications pertained to the environmental risk assessment (ERA) or risk management (studies assessing the interaction of maize MON 810 with target organisms (TOs) and non-target organisms (NTOs)). Two of these publications, Campagne et al. (2013) and Twardowski et al. (2014), had already been assessed by EFSA (2014) and its GMO Panel (EFSA GMO Panel, 2015) and did not reveal safety concerns.

The EFSA GMO Panel noted that 12 relevant scientific publications related to Bt-maize/maize MON 810 and/or the Cry1Ab protein (published between June 2013 and beginning June 2014) were not reported by the applicant (see section 3.3.2). Therefore, the EFSA GMO Panel makes the following recommendations:

* The literature search performed by the applicant should adhere to some fundamental principles of systematic review, i.e. methodological rigour and coherence in the retrieval and selection of scientific publications, transparency and reproducibility of the performed literature

<footnote n="11">¹¹ The list of keywords used to query specific literature databases when performing the literature search are given in Table 1 of the revised annual 2013 PMEM report on maize MON 810, Appendix 5.</footnote>

<footnote n="12">¹² http://apps.webofknowledge.com/ (accessed on 10 October 2015).</footnote>
search (EFSA, 2010). In this respect, the applicant did not provide information on the date of the search, the full list of retrieved scientific publications and clear criteria for exclusion/inclusion of relevant scientific publications.

* In addition, the EFSA GMO Panel considers that additional scientific literature databases such as CAB Abstracts®<fnref n="13" /> should have been used to increase the likelihood to retrieve all relevant scientific publications (EFSA, 2015b).

* The protocol for the literature search by the applicant is revised accordingly and supplied to EFSA annually.

The EFSA GMO Panel assessed all the scientific publications selected by the applicant, and acknowledged that these were adequately discussed and put into the context of the overall safety assessment of maize MON 810 in line with previous recommendations made by the EFSA GMO Panel. The EFSA GMO Panel considered that none of the selected publications relating to maize MON 810 and/or the Cry1Ab protein reported adverse effects on human and animal health or the environment. However, the findings reported by Crava et al. (2013) and González-Cabrera et al. (2013) reinforce previous EFSA GMO Panel recommendations to adapt the IRM strategy for target ECB and MCB populations, and to use farmer questionnaires as a tool to capture early warning signs indicating increases in tolerance in populations of non-target lepidopteran pests in the field (for further details, see section 3.3.3).

### 3.3.2. Additional publications identified by the EFSA GMO Panel

EFSA conducted two separate, independent, literature searches to identify additional relevant scientific publications. Several bibliographic databases were queried simultaneously to identify as many relevant peer-reviewed scientific publications as possible. These were BIOSIS Citation Index<sup>SM14</sup>, CAB Abstracts®, Current Contents Connect<sup>®15</sup>, Medline<sup>®16</sup> and Web of Science Core Collection<sup>TM17</sup>.

One literature search was performed using the same search terms and Boolean operators as the applicant, and the second one with a combination of generic terms previously used by the EFSA GMO Panel (2012b). Both searches targeted scientific publications in the peer-reviewed scientific literature between June 2013 and June 2014, which was the time interval covered by the literature search conducted by the applicant. The searches were subsequently refined by selecting only those publications (as 'document type') that were written in English.

Both searches were conducted on 7 July 2015 and identified a total of 469 references. References from both searches were exported into a single EndNote X5 database (Thomson Reuters, New York, NY, USA). Then, duplicated references were deleted. The 312 remaining scientific publications were screened and assessed by title and abstract by EFSA. Only peer-reviewed publications containing evidence specific to the risk assessment and/or management of maize MON 810 were considered for further assessment.

The EFSA GMO Panel identified a total number of 37 relevant scientific publications,<fnref n="18" /> of which 25 were reported by the applicant in its revised annual 2013 PMEM report on maize MON 810, and/or which were previously discussed in EFSA GMO Panel scientific outputs (EFSA, 2014; EFSA GMO Panel, 2015). From the remaining 12 scientific publications, two are relevant to the FF safety assessment and 10 for the ERA and/or risk management of maize MON 810 (Appendix A).

The relevance of the 12 scientific publications for the risk assessment and/or risk management of maize MON 810 is discussed below. Scientific publications are grouped per area of concern.

<footnote n="13">¹³ http://www.cabi.org/publishing-products/online-information-resources/cab-abstracts/ (accessed 10 October 2015).</footnote>

<footnote n="14">¹⁴ http://wokinfo.com/products_tools/specialized/bci/ (accessed 10 October 2015).</footnote>

<footnote n="15">¹⁵ http://thomsonreuters.com/en/products-services/scholarly-scientific-research/scholarly-search-and-discovery/current-contents-connect.html (accessed 10 October 2015).</footnote>

<footnote n="16">¹⁶ https://www.nlm.nih.gov/bsd/pmresources.html (accessed 10 October 2015).</footnote>

<footnote n="17">¹⁷ http://thomsonreuters.com/en/products-services/scholarly-scientific-research/scholarly-search-and-discovery/web-of-science-core-collection.html (accessed 10 October 2015).</footnote>

<footnote n="18">¹⁸ Three additional relevant publications were identified by the EFSA GMO Panel (i.e. Erasmus and van den Berg, 2014; Grabowski et al., 2014; Truter et al., 2014). These publications have been reported by the applicant in the annual 2014 PMEM report (they were most likely not retrieved by the applicant when the literature search was performed), and they will be assessed by the EFSA GMO Panel in the corresponding scientific opinion.</footnote>
## Publications relevant to FF safety assessment of maize MON 810

One publication focuses on the toxicity of maize MON 810:

* Abdo et al. (2014) conducted a 90-day feeding study to evaluate the effect of maize MON 810 diet (standard diet containing 30 % Bt-maize) on rats.

One publication focuses on the nutritional assessment of maize MON 810:

* Korwin-Kossakowska et al. (2013) assessed the effect of maize MON 810 grains on the performance of the Japanese quail, and the accumulation of recombinant DNA in its eggs, muscle and internal organs.

The EFSA GMO Panel assessed both FF-related scientific publications, and concluded that no safety concerns owing to maize MON 810 were identified in the Korwin-Kossakowska et al. (2013) study. However, because of major methodological limitations in the experimental design of the study by Abdo et al. (2014),<fnref n="19" /> it is not possible to conclude whether the changes observed in the weight of some organs and in blood haematology, clinical chemistry analysis and liver histopathology in rats are due to the endotoxins produced in Bt-maize, as claimed by the authors. The EFSA GMO Panel is therefore not in a position to conclude on the relevance of the reported findings for the FF risk assessment of maize MON 810.

## Publications relevant to ERA of maize MON 810

Six publications focus on interactions of maize MON 810 with TOs:

* Beres et al. (2013) determined the oviposition dynamics and egg hatching of *O. nubilalis* on maize MON 810 and its conventional counterpart in a field study in Poland over four consecutive growing seasons.

* Bohnenblust et al. (2013) assessed the efficacy of several Bt-maize varieties for controlling the corn earworm, *Helicoverpa zea* (Lepidoptera: Noctuidae), a secondary pest of maize in the USA, at 26 field sites in two growing seasons.

* El Shazly et al. (2013) investigated the effectiveness of two MON 810 hybrids to control the pink corn borer, *Sesamia cretica* (Lepidoptera: Noctuidae), in the El-Ayat region (Egypt) in 2011.

* López et al. (2013) studied the dispersal capacity of larvae and adults of *S. nonagrioides*, and the role of antixenosis in oviposition on Bt-maize plants.

* Obopile and Hammond (2013) investigated the effects of two MON 810 hybrids of different maturities and their corresponding conventional counterparts on oviposition preference and population dynamics of *O. nubilalis*.

* White et al. (2014) evaluated the mating status and *Nosema pyrausta* infection of *O. nubilalis* as a function of population density and sex ratio over two growing seasons across several US states.

<footnote n="19">¹⁹Abdo et al. (2014) described observation in organ weights, haematological and serum chemistry parameters and liver histopathology in rats fed up to 90 days with a standard diet alone (unspecified), a diet containing 30 % maize MON 810, or a diet containing 30 % non-GM maize. In this subchronic feeding study, 36 rats (18 males and 18 females) were divided into three groups each of six males and six females, receiving one of the three diets. Three males and three females from each group were sacrificed after 1.5 and 3 months. At the end of the study, their offspring were also sacrificed. The authors concluded that there were many alterations in the parameters assessed in the GM group, including severe changes in the histopathological examination of liver tissues. Based on the data presented, the EFSA GMO Panel identified weaknesses in the study design and in the interpretation of the results which make difficult a clear interpretation: (1) the number of animals tested is too low to achieve an adequate statistical power, and is not in accordance with the relevant OECD guideline (OECD, 1998); (2) organ weights are given only as relative mean weights (as per cent of body weight), and no description of the mean absolute values and the body weights is provided; (3) since the study design includes mating to generate offspring, with increase of the body weights in females, the mean relative organ weights as a percentage of brain weights would have been more appropriate; (4) the photomicrographs of liver sections shown in Figure 1 do not support the description of the histopathological changes, as described in the relative paragraph; (5) no reference to a peer review by a second pathologist is made, as is recommended to verify and improve the accuracy and quality of pathology diagnoses and interpretations (Morton et al., 2010); (6) organ weights, haematological and serum chemistry variations, in the absence of a full histopathological assessment, cannot be interpreted. Consequently, owing to the methodological weaknesses observed in the Abdo et al. (2014) study, it is not possible to conclude on the relevance of these findings.</footnote>
Three publications focus on the interaction of maize MON 810 or the Cry1Ab protein with NTOs:

* Hurej et al. (2014) assessed the impact of maize MON 810 on the bird cherry-oat aphid, *Rhopalosiphum padi* (Homoptera: Aphididae), under field conditions.

* Li et al. (2014) investigated the effects of the purified Cry1Ab protein on larvae of the predatory green lacewing *Chrysoperla sinica* (Neuroptera: Chrysopidae) under laboratory conditions.

* Zhang et al. (2014) assessed the toxicity of purified Cry1Ab on the ladybird beetle, *Propylea japonica* (Coleoptera: Coccinellidae), in a laboratory assay.

One publication focuses on the interaction of maize MON 810 with biogeochemical processes:

* Kamota et al. (2014) evaluated the decomposition of surface-applied and soil-incorporated maize MON 810 leaf litter and Cry1Ab degradation in winter fallow field conditions at two sites in South Africa during 2008.

The EFSA GMO Panel assessed the ERA-related scientific publications listed above, and concludes that no environmental safety concerns owing to maize MON 810 or Cry1Ab were identified.

### 3.3.3. Conclusions of the literature review

The results reported in the relevant peer-reviewed scientific publications included by the applicant in the revised annual 2013 PMEM report on maize MON 810 and those additionally identified by the EFSA GMO Panel do not provide new information that would invalidate the previous FF and ERA conclusions on maize MON 810 made by the Panel. Therefore, the EFSA GMO Panel considers that its previous conclusions on the safety of maize MON 810 remain valid and applicable (EFSA, 2009; EFSA GMO Panel, 2012b,c).

However, the findings reported in some of the scientific publications listed above reinforce previous recommendations on insect resistance monitoring as part of the IRM plan given by the EFSA GMO Panel:

* Crava et al. (2013) reported that the occurrence of a Cry1Ab tolerance trait in *O. nubilalis* field populations in Spain is common. This reinforces the need for the adaptation of the IRM strategy as previously suggested by the EFSA GMO Panel, i.e. detection limit for resistance allele frequency at 1 % for areas with 80 % maize MON 810 adoption rate or 3 % for areas with 60 % maize MON 810 adoption rate, and annual sampling of bi-/multi-voltine target pest populations in areas where the maize MON 810 adoption rate is at least 60 % of the total cultivated maize (EFSA, 2015a).

* In their study, González-Cabrera et al. (2013) emphasised the risk that Cry1Ab resistance will evolve in field populations of *Mythimna unipuncta* (Lepidoptera: Noctuidae), and suggested that this secondary maize pest should be considered in IRM strategies. The EFSA GMO Panel previously addressed this issue, and therefore reiterates its previous recommendation that the applicant uses farmer questionnaires as a tool to give early warning of the potential for resistance evolution in populations of these non-target pests by reporting observations on the occurrence and unexpected survival (and possible damage) of their larvae on Bt-maize plants. In the event that farmers report indications of possible resistance evolution in regionally occurring non-target lepidopteran pests, they should become subject to routine IRM (for further details, see EFSA GMO Panel, 2012c).

* The results reported by López et al. (2013) on dispersal stimulation of *O. nubilalis* larvae by Bt-maize plants might compromise the use of the IRM strategy. Increased dispersal of adults may accelerate the speed of resistance development in the target pest, and increased mobility of the larvae would produce low-dose effects when mixed seeds are used as the refuge strategy. However, the EFSA GMO Panel does not consider the seed mixture an appropriate strategy for managing resistance evolution in lepidopteran target pests in the EU, considering their biology (EFSA GMO Panel, 2012a).

* White et al. (2014) collected field data indicating that a correlation might exist between surfaces cropped with Bt-maize and female-biased sex ratios, and suggested that models
developed to determine the risk of resistance evolution to Cry1Ab in *O. nubilalis* populations should consider this. However, more robust evidence about the extent and the cause of this phenomenon are still lacking.

4. Conclusions

The data reported in the revised annual 2013 PMEM report on maize MON 810 do not indicate any unanticipated adverse effects on human and animal health or the environment arising from the cultivation of maize MON 810 during the 2013 growing season. The EFSA GMO Panel therefore concludes that there are no new data from the GS of maize MON 810 grown in 2013 that would invalidate previous EFSA GMO Panel evaluations on the safety of maize MON 810 (EFSA, 2009; EFSA GMO Panel, 2012b,c).

5. Recommendations

The EFSA GMO Panel identified shortcomings in the methodology followed by the applicant to analyse the farmer questionnaires similar to those found in previous reports. Therefore, the Panel reiterates its recommendations on the survey design and reporting to provide more detailed information on the sampling methodology and to reduce the possibility of selection bias (for further details on the recommendations, see Annex A), as this would give more confidence in the conclusion on the absence of adverse effects. In order to improve the sampling frame of the farmer survey, the EFSA GMO Panel reiterates the importance of national GMO cultivation registers, as referred to in Article 31.3 (b) of Directive 2001/18/EC (EC, 2001); and its recommendations to applicants to consider how they may make best use of the information recorded in national registers and foster dialogue with those responsible for the administration of these registers where maize MON 810 is cultivated.

The outcome of the literature review confirms the previous conclusions on the safety of maize MON 810 made by the EFSA GMO Panel. However, the EFSA GMO Panel reiterates some of its previous recommendations, i.e. to adapt the IRM strategy for ECB and MCB populations and to use farmer questionnaires as a tool to give early warning of the potential for resistance evolution in populations of non-target lepidopteran pests. In addition, considering the relevant publications identified as missing, the EFSA GMO Panel advises the applicant to improve the methodology followed in the literature review to ensure that all relevant publications are identified and assessed (e.g. use of additional databases, provide inclusion/exclusion criteria).

No information collected from existing monitoring networks in the EU was provided by the applicant. However, the EFSA GMO Panel notes that initiatives have been taken to develop a methodological framework to use existing networks in the broader context of environmental monitoring, and encourages the relevant parties to continue to use these.

With regard to rove beetles (Coleoptera: Staphylinidae), the applicant is requested to continue screening, reviewing and discussing relevant scientific publications on possible adverse effects of maize MON 810 on rove beetles.

Documentation provided to EFSA

1. Letter from the EC, dated 18 June 2015, to the EFSA Executive Director requesting the assessment of the revised annual 2013 PMEM report on maize MON 810 cultivation during the 2013 growing season report provided by Monsanto; the PMEM report was annexed to the letter.

2. Comments from Member States on the revised annual 2013 PMEM report on the cultivation of maize MON 810 during the 2013 growing season.

3. Acknowledgment letter, dated 28 July 2015, from the EFSA Executive Director to the EC.
## References

Abdo EM, Barbary OM and Shaltout OE, 2014. Feeding study with Bt corn (MON810: Ajeeb YG) on rats: biochemical analysis and liver histopathology. Food and Nutrition Sciences, 5, 185-195.

Beckie HJ, Hall LM, Simard M-J, Leeson JY and Willenborg CJ, 2010. A framework for postrelease environmental monitoring of second-generation crops with novel traits. Crop Science, 50, 1587-1604.

Beres PK, Dabrowski ZT and Sowa S, 2013. Comparison of some aspects of the bionomy of *Ostrinia nubilalis* Hbn. (Lep., Crambidae) on Bt and non-Bt maize in south-eastern Poland. Journal of Central European Agriculture, 14, 672-690.

Bohnenblust E, Breining J, Fleischer S, Roth G and Tooker J, 2013. Corn earworm (Lepidoptera: Noctuidae) in northeastern field corn: Infestation levels and the value of transgenic hybrids. Journal of Economic Entomology, 106, 1250-1259.

Campagne P, Kruger M, Pasquet R, Le Ru B and Van den Berg J, 2013. Dominant inheritance of field-evolved resistance to Bt corn in *Busseola fusca*. PLOS ONE, 8, e69675.

Carringer RD, Marcinkiewicz BR and Schottel BS, 2004. Collection of plant tissue and grain samples from test corn hybrids containing M810xMON863 and MON863xNK603, parental controls and references in 2002-2003 Argentina field production. Report No MSL-18567.

Centre for Ecology and Hydrology, Perseus, Rijksinstituut voor Volksgezondheid en Milieu; Review of statistical methods and data requirements to support post market environmental monitoring of agro ecosystems. Supporting Publications 2014:EN-582, 151 pp.

Crava CM, Farinós GP, Bel Y, Castañera P and Escriche B, 2013. Quantitative genetic analysis of Cry1Ab tolerance in *Ostrinia nubilalis* Spanish populations. Journal of Invertebrate Pathology, 113, 220-227.

El-Shazly EA, Ismail IA, El Shabrawy HA, Abdel-Moniem ASH, and Abdel-Rahman RS, 2013. Transgenic maize hybrids as a tool to control *Sesamia cretica* Led. compared by conventional method of control on normal hybrids. Archives of Phytopathology and Plant Protection, 46, 2304-2313.

EFSA (European Food Safety Authority), 2005. Opinion of the Scientific Panel on Genetically Modified Organisms on an application (reference EFSA-GMO-BE-2004-07) for the placing on the market of insect-protected glyphosate- tolerant genetically modified maize MON 863 x MON 810 x NK603, for food and feed use, import and processing under Regulation (EC) No 1829/2003 from Monsanto, The EFSA Journal 2005, 256, 1-25. doi:10.2903/j.efsa.2005.256

EFSA (European Food Safety Authority), 2009. Scientific Opinion of the Panel on Genetically Modified Organisms on applications (EFSA-GMO-RX-MON 810) for the renewal of authorisation for the continued marketing of (1) existing food and food ingredients produced from genetically modified insect resistant maize MON810; (2) feed consisting of and/or containing maize MON810, including the use of seed for cultivation; and of (3) food additives, and feed materials produced from maize MON810, all under Regulation (EC) No 1829/2003 from Monsanto. The EFSA Journal, 1149, 1-85. doi:10.2903/j.efsa.2009.1149

EFSA (European Food Safety Authority), 2010. Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA Journal 2010;8(6):1637, 90 pp. doi:10.2903/j.efsa.2010.1637

EFSA (European Food Safety Authority), 2014. Statement on a request from the European Commission related to an emergency measure notified by France under Article 34 of Regulation (EC) 1829/2003 to prohibit the cultivation of genetically modified maize MON 810. EFSA Journal 2014;12(8):3809, 18 pp., doi:10.2903/j.efsa.2014.3809

EFSA (European Food Safety Authority), 2015a. Clarifications on EFSA GMO Panel recommendations on the Insect Resistance Management plan for genetically modified maize MON 810. EFSA supporting publication 2015; EN-842. 14 pp.
EFSA (European Food Safety Authority), 2015b. Tools for critically appraising different study designs, systematic review and literature searches. EFSA supporting publication 2015; EN-836. 65 pp.

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2011a. Scientific Opinion on the annual Post-Market Environmental Monitoring (PMEM) report from Monsanto Europe S.A. on the cultivation of genetically modified maize MON 810 in 2009. EFSA Journal 2011;9(10):2376, 66 pp. doi:10.2903/j.efsa.2011.2376

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2011b. Scientific Opinion providing guidance on the Post-Market Environmental Monitoring (PMEM) of genetically modified plants. EFSA Journal 2011; 9(8):2316, 40 pp. doi:10.2903/j.efsa.2011.2316

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2012a. Scientific Opinion on the annual Post-Market Environmental Monitoring (PMEM) report from Monsanto Europe S.A. on the cultivation of genetically modified maize MON 810 in 2010. EFSA Journal 2012;10(3):2610, 35 pp. doi:10.2903/j.efsa.2012.2610

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2012b. Scientific Opinion updating the risk assessment conclusions and risk management recommendations on the genetically modified insect resistant maize MON 810. EFSA Journal 2012;10(12):3017, 98 pp. doi:10.2903/j.efsa.2012.3017

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2012c. Scientific Opinion supplementing the conclusions of the environmental risk assessment and risk management recommendations for the cultivation of the genetically modified insect resistant maize Bt11 and MON 810. EFSA Journal 2012;10(12):3016, 32 pp. doi:10.2903/j.efsa.2012.3016

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2012d. Scientific Opinion on the annual Post-Market Environmental Monitoring (PMEM) report from BASF Plant Science Company GmbH on the cultivation of genetically modified potato EH92-527-1 in 2010. EFSA Journal 2012;10(2):2558, 38 pp. doi:10.2903/j.efsa.2012.2558

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2012e. Scientific Opinion on the annual Post-Market Environmental Monitoring (PMEM) report from BASF Plant Science Company GmbH on the cultivation of genetically modified potato EH92-527-1 in 2011. EFSA Journal 2012;10(12):3015, 35 pp. doi:10.2903/j.efsa.2012.3015

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2013. Scientific Opinion on the annual Post-Market Environmental Monitoring (PMEM) report from Monsanto Europe S.A. on the cultivation of genetically modified maize MON 810 in 2011. EFSA Journal 2013; 11(12):3500, 38 pp. doi:10.2903/j.efsa.2013.3500

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2014a. Scientific Opinion on the annual post-market environmental monitoring (PMEM) report from Monsanto Europe S.A. on the cultivation of genetically modified maize MON 810 in 2012. EFSA Journal 2014;12(6):3704, 29 pp. doi:10.2903/j.efsa.2014.3704

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2014b. Scientific Opinion on the use of existing environmental surveillance networks to support the post-market environmental monitoring of genetically modified plants. EFSA Journal 2014;12(11):3883, 24 pp. doi:10.2903/j.efsa.2014.3883

EFSA GMO Panel (EFSA Panel on Genetically Modified Organisms), 2015. Scientific Opinion on the annual post-market environmental monitoring (PMEM) report from Monsanto Europe S.A. on the cultivation of genetically modified maize MON 810 in 2013. EFSA Journal 2015;13(3):4039, 11 pp. doi:10.2903/j.efsa.2015.4039

Erasmus A and Van den Berg J, 2014. Effect of Bt-maize expressing Cry1Ab toxin on non-target Coleoptera and Lepidoptera pests of maize in South Africa. African Entomology, 22, 167–179.

Galyean ML and Wester DB, 2010. Comparison of multinomial and binomial proportion methods for analysis of multinomial count data. Journal of Animal Science, 88, 3452–3463.
González-Cabrera J, García M, Hernández-Crespo P, Farinós GP, Ortego F and Castañera P, 2013. Resistance to Bt maize in *Mythimna unipuncta* (Lepidoptera: Noctuidae) is mediated by alteration in Cry1Ab protein activation. Insect Biochemistry and Molecular Biology, 43, 635-643.

Grabowski M, Lipska A, Żmijewska E, Kozak M and Dąbrowski ZT, 2014. Transfer of the Cry1Ab toxin in tritrophic bioassays involving transgenic maize MON810, the herbivore *Tetranychus urticae* Koch and the predatory ladybird beetle *Adalia bipunctata* L. (Coleoptera: Coccinellidae). Egyptian Journal of Biological Pest Control, 24, 11-16.

Hollis JM, Jones RJA, Marshall CJ, Holden A, Van de Veen JR and Montanarella L, 2006. SPADE-2: The soil profile analytical database for Europe, version 1.0. European Soil Bureau Research Report No 19, EUR 22127 EN, Office for Official Publications of the European Communities, Luxembourg, 38 pp.

Hurej M, Twardowski JP, Beres P and Klukowski Z, 2014. Population development of bird cherry-oat aphid *Rhopalosiphum padi* (L.) (Hemiptera, Aphididae) on conventional and Bt-maize expressing the insecticidal protein Cry1Ab. Bulgarian Journal of Agricultural Science, 20, 658-665.

Kamota A, Muchaonyerwa P and Mnkeni PNS, 2014. Decomposition of surface-applied and soil-incorporated Bt maize leaf litter and Cry1Ab protein during winter fallow in South Africa. Pedosphere, 24, 251-257.

Korwin-Kossakowska A, Sartowska K, Linkiewicz A, Tomczyk G, Prusak B and Sender G, 2013. Evaluation of the effect of genetically modified Roundup Ready soya bean and MON 810 maize in the diet of Japanese quail on chosen aspects of their productivity and retention of transgenic DNA in tissues. Archiv Tierzucht, 56, 597-606.

Li Y, Romeis J, Wang Y, Han L, Chen X and Peng Y, 2014. Use of an artificial diet system to study the toxicity of gut-active insecticidal compounds on larvae of the green lacewing *Chrysoperla sinica*. Biological Control, 69, 45-51.

López C, Hernández-Escareño G, Eizaguirre M and Albajes R, 2013. Antixenosis and larval and adult dispersal in the Mediterranean corn borer, *Sesamia nonagrioides*, in relation to Bt maize. Entomologia Experimentalis et Applicata, 149, 256-264.

Morton D, Sellers RS, Barale-Thomas E, Bolon B, George C, Hardisty JF, Irizarry A, McKay JS, Odin M and Teranishi M, 2010. Recommendations for pathology peer review. Toxicologic Pathology, 38, 1118-1127.

Obopile M and Hammond RB, 2013. The influence of planting date, transgenic Bt maize and hybrid relative maturity on European corn borer *Ostrinia nubilalis* (Lepidoptera: Crambidae) ovipositional patterns. Entomological Research, 43, 299-305.

OECD (Organisation for Economic Co-operation and Development), 1998. Guideline for the testing of chemicals—Repeated dose 90-day oral toxicity study in rodents, 408.

Rasch D, Herrendörfer G, Bock J, Victor N and Guiard V, 2007. Verfahrensbibliothek Versuchsplanung und -auswertung. Oldenbourg Verlag, Munich, Germany.

Smets G, Alcalde E, Andres D, Carron D, Delzenne P, Heise A, Legris G, Martinez Parrilla M, Verhaert J, Wandelt C, Ilegems M and Rüdelsheim P, 2014. The use of existing environmental networks for the post-market monitoring of GM crop cultivation in the EU. Environmental Science: Processes and Impacts, 16, 1754-1763.

Truter J, Van Hamburg H and Van den Berg J, 2014. Comparative diversity of arthropods on Bt maize and non-Bt maize in two different cropping systems in South Africa. Environmental Entomology, 43, 197-208.

Twardowski J, Beres P, Hurej M and Klukowski Z, 2014. A quantitative assessment of the unintended effects of Bt-maize (MON 810) on rove beetle (Col., Staphylinidae) assemblages. Polish Journal of Environmental Studies, 23, 215-220.

White JA, Burkness EC and Hutchison WD, 2014. Biased sex ratios, mating frequency and *Nosema* prevalence in European corn borer, at low population densities. Journal of Applied Entomology, 138, 195-201.
Wilhelm R, Beißner L, Schmidt K, Schmidtke J and Schiemann J, 2004. Monitoring des Anbaus gentechnisch veränderter Pflanzen—Fragebögen zur Datenerhebung bei Landwirten. Nachrichtenblatt des Deutschen Pflanzenschutzdienstes, 56, 184–188.

Zhang X, Li Y, Romeis J, Yin X, Wu K and Peng Y, 2014. Use of a pollen-based diet to expose the ladybird beetle *Propylea japonica* to insecticidal proteins. PLOS ONE, 9, e85395.
Abbreviations

<table><tr><td>AMU Unit</td><td>Unit for Assessment and Methodological Support</td></tr><tr><td>CSM</td><td>case-specific monitoring</td></tr><tr><td>EC</td><td>European Commission</td></tr><tr><td>ECB</td><td>European corn borer</td></tr><tr><td>EFSA GMO Panel</td><td>Scientific Panel on Genetically Modified Organisms of the European Food Safety Authority</td></tr><tr><td>ERA</td><td>environmental risk assessment</td></tr><tr><td>FF</td><td>food and feed</td></tr><tr><td>GM</td><td>genetically modified</td></tr><tr><td>GMO</td><td>genetically modified organism</td></tr><tr><td>GS</td><td>general surveillance</td></tr><tr><td>ha</td><td>hectare</td></tr><tr><td>IPM</td><td>integrated pest management</td></tr><tr><td>IRM</td><td>insect resistance monitoring</td></tr><tr><td>MCB</td><td>Mediterranean corn borer</td></tr><tr><td>NTO</td><td>non-target organisms</td></tr><tr><td>PMEM</td><td>post-market environmental monitoring</td></tr><tr><td>TO</td><td>target organism</td></tr></table>
## Appendix A – Peer-reviewed scientific publications relevant to the risk assessment and/or management of maize MON 810 assessed by the EFSA GMO Panel as part of the revised annual 2013 PMEM report on the cultivation of maize MON 810

<table><thead><tr><td>Authors</td><td>Title</td><td>Journal</td><td>Year</td><td>Identified by</td><td>Relevant field<sup>(a)</sup></td></tr></thead><tbody><tr><td>Abdo EM, Barbary OM, Shaltout OE</td><td>Feeding study with Bt corn (MON810: Ajeeb YG) on rats: biochemical analysis and liver histopathology</td><td>Food and Nutrition Sciences</td><td>2014</td><td>EFSA</td><td>FF—Toxicology</td></tr><tr><td>Agapito-Tenfen SZ, Guerra MP, Wikmark O-G, Nodari RO</td><td>Comparative proteomic analysis of genetically modified maize grown under different agroecosystems conditions in Brazil</td><td>Proteome Science</td><td>2013</td><td>Applicant</td><td>MC</td></tr><tr><td>Bednarek D, Dudek K, Kwiatek K, Swiatkiewicz M, Swiatkiewicz S, Strzetelski J</td><td>Effect of a diet composed of genetically modified feed components on the selected immune parameters in pigs, cattle, and poultry</td><td>Bulletin of the Veterinary Institute in Pulawy</td><td>2013</td><td>Applicant</td><td>FF—Toxicology</td></tr><tr><td>Beres PK, Dabrowski ZT, Sowa S</td><td>Comparison of some aspects of the bionomy of <i>Ostrinia nubilalis</i> Hbn. (Lep., Crambidae) on Bt and non-Bt maize in south-eastern Poland</td><td>Journal of Central European Agriculture</td><td>2013</td><td>EFSA</td><td>ERA—TO</td></tr><tr><td>Bohnenblust, E, Breining J, Fleischer S, Roth G, Tooker J</td><td>Corn earworm (Lepidoptera: Noctuidae) in northeastern field corn: infestation levels and the value of transgenic hybrids</td><td>Journal of Economic Entomology</td><td>2014</td><td>EFSA</td><td>ERA—TO</td></tr><tr><td>Buzoianu SG, Walsh MC, Rea MC, Quigley L, O'Sullivan O, Cotter PD, Ross RP, Gardiner GE, Lawlor PG</td><td>Sequence-based analysis of the intestinal microbiota of sows and their offspring fed genetically modified maize expressing a truncated form of <i>Bacillus thuringiensis</i> Cry1Ab protein (Bt maize)</td><td>Applied and Environmental Microbiology</td><td>2013</td><td>Applicant</td><td>FF—Toxicology</td></tr><tr><td>Buzoianu SG, Walsh MC, Rea MC, Cassidy JP, Ryan TP, Ross RP, Gardiner GE, Lawlor PG</td><td>Transgenerational effects of feeding genetically modified maize to nulliparous sows and offspring growth and health</td><td>Journal of Animal Science</td><td>2013</td><td>Applicant</td><td>FF—Toxicology</td></tr><tr><td>Cheeke TE, Cruzan MB, Rosenstiel TN</td><td>Field evaluation of arbuscular mycorrhizal fungal colonization in <i>Bacillus thuringiensis</i> toxin-expressing (Bt) and non-Bt maize</td><td>Applied and Environmental Microbiology</td><td>2013</td><td>Applicant</td><td>ERA—NTO</td></tr><tr><td>Crava C, Farinós GP, Bel Y, Castañera P, Escriche B</td><td>Quantitative genetic analysis of Cry1Ab tolerance in <i>Ostrinia nubilalis</i> Spanish populations</td><td>Journal of Invertebrate Pathology</td><td>2013</td><td>Applicant</td><td>ERA—TO</td></tr><tr><td>El-Shazly EA, Ismail IA, El Shabrawy HA, Abdel-Moniem ASH, Abdel-Rahman S</td><td>Transgenic maize hybrids as a tool to control <i>Sesamia cretica</i> Led. compared by conventional method of control on normal hybrids</td><td>Archives of Phytopathology and Plant Protection</td><td>2013</td><td>EFSA</td><td>ERA—TO</td></tr><tr><td>Furgal-Dierzuk I, Strzetelski J, Kwiatek K, Twardowska M, Mazur M, Sieradzki Z, Kozaczyński W, Reichert M</td><td>The effect of genetically modified maize (MON 810) and soybean meal (Roundup Ready) on rearing performance and transfer on rearing performance and transfer of transgenic DNA to calf tissues</td><td>Journal of Animal and Feed Sciences</td><td>2014</td><td>Applicant</td><td>FF—Toxicology</td></tr></tbody></table>
<table><thead><tr><td>Authors</td><td>Title</td><td>Journal</td><td>Year</td><td>Identified by</td><td>Relevant field<sup>(a)</sup></td></tr></thead><tbody><tr><td>García-Villaraco Velasco A, Kowalchuk GA, Gutierrez Mañero FJ, Ramos B, Yergeau E, Lucas García JA</td><td>Increased microbial activity and nitrogen mineralization coupled to changes in microbial community structure in the rhizosphere of Bt corn</td><td>Applied Soil Ecology</td><td>2013</td><td>Applicant</td><td>ERA—NTO</td></tr><tr><td>González-Cabrera J, García M, Hernández-Crespo P, Farinós GP, Ortego F, Castañera P</td><td>Resistance to Bt maize in <i>Mythimna unipuncta</i> (Lepidoptera: Noctuidae) is mediated by alteration in Cry1Ab protein activation</td><td>Insect Biochemistry and Molecular Biology</td><td>2013</td><td>Applicant</td><td>ERA—TO</td></tr><tr><td>Habuštová O, Doležal P, Spitzer L, Svobodová Z, Hussein H, Sehnal F</td><td>Impact of Cry1Ab toxin expression on the non-target insects dwelling on maize plants</td><td>Journal of Applied Entomology</td><td>2014</td><td>Applicant</td><td>ERA—NTO</td></tr><tr><td>Hurej M, Twardowski JP, Beres P, Klukowski Z</td><td>Population development of bird cherry-oat aphid <i>Rhopalosiphum padi</i> (L.) (Hemiptera, Aphididae) on conventional and Bt-maize expressing the insecticidal protein Cry1Ab</td><td>Bulgarian Journal of Agricultural Science</td><td>2014</td><td>EFSA</td><td>ERA—NTO</td></tr><tr><td>Kamota A, Muchaonyerwa P, Mnkeni PNS</td><td>Decomposition of surface-applied and soil-incorporated Bt maize leaf litter and Cry1Ab protein during winter fallow in South Africa</td><td>Pedosphere</td><td>2014</td><td>EFSA</td><td>ERA—Biogeochemical processes</td></tr><tr><td>Kocourek F, Saska P, Řezáč M</td><td>Diversity of carabid beetles (Coleoptera: Carabidae) under three different control strategies against European corn borer in maize</td><td>Plant Protection Science</td><td>2013</td><td>Applicant</td><td>ERA—NTO</td></tr><tr><td>Korwin-Kossakowska A, Sartowska K, Linkiewicz A, Tomczyk G, Prusak B, Sender G</td><td>Evaluation of the effect of genetically modified Roundup Ready soya bean and MON 810 maize in the diet of Japanese quail on chosen aspects of their productivity and retention of transgenic DNA in tissues</td><td>Archiv Tierzucht</td><td>2013</td><td>EFSA</td><td>FF—Nutritional assessment</td></tr><tr><td>Kruger M, Van Rensburg JBJ, Van den Berg J</td><td>No fitness costs associated with resistance of <i>Busseola fusca</i> (Lepidoptera: Noctuidae) to genetically modified Bt maize</td><td>Crop Protection</td><td>2014</td><td>Applicant</td><td>ERA—TO</td></tr><tr><td>Kuramae EE, Verbruggen E, Hillekens R, de Hollander M, Röoling WFM, van der Heijden MGA, Kowalchuk GA</td><td>Tracking fungal community responses to maize plants by DNA- and RNA-based pyrosequencing</td><td>PLOS ONE</td><td>2013</td><td>Applicant</td><td>ERA—NTO</td></tr><tr><td>Li Y, Hu L, Romeis J, Wang Y, Han L, Chen X, Peng Y</td><td>Use of an artificial diet system to study the toxicity of gut-active insecticidal compounds on larvae of the green lacewing <i>Chrysoperla sinica</i></td><td>Biological Control</td><td>2014</td><td>EFSA</td><td>ERA—NTO</td></tr><tr><td>Lopez C, Hernandez-Escareno G, Eizaguirre M, Albajes R</td><td>Antixenosis and larval and adult dispersal in the Mediterranean corn borer, <i>Sesamia nonagrioides</i>, in relation to Bt maize</td><td>Entomologia Experimentalis et Applicata</td><td>2013</td><td>EFSA</td><td>ERA—TO</td></tr><tr><td>Mesnage R, Clair E, Gress S, Then C, Székács, Séralini G-E</td><td>Cytotoxicity on human cells of Cry1Ab and Cry1Ac Bt insecticidal toxins alone or with a glyphosate-based herbicide</td><td>Journal of Applied Toxicology</td><td>2013</td><td>Applicant</td><td>FF—Toxicology</td></tr></tbody></table>
<table><thead><tr><td>Authors</td><td>Title</td><td>Journal</td><td>Year</td><td>Identified by</td><td>Relevant field<sup>(a)</sup></td></tr></thead><tbody><tr><td>Obopile M, Hammond HB</td><td>The influence of planting date, transgenic Bt maize and hybrid relative maturity on European corn borer <i>Ostrinia nubilalis</i> (Lepidoptera: Crambidae) ovipositional patterns</td><td>Entomological Research</td><td>2013</td><td>EFSA</td><td>ERA-TO</td></tr><tr><td>Ondreičková K, Mihálik D, Ficek A, Hudcovicová M, Kraic J, Drahovská H</td><td>Impact of genetically modified maize on the genetic diversity of rhizosphere bacteria: a two-year study in Slovakia</td><td>Polish Journal of Ecology</td><td>2014</td><td>Applicant</td><td>ERA-NTO</td></tr><tr><td>Pérez-Hedo M, Reiter D, López C, Eizaguirre M</td><td>Processing of the maize Bt toxin in the gut of <i>Mythimna unipuncta</i> caterpillars</td><td>Entomologia Experimentalis et Applicata</td><td>2013</td><td>Applicant</td><td>ERA-TO</td></tr><tr><td>Sanden M, Ornsrud R, Sissener NH, Jorgensen S, Gu J, Bakke AM, Hemre G-I</td><td>Cross-generational feeding of Bt (<i>Bacillus thuringiensis</i>)-maize to zebrafish (<i>Danio rerio</i>) showed no adverse effects on the parental or offspring generations</td><td>British Journal of Nutrition</td><td>2013</td><td>Applicant</td><td>FF-Toxicology</td></tr><tr><td>Sieradzki Z, Mazur M, Kwiatek K, Swiatkiewicz S, Swiatkiewicz M, Koreleski J, Hanczakowska E, Arczewska-Wlosek A, Goldsztejn M</td><td>Assessing the possibility of genetically modified DNA transfer from GM feed to broiler, laying hen, pig and calf tissues</td><td>Polish Journal of Veterinary Sciences</td><td>2013</td><td>Applicant</td><td>FF-Toxicology</td></tr><tr><td>Swiatkiewicz M, Bednarek D, Markowski J, Hanczakowska E, Kwiatek K</td><td>Effect of feeding genetically modified maize and soybean meal to sows on their reproductive traits, haematological indices and offspring performance</td><td>Bulletin of the Veterinary Institute in Pulawy</td><td>2013</td><td>Applicant</td><td>FF-Toxicology</td></tr><tr><td>White JA, Burkness EC, Hutchison WD</td><td>Biased sex ratios, mating frequency and <i>Nosema</i> prevalence in European corn borer, at low population densities</td><td>Journal of Applied Entomology</td><td>2014</td><td>EFSA</td><td>ERA-TO</td></tr><tr><td>Zhang X, Li Y, Romeis J, Yin X, Wu K, Peng Y</td><td>Use of a pollen-based diet to expose the ladybird beetle <i>Propylea japonica</i> to Insecticidal Proteins</td><td>PLOS ONE</td><td>2014</td><td>EFSA</td><td>ERA-NTO</td></tr></tbody></table>

<footnote>(a): ERA, environmental risk assessment; FF, food and feed; MC, molecular characterisation; NT, non-target organism; TO, target organism.</footnote>
# Annex A – EFSA AMU Unit technical report on the evaluation of farmer questionnaires

## 1. Background

This annex was prepared by the EFSA Unit for Assessment and Methodological Support (EFSA AMU Unit) to support the Scientific Panel on Genetically Modified Organisms of the European Food Safety Authority (hereafter referred to as EFSA GMO Panel) in its evaluation of the revised annual post-market environmental monitoring (PMEM) report on maize MON 810 for the 2013 growing season, specifically to provide methodological guidance on evaluation of the farmer questionnaires submitted as part of the general surveillance (GS) programme, which aimed to identify adverse effects of maize MON 810 or its use on human and animal health or the environment that had not been anticipated in the environmental risk assessment (ERA).

## 2. Method

Evaluation criteria were developed based on the principles of design for cross-sectional studies, and in particular surveys (Table 1). The evaluation grid can be applied to surveys used for GS of GM plants. In July 2011, the EFSA GMO Panel updated its guidance on the PMEM of genetically modified (GM) plants (EFSA GMO Panel, 2011b). The criteria reflect the recommendations in this guidance document. These criteria were previously applied in the assessment of the annual 2009–2013 PMEM reports on maize MON 810 (EFSA GMO Panel, 2011, 2012a, 2013, 2014a, 2015) and the 2010–2011 ones on the potato Amflora (EFSA GMO Panel, 2012d, e).

<table><caption>**Table 1:** Criteria used to assess the methodology of farmer questionnaires.</caption><thead><tr><td>Study design principle</td><td>Criteria</td></tr></thead><tbody><tr><td>Sampling frame</td><td>1. The sampling frame used is specified<br>2. The total population included in the sampling frame is specified<br>3. The characteristics of the population included in the sampling frame are described, including region, agricultural practices, GM plant cultivation<br>4. The sampling frame coverage is appropriate for GM cultivation in the EU</td></tr><tr><td>Sampling method (sample bias)</td><td>1. The sampling method to select sample units from the sampling frame is described<br>2. The sampling method ensures that sampling units from representative environments, reflecting the range and distribution of plant production systems and environments exposed to the GM plants and their cultivation, are sampled<br>3. A list of sample units selected from the sample frame is provided<br>4. The sampling method minimises selection bias</td></tr><tr><td>Sample size (sample precision)</td><td>1. The size of the adverse effect to be measured is specified and scientifically justified and is within an acceptable limit of change<br>2. The significance level is specified and the chosen level is scientifically justified (Type I error rate)<br>3. The power is specified and the chosen level is scientifically justified (Type II error rate)<br>4. A literature reference for the sample size method is provided<br>5. The sample size calculation method is appropriate for a proportion in a cross-sectional study<br>6. The sample size is sufficient to detect an adverse effect related to GM plant cultivation</td></tr></tbody></table>
<table><thead><tr><th>Study design principle</th><th>Criteria</th></tr></thead><tbody><tr><td>Survey response rate (nonresponse bias)</td><td><ol><li>Follow-up method for non-responders is described and appropriate</li><li>Response rate is specified</li><li>Details of losses in sampling are described</li><li>The number of partial responses and reasons for non-completion are specified</li><li>Comparison is made between the characteristics of the responder group and the non-responder group</li><li>Comparison is made between the characteristics of the responder group and independent sources of information about the target population</li><li>The effects of non-response bias have been minimised</li></ol></td></tr><tr><td>Instrument design</td><td><ol><li>The study design includes considerations to avoid interviewer bias</li><li>Where interviewers are used the interviewer training is described</li><li>The selection of open and closed questions is appropriate for the question type</li><li>The questions are clearly phrased and not open to misinterpretation</li><li>The questions encourage independent and objective responses</li><li>The comparator used in the study is described and appropriate for general surveillance</li><li>The instrument has been previously tested and validated</li></ol></td></tr><tr><td>Instrument validity</td><td><ol><li>Content validity—the survey includes questions relevant to assess<ul><li>Background data</li><li>Identifier of location of monitoring site and comparator site, surrounding landscape, type of field margins, proximity to conservation areas, cultivation and management of the GM plant field including recent history and previous cropping, soil (type, structure, quality), nutrient status, fertilisation, irrigation</li><li>Data on possible change in behaviour and performance of the GM plant</li></ul>Other GM plants cultivated, number of years of cultivation of the GM plant, cultivation and tillage from the removal of the previous crop to seed sowing, crop husbandry including sowing/planting date, post-planting management, crop emergence, growth (vigour, height), pest, disease and weed management, flowering, standing ability, harvesting date and methods, yield, post-harvest management and subsequent cropping of the site, post-harvest storage, handling, processing, feeding.</li><li>Data on possible ecological/environmental impacts of the GM plant on the protection goals and measurement</li><li>Weed and pest populations, observations of other flora and fauna such as insects, birds and mammals, pollination and presence of pollinators, health of humans and performance of livestock.</li><li>Implementation of specific management requirements</li></ul>Implementation of risk management measures, coexistence segregation measures, stewardship recommendations, specific management due to regional environmental requirements</li><li>Criterion validity—agronomy parameters reported in the survey are compared with field trial data to test for concurrency</li><li>External consistency—results from survey are compared to and conform with independent external data sources (for example</li></ol></td></tr></tbody></table>
<table><thead><tr><th>Study design principle</th><th>Criteria</th></tr></thead><tbody><tr><td></td><td>pest/weed occurrence reports, soil characteristics from geological surveys, authorisations and use reports for plant protection products)</td></tr><tr><td></td><td>4. Plausibility of responses - results for cultivation methods, agronomy parameters and weed/pest management practices reported in the survey conform to European agricultural practices</td></tr><tr><td></td><td>5. Construct validity—consistency and agreement between outcome variables are examined</td></tr><tr><td>Data validation</td><td>1. Data validation procedures are documented</td></tr><tr><td></td><td>2. Results excluded from the statistical analysis during validation are reported</td></tr><tr><td></td><td>3. Missing values are reported</td></tr><tr><td>Longitudinal aspects</td><td>Comparison with survey results from previous years</td></tr><tr><td></td><td>1. The survey is applied to the sample unit for multiple years in order to assess residual effects</td></tr><tr><td>Statistical analysis</td><td>1. Objective and hypotheses for analysis are clearly stated</td></tr><tr><td></td><td>2. A statistical analysis plan is provided</td></tr><tr><td></td><td>3. Statistical analysis includes analysis of pre-defined subgroups in accordance with PMEM guidance, e.g. country</td></tr><tr><td></td><td>4. Statistical analysis is appropriate for the data types</td></tr><tr><td></td><td>5. Results are clearly and consistently presented</td></tr><tr><td></td><td>6. The report should include descriptive statistics for the outcome variables</td></tr><tr><td></td><td>7. The issue of multiplicity is addressed</td></tr><tr><td></td><td>8. Methods for handling missing values are described</td></tr><tr><td></td><td>9. Where appropriate, confidence intervals should be provided</td></tr><tr><td></td><td>10. The results of post-hoc analysis should be identifiable</td></tr><tr><td>Report conclusions</td><td>The report conclusions are clearly stated</td></tr><tr><td></td><td>1. The study design is appropriate to assess the conclusions</td></tr><tr><td></td><td>2. The data provided support the conclusions presented in the report</td></tr></tbody></table>

# 3. Results

## 3.1. Sampling frame

### 3.1.1. Sampling frame specification

Appendix 1 of the revised annual 2013 PMEM report on maize MON 810 specifies that in Portugal and Romania the sampling frame for the survey was a public register. It is indicated that in the Czech Republic and Slovakia, customer lists obtained from companies selling seeds were used (however, no planted maize MON 810 surfaces were monitored in Slovakia for the 2013 survey). In Spain, the country with highest cultivation of maize MON 810, and therefore the largest number of surveyed farmers, no suitable sampling frame was available as a consequence surveyors used previous contacts (the report states that 'the interviewers identify MON 810 cultivating farmers by knowledge from previous surveys or search in the region').

### 3.1.2. Population included in the sampling frame

Appendix 1 of the revised annual 2013 PMEM report on maize MON 810 did not include information on the number of farmers in the sampling frame. The report states that 'the total number of growers (and of fields and field sizes) is not known, but only the total cultivated area (in ha)'. The report
continues: 'Therefore the sampling frame for this survey cannot be based on the total population of fields with MON 810 cultivation in Europe. Instead of this, a quota considering the magnitude (ha planted per country/ ha planted in the EU) and product situation (average field size in the country) of MON 810 cultivation will be applied, resulting in certain numbers of farmers to be monitored per year and country'.

**3.1.3. Characteristics of the population included in the sampling frame**

Appendix 1 of the revised annual 2013 PMEM report on maize MON 810 did not include information on the characteristics of farmers included in the sampling frame. Information on the number of farmers in the sampling frame according to country, region, agricultural practices, size of farm/number of fields and previous cultivation of GM plants is important.

**3.1.4. Sampling frame coverage**

Information on the sampling frame was not provided in Appendix 1 of the revised annual 2013 PMEM report on maize MON 810, which hampers its assessment. The report states that 'The customer lists of the seed selling companies do not completely cover all MON 810 cultivating farmers, so that some are missing', but does not try to characterise the missing farmers. Table 3.2 indicates that farmers from four out of five countries growing maize MON 810 in 2013 were included in the survey (no monitored surfaces in Slovakia). The percentage of maize MON 810-planted surfaces surveyed ranged from 3.8% in Spain to 67.7% in the Czech Republic. For Europe as a whole, 6.8% of maize MON 810-planted surfaces were surveyed. Full details on the source of the sampling frame, the number of farmers and the major characteristics of the farmers should be included in the survey report. The national registers set by Member States on the cultivation of GM plants would be the optimum sampling frame; however, the report notes that when using public registers they 'do not necessarily contain the contact data of the farms so it is often very difficult to identify them'. Both in cases of incomplete customer lists of the seed selling companies and of incomplete contact data in public registers, it needs to be considered whether the data might be missing systematically.

**3.2. Sampling method**

**3.2.1. Selection of sample units**

Appendix 1 of the revised annual 2013 MON 810 report states public registers and customer lists of the seed selling companies were used as sampling frames in 2013, but in one country no sampling frame was available. For this country, the report states that 'the interviewers identify MON 810 cultivating farmers by knowledge from previous surveys or search in the region'. Survey design methodology requires the sampling frame to be representative for the target population, in this case European farmers growing MON 810, and that the random selection process is applied to the sample units in the sampling frame prior to proceeding with the interviews. A description of the method to ensure that units are randomly selected from the sampling frame should be included in the report, including where relevant the statistical software and/or the program code used for this procedure.

**3.2.2. Sampling of units from representative environments**

Appendix 1 of the revised annual 2013 MON 810 report states 'Sampling of these 2500 fields should ensure to reflect the range and distribution of plant production systems and environments exposed to GM plant cultivation. This range, on the one hand, is characterized by the growing season (year and its climatic, environmental conditions). On the other hand, it is characterized by the geographic regions where GM cultivation takes place. Regions may vary by production systems, regulatory requirements, agro-political and socio-economic conditions and therefore are best described by European countries. Consequently, sampling takes place within strata (defined by years and countries of cultivation)'. It is also further stated that: 'Splitting the number per year to the countries considers fluctuant adoption of the GM plant (grade of market maturity) and therefore is performed yearly for the actual situation' and that: 'Consequently, cultivation areas with a high uptake of the GM plant will be over-represented by a high number of fields to be monitored. Within each stratum (per year and country) the determined number of monitoring units is selected randomly where each field has the
same chance to be surveyed. The whole sampling procedure ensures that the monitoring area will be proportional to and representative of the total regional area under GM cultivation'.

This is similar to the 2012 MON 810 report, but differs from the 2011 MON 810 report, which stated: 'two strategies for selecting farmers are applied: in MS with a high rate of market penetration a certain number of farms will be selected whereas in MS with low cultivation rates preferably all MON 810 cultivating farmers are interviewed', and from the 2010 MON 810 report, which stated 'For selecting farmers in countries with higher market penetration a procedure is applied to select: at least 10 % of farmers and 10 % GM area per region and at least 20 % of new farmers each year'. To ensure that units are selected from representative environments (regions with high uptake of maize MON 810), a proportion of farmers to be selected from each stratum (e.g. country) should be clearly defined and consistently applied in each year of the survey. It is reported that the number of farmers to be monitored per year and country is determined based on two factors ('the magnitude (ha planted per country/ ha planted in the EU) and product situation (average field size in the country) of MON 810 cultivation') but it is not entirely clear how this is done. For example, based on table 3.2, Spain seems to account for 92.2 % of the total European MON 810-planted surface area, but contributed only about 74.2 % of the questionnaires for 2013. Moreover, the claim that each field within a stratum has the same chance of being surveyed is not entirely substantiated, since sampling selection considers farmers and not fields and also it is stated that 'Actually, the sampling procedure is afflicted by the problem that the total number of growers (and of fields and field sizes) is not known, but only the total cultivated area (in ha). Therefore the sampling frame for this survey cannot be based on the total population of fields with MON 810 cultivation in Europe'.

### 3.2.3. Proportion of sample units selected

The number of farmers surveyed in each country is provided, but no indication of the total number of farmers in each country and region included in the sampling frame is given. Table 3.2 describes the proportion of maize MON 810-planted area covered in the survey. Information on the farmers included in the sampling frame and selected from the sampling frame should be provided as evidence that the sampling method has been successfully implemented.

### 3.2.4. Selection bias

If the number of farmers cultivating maize MON 810 increases, it becomes difficult to ensure that all farmers within a region are interviewed and, as a consequence, an appropriate sampling methodology becomes more important. The report provides limited information on the sampling methodology, and the possibility of selection bias and achievement of inadequate power in the survey cannot be excluded. The grouping of sample units according to the strata and random selection of sample units from within the strata should be performed using the specified sampling frame prior to conducting the interviews and consistently applied over the 10 years of the survey. A description of the method used to ensure that units are randomly selected from the sampling frame should be included in the report, including where relevant the statistical software and/or the program code used for this procedure. The proportion of new farmers and farmers with previous experience of maize MON 810 selected from the sampling frame for each region should be presented in the report to provide evidence that the sampling method ensures that areas of intensive maize MON 810 cultivation are appropriately covered in the survey. If the used sampling frames (public registers and customer lists obtained from companies selling seeds) were missing information in a systematic way (i.e. specific subsections of the farmers' populations) then bias could be introduced in the study if the reasons for the missing information were related to the study outcomes. Moreover, in the case of Spain, where no suitable sampling frame was available, it was noted that 'Here, the interviewers identify MON 810 cultivating farmers by knowledge from previous surveys or search in the region'. This approach cannot guarantee a representative sample and, therefore, there is a possibility of introducing bias (the direction of which cannot be predicted). It is not explained how exactly this selection is done; however, it is possible that it might make it more likely that some of the same farmers are sampled in consecutive years, perpetuating a possible bias, if some bias existed in the first place.
3.3. Sample size

3.3.1. Size of the adverse effect

Appendix 1 of the revised annual 2013 MON 810 report states that 'the **baseline** for the analysis of monitoring characters with categories As usual and Different is 90% - 10%, where Minus- and Plus- answers are balanced and both are about 5%. An effect of the cultivation of MON 810 or any other influencing factor would arise in a greater percentage of Different (i.e. Plus or Minus) answers, where 'greater' or an **effect**, was quantitatively defined by exceeding a threshold of 10%', and 'Therefore, to identify an effect within the data means to test the frequencies of the Plus- or Minus- answers statistically against the threshold of 10%'.

No specific reference from the scientific literature was provided to support the selection of 10 %; however, for this type of study 10 % represents an acceptable limit of change. A 10 % effect size has also been selected in a framework proposal for post-release monitoring of second-generation crops with novel traits in Canada (Beckie et al., 2010).

3.3.2. Type I error rate

The Type I error rate is α = 0.01 in Appendix 1 of the revised annual 2013 MON 810 report. This denotes that for each of the three testing procedures described in section 2.4 there is a 1 % probability of rejecting the null hypothesis that 'the probability of As usual-answers is smaller than 90%' or that 'the probability of Plus- or Minus- answers is larger than 10%' when it is true, i.e. failure to detect a true adverse effect. A Type I error rate of 1 % is conservative and acceptable. However, it needs to be noted that the error rates are specified for each one-sided test, separately, but it is not clear what these error probabilities would be if all hypothesis tests (two or three for each question) were considered together. It is indicated that 'to keep the experiment-wise type I error rate a closed principle test procedure is performed by testing all three probabilities subsequently in descending order'. If the closed principle test procedure were followed appropriately, it would indeed address the issue of the overall Type I error rate in multiple testing. However, in the monitoring characteristics comparisons presented in the report (see table 3.1), the results of all possible tests (two or three, depending on the structure of the question) are presented.

3.3.3. Type II error rate

The Type II error rate is β = 0.01 in Appendix 1. This denotes that there is a 1 % probability of not rejecting the null hypothesis that there is a 'proportion of adverse effects equal to or greater than 10%' when it is false, i.e. falsely detecting an adverse effect. The selection of 0.01 will result in a large sample size. However, it needs to be noted that this error rate will be realised only with the overall calculated sample size of 2 500.

3.3.4. Reference for the sample size method

The sample size calculation was performed using the methodology described in Rasch et al. (2007).

3.3.5. Sample size calculation

The sample size is calculated assuming difference testing.

3.3.6. Sample size

As concluded for the 2011 and 2012 monitoring reports (EFSA GMO Panel, 2013, 2014a), the selection of parameters for the sample size calculation is conservative. In 2013, 256 farmers were sampled—this is six farmers more than the planned 250 farmers per year. Nonetheless, it is likely that the same farmer may be surveyed in different years and therefore sample units may not be independent of each other. Consideration of this factor should be included in the sample size calculation. Most importantly, the power of the study will be achieved only when the sample size of 2 500 farmers/fields surveyed is achieved after 10 years. Concerning the 'allocation' of the calculated sample size among the participating countries, it is reported that the number of farmers to be monitored per year and
country is determined based on two factors (mentioned above), but it is not entirely clear how this is
done.

3.4. Survey response rate

3.4.1. Follow-up for non-responders

Appendix 1 of the revised annual 2013 MON 810 report states: 'The surveys are performed after the planting season, the farmers are provided with a copy of the questionnaire at least two weeks before a telephone interview or interviewed face-to-face'. This should reduce the number of non-responders in comparison with other survey methods. No information is provided in the report on the follow-up for non-responders.

3.4.2. Response rate

The response rate is provided (90 % for Czech Republic and 100 % for Spain, Portugal and Romania).
In Appendix 1 of the revised annual 2013 MON 810 report, the fact that two farmers from the Czech
Republic refused to participate is recorded.

3.4.3. Losses in sampling

No details of losses in sampling are included in the report. The number of farmers selected from the sampling frame but not contacted by the interviewers should be stated in the report.

3.4.4. Partial responses and reasons for non-completion

This information was not presented in the report; however, the use of trained interviewers may have
resulted in no cases of partial completion of the survey.

3.4.5. Characteristics of responder group and non-responder group

Two farmers from the Czech Republic declined to participate. It is important to know if a specific subgroup of farmers is not participating in the survey and therefore is not represented in the survey findings; consequently, this information should be presented in the report.

3.4.6. Characteristics of responder group compared with the target population

No comparison between the responder group and the target population is provided in the report.
Where available, national registers for the cultivation of GM crops should be compared with the
characteristics of the farmers surveyed in terms of geographical location and farming practices to
ensure that the farmers surveyed are representative of the target population.

3.4.7. Non response bias

The losses to sampling should be fully documented in the report to provide evidence that there is no non-response bias. It is important to know if a specific subgroup of farmers is not participating in the survey and therefore is not represented in the survey findings.

3.5. Instrument design

3.5.1. Interviewer bias

The 2013 MON 810 survey used third parties to perform the interviews, with the exception of Romania, where Monsanto field representatives helped the farmers to fill in the questionnaire. The use of third-party interviewers can prevent interviewer bias. A lot of attention should be paid to the standardisation of the delivery of the questions from all interviewers, since the interviews should be conducted in the same way for all participants. This means also that the administration of the questionnaire should be done in the same way for all participants; in 2013, some interviews were done during personal visits, while others were done by telephone communication (in which case the farmers were sent the questionnaire two weeks in advance).
### 3.5.2. Interviewer training

Appendix 1 states that 'all interviewers have been trained to understand the background of the questions', and mentions that the interviewers also draw on previous experience in administering the questionnaire to ensure that the questions are answered correctly. In addition, a 'user's manual' is provided to the interviewers.

### 3.5.3. Question type

The questionnaire contains 27 closed questions, which require a comparison between the representative GM maize field and the representative conventional maize field. For these questions the response options are of the type "'the same' or 'different/changed'" or "'as usual' or 'worse' or 'better'". It is these questions that are primarily analysed in the report. Where the response is not 'same/as usual', there is an option to provide more details as free text. There is also a mix of closed and open questions to gather additional information about the farming practices on the farm and five closed questions to gather information about good agricultural practice and implementation of non-Bt refuge(s). The combination of open and closed questions allows quantitative analysis of the comparisons between the GM maize field and the conventional maize field, and, where differences occur between the two field types, explanatory analysis can be performed using the information from the free text questions.

### 3.5.4. Phrasing of questions

The questionnaire uses questions based on farm records and should be understood by a grower.

### 3.5.5. Independent and objective responses

Overall, the questionnaire seeks to obtain an objective set of responses to summarise the results and experiences during the growing season for maize. Nevertheless, the questionnaire could be improved by adjusting the balance between crop performance questions and questions on the general farm environment by addressing the latter more fully. Furthermore, qualitative responses may sometimes relate to a subjective assessment on the part of the farmer. An effort should be made to use objective measurable outcomes, whenever this is possible.

### 3.5.6. Comparator

The questionnaire relies on a comparison between a representative GM maize field and a representative conventional field in order to detect unanticipated adverse effects. Consequently, the choice of representative fields and the recollection of similarities and differences are crucial to the success of the survey. The report provides no indication about the comparator fields selected by the farmer for comparison in the survey; however, the inclusion in table 3.5 and figure 3.4 of the mean MON 810 cultivation area as a percentage of total maize area on each farm appears to indicate that some non-GM that may be suitable as a comparator is probably grown on all farms. It is recommended that the questionnaire contains questions record clearly whether the comparator field is on the same farm in the same growing season and the variety of the comparator. If no comparators are being grown spatially or temporally close to the GM crop, then the rationale for selecting another comparator (e.g. maize grown in previous years) should be fully described. The comparators selected by the farmers for the survey should be summarised in the report.

### 3.5.7. Validation of the instrument

The questionnaire was developed by the German Federal Biological Research Centre for Agriculture and Forestry and maize breeders and statisticians in Germany (Wilhelm et al., 2004). The questionnaire was used in annual monitoring reports in the period 2006-2013. Any future amendments to the questions should be made giving consideration to pooled analysis of the results over 10 years. The report mentions: 'The format of the questionnaire is reviewed on a yearly basis based on the outcome of the latest survey. As appropriate, adjustments are made to improve the statistical relevance of the collected data'. As a consequence of this approach, increased care should be taken to assure the comparability of the obtained data from year to year.
3.6. Instrument validity

3.6.1. Content validity

Background data

Background data relating to geographical location at country and county level, surrounding environment, soil type, crop rotations in the previous two years and fertiliser treatments and irrigation are collected by the questionnaire. It would be of value to take longitude and latitude measurements of the representative GM maize field; information of this nature would facilitate linkage with other spatial monitoring datasets. In addition, the questionnaire should record for how many years the farmer has been growing maize MON 810 on the farm, and the question on crop rotation should also record, for rotations in which maize was grown, whether this was GM or conventional maize.

Data on possible change in behaviour and performance of the GM crop

The following characteristics were monitored to obtain data on any change in the behaviour and
performance of maize MON 810: crop rotation, time of planting, tillage and planting technique, insect
control practices, weed control practices, fungal control practices, fertiliser application, irrigation
practice, time of harvest, germination vigour, time to emergence, time to male flowering, plant growth
and development, incidence of stalk/root lodging, time to maturity, and yield. It is noted that
information on plant protection products applied to the GM maize field was collected, but the same
information was not supplied for the conventional field. In order to fully explain changes in plant
protection product use, the products applied to the conventional field should also be recorded, and
the quantities applied over the season to the GM maize field and the comparator field should be
recorded.

Data on possible ecological/environmental impacts of the GM crop on the protection goals and measurement

The following characteristics were monitored to obtain information on possible ecological/environmental impacts of maize MON 810 on protection goals: occurrence of MON 810 volunteers, disease susceptibility, insect pest control (Ostrinia nubilalis), insect pest control (Sesamia spp.), pest susceptibility, weed pressure, occurrence of insects, occurrence of birds, occurrence of mammals. For some closed questions, e.g. on occurrence of insects, birds and mammals, the option 'Do not know' is included; however, it has been excluded in other closed analysis questions, forcing the farmer to make a clear assessment. Allergenicity in people handling the GM crop during production and harvesting could be an adverse effect: a question to assess this should be included in the questionnaire. It is important that the question is phrased in such a way that it discriminates between allergenicity to the GM crop and background levels of hay fever-type symptoms.

Compliance with good agricultural practice

Section 4 requests information on compliance with good agricultural practice, and in this case the
planting of non-Bt refuge(s).

3.6.2. Criterion validity

The scientific opinion of the EFSA GMO Panel on the renewal of the authorisation for maize MON 810 commercialisation in the EU (EFSA, 2009) states that ‘The information available in the renewal applications gives no reason to change the opinion that maize MON 810 is agronomically and phenotypically equivalent to currently grown non-GM maize varieties, with exception of the insect resistance conferred by the Cry1Ab protein.’ The 2005 scientific opinion on maize MON 863 × MON 810 × NK603 (EFSA, 2005) states ‘Plants of the same field trials as for compositional analysis, except for a difference in glyphosate treatment (see section 3.2.2) were compared for their agronomic and phenotypic characteristics. These characteristics included seedling vigour, crop growth stages (for example, the stage at which silking and pollination occurred), height of the plant and ear (attachment containing the cob and kernels), root lodging (plants leaning to the surface), stalk lodging (plants with stalks broken below the ear), dropped ears, final stand count, stay-green and kernel yield. The plants tested showed no particular deviations in any of these parameters. In
addition, plant damage due to insect feeding in two locations and due to weather in one location appeared to occur preferentially in plots planted with reference lines.' Report MSL-18567 (Carringer et al., 2004) includes data on the agronomic parameters assessed in the above opinion. In the case of seedling vigour, both maize MON 810 and the reference varieties had 'excellent' vigour, with the exception of one site where one reference variety was classed as poor and one as average. Stalk lodging in plants near harvest was observed more frequently in the reference varieties, and at one site root lodging in plants close to harvest was observed more frequently in the reference varieties. In the case of the other agronomic parameters, there was no particular deviation between maize MON 810 and the reference varieties. Appendix 1 of the revised annual 2013 MON 810 report assessing the characteristics of maize MON 810 reported 'slightly more vigorous germination, unchanged time to emergence, unchanged time to male flowering, unchanged plant growth and development, lower incidence of stalk/root lodging, delayed time to maturity, higher yield and unchanged occurrence of MON 810 volunteers'. Comparing the field trial data with the farmer survey data provides an opportunity to check the validity of the farmers' responses. It appears that there may be differences between field trial data and the questionnaire: there are a number of possible explanations for this, e.g. the conventional crops grown on the farms differ from the comparator variety used in the field trials, the information provided by the farmers is biased or erroneous or the GM crop is performing differently on farm-scale cultivation (possibly performing better when the cultivation conditions are less than optimal). It is of value to select parameters measured using a 'gold standard' methodology and to contrast these with the responses in the survey to ensure the validity of the reported responses.

### 3.6.3. External consistency

Comparison of the data reported in the survey with information from independent data sources provides a further opportunity to test the validity of the responses. The information on soil quality offers the opportunity to compare it with the information held in the Soil Profile Analytical Database for Europe (SPADE-2) (Hollis et al., 2006). Figure 1 shows the information on top soil organic carbon contained in this database. The MON 810 survey reports organic carbon content values between 0.6 % and 6.5 % with a mean of 1.6 %. It can be seen that this range falls within that of the SPADE-2 range for organic carbon content. It should be noted that the SPADE-2 database provides a useful dataset for European soil properties but that the values are based on a limited set of soil samples for each EU country.

<figure><img src="image_2.png" /></figure>

<table><caption>**Figure 1:** Distribution and descriptive statistics of top soil organic carbon contents in SPADE-2 for free-draining non-organic soils (from Hollis et al., 2006)</caption><tr><td>Mean</td><td>1.983665</td></tr><tr><td>Standard Error</td><td>0.051215</td></tr><tr><td>Median</td><td>1.5</td></tr><tr><td>Mode</td><td>1.5</td></tr><tr><td>Standard Deviation</td><td>1.494919</td></tr><tr><td>Sample Variance</td><td>2.234783</td></tr><tr><td>Kurtosis</td><td>9.998333</td></tr><tr><td>Skewness</td><td>2.648682</td></tr><tr><td>Range</td><td>13.1</td></tr><tr><td>Minimum</td><td>0.3</td></tr><tr><td>Maximum</td><td>13.4</td></tr><tr><td>Sum</td><td>1690.082</td></tr><tr><td>Count</td><td>852</td></tr><tr><td>Confidence Level(95.0%)</td><td>0.100523</td></tr></table>

### 3.6.4. Plausibility of responses

The report states that: 'All data are entered and controlled for their quality and plausibility. A quality control check first checks the completeness of the data. Some data fields (especially the monitoring characters or comments in case of farmer's assessments differ from As usual) are defined to be obligatory, therefore missing values or unreadable entries are not accepted. Furthermore the values
are checked for correctness (quantitative values within a plausible min-max range, qualitative values meeting only acceptable parameter values). Plausibility control checks the variable values for their contents, both to find incorrect answers and to prove the logical connections between different questions. It also looks for the consistency between Plus/Minus-answers and specifications, i.e. whether all these answers were provided with a specification and whether the specifications really substantiated the Plus/Minus-answers'.

It is indicated that the sowing and harvest times were used to check the plausibility of the responses provided by the farmers: the sowing period lasted from 1 March 2013 to 30 July 2013 and the harvest period for maize grain lasted from 15 August 2013 to 30 December 2013 and for maize forage from 20 July 2013 to 5 November 2013.

3.6.5. Construct validity

The questionnaire is able to detect changes in characteristics of the GM maize field compared with the conventional field that could be predicted when the nature of the genetic event in MON 810 is considered. Maize MON 810 expresses the *cry1Ab* coding sequence, which encodes an insecticidal protein, Cry1Ab. The responses to the survey indicated that, for the maize MON 810 field, insecticide application and corn borer control practices were different: owing to a reduction in insecticides applied to control corn borers, the yield was higher, there was a lower incidence of root and stalk lodging and less susceptibility to diseases and pests. The questionnaire also indicated that the control of *O. nubilalis*, and *Sesamia* spp. in maize MON 810 fields was very good, compared with conventional maize. The report proposes that the change in previously mentioned characteristics is due to the increased protection from corn borer damage. This hypothesis is credible and indicates consistency and agreement among outcome variables.

3.7. Data validation

3.7.1. Validation procedures

Section 2.7 of Appendix 1 describes the data management and quality control procedures. It states that 'For not readable entries in the questionnaires, queries were formulated and the field representatives or farmers were asked for clarification. These entries in the database were corrected'. In section 3.1 it is further explained that 'After the first quality and plausibility control, 8 farmers were contacted again to provide additional clarifications (2 from Spain, 4 from Portugal and 2 from Czech Republic). Examples of items that had to be clarified were incorrect variety names and missed answers (surrounding environment, weed and pest control practices in conventional maize). Two farmers were also asked to clarify some inconsistencies between weed and pest control practices in conventional maize compared to MON 810, and plant protection products used in MON 810. After including the corrections, the quality and plausibility control confirmed that all 256 questionnaires could be considered for analysis'. In this report, the number of questionnaires that require further clarification with the farmers is included; however, a classification by error types should also be presented.

3.7.2. Exclusion of results

All completed questionnaires (256) were included in the analysis.

3.7.3. Missing values

It is stated in the report: 'When farmers gave no statement, these answers are accounted as missing values and therefore not considered valid'. There are several questions for which the number of valid answers was less than 256. In the analysis of each of the monitoring characteristics, the number of responses for each value was shown in the table, including the missing values where they occur. In general, the number of missing values was low.
3.8. Longitudinal aspects

3.8.1. Sampling over multiple years

The possible repeated sampling of a sample unit needs to be considered in the sample size calculations and in the statistical analysis of the results. It is important that a mechanism for recording possible repeated sampling is introduced and the numbers of sample units repeatedly sampled are included in the report. If this information were available, it would allow an analysis considering the intensity of maize MON 810 cultivation and the possible changes in monitoring characteristic assessment as maize MON 810 cultivation is repeated in consecutive years.

3.9. Statistical analysis

3.9.1. Objective and hypotheses

Appendix 1 states: 'The aim of the survey is to identify potential adverse effects that might be related to MON 810 plants and their cultivation. For that reason, most questions are formulated to identify deviation from the situation with conventional maize. Farmers are asked to assess the situation compared to conventional cultivation. If the farmer assesses the situation to be different he is additionally asked to specify the direction of the difference, hence the category different is divided into two subcategories. To simplify this two-stage procedure in the questionnaire for most questions three possible categories of answers are given: As usual, Plus (e.g., later, higher, more) or Minus (e.g., earlier, lower or less). High frequency (> 10 %) of Plus- or Minus-answers would indicate possible effects'.

3.9.2. Statistical analysis plan

Section 2.4 of Appendix 1 describes the statistical test procedure. The effect is defined as: 'An effect of the cultivation of MON 810 or any other influencing factor would arise in a greater percentage of Different (i.e. Plus or Minus) answers, where "greater" or an **effect**, was quantitatively defined by exceeding a threshold of 10%'. It would be expedient to provide scientific references to support the selection of the 10 % threshold. Additionally, for certain responses, 10 % may be greater than the acceptable limit of change. Alternative statistical analyses allowing the exploration of different effect sizes for certain monitoring characteristics would assist in the interpretation of the results. A closed principle testing procedure of sequential hypothesis testing is described: first the null hypothesis that the proportion of responses 'as usual' is smaller or equal to 90 % is tested. If this is not rejected (indicating an effect), then two more hypothesis tests can be conducted: for the major and for the minor of the probabilities (of plus or minus answers), the null hypothesis being each time that the proportion of the answers is greater than or equal to 10 %. If p < 0.01, then the null hypothesis that the proportion of minus/plus responses is equal to or greater than 10 % is rejected and therefore no effect can be identified. The proposed approach should be described more clearly, especially the proposed sequence of testing and the criteria to continue or stop testing after each hypothesis test.

3.9.3. Pre-defined subgroups

The analysis was performed for all fields surveyed in 2013. There was no analysis of country-level
data. Given the number of farmers surveyed in some countries, analyses of country-level subgroup
may not have been statistically valid; however, consideration should be given to the fact that Member
States may require country-level results. Moreover, in the report it is mentioned: 'Consequently,
sampling takes place within strata (defined by years and countries of cultivation)'. This stratified
sampling should also be taken into consideration in the statistical analysis, which currently considers
the sample as 'homogeneous', being composed of independent units (farmers). In addition, analysis of
the assessment of monitoring characteristics by new farmers compared with farmers with previous
experience of cultivation of maize MON 810 would be of interest. This could assist in detecting
residual effects.

3.9.4. Statistical analysis

The report states that plus responses and minus responses were statistically tested by using the exact binomial test. This test is appropriate for the 'same/different' type of question. However, for questions
of the 'as usual or worse or better' type, where there are three outcomes, an analysis using a multinomial test should be performed (in this case a trinomial test). Galyean and Wester (2010) used simulation methods to generate experimental count data from multinomial distributions in order to compare multinomial and binomial proportion methods for analysis. It was concluded that analysing multinomial data as a series of binomial proportions increased the survey-wise Type I error rate and, therefore, they recommended using multinomial analysis to test for the distributional difference with a subsequent binomial approach used to test for differences in a specific category or to correct for the multiplicity of testing. Section 2.4 of Appendix 1 proposes using a closed principle test procedure to address the issue of the overall Type I error. However, in practice, in the monitoring characteristics comparisons presented in the report (see table 3.1), the results of all possible tests are presented, even if in the first hypothesis test the null hypothesis $p_{AS usual} \le 0.9$ is rejected. Hence, the closed principle test procedure is not followed and the issue of the overall Type I error in multiple testing is still relevant.

### 3.9.5. Results presentation

For most analysed monitoring characteristics measured by the survey, a table of the responses was provided with percentage and 'valid percentages' (the proportion of answers excluding missing values) plus, often, a bar chart of the frequency of responses. The valid percentages were used in the binomial test. The presentation of the valid percentages in the table of responses and the table of the results of the binomial test for different 'treatments/practices' should be further explained by the applicant in order to facilitate interpretation of the results. Concerning the characteristic 'Maize Borer control practice', for which hypothesis testing results are provided in table 3.1, the comparisons are not described in the text of the report. Moreover, the presentation of a power calculation in situations in which the null hypothesis has been rejected is not necessary.

### 3.9.6. Descriptive statistics

Descriptive statistics were provided for the continuous outcome values number of fields, maize area in hectares, percentage humus content, sowing date and harvest date. The analysis of the categorical values was provided as frequency tables.

### 3.9.7. Multiplicity

A significance level of 0.01 was used, but the issue of multiplicity of testing was not addressed adequately. A closed principle test procedure is proposed to be used in order to address the issue of the overall type I error. However, in practice, in the monitoring characteristics comparisons presented in the report (see table 3.1), the results of all possible tests are being presented, even if in the first hypothesis test the null hypothesis $p_{AS usual} \le 0.9$ is rejected. Another major problem is related to the fact that the analysis needs to be pooled after 10 years to achieve the statistical power described in the sample size calculations. Each annual report represents an interim analysis, and the statistical analysis plan needs to compensate for these interim analyses, considering also possible situations in which the same farmer(s) is (are) sampled in more than one year.

### 3.9.8. Handling missing values

In the tables two percentages are presented: 'Per cent', which includes missing values, and 'Valid percentages', from which missing data or 'Don't know' responses were excluded.

### 3.9.9. Confidence intervals

For statistical tests it is standard practice to use confidence intervals. Confidence intervals have been provided graphically in figure 3.1, but only for the probabilities of 'as usual' answers. In the table summarising the analysis of the monitoring characteristics (i.e. table 3.1 in Appendix 1) the confidence intervals should be included. The inclusion of all confidence intervals would allow an understanding of the sensitivity of the analysis to the choice threshold.
3.9.10. Post-hoc analysis

Post-hoc analysis was performed only in situations in which an effect was identified and further
explanatory analysis was possible using less structured information, e.g. free text collected in the
questionnaire.

4. Report conclusions

**4.1. Report conclusions**

Appendix 1 contains the following conclusions:

'2013 data indicates that in comparison to conventional maize plants, MON 810 plants

- received less insecticides caused by their inherent protection against certain lepidopteran pests,

- germinated more vigorously caused by the high quality germplasm,

- had less incidence of stalk/root lodging caused by the inherent protection against certain lepidopteran pests,

- had a longer time to maturity caused by the absence of pest pressure of certain lepidopteran pests,

- gave a higher yield caused by the better fitness of the plant,

- were less susceptible to diseases caused by hardly any insect feeding damage,

- controlled corn borers very well caused by the inherent protection against certain lepidopteran pests, and

- were less susceptible to pests, other than corn borers, especially lepidopteran pests caused by the inherent protection against certain lepidopteran pests and the resulting better fitness of the plants'.

**4.2. Study design**

The study design was appropriate to evaluate whether a set of monitoring characteristics relating to plant performance and management practices for maize MON 810 cultivation in the current year of the survey differed from a comparator variety by a threshold of 10 %. However, there are indications of weaknesses in the sampling methodology applied for the survey and, as a consequence, the possibility of selection bias in the survey cannot be excluded. In addition, the result of this assessment was very much dependent on the selection of an appropriate comparator.

Certain effects may reach a sufficient magnitude for detection only with repeated cultivation of a GM crop, and so amendments to study design and the analysis plan should be considered in order to assess the effect of multiple years of GM crop cultivation. Tables 4.1 and 4.2 in Appendix 1 present the results from the previous seven years and the 2013 results. The inclusion of the pooled results would be of interest.

**4.3. Substantiation of results**

Forty-two farmers (16.4 %) indicated that they had changed their insect control practices in the maize MON 810 compared with conventional maize in 2013; The report states: 'The difference arises from farmers using less insecticide applications in general' and 'as well as from farmers not controlling corn borers any more with conventional insecticide applications'. Nineteen farmers (7.4 %) assessed the germination of MON 810 to be more vigorous than that of conventional maize. Even though the percentage is lower than 10 %, the null hypothesis of p<sub>More vigorous</sub> ≥ 0.1 was not rejected (the 99 % confidence interval for this parameter was between 0.032 and 0.116). The report indicates: 'The more vigorous germination is likely associated to the quality of the germplasm'. Forty-four farmers (17.2 %) reported a reduction in stalk and root lodging in the maize MON 810 compared with the conventional field. A reduction in stalk and root lodging was also observed in the field trial studies. Thirty-two farmers (12.5 %) reported delayed maturity. Thirty-two farmers (12.5 %) reported that the maize MON 810 field was less susceptible to diseases, with associated reports of reduced susceptibility
predominantly to *Fusarium* spp. (17 farmers) and *Ustilago maydis* (14 farmers). The reports of reduced susceptibility to fungal infections were substantiated with similar findings from the scientific literature. The number of farmers who reported that maize MON 810 provided 'very good' control of *O. nubilalis* and *Sesamia* spp. was 208 (89.1 %) and 209 (valid percentage 81.6 %), respectively. These results are to be expected, as the genetic modification provides protection from corn borers and therefore should result in a healthier crop. Forty-six farmers (18.0 %) reported maize MON 810 to be less susceptible to pests other than the two borers mentioned above. An increased yield was reported by 89 farmers (34.8 %); as maize MON 810 suffers less insect damage, an increased yield is not unexpected. Concerning the characteristic 'Maize borer control practice', table 3.1 indicates statistically non-significant results for testing of the null hypotheses $p_{As \text{ usual}} \le 0.9$ and $p_{Plus} \ge 0.1$; however, the comparisons are not described in the text of the report.

For the monitoring characteristics above, the null hypothesis that an effect was evident could not be rejected. One farmer (0.4 %) reported an assessment of the MON 810 maize as more susceptible than conventional maize to pests other than the two borers mentioned above (*O. nubilalis* and *Sesamia* spp.) (however, the corresponding null hypothesis of $p_{More susceptible} \ge 0.1$ could be rejected). Two farmers (0.8 %) assessed the germination of MON 810 as less vigorous than conventional maize, but the corresponding null hypothesis of $p_{Less vigorous} \ge 0.1$ could be rejected. Moreover, a lower yield, in comparison with conventional maize, was reported by five farmers (2.0 %—however, the corresponding null hypothesis of $p_{Lower} \ge 0.1$ could be rejected).

Confidence intervals have been provided graphically, but only for the probabilities of 'as usual' answers. Presenting all results with confidence intervals would have facilitated the interpretation of the results and would have allowed the effect of the selection of alternative threshold values other than the arbitrarily selected 10 % to be explored.

A closed principle test procedure is proposed in the report to be used in order to address the issue of the overall Type I error. However, in practice, in the monitoring characteristics comparisons presented in the report (see table 3.1), the results of all possible tests are presented, even if in the first hypothesis test the null hypothesis $p_{As usual} \le 0.9$ is rejected. Hence, the closed principle test procedure is not followed and the issue of the overall Type I error in multiple testing is still relevant.

The monitoring characteristics that were not 'as usual' described above were also observed in the 2012 monitoring reports, except for the germination vigour characteristic. The consistency of the results in each year of survey indicates the stability of the observed effects. Interpretation of the results should be viewed with caution as there are indications of weaknesses in the sampling methodology applied for the survey and, as a consequence, the possibility of selection bias in the survey cannot be excluded. It is important that an appropriate and consistent sampling methodology is used. The grouping of sample units according to the strata and random selection of sample units from within the strata should be performed using the specified sampling frame prior to conducting the interviews. A description of the method to ensure that units are randomly selected from the sampling frame should be included in the report, including, where relevant, the statistical software and/or the program code used for this procedure. The proportion of new farmers and farmers with previous experience of maize MON 810 selected from the sampling frame for each region should be presented in the report to provide evidence that the sampling method ensures that areas of intensive maize MON 810 cultivation are appropriately covered in the survey.

**5. Recommendations and conclusions**

From the data provided in the 2013 survey from the farmer questionnaire to monitor adverse effects associated with the cultivation of maize MON 810, no adverse effect can be identified. However, the revised annual 2013 MON 810 report provides limited information on the sampling methodology and the possibility of selection bias in the survey cannot be excluded. Therefore, the following improvements to the survey design and reporting are recommended:

* Full details on the source of the sampling frame, the number of farmers and the major characteristics (e.g. previous cultivation of maize MON 810) of the farmers should be included in the survey report. The national registers set by Member States for the cultivation of GM crops would be the optimum sampling frame, if available.
* A full description of the method by which the number of farmers to be monitored per year and country is determined, should be provided. It is currently indicated that this is based on 'the magnitude (ha planted per country/ ha planted in the EU) and product situation (average field size in the country) of MON 810 cultivation'; however, inadequate clarification is given on how this is actually done.

* A description of the method to ensure that units are randomly selected from the sampling frame should be included in the report, including, where relevant, the statistical software and/or the program code used for this procedure. The proportion of new farmers and farmers with previous experience of maize MON 810 selected from the sampling frame for each region should be presented in the monitoring report to provide evidence that the sampling method ensures that areas of intensive maize MON 810 cultivation are appropriately covered by the survey.

* The losses to sampling should be fully documented in the report to provide evidence that there is no non-response bias. It is important to know if a specific subgroup of farmers is not participating in the survey and therefore is not represented in the survey findings.

* It is recommended that independent trained interviewers are used to reduce interviewer bias. It is also recommended that the interviews are conducted in the same way for all participating farmers, for example either by telephone or by personal interview. In cases where this is not possible, additional attention should be given to the uniformity of the delivery of the questions and the achievement of results of comparable quality.

* It is recommended that the farmer questionnaire contain questions to record whether the comparator field is on the same farm in the same growing season and the variety of the comparator. If no comparators are being grown spatially or temporally close to the GM crop, then the rationale for selecting another comparator (e.g. maize grown in previous years) should be fully described. The comparators selected by the farmers for the survey should be summarised in the monitoring report.

* Farmer questionnaires should focus only on changes that would be recognised by the farmer during the daily management of the farm. However, additional questions could be included to gain a better understanding of the intensity of GM maize cultivation on the farm (number of years of maize MON 810 cultivation and frequency of maize MON 810 in crop rotations), and further information on plant protection product usage (in particular, in the comparator field) should be obtained to facilitate a full understanding of any observed changes. Moreover, qualitative responses may sometimes relate to a subjective assessment on the part of the farmer. An effort should be made to use objective measurable outcomes, whenever this is possible.

* The choice of statistical test should be based on the number of possible outcomes, since the use of a series of binomial tests for multinomial distributions would increase the experiment-wise Type I error rate (i.e. failure to detect a true adverse effect). In the current analysis, a closed principle test procedure is proposed to be used in order to address the issue of the overall type I error. This approach is acceptable, and can be effective for this purpose, if applied correctly, with the relevant not 'as usual' effects being assessed only when the null hypothesis $p_{As \text{ usual}} \le 0.9$ is not rejected. However, in practice, in the monitoring characteristics comparisons presented in the report (see table 3.1), the results of all possible tests (two or three, depending on the structure of the question) are presented, even if in the first hypothesis test the null hypothesis $p_{As \text{ usual}} \le 0.9$ is rejected. Hence, the closed principle test procedure is not followed and the issue of the overall type I error in multiple testing is still relevant. Statistical analysis should also account for the stratified nature of the sample.

* The statistical analysis should be planned to allow an analysis of the monitoring characteristics according to the length of GM crop cultivation in order to assess residual effects. As the statistical power of the study will be achieved only after 10 years, this will require a pooled analysis. Consequently, when conducting the survey, consideration should be given to the consistency of questions to assess monitoring characteristics, the inclusion of the same farmers in consecutive years in the survey (and the enumeration of these farmers in the report) and the interim analyses performed for the annual reports.
* The presentation of the results reported in Appendix 1 of the report should be improved in order to facilitate their interpretation.