# Multi-country perspectives on best practices and barriers to preference elicitation lab-in-the-field experiments with farmers

Marieke Cornelia Baaken, Laure Kuhfuss, Douadia Bougherara, Mikolaj

Czajkowski, Macario Rodriguez-entrena, Julia Höhler, Carl-johan Lagerkvist, Antonio Paparella, Erika Quendler, Jens Rommel, et al.

► To cite this version:

Marieke Cornelia Baaken, Laure Kuhfuss, Douadia Bougherara, Mikolaj Czajkowski, Macario Rodriguez-entrena, et al.. Multi-country perspectives on best practices and barriers to preference elicitation lab-in-the-field experiments with farmers. Applied Economic Perspectives and Policy, 2025, 47 (2), 10.1002/aepp.13488. hal-04950917

HAL Id: hal-04950917

https://hal.science/hal-04950917v1

Submitted on 17 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multi-country perspectives on best practices and barriers to preference elicitation lab-in-the-field experiments with farmers

Marieke Cornelia Baaken<fnref n="1" /> | Laure Kuhfuss<fnref n="2" /> |

Douadia Bougherara<fnref n="3" /> | Mikołaj Czajkowski<fnref n="4" /> |

Macario Rodriguez-Entrena<fnref n="5" /> | Julia Höhler<fnref n="6" /> |

Carl-Johan Lagerkvist<fnref n="7" /> | Antonio Paparella<fnref n="8" /> |

Erika Quendler<fnref n="9" /> | Jens Rommel<fnref n="7" /> | Julian Sagebiel<fnref n="10" /> |

Christoph Schulze<fnref n="11" /> | Tanja Šumrada<fnref n="12" /> |

Annika Francesca Tensi<fnref n="6" /> | Sophie Thoyer<fnref n="3" /> |

Riccardo Vecchio<fnref n="8" /> | Katarzyna Zagórska<fnref n="4" />

## Correspondence

Marieke Cornelia Baaken, Department of Environmental Economics, School of Business Administration and Economics, Bar barastraße 12, 49076 Osnabrück,

## Abstract

In this study, we define, categorize, and systematically describe best practices and barriers to implementing

This is an open access article under the terms of the [Creative Commons Attribution License](https://creativecommons.org/licenses/by/4.0/), which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

© 2024 The Author(s). *Applied Economic Perspectives and Policy* published by Wiley Periodicals LLC on behalf of Agricultural & Applied Economics Association.

<footnote n="1">¹Department of Environmental Economics, Institute of Environmental Systems Research and Faculty of Economics and Business Administration, Osnabrück University, Osnabrück, Germany</footnote>

<footnote n="2">²Social Economic and Geographical Sciences Department, The James Hutton Institute, Dundee, Scotland, UK</footnote>

<footnote n="3">³CEE-M, University of Montpellier, CNRS, INRAE, Institut Agro, Montpellier, France</footnote>

<footnote n="4">⁴Faculty of Economic Sciences, University of Warsaw, Warsaw, Poland</footnote>

<footnote n="5">⁵WEARE – Water, Environmental and Agricultural Resources Economics Research Group, Universidad de Córdoba, Córdoba, Spain</footnote>

<footnote n="6">⁶Business Economics Group, Wageningen University & Research, Wageningen, The Netherlands</footnote>

<footnote n="7">⁷Department of Economics, Swedish University of Agricultural Sciences, Uppsala, Sweden</footnote>

<footnote n="8">⁸Department of Agricultural Sciences, University of Naples Federico II, Naples, Italy</footnote>

<footnote n="9">⁹Federal Institute of Agricultural Economics, Rural and Mountain Research, Vienna, Austria</footnote>

<footnote n="10">¹⁰German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig, Leipzig, Germany</footnote>

<footnote n="11">¹¹Leibniz Centre for Agricultural Landscape Research (ZALF) e.V., Müncheberg, Germany</footnote>

<footnote n="12">¹²Biotechnical Faculty, University of Ljubljana, Ljubljana, Slovenia</footnote>
<figure><img src="image_2.png" /></figure>

Germany.

Email: marieke.baaken@uos.de

**Funding information**

Swedish Research Council for Sustainable
Development (FORMAS); University of
Montpellier (SRUM); French National
Research Agency, Grant/Award Number:
ANR-16-CONV-0004; Wageningen
University & Research and the Business
Economics Group; Faculty of Economic
Sciences, University of Warsaw; Czech
Science Foundation, Grant/Award
Number: 19-26812X; Scottish
Government RESAS; Pays de Loire
Region through RFI Alliance Europa;
Slovenian Research Agency; European
Union's Horizon 2020, Grant/Award
Number: 952303; Italian Ministry of
Education, University and Research
(MIUR), Grant/Award Number:
2017JYRZFF; European Commission
(EC); Universidades de Córdoba,
Grant/Award Number: JRC/SVQ/2021/
VLVP/0333

Editor in charge: Mindy Mallory

preference elicitation lab-in-the-field experiments with farmers based on qualitative, in-depth interviews with researchers from the Research Network on Economic Experiments for the Common Agricultural Policy (REECAP). Following an assessment of the challenges the researchers faced and the solutions they adopted, we provide a step-by-step guide to conducting such experiments. We elaborate on limitations and provide recommendations and suggestions for future research.

**KEYWORDS**

experimental economics, farmer recruitment, lab-in-the-field experiments, replication, stakeholder fatigue

**JEL CLASSIFICATION**

Q18, C90, C83

Exploring and understanding the behavior and decision-making of farmers is crucial for designing agricultural policies that can transform the agricultural sector at the scale and pace required to achieve environmental sustainability in Europe and beyond (Brown et al., 2021; Campbell et al., 2017; Pe'er et al., 2020; Rockström et al., 2017). Economic experiments are a useful tool to obtain robust and reliable information on farmers' preferences that can, in turn, help researchers and policymakers understand and anticipate farmers' responses to incentives and regulations (Colen et al., 2016; Herberich et al., 2009; Lefebvre et al., 2021; Palm-Forster & Messer, 2021; Thoyer & Préget, 2019).

In this study, we gather best practices and identify potential barriers to implementing preference elicitation lab-in-the-field experiments with farmers. We consider each step of the process, from design to participant recruitment to data collection and follow-up. There are a range of approaches that fall under the definition of economic experiments (Colen et al., 2016), but lab-in-the-field experiments¹ have gained particular popularity (Palm-Forster & Messer, 2021). Lab-in-the-field experiments occupy a middle ground between lab experiments—in which protocols are implemented in a controlled setting with participants from a maintained experimental subject pool (often students)—and the collection of data occurring naturally in the field.

Lab-in-the-field experiments are similar to lab experiments because in both cases participants perform tasks (i.e., make decisions) following a controlled experimental protocol. However, lab-in-the-field experiments are implemented with the population of interest (here, farmers), often outside of the lab, with different degrees of context introduced in the protocol (Harrison & List, 2004; Lefebvre et al., 2021). By focusing on incentivized tractable decision tasks, lab-in-the-field experiments maintain a link to economic theory while addressing how
<figure><img src="image_3.png" /></figure>

diverse contexts and subject experiences interact with economic decision-making (e.g., Henrich et al., 2004 for the importance of cultural norms for bargaining). Using standardized approaches such as controlled experimental protocols in multiple contexts can help researchers in developing a comprehensive understanding of decision-making behaviors. Standardized approaches are especially valuable when the goal is to understand the behavior of specific populations in their natural context while maintaining a degree of experimental control. Lab-in-the-field experiments allow researchers to elicit information on participants' behavioral traits and preferences in environments that are more representative of their real-life decision-making contexts than is possible in a traditional laboratory setting. This feature of the approach is important when exploring behaviors and decisions, such as farmers' risk preferences and risk management decisions, that are deeply embedded in specific cultural, social, or economic contexts. Therefore, lab-in-the-field experiments usually display good internal validity but also satisfactory external validity (Cason & Wu, 2019).

Lab-in-the-field experiments can have different aims. Some are designed to elicit information on behavioral characteristics (e.g., risk aversion, time preferences, and other-regarding preferences)—which we refer to as preference elicitation experiments—while others are used for ex ante evaluation of interventions. We build on Rommel et al. (2023) and focus on the implementation of preference elicitation experiments, the results of which can provide useful values for the parametrization of other behavioral models (e.g., farmer's decision-making). These experiments also help us better understand and characterize the heterogeneity observed in the impact of programs on farmers' behavior.

Implementing experiments outside of the laboratory adds additional challenges that must be overcome; we highlight these in this paper. Being able to recruit farmers and implement preference elicitation lab-in-the-field experiments to the best standards is important for the reliability of parameters intended to be used in behavioral models, and our paper is intended to establish what these standards are. We build on the experience of researchers participating in a large replication study that elicited the risk attitudes of farmers in various EU countries using a lab-in-the-field experiment. All researchers in the REECAP study implemented the same protocol in parallel using a range of approaches and contexts to gain insights into diverse challenges and solutions that are useful for future implementation of lab-in-the-field experiments; this approach also addressed concerns regarding the impact of single study designs on experimental outcomes (Huber et al., 2023).

Based on qualitative interviews with researchers, we offer a systematic exploration of best practices and barriers to the implementation of lab-in-the-field experiments with farmers. However, our objective is not to disentangle the causes of success in this implementation in a quantitative sense. Rather, we are interested in collecting the knowledge and opinions of a group of researchers based on their experience. In addition, since we draw on the analysis on a lab-in-the-field experiment eliciting individual preferences, we cannot address specific challenges that might arise where the experiments' participants are required to interact with one another.

We gathered the required information through qualitative, in-depth interviews with researchers from the Research Network on Economic Experiments for the Common Agricultural Policy (REECAP), who participated in the multi-country replication project in 2021 (Rommel et al., 2023).² For this purpose, we conduct a qualitative content analysis using recorded interview data. We also analyze the anonymous feedback provided by farmers after they completed the required tasks in the REECAP online experiment. The results of this analysis offer the scientific community information on drivers and barriers to the implementation
<figure><img src="image_4.png" /></figure>

and management of preference elicitation lab-in-the-field experiments with farmers. The results also form the basis of our proposed step-by-step guideline.

Some of the challenges identified for the implementation of these experiments are shared with all types of empirical research relying on surveys to collect data from farmers; for example, respondent fatigue arises from the burden of too many requests and results in low response rates. Pennings et al. (2002) provide valuable guidance to increase response rates, and these recommendations apply here. However, we also identify the specific characteristics of lab-in-the-field experiments that raise unique challenges and propose a range of solutions and guidance in response. The characteristics we identify are the perceived game-like nature of the approach, the decontextualization and abstract nature of some protocols, and the incentivization of participants according to the choices they make during the experiment. In that regard, and as detailed in the discussion section, we complement the initial recommendations provided by Lefebvre et al. (2021), Rosch et al. (2021), and Weigel et al. (2021) for the implementation of preference elicitation lab-in-the-field experiments with farmers.

The structure of the study is as follows. In Section 1, we explain the methodology, including the sample selection, data collection, and qualitative methods used to answer the research question “What are the best practices and barriers to implementing lab-in-the-field experiments with farmers?” In Section 2, we present the main findings of the qualitative content analysis and the anonymized farmer feedback from the REECAP replication project. We discuss the results and limitations of the present study and present policy recommendations and suggestions for future research in Section 3.

# METHODOLOGY AND DATA

## Context

Initial debates on low replicability and inflated risks of false positives arose in psychology (Simmons et al., 2011) and have since spread to the economics and agricultural economics communities (Ferraro & Shukla, 2022; Finger et al., 2023; Heckelei et al., 2023). In a large-scale replication project, Camerer et al. (2016) attempt to directly replicate 18 laboratory experiments in “top 5” journals and show there is room for improvements in their replicability. Replication in research entails the utilization of diverse methodologies to reanalyze original data via identical (verification) or distinct (reanalysis) techniques. Additionally, it involves the acquisition of new data through either identical (direct replication) or alternative (extension) methods. This multifaceted approach is discussed in Christensen et al. (2019).

We are concerned with describing our experiences with what, to the best of our knowledge, is the first attempt to replicate an agricultural economics experiment across farming and country contexts. Between 2020 and 2022, a team of 28 researchers from 11 countries in geographical Europe participated in a project to replicate Bocquého et al.'s (2014) well-known study of farmers' attitudes to risk. The original study uses multiple price lists developed by Tanaka et al. (2010), embedded in 1-on-1 in-person interviews. The researchers from the replication project used the same protocol, which was adapted for online or in-person implementation. The researchers, all experienced in implementing lab-in-the-field experiments with farmers, used a range of recruitment and data collection strategies to reach 1430 farmers from 10 countries (the UK team was unable to recruit sufficient farmers). More details about the replication project and its results are available in Rommel et al. (2023). We build on the
<figure><img src="image_5.png" /></figure>

research team's experience from the replication project and, more broadly, assess the best practices and barriers to implementing lab-in-the-field experiments with farmers.

## Interview sample

The data for this study were collected through interviews with the researchers who collaborated on the REECAP replication project (Rommel et al., 2023).³ We ensured coverage of all aspects of the project by having the two lead authors collect data through qualitative in-depth semi-structured interviews. In that sense, this paper is not typical qualitative research, although qualitative methods are used to improve its rigor. We conducted online interviews in May 2022 using the Webex software (n = 11), with each interview lasting 32 to 70 min. All interviews were digitally recorded and subsequently anonymized and transcribed following the practical transcription guideline developed in Azevedo et al. (2017). We conducted eight interviews with single participants, two interviews (ID 02; ID 08) with two participants, and one (ID 03) with three participants from the same country team. We used purposive rather than random sampling for participant selection as we contacted researchers from the REECAP project only.

## Data

The complete interview guide is available in Supplementary Material 1. The collaborative work and thus trusting relationship, fostered by the confidentiality maintained among participants, enhances the validity of question-answer interactions (Roller & Lavrakas, 2015).

Part 1 of the interview consisted of an introduction including (i) information about the interview (e.g., content, duration, and dates of the interviews), (ii) a privacy and impartiality statement, and (iii) a space to pose questions. Part 2 followed a semi-structured procedure; this approach was used to obtain a nuanced understanding of the best practices and barriers that researchers face when implementing lab-in-the-field experiments with farmers across 11 European countries. During the semi-structured interview, we asked nine open-ended questions concerning the following: the data collection process, the representativeness of the sample, the sampling process, including relevant barriers and best practices, recommendations for future preference elicitation lab-in-the-field experiments with farmers, main lessons learned, feedback from farmers and payment procedures of the experiment. The use of a semi-structured approach coupled with an interview guideline allowed us to respond to emerging topics in the course of the session. We employed probing questions strategically to modify the phrasing or sequence of inquiries, thereby ensuring consistency and comparability across interviews (Roller & Lavrakas, 2015). In Part 3 of the interview, participants could give feedback, clarify open questions, make comments, and discuss aspects that they thought had not been well covered in the interview. Sections 1 and 2 generated the data for the qualitative content analysis. We increase the transparency and reproducibility of the results by following the interview method of Roller and Lavrakas (2015); the method covers issues such as determining research questions and obtaining consent from participants.
<figure><img src="image_6.png" /></figure>

Qualitative content analysis

The transcripts provide a large amount of qualitative data that is highly context-specific and requires interpretation. We use qualitative content analysis and the MAXQDA 2022 software to structure the data. Data are deductively coded into categories and according to the phases of the experiment. We use the framework analysis method adapted from Espinosa-González and Normand (2019). In the first round of data review, we build initial categories inductively based on clusters of similar information (i.e., “challenges,” “recommendations,” “positives,” “lessons learned,” and “feedback”). In subsequent rounds of data review, we refine the initial categories and identify additional sub-categories. We then compare categories across cases and assign categories (i.e., distinct barriers and best practices) to the different phases in preference elicitation lab-in-the-field experiments with farmers. The data are coded by the lead author and cross-checked by the second author, with discrepancies resolved by agreement.

Sample characteristics

Table 1 shows that most of the participants' lab-in-the-field experiments were conducted online (n = 7), and four complemented online data collection with face-to-face interactions. The farmer response rate was heterogeneous from under 1% to around 92%, depending on the recruitment approach taken. Five country teams hired a local marketing agency for recruitment and/or to handle the payment procedure. The sample selection strategy differs among countries, with different country teams targeting different types of farmers: arable farmers in Austria, Germany, the Netherlands, and Sweden, wine growers in Croatia, potato farmers in northern France and Scotland (UK), olive growers in Apulia (Italy) and Andalusia (Spain), young farmers in Slovenia and farmers in a cross-section of specializations in Poland. None of the final samples are random samples, and farmers self-selected into the experiment.

RESULTS

Several participants noted that although they had previously conducted many lab-in-the-field experiments, the issues encountered were always different (e.g., difficulties with sampling and data collection and low participation rates). The country teams rated the perceived difficulty of data collection in their studies on a 5-point Likert scale ranging from 1, “very easy,” to 5, “very difficult.” One country team (Sweden) assessed their data collection process as very easy, two as rather easy (Germany and the Netherlands), three as neither difficult nor easy (Croatia, Austria, and Poland), four as rather difficult (Slovenia, France, Italy, and Spain) and one as very difficult (Scotland).

All country teams that stated that their process was rather easy or very easy either hired a
marketing agency or had access to a sampling frame (e.g., through the respective country's
national statistics office). Vice versa, none of the country teams that stated that their process
was rather difficult or very difficult hired a marketing agency for sampling; only Spain did so to
process the payment. Our best practice guideline contrasts the best practices of groups with a
relatively easy data collection process and with the challenges faced by those employing a rela-
tively difficult process for data collection. Our intention is to help researchers prepare and over-
come possible difficulties encountered in conducting preference elicitation lab-in-the-field
experiments with farmers. The steps, challenges, best practices, and phases are depicted in
Table 2 and further explained in the following section.
<table><caption>TABLE 1 Selected sample statistics of the REECAP experiment.</caption><thead><tr><th>Country</th><th>Marketing agency hired</th><th>Sampling approach</th><th>Response rate (approx.)</th><th>Mode</th><th>Sample size, n = 1430</th></tr></thead><tbody><tr><td>Sweden</td><td>Yes (only for payment)</td><td>Random sample from a list of farmers (emails via the national statistics office)</td><td>1%</td><td>Self-administered online</td><td>218</td></tr><tr><td>The Netherlands</td><td>Yes (for recruitment & payment)</td><td>Pre-defined list of farmers (emails via marketing agency)</td><td>3%</td><td>Self-administered online</td><td>160</td></tr><tr><td>Spain</td><td>Yes (only for payment)</td><td>Agricultural cooperatives (olive oil mills) & snowballing</td><td>64%</td><td>Face-to-face group sessions</td><td>130</td></tr><tr><td>Italy</td><td>No</td><td>Agricultural cooperatives (olive mills) & emails</td><td>47%</td><td>Face-to-face group sessions</td><td>130</td></tr><tr><td>France</td><td>No</td><td>Pre-defined list of farmers & newsletter potato grower association & social media</td><td>1% (estimated)</td><td>Self-administered online</td><td>124</td></tr><tr><td>Poland</td><td>Yes (for recruitment)</td><td>Pre-defined list of farmers (via agricultural advisors and marketing agency)</td><td>30%</td><td>Face-to-face individual session & self-administered online</td><td>169</td></tr><tr><td>Austria</td><td>No</td><td>Pre-defined list of farmers (agricultural chamber meetings)</td><td>92%</td><td>Face-to-face individual</td><td>128</td></tr><tr><td>Scotland</td><td>No</td><td>Pre-defined list of farmers, newsletter & social media</td><td>&lt;1%</td><td>Self-administered online</td><td>0</td></tr><tr><td>Slovenia</td><td>No</td><td>Young farmer events and agricultural fairs, website of national rural network & young farmers association newsletter</td><td>3%</td><td>Face-to-face individual sessions & self-administered online</td><td>114</td></tr><tr><td>Croatia</td><td>No</td><td>Pre-defined list of farmers</td><td>17%</td><td>Self-administered online</td><td>104</td></tr><tr><td>Germany</td><td>Yes (for recruitment & payment)</td><td>Pre-defined list of farmers (emails via marketing agency)</td><td>&lt;1%</td><td>Self-administered online</td><td>153</td></tr></tbody></table>

Note: “Country” refers to the country of data collection.

## Pre-experiment phase: Awareness raising, promotion, quality assurance, and collaboration

Ensure that experiments are perceived as highly relevant from the farmers' perspective and address farmers' skepticism through transparent communication and engagement

Before designing the experiment, researchers should acknowledge that farmers may be skeptical of researchers. Individuals designing experiments can address the skepticism by researching the demographics and characteristics of the farmers with whom they will be working. These
<figure><img src="image_7.png" /></figure>

<table><caption>TABLE 2 Clustered topics and best practices, ordered by stages in the experimental process.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Pre-experiment phase: Awareness raising and promotion, quality assurance and collaboration</td><td>Ensure that experiments are perceived as highly relevant from the farmers' perspective and address farmers' skepticism through transparent communication and engagement<br>• Read up on farmers' habits, including language, customs, working times, current agricultural political events, and changes in political narratives and contexts.<br>Cultivate strong motivation and build the participation capacity of farmers and stakeholders<br>• Network and build a culture of trust with key stakeholders (e.g., elected representatives in farm unions or cooperatives, farm advisors, management staff in inter-professional structure) to increase know-how, access sampling frames, get help with distributing the experiment, or obtain permission to conduct the experiment on stakeholder premises.<br>• As with economic labs, establish a pool of participants (farmers) for selection who will actively engage in various activities such as experiments, surveys, discussions, or consultations.<br>• Contact other institutions and universities to network.<br>Ensure high data quality<br>• Ensure the sample is representative by, for example, collecting probabilistic samples.<br>• For samples provided by a professional market research company, ensure the accuracy of farmer panels and database.<br>• Ensure proper training of researchers by, for example, actively engaging protocols in research networks or training in (statistical) training sessions.<br>• Ensure quality of research by presenting protocol to other researchers prior to data collection.<br>• Identify additional research partners as necessary to increase sample size and replication opportunities in different contexts.<br>• Employ streamlined and targeted approaches to avoid overwhelming farmers with surveys and experiments. Contact other institutions and universities working in the same area to possibly coordinate experiments and resources.</td></tr><tr><td>Design phase: Language, practicability and innovativeness</td><td>Implement strategies to mitigate farmer fatigue from experiment participation<br>• Create appealing recruitment materials.<br>• Add a brief introduction (e.g., a short video or figures) that includes, for example, advantages of participation, intent, purpose, and relevance of the study—farmers can be expected to participate if they perceive the study to be relevant to them, their peers and/or society.<br>• Offer to share the survey results and conclusions with respondents once the research project is completed (either by offering a registration link or by indicating a date on which conclusions will be made public via appropriate media).<br>Ensure a clear understanding of the experiment and foster strong motivation<br>• Create clear, simple, short, and motivating experiments; the title of the survey can serve as an enticement.<br>• Carefully assess the appropriate level of contextualization required.<br>Enhance farmers' trust in research endeavors<br>• Design the experiment to include a mechanism by which potential comments left by participants can be followed up.<br>• Check the need for and characteristics of performance-based payment systems.<br>• Establish a reliable payment procedure (i.e., payments should be delivered on time) and ensure that the payment vehicle is incentive-compatible.<br>• Consider co-designing the experiment with farmers and relevant stakeholders (see, e.g., Höhler et al. (2023) and Hölting et al. (2022) for details on co-designing processes with farmers).<br>• Receive experiment approval (e.g., through mock interviews, co-design, farmers associations, ethics approval).<br>Implement strategies to attain a sufficiently large pool of farmers<br>• Add recruitment questions to create a farmer panel for future studies.</td></tr></tbody></table>
<figure><img src="image_8.png" /></figure>

TABLE 2 (Continued)

* Add screening questions to ensure responses from the desired target group only.

* Set a maximum number of participants within the experiment software to remain within the budget and prevent extensive fraud.

Implement strategies to maintain participant engagement and minimize attrition

* Incorporate/add a comment box or drop-down question alongside a registration form, for example, by implementing a pop-up window if the farmer attempts to close the experiment software prematurely.

Data collection phase: Timing, experiment method, and payment

Implement effective farmer recruitment strategies and carefully select an appropriate sampling strategy

* Carefully consider the timing of data collection. When choosing the time of year, busy phases of the agricultural season should be avoided; weather forecasts can be indicators of when farmers are more likely to be available. Allowing the possibility of evening interviews can help ensure availability.

* Working with farm advisors or cooperatives has led to successful farmer recruitment, making it a reliable method for obtaining a representative sample.

* Farmers' events, such as agricultural fairs or general assemblies of cooperatives, are also good opportunities to collect data, and presentations or training sessions can be organized to run in tandem with the experimental session.

* Predetermine an appropriate sampling strategy (i.e., representativeness).

* If considering panel providers, ensure they are transparent regarding their panel recruitment, have experience in farm research, respond quickly to farmers' questions, and are able to administer payment of monetary rewards.

* Send reminders with parsimony.

For better comprehension and feedback

* Face-to-face data collection enables direct feedback, which is particularly beneficial for experiments requiring clarity, such as those involving abstract tasks.

Increase the perceived legitimacy of the experiment

* Establish trust in the study by, for example, creating a project web page affiliated with a trusted research institute or partner website.

* Collaborate with farm advisors and cooperatives.

Implement streamlined payment procedures to facilitate ease of transaction and tackle payment-related challenges

* If earnings of experiment participants are relatively low, it may help to only pay a share of participants and increase the stakes (see detailed information in Charness et al. (2016))

* Utilize coupons or vouchers for cooperative shops; these may be more widely accepted and more convenient than cash.

* Consider alternative payment methods such as VISA gift cards or big vendor codes via email.

Ensure IT safety of experiments

* Be wary of recruitment via social media (not all social media and other online platforms are equally affected) because monetary incentives encourage fraudulent participation of non-farmers. Unique links should be sent to verified farmers, and bot-screening mechanisms can be included on the data collection platform.

* Possibly include a link for farmers to register interest and provide details to be contacted with a unique and personal link to the actual experiment

* Make sure to set a maximum number of completed interviews within the survey software used for the experiment

Post-experiment phase: Sharing results and critical evaluation

Increase the reputation of the research group and the trust of participants

* Provide summary of study to partners and farmers with contact details obtained, for example, through an opt-in opportunity for farmers interested in receiving results or being involved in further research.

* Store anonymized data (i.e., contact details need to be stored separately from the dataset).

(Continues)
<figure><img src="image_9.png" /></figure>

<caption>TABLE 2 (Continued)</caption>

* Where possible, negative and positive feedback should be addressed for participants who have provided contact details and agreed to contact following data collection.

include factors such as language, customs, working schedules, and current agricultural policies and events. Additionally, participants emphasized the importance of carefully adapting any survey questions embedded in the experiment to reflect on local conditions.

ID04: If you want to work with almond producers, you should find someone that knows very well the almond producer world. Being conscious of the particularity of the sample you are going to face. You find people, also, it helps you approach people in the correct way and correct timing also. For example, like, when the olive oil harvesting period would have started it would have been impossible for me to reach the farmers. So, the data collection takes place in a period, when the farmers are not full of work.

One survey participant mentioned that the political atmosphere regarding research and policy prevailing during the experiment is crucial for farmers' participation.

## Cultivate strong motivation and build the participation capacity of farmers and stakeholders

Where researchers create a sample directly from a farmer population, it is essential to establish contacts with farmers' associations and cooperatives, as well as with peers early on in the study design phase to (a) understand the current situation of farmers and gain information about farmer events (e.g., agricultural conferences, agricultural fairs, agricultural consultant days), (b) network before conducting the experiment, (c) secure access to sampling frames, and (d) obtain necessary assistance with distributing the experiment, including on the researcher's behalf, and with obtaining permission to conduct the experiment at a stakeholder's premises. Collaborate with different stakeholders to increase social capital by, for example, increasing the commitment of farmers and stakeholders. For instance, in the Spanish case, the cooperative manager, an individual trusted by farmers and in charge of administrative tasks, committed to collecting some of the observations missed in the live guided session when farmers visited the cooperative's facilities.

In the interests of avoiding farmers becoming overwhelmed with researchers' surveys and experiments, it may be helpful to contact other universities and institutions working in the same area to combine experiments and resources.

ID02: Probably a critical question for us as researchers is also what are we doing to our future chances of being able to recruit farmers by sending more and more requests to take part in surveys. So, I think it's very relevant for us to really highlight the relevance and to kind of sell what we are doing.
<figure><img src="image_10.png" /></figure>

ID02: There are a lot of surveys done. A lot of data collected. And I think we all need to be careful that we don't overflow the farmers with surveys. And then there is no interest of farmers in participating. And it's really about which service do we need and for what.

## Ensure high data quality

If the sample is provided by a professional market research company, the quantity and quality of responses also depend on the size and reliability of farmer panels and related databases.

ID02: I think it's also important that the agency has some experience in that particular field because it's different when you do a survey among farmers. And then the next thing was to be clear about what we wanted from them.

One team stated the difficulty of convincing marketing agencies to work with them, given the scope of economic experiments.

ID11: It was more about convincing the marketing agency to participate than the farmers. Or like once they give their consent, I think it's easy for them to send the link and get the response. But in the first place, they have the power to say: "No, we don't do something like this. This is too crazy. What you want to do with this? If we do this, we destroy our panel or scare off the farmers." ... But since there was an individual payment and everything, they [the marketing agency] kind of liked it, and also, I think it was a learning experience for them, like how this works.

We also recommend that before designing the experiment, researchers should carefully consider how to increase data quality, for example, by collecting probabilistic samples, participating in research networks and in (statistical) training sessions.

ID01: There's this huge challenge, I think, in this line of work, in terms of access to participants, random samples in particular. Power issues, I think, very small sample sizes, very biased samples. So, we should really make an effort to work toward a better data quality. And I think that that problem might be easier to address in some developing countries actually, than in Europe. There, you also have quite good data, and you can also have good random samples. Farmers are often living much closer when they are smallholders, so often, you find a village where you have 20-30 farmers that you can easily sample. So, I assume in Europe, it is really difficult to just travel to an area and sample farmers randomly; that is almost impossible.

ID09: Like, even in the faculty of economics, they don't have a subject for students on experimental economics at all. So, you know, you can't really get this knowledge as part of the educational process.
<figure><img src="image_11.png" /></figure>

# Design phase: Language, practicability, and innovativeness

## Implement strategies to mitigate farmer fatigue from experiment participation

A brief but catchy introduction, which can include a short video or figures, can increase farmers' willingness to participate and their understanding of the purpose of the experiment. The introduction may include the advantages of participation for farmers and could set out the intent and purpose of the study. This best practice applies to the experiment itself and the recruitment material (e.g., email, texts, and advertisements). As mentioned above, farmers must trust that the experiment has an aim (e.g., lotteries are an established mechanism to study risk preferences) and that they should respond truthfully (i.e., they need to understand the project's relevance). For example, Stantcheva (2022) offers guidance on the design of a survey. In the case of the replication project, several farmers asked about the purpose of the experiment, which can be directly addressed in the case of face-to-face data collection but needs to be anticipated in the case of online surveys. Moreover, face-to-face data collection, by guiding participants through the experiment's successive steps, allows for the participation of farmers who may not have the technological skills or availability to successfully complete the experiment online.

## Ensure a clear understanding of the experiment and foster strong motivation

Designing an experiment for farmers is challenging because it must be relevant to the participants, innovative, and as short as possible. Experiment questions⁴ and texts that are easy to understand and clearly written, including agrarian vernacular, are more likely to be well received by farmers.

**ID07:** And always the challenge is to use the language of the farmers. Because scientific language is different. And the next thing is how you contact the farmer, what is the invitation like, and how do you speak to them. If you can do it in a way, where they can say that [the survey] is interesting, and I can learn something and I can share my knowledge, then they will do it.

The replication project experience shows that experiments should be as close to the farmers' reality as possible (Palm-Forster & Messer, 2021), and their relevance should be clear and closely linked to the design. Lotteries and other lab-in-the-field experiments are usually fairly decontextualized, which may challenge farmers' comprehension and motivation. For example, conducting the experiment alone at home can be challenging and may decrease the (lottery) choice consistency. Participants mentioned that farmers often question the experiment's purpose and sense when it is not comprehensible, too complicated, and decontextualized (i.e., too academic and technical and not transferable to the farmer's agricultural context).

**ID10:** So for some of them [i.e., farmers], it [the experiment] was not serious work. It was like a public lottery. So, what does it have to do with the scientific research?

**ID09:** I think, definitely, I don't think they made a clear connection with their farming situation or something like this. So, it was more of a like a thought
<figure><img src="image_12.png" /></figure>

[experiment]. So, what would I do in this kind of situation? It was not like, I now imagine my farm investments; it was like ... I think it was possible once they understood the contents. And this is what we made sure, that they understood the context, and my impression was that they did. But maybe the procedure is something remote, oftentimes they asked, "Oh, so it's a lottery like what are the results?" It was difficult to make the connection between the decision they made and the final results.

Some of the farmer feedback confirms this view: participants criticized the survey's lack of seriousness and considered it a “waste of time” because of the underlying “game” approach of the preference elicitation lab-in-the-field experiment. One farmer mentioned that a farmer's decision-making process cannot be compared to lotteries at all. Another farmer wrote that they are against lotteries in principle, but the approach nevertheless brought them joy. Others said that a lottery simplifies their decisions too much and that many other factors must be considered.

## Enhance farmers' trust in research endeavors

The description and procedure for payments should not increase farmers' skepticism toward experiments. Importantly, delivering payments for survey participation on time and seeking and abiding by ethics approval processes are good practices that help maintain participant trust.

ID06: Because they don't trust some outsider who tells them that he will pay them the money if they provide their bank account. There is a lot of such fraud procedure on the Internet already, so it's quite complicated with paying out the incentives.

One team, for example, used coupons that could be spent at the cooperative's shop (e.g., to purchase gloves, boots, and respirator masks for phytosanitary treatments); these were more well received than cash, which farmers were sometimes embarrassed to accept.

ID03: And that's another important thing. I think farmers prefer vouchers instead of cash. In some way, they feel, I will say, uncomfortable, because of the cash. We used a voucher as a strategy mechanism. They could spend the money in the cooperative shop.

Co-designing the experiment with farmers increases their trust and their understanding of the experiment. Ideally, approval for the experiment is obtained by important farmer associations and through mock sessions (see also Höhler et al., 2023). Moreover, it is advisable that follow-up surveys include a question enquiring whether the farmer wishes to receive a response to the feedback. In addition, providing space for comments is a good way for researchers to address some of the critical feedback and to maintain reputation and trust between research teams and experiment participants in the long term.
<figure><img src="image_13.png" /></figure>

## Implement strategies to attain a sufficiently large pool of farmers

First, screening questions ensure that researchers only receive responses from the desired target group, which could be farmers in general or specialized farmers. These questions qualify or dis-qualify participants from participating and should be carefully designed and pre-tested. A recruitment question can be added at the end of the experiment to increase the database of farmer contacts and create a pool of potential farmer volunteers who can be contacted for future experiments. Note that this may increase the likelihood of self-selection bias. When using experiments embedded in online surveys, it is helpful to set a maximum number of participants within the survey software to (i) ensure the experiment remains within budget and (ii) limit the number of false responses if the experiment is hacked.

## Implement strategies to maintain participant engagement and minimize attrition

A comment box or a short follow-up question should be added to address possible participant attrition before completion (e.g., installing a pop-up window that appears if the farmer indicates they wish to close the experiment software). This follow-up question should investigate the reasons for dropping out.

## Data collection phase: Timing, survey method, and payment

### Implement effective farmer recruitment strategies and carefully select an appropriate sampling strategy

Most researchers note that recruiting farmers to participate in experiments is very difficult and exceptionally time consuming. In addition, timing matters when collecting data from farmers. There are two timing dimensions: when farmers are contacted for recruitment and when they actually participate in the experiments (actual data collection). When choosing the time of year for data collection, researchers should avoid busy phases of the agricultural season (e.g., sowing or harvest time). Weather forecasts can indicate when farmers are more likely to be in the fields (e.g., if they have several sunny days in a row, these are likely useful for mowing, turning, raking, and baling hay). Researchers can provide flexibility for participation in the evenings, which can help ensure availability whether data are collected in person or by phone. When recruiting farmers, sending the invitation email at times when farmers are most likely to be in the farm office increases the chances that they will be available to read it.

ID02: The email, or sending the emails he [a marketing agency employee] tried to time, quite nicely, so that he was sending it early in the morning or late in the afternoon.

While the ideal approach for researchers is to work with a comprehensive list of farmer contacts and select a large representative sample, this is only feasible in countries where research organizations are provided with this information from relevant authorities (e.g., government agencies may hold farm census data or manage agricultural payments as is the case of the
<figure><img src="image_14.png" /></figure>

statistics offices in Sweden and Slovenia). The use of general public panels is problematic due to the limited availability of farmers within these panels (e.g., only 3.9% of the panel members were farmers in the largest Spanish panel). Therefore, panel providers cannot guarantee the minimum sample size required (Rommel et al., 2023). However, there may be low response rates even when census contact details are available (see Table 1); recruitment through trusted partners such as cooperatives yields much higher response rates.

Most research teams in the replication study needed to rely on snowballing or on-site sampling and then check the representativeness of the sample ex-post. This sampling approach leads to self-selection bias, which may also occur within representative samples, but farmers interested in the research are unlikely to be representative of the population. Collaborations with farm advisors or cooperatives have proven effective in recruiting farmers. The Italian research team collected data when farmers delivered their harvest to the cooperative and needed to wait or were free of further harvest-related commitments for the day. The Spanish team organized a successful session at the cooperative where they provided practical information about the recent Common Agricultural Policy (CAP) reform to the farmers and then conducted the experiment at the end of the event. Other researchers collected data at the farm advisors' office, for example, when farmers visited their advisors to fill in applications (e.g., for CAP subsidies).

Farmer events, such as agricultural fairs, are also good opportunities to collect data. Presentation or training sessions can be organized and coupled with the experimental session. In one instance, farm advisors agreed to administer the survey themselves in exchange for a fee. In such cases, advisors should be carefully briefed and trained, as experience in other contexts has shown that advisors can introduce bias by getting too involved in helping farmers fill out the survey. Concerns about non-probabilistic samples are warranted in these instances, as standard errors do not reflect sampling error, and selection bias may occur (see Heckelei et al., 2023 for a discussion).

Market research agencies were hired in some cases and provided farmer panels. In such cases, the agencies relay the link to the online experiment to a panel of farmers or assist participants through computer-assisted telephone interviews. While this approach saves time for the research team, it can be expensive. In the interests of data quality, we recommend selecting a company that is transparent concerning its panel recruitment processes, has experience in farm research, and is able to administer payments of monetary rewards. They also need to be able to respond quickly to farmers' questions on the survey.

ID02: I worked with him [an employee of the marketing agency] before, and the communication was quite good. So, he replied quickly to the emails and the questions, and he also answered the emails that the farmers sent.

ID02: I asked him [agency employee] where he got the contact addresses of his sample from. He replied that he collected these email addresses [of the farmer panel] himself by going on websites, but also, when he conducted research with farmers, then he asked the farmers for their contact details, and that's how he got the email addresses from.

It is important to mention that when contacting a market research company, one should allow sufficient time; these companies often conduct multiple surveys at the same time and have only limited resources, as it has been the case for the German team. Moreover, trustworthy market research agencies (specialized in farmers) are only available in some countries.
<figure><img src="image_15.png" /></figure>

When circulating the online version of the experiment by email, reminders need to be sent, but with parsimony.

## For better comprehension and feedback

Our study shows that if the data collection were conducted face-to-face, researchers could directly ask for and address farmers' feedback, especially for less self-explanatory experiments (e.g., more abstract tasks which are more often the case in preference elicitation lab-in-the-field experiments). Thus, as stated above, face-to-face data collection facilitates the understanding of the experiment by guiding participants through the successive steps.

## Increase the perceived legitimacy of the experiment

A frequently mentioned concern of farmers was the increasingly high risks of scams and phishing emails. The concerns might explain the meager response rates when trying to recruit participants by email. Communicating about payments during online recruitment may also be treated with suspicion, and increases the probability that the experiment is seen as a scam. A project web page affiliated with a trusted research institute or partner's website increases perceived legitimacy. It can also increase if farmer associations or other trusted organizations spread the link. The web page can include a registration link for farmers to provide contact details that can be used to send a further unique and personal link to the actual experiment. Collaborating with farm advisors and cooperatives will also increase the perceived legitimacy of the experiment.

## Implement streamlined payment procedures to facilitate ease of transaction and tackle payment-related challenges

Participant payments are another critical aspect of preference elicitation lab-in-the-field experiments with farmers, who often refused the payment they earned through the experiment, especially if it was conducted online and not face-to-face. For instance, one farmer even stated that they felt humiliated to be offered a reward. Several farmers requested that the payment be transferred to a charity. Some farmers did not see the benefit of filling in the required forms to receive the payment when their earnings were relatively low. The relatively low earning of participants can be avoided by paying only a share of participants and increasing the stakes (e.g., an expected payoff of €100 for 1 in 10 participants instead of an expected payoff of €10 to all). Further discussion on whether all or a selection of participants should be paid is available in Charness et al. (2016). Consider using coupons or vouchers that can be spent, for example, at cooperative shops; these are likely to be more acceptable than cash (see Section 2.2).

## Ensure IT safety of experiments

Monetary incentives encourage fraudulent participation of non-farmers, who claim to be farmers, to receive the payment. This is particularly true when social media is used to advertise an experiment, as it can be circulated beyond the close, trusted farmer networks. For example,
<figure><img src="image_16.png" /></figure>

as reported in Rommel et al. (2023), during the data collection phase in Scotland, the online platform on which the experiment was hosted was hacked, and bots automatically filled in surveys until the quota of responses was filled overnight.

ID08: Our survey was hacked: as we had initially advertised a general link to the online survey, referring to the fact that participants would receive a payment, we, unfortunately, have had our survey hacked. After a week, we only had three responses from farmers, and suddenly, overnight, our quota of 100 participants was filled by fake responses. We identified them as they were filled in in the middle of the night, with very short response time, the email addresses provided for payment were unrealistic, and the IP addresses were not located within rural areas or even within Scotland. We sent an email to all “participants” to ensure we did not miss any real farmers who participated, asking farmers to get in touch with us if they were genuine farmers so that we could validate their answers and proceed to their payment. None responded. We then started again the data collection by including a first stage of registration to the survey. We replaced what was initially the link to the survey on the project webpage (hosted on the [Institute]'s website) with a link to a short “registration” survey in which we asked volunteers to provide some initial information on their farm so that we could check that they were genuine farms and an email address at which we could send a unique personalized link to the online survey. This process was quite time-consuming, but only very few farmers actually registered this way.

However, Goodrich et al. (2023) find that not all social media and other online platforms are equally affected. In particular, when the link to the experiment is posted to private groups, the chance of the link being inundated by fake responses is rather low. Unique links should be sent to verified farmers, and bot-screening mechanisms should be included on the data collection platform to prevent such fraudulent activities and disingenuous responses. When a snowballing approach is used for sampling for an online experiment, farmers could be required to first register their interest and provide minimal information to validate their status as farmers before they are then sent a unique link for participation. These (time-consuming) steps are essential to guarantee the quality of the data collected.

## Post-experiment phase: Sharing results and critical evaluation

### Increase the reputation of the research group and the trust of participants

Generally, it is a good practice to provide a synthesis of study results to partners and farmers. Doing so can help establish the research group's reputation and maintain trust and good relationships with participants. At the end of the experiment, contact details can be gathered from participants who are interested in receiving a synthesis of the results or would like to be involved in further research (an opt-in approach). The list of contact details needs to be separated from the dataset to ensure anonymity.

ID05: That's another problem when you promise that you're going to send them [farmers] a summary, but how do you reach these people if you promised that [the survey] would be anonymous so you don't have their email address or anything.
The results can also be circulated through farmers' associations, cooperatives, a web page, or by publication in a widely read farmers' magazine.

The survey participants reported that it was common to receive critical farmer feedback but that they also received positive feedback on the innovativeness of the format compared to standard questionnaires. The most critical reactions were directed at the lottery format and the abstract tasks. As mentioned above, several farmers asked about the purpose of the experiment. One country's research team noted that in their face-to-face implementation of the experiment, they addressed these concerns through discussions and explanations. In an online format, farmers are more likely to use the comment box to communicate their criticisms. The advice is to address negative and positive feedback where possible when contact details have been provided and farmers have agreed to be contacted after the data collection phase.

## DISCUSSION

In line with previous literature (Lefebvre et al., 2021; Rosch et al., 2021; Weigel et al., 2021), our study highlights the recruitment of farmers as a challenge for implementing preference elicitation lab-in-the-field experiments in the agricultural sector. Weigel et al. (2021) point to the high recruitment costs and low participation rates in the farming sector as a key challenge in recruiting U.S. farmers for (policy-relevant) economic field experiments. Strikingly, in their study testing the cost-effectiveness of alternative strategies to recruit farmers into experiments, the authors note that none of the more than 4000 farmers contacted by email were willing to participate in the experiment. Rosch et al. (2021) argue that researchers should investigate barriers to economic farmer experiments for agricultural policymaking, with access to participants being a main issue. The authors recommend using student participants in pre-test experiments to decrease the burden on farmers participating and to increase response rates.

Like Pennings et al. (2002), we identify the length of the experiment, the period in which the experiment is sent, the sender of the experiment, and the form and amount of payment as determinants of farmer participation. Our study also adds important dimensions. For instance, it is not only the (harvesting) period during which the experiment is conducted that is important. Other considerations include the time of day and the weather at the time the experiment is sent (or the farmer is visited). We provide additional recommendations to decrease farmers' fatigue regarding participation in economic experiments. We suggest, among others, (i) offering a voucher as payment for participants completing the experiment, (ii) creating appealing recruitment material, (iii) developing a clear, simple, and interesting introduction that includes, for example, advantages of participation, intent, purpose and relevance of the study, (iv) focusing on the relevance and purpose of the experiment for farmers, (v) offering to share survey results and conclusions with participants once the research project is completed, and (vi) enhancing farmers' trust in research endeavors.

Our results show that the recruitment of farmers is often expensive and time consuming. Probabilistic representative samples, for example, of “hobby farmers,” are difficult to obtain as a result of various limiting factors including access to sampling frames, costs, and attrition rates (Avemegah et al., 2021; Johansson et al., 2017; Pennings et al., 2002; Sutherland, 2019; Weigel et al., 2021). Marketing agencies are increasingly offering to provide panels of farmers for research, and this can be convenient, though it is often costly. The Spanish team secured quotes indicating that the cost per response for a 20-min online survey increased significantly, from 10 euros for the general population to 75 euros for a sample of farmers. The long-run time and
<figure><img src="image_17.png" /></figure>

financial burden can be decreased by creating a pool of farmers through follow-up questions regarding willingness to participate in future experiments. However, if the only farmers contacted are those who were willing to participate and gave their email addresses for future contact, it has the potential to amplify problems of self-selection. Experience shows that steps should be taken to make the recruitment, data collection, and payment processes as transparent as possible to ensure high data quality.

A standard should be developed for the selection of sociodemographic variables that can be employed to analyze variations in farmers' behavioral characteristics across population subgroups (Rosch et al., 2021). It is important to increase the quality of data by establishing standards and representative sample pools and by securing the approval of various stakeholders (e.g., ethics committees and farmer associations) for the experiment. There is also a need for substantial funding for a research agenda that supports economic experiments for agricultural policymaking. These funding efforts could be supported by improving research funders and research proposal reviewers' awareness of the potential of (well designed) experiments to yield valuable findings (Rosch et al., 2021). This aspect was, however, not part of this study.

Compared with studies with more standard subjects or surveys of farmers, conducting experiments with farmers presents unique challenges. Specifically, farmers' reaction to incentives and their willingness to participate in incentivized experiments, as well as the logistical issue of payments (i.e., who can pay, keeping the anonymity of participants), have an impact on the incentives provided. Moreover, lab-in-the-field experiments sometimes include rather abstract tasks (in the case of decontextualized games). Finally, the "game-like" nature of some experiments can be an advantage or a challenge depending on how farmers perceive the experiment; for example, do they see a lottery as a more engaging approach to data collection or as indicating a lack of seriousness? If the latter, then farmers may be skeptical about participation, hampering recruitment.

As our study shows, farmers may have a negative reaction to payments and decontextualized games. The large share of farmers that refused payment suggests the need for further investigation of how this impacts the incentive-compatibility dimension of lab-in-the-field experiments. Standard practice requires the inclusion of performance-based incentives, and cash incentives are seen as the least subject to bias (Read, 2005). Performance-based incentives are considered particularly important to motivate careful decision-making (Voslinsky & Azar, 2021). However, farmers' disinterest in payment may signal that the nature of monetary payments is different in lab-in-the-field experiments and those with more standard participants. Voslinsky and Azar (2021) note that the need for performance-based incentives depends on the question being addressed and they are not always adequate or necessary, for example, when participants have no interest in lying. When incentives are included in the experiment design, the experiment's requirements must be included and explained to first-time participants.

Some specific challenges in our study came from the decontextualized nature of the lotteries used to elicit risk attitudes, which were perceived by some as cognitively demanding. As mentioned above, experience from other projects and some of the feedback received in the replication project show that the "game" approach that underlies lab and lab-in-the-field experiments was perceived as innovative and fun by some farmers but as lacking seriousness or even being suspected of being spam by others. This more negative feedback would call for more context to be provided to respondents so they can understand not only the task required but also the purpose of the research being undertaken. However, contextualizing an experimental protocol has consequences. Alekseev et al. (2017) discuss the advantages and drawbacks of using context-framed instructions instead of abstract language. They distinguish two levels of
additional context: the “meaningful” framing, which employs terms that are more directly related to the real-life situations of respondents. It can enhance the understanding of experimental tasks and, therefore, reduce confusion; the “evocative” framing adds vocabulary that can trigger emotional responses and sometimes better capture psychological biases. For example, in the experiment presented in our paper, adding meaningfulness could have been done by presenting choices of lotteries as choices of crops with different yield risks. Adding words such as crop failure or harvest losses could carry more weight, especially for those who have experienced bad harvests in the recent past. Alekseev et al. (2017) suggest, therefore, that introducing some context in the instructions can improve the understanding of respondents, avoid confused answers, and increase the external validity of the results.

It should be noted that the interviews of this study were solely conducted with authors of Rommel et al. (2023) without aiming for the representativeness of experts in this research area. However, the character of this study is explorative; all of the participants were experts in the same field and had conducted the same survey and lottery, and the study builds on their previous experience in a range of contexts.

## CONCLUSION

In this paper, we use semi-structured in-depth interviews to analyze, reflect on, and summarize the experiences in conducting lab-in-the-field experiments of selected researchers from 10 distinct countries in the REECAP network. Our findings complement the existing literature but highlight the need to focus on (i) raising awareness and promoting experiments, (ii) ensuring high data quality and IT safety of experiments, (iii) collaborating, (iv) taking into account participant's language needs, practicability and innovativeness, (v) enhancing farmers' trust in research endeavors, (vi) the party responsible for distributing the experiment, (vii) the form and amount of payment, and (viii) the adequate sharing of results and critical evaluation of feedback received.

Since the implementation of preference elicitation lab-in-the-field experiments is challenging and many issues are likely to arise in the process, we offer our recommendations for the pre-experiment, design, data collection, and post-experiment phases to help guide the work of scientists who want to conduct and facilitate these.

Based on this study, we have several further research recommendations. First, we have articulated practical guidelines for researchers that set out phases, typical challenges, and how these can be overcome. The guidelines could be further refined through additional research. Second, we recommend conducting further research on which and how incentives can be used to work with farmers, as various farmers are not interested in such an incentive or refuse the payment.

In closing, we believe that the continued growth in popularity of (preference elicitation) lab-in-the-field experiments with farmers will lead to fruitful collaborations and further research that is both theoretical and empirical. Nevertheless, it is clear that regardless of the increase in popularity, this study is a strong complement to existing studies and offers insights from an upcoming field of research that will permit researchers to develop a deeper understanding of farmers' behavior and their methodology.

## AUTHOR CONTRIBUTIONS

**Marieke Cornelia Baaken:** Conceptualization; methodology; formal analysis; investigation; writing – original draft; writing – review & editing. **Laure Kuhfuss:** Conceptualization;
<figure><img src="image_18.png" /></figure>

methodology; investigation; writing – original draft; writing – review & editing. All others: Investigation; writing – review & editing.

## ACKNOWLEDGMENTS

We are indebted to Marija Cerjak, Marina Tomić Maksan, and Tajana Čop for their participation in the interviews used in this study. This paper builds on the efforts made by the team of the REECAP replication project (Rommel et al., 2023), thus we wish to reiterate our thanks to the authors of Bocquého et al. (2014). We would like to thank Ciaran Blanchflower, Olivia Stopek, Nino Cavallaro, and the REECAP network for their continued support of the project. Jens Rommel and Julian Sagebiel are grateful for funding received from the Swedish Research Council for Sustainable Development (FORMAS) for the project “Building resilience for Swedish food production under increasing risk of extreme weather” under registration number 2019-00993. Douadia Bougherara, Sophie Thoyer, and Marc Willinger acknowledge financial support from the University of Montpellier (SRUM) and from the French National Research Agency under the Investments for the Future Program, referred to as ANR-16-CONV-0004 (#DigitAg), and are grateful to F. Brun and N. Bernard Le Gall (ACTA), M. Beurey (APCA), E. Gourdain and C. Hannon (Arvalis), and L. Lemeur (UNPT) for their contribution to survey dissemination. Julia Höhler and Annika Tensi acknowledge financial support from Wageningen University & Research and the Business Economics Group. Katarzyna Zagorska would like to thank the Faculty of Economic Sciences, University of Warsaw, which funded the data collection in Poland with a grant for young scholars (Badania Młodych 2020, resources from the Ministry of Science and Higher Education). Mikolaj Czajkowski acknowledges funding from the Czech Science Foundation (grant 19-26812X) within the EXPRO Program “Frontiers in Energy Efficiency Economics and Modelling (FE3M).” Laure Kuhfuss acknowledges funding received from the Scottish Government RESAS division in the framework of the 2016–2021 Strategic Research Programme. Marianne Lefebvre acknowledges funding received from the Pays de Loire Region through RFI Alliance Europa (2020–2022). She is also grateful to Alexandre Perais and Ivan Dufeu for their contributions to survey dissemination. Tanja Sumrada acknowledges financial support from the Slovenian Research Agency (research programme P4-0022 (B)) and is grateful to the Association of Slovenian Rural Youth for their help with data collection. The Croatian and Swedish teams acknowledge funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 952303 (AgriFoodBoost). The Italian team received a grant from the Project PRIN DRASTIC “Driving The Italian Agri-Food System Into A Circular Economy Model,” PRIN-MIUR – Call 2017, funded by the Italian Ministry of Education, University and Research (MIUR) under grant number 2017JYRZFF. Macario Rodriguez-Entrena, María Espinosa-Goded, and Jesús Barreiro-Hurlé acknowledge the funding via a specific contract between the European Commission (EC) and the Universidades de Córdoba (JRC/SVQ/2021/VLVP/0333 – Programming and data-gathering for replication of the Bocquého et al. (2013) experiment). Open Access funding enabled and organized by Projekt DEAL.

## CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

## ENDNOTES

<footnote n="1">¹ In this paper, we use the term “lab-in-the-field experiments” to refer to a variety of approaches in the literature. For example, in Harrison and List (2004), an “artefactual field experiment” is the same as a conventional lab experiment but with a non-standard subject pool (i.e., participants are not students), and a “framed field</footnote>
<figure><img src="image_19.png" /></figure>

experiment" is the same as an artefactual field experiment but with field context in either the commodity, task, or information set that can be used by the subjects. Charness et al. (2013) refer to "extra-laboratory experiments," for what Harrisson and List call "artefactual field experiments" and Viceisza (2016) refers to these and "framed field experiments" as "lab-like field experiments."

² https://sites.google.com/view/reecap/our-research/reecap-replication-project-2020-2022.

³ Since this article is another outcome of the REECAP replication initiative, all 28 authors of the 2023 paper (Rommel et al., 2023) who agreed to be interviewed and contribute to the recommendations are co-authors of this article. We were able to secure representation of all 11 countries.

⁴ Typical economic experiment sessions consist of an experiment and a follow-up survey and include, for example, socio-demographic questions.

REFERENCES

Alekseev, A., G. Charness, and U. Gneezy. 2017. "Experimental Methods: When and Why Contextual Instructions Are Important." *Journal of Economic Behavior and Organization* 134: 48–59. https://doi.org/10.1016/j.jebo.2016.12.005.

Avemegah, E., W. Gu, A. Abulbasher, K. Koci, A. Ogunyiola, J. Eduful, S. Li, et al. 2021. "An Examination of Best Practices for Survey Research with Agricultural Producers." *Society & Natural Resources* 34(4): 538–549. https://doi.org/10.1080/08941920.2020.1804651.

Azevedo, V., M. Carvalho, F. Costa, S. Mesquita, J. Soares, F. Teixeira, and A. Maia. 2017. "Interview Transcription: Conceptual Issues, Practical Guidelines, and Challenges." *Revista de Enfermagem Referência IV Série (No. 14)*: 159–168. https://doi.org/10.12707/riv17018.

Bocquého, G., F. Jacquet, and A. Reynaud. 2014. "Expected Utility or Prospect Theory Maximisers? Assessing Farmers' Risk Behaviour from Field-Experiment Data." *European Review of Agricultural Economics* 41(1): 135–172. https://doi.org/10.1093/erae/jbt006.

Brown, C., E. Kovács, I. Herzon, S. Villamayor-Tomas, A. Albizua, A. Galanaki, I. Grammatikopoulou, D. McCracken, J. A. Olsson, and Y. Zinngrebe. 2021. "Simplistic Understandings of Farmer Motivations Could Undermine the Environmental Potential of the Common Agricultural Policy." *Land Use Policy* 101: 1–15. https://doi.org/10.1016/j.landusepol.2020.105136.

Camerer, C. F., A. Dreber, E. Forsell, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, et al. 2016. "Evaluating Replicability of Laboratory Experiments in Economics." *Science* 351(6280): 1433–36. https://doi.org/10.1126/science.aaf0918.

Campbell, B. M., D. J. Beare, E. M. Bennett, J. M. Hall-Spencer, J. S. I. Ingram, F. Jaramillo, R. Ortiz, N. Ramankutty, J. A. Sayer, and D. Shindell. 2017. "Agriculture Production as a Major Driver of the Earth System Exceeding Planetary Boundaries." *Ecology and Society* 22(4): 8. https://doi.org/10.5751/ES-09595-220408.

Cason, T. N., and S. Y. Wu. 2019. "Subject Pools and Deception in Agricultural and Resource Economics Experiments." *Environmental and Resource Economics* 73(3): 743–758. https://doi.org/10.1007/s10640-018-0289-x.

Charness, G., U. Gneezy, and B. Halladay. 2016. "Experimental Methods: Pay One or Pay All." *Journal of Economic Behavior and Organization* 131(A): 141–150. https://doi.org/10.1016/j.jebo.2016.08.010.

Charness, G., U. Gneezy, and M. A. Kuhn. 2013. "Experimental Methods: Extra-Laboratory Experiments-Extending the Reach of Experimental Economics." *Journal of Economic Behavior and Organization* 91: 93–100. https://doi.org/10.1016/j.jebo.2013.04.002.

Christensen, G., J. Freese, and E. Miguel. 2019. *Transparent and Reproducible Social Science Research: How to Do Open Science* 1st ed. California: University of California Press.

Colen, L., S. Gomez y Paloma, U. Latacz-Lohmann, M. Lefebvre, R. Préget, and S. Thoyer. 2016. "Economic Experiments as a Tool for Agricultural Policy Evaluation: Insights from the European CAP." *Canadian Journal of Agricultural Economics* 64(4): 667–694. https://doi.org/10.1111/cjag.12107.

Espinosa-González, A. B., and C. Normand. 2019. "Challenges in the Implementation of Primary Health Care Reforms: A Qualitative Analysis of Stakeholders' Views in Turkey." *BMJ Open* 9(7): 1–11. https://doi.org/10.1136/bmjopen-2018-027492.
<figure><img src="image_20.png" /></figure>

Ferraro, P. J., and P. Shukla. 2022. "Credibility Crisis in Agricultural Economics." *Applied Economic Perspectives and Policy* 45(3): 1275-91. https://doi.org/10.1002/aepp.13323.

Finger, R., C. Grebitus, and A. Henningsen. 2023. "Replication in Agricultural Economics." *Applied Economic Perspectives and Policy* 45(3): 1258-74. https://doi.org/10.1002/aepp.13386.

Goodrich, B., M. Fenton, J. Penn, J. Bovay, and T. Mountain. 2023. "Battling Bots Experiences and Strategies to Mitigate Fraudulent Responses in Online Surveys." *Applied Economic Perspectives and Policy* 45(2): 762-784. https://doi.org/10.1002/aepp.13353.

Harrison, G. W., and J. A. List. 2004. "Field Experiments." *Journal of Economic Literature* 42(4): 1009-55.

Heckelei, T., S. Hüttel, M. Odening, and J. Rommel. 2023. "The P-Value Debate and Statistical (Mal) Practice - Implications for the Agricultural and Food Economics Community." *German Journal of Agricultural Economics* 72(1): 47-67. https://doi.org/10.30430/gjae.2023.0231.

Henrich, J., R. Boyd, S. Bowles, C. Camerer, E. Fehr, and H. Gintis. 2004. *Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-Scale Societies*. Oxford: Oxford University Press. https://doi.org/10.1093/0199262055.001.0001.

Herberich, D. H., S. D. Levitt, and J. A. List. 2009. "Can Field Experiments Return Agricultural Economics to the Glory Days?" *American Journal of Agricultural Economics* 91(5): 1259-65. https://doi.org/10.1111/j.1467-8276.2009.01294.x.

Höhler, J., J. Barreiro-Hurlé, M. Czajkowski, F. J. Dessart, P. J. Ferraro, T. Li, K. D. Messer, et al. 2023. "Perspectives on Stakeholder Participation in the Design of Economic Experiments for Agricultural Policymaking: Pros, Cons, and Twelve Recommendations for Researchers." *Applied Economic Perspectives and Policy* 46(1): 338-359. https://doi.org/10.1002/aepp.13385.

Hölting, L., M. Busse, S. Bülow, J. O. Engler, N. Hagemann, I. Joormann, M. L. Kernecker, et al. 2022. "Co-Design: Working with Farmers in Europe to Halt the Loss of Biological Diversity." *Ecological Solutions and Evidence* 3(3): 1-8. https://doi.org/10.1002/2688-8319.12169.

Huber, C., A. Dreber, J. Huber, M. Johannesson, M. Kirchler, U. Weitzel, M. Abellán, et al. 2023. "Competition and Moral Behaviour: A Meta-Analysis of Forty-Five Crowd-Sources Experimental Designs." *Proceedings of the National Academy of Sciences* 120(3): 1-10. https://doi.org/10.1073/pnas.2215572120.

Johansson, R., A. Effland, and K. Coble. 2017. "Falling Response Rates to USDA Crop Surveys: Why it Matters." *Farmdoc Daily* 7(9): 1-6 Department of Agricultural and Consumer Economics, University of Illinois at Urbana-Champaign. http://farmdocdaily.illinois.edu/2017/01/falling-response-rates-to-usda-crop-surveys.html.

Lefebvre, M., J. Barreiro-Hurlé, C. Blanchflower, L. Colen, L. Kuhfuss, J. Rommel, T. Šumrada, F. Thomas, and S. Thoyer. 2021. "Can Economic Experiments Contribute to a More Effective CAP?" *EuroChoices* 20(3): 42-49. https://doi.org/10.1111/1746-692X.12324.

Palm-Forster, L. H., and K. D. Messer. 2021. "Experimental and Behavioral Economics to Inform Agricultural Environmental Programs and Policies." In *Handbook of Agricultural Economics*, Vol 5 4331-4406. North Holland: Elsevier B.V. https://doi.org/10.1016/bs.hesagr.2021.10.006.

Pe'er, G., A. Bonn, H. Bruelheide, P. Dieker, N. Eisenhauer, P. H. Feindt, G. Hagedorn, et al. 2020. "Action Needed for the EU Common Agricultural Policy to Address Sustainability Challenges." *People and Nature* 2(2): 305-316. https://doi.org/10.1002/pan3.10080.

Pennings, J. M. E., S. H. Irwin, and D. L. Good. 2002. "Surveying Farmers: A Case Study." *Review of Agricultural Economics* 24(1): 266-277.

Read, D. 2005. "Monetary Incentives, What Are They Good For?" *Journal of Economic Methodology* 12(2): 265-276. https://doi.org/10.1080/13501780500086180.

Rockström, J., J. Williams, G. Daily, A. Noble, N. Matthews, L. Gordon, H. Wetterstrand, et al. 2017. "Sustainable Intensification of Agriculture for Human Prosperity and Global Sustainability." *Ambio* 46(1): 4-17. https://doi.org/10.1007/s13280-016-0793-6.

Roller, M. R., and P. J. Lavrakas. 2015. *Applied Qualitative Research Design – A Total Quality Framework Approach*. New York City: The Guilford Press.

Rommel, J., J. Sagebiel, M. C. Baaken, J. Barreiro-Hurlé, D. Bougherara, L. Cembalo, M. Cerjak, M., et al. 2023. "Farmers' Risk Preferences in 11 European Farming Systems: A Multi-Country Replication of Bocquého et al. (2014)." *Applied Economic Perspectives and Policy* 45(3): 1374-99. https://doi.org/10.1002/aepp.13330.
Rosch, S., S. R. Skorbiansky, C. Weigel, K. D. Messer, and D. Hellerstein. 2021. "Barriers to Using Economic Experiments in Evidence-Based Agricultural Policymaking." *Applied Economic Perspectives and Policy* 43(2): 531–555.

Simmons, J. P., L. D. Nelson, and U. Simonsohn. 2011. "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant." *Psychological Science* 22(11): 1359–66. https://doi.org/10.1177/0956797611417632.

Stantcheva, S. 2022. "How to Run Surveys: A Guide to Creating your Own Identifying Variation and Revealing the Invisible." *National Bureau of Economic Research* 1–51. https://doi.org/10.3386/w30527.

Sutherland, L. A. 2019. "Finding ‘Hobby’ Farmers: A ‘Parish Study’ Methodology for Qualitative Research." *Journal of the European Society for Rural Sociology* 60(1): 129–150. https://doi.org/10.1111/soru.12262.

Tanaka, T., C. F. Camerer, and Q. Nguyen. 2010. "Risk and Time Preferences: Linking Experimental and House-
hold Survey Data from Vietnam." *American Economic Review* 100(1): 557–571. https://doi.org/10.1257/aer.
100.1.557.

Thoyer, S., and R. Préget. 2019. "Enriching the CAP Evaluation Toolbox with Experimental Approaches: Intro-
duction to the Special Issue." *European Review of Agricultural Economics* 46(3): 347–366. https://doi.org/10.
1093/erae/jbz024.

Viceisza, A. C. G. 2016. “Creating a Lab in the Field: Economic Experiments for Policymaking.” *Journal of Economic Surveys* 30(5): 835–854. https://doi.org/10.1111/joes.12118.

Voslinsky, A., and O. H. Azar. 2021. "Incentives in Experimental Economics." *Journal of Behavioral and Experimental Economics* 93: 1–6. https://doi.org/10.1016/j.socec.2021.101706.

Weigel, C., L. A, P. J. Paul, L. A. Ferraro, and K. D. Messer. 2021. "Challenges in Recruiting U.S. Farmers for Policy-Relevant Economic Field Experiments." *Applied Economic Perspectives and Policy* 43(2): 556–572. https://doi.org/10.1002/aepp.13066.

SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section
at the end of this article.

**How to cite this article:** Baaken, Marieke Cornelia, Laure Kuhfuss, Douadia Bougherara, Mikołaj Czajkowski, Macario Rodriguez-Entrena, Julia Höhler, Carl-Johan Lagerkvist, et al. 2024. “Multi-Country Perspectives on Best Practices and Barriers to Preference Elicitation Lab-In-The-Field Experiments with Farmers.” *Applied Economic Perspectives and Policy* 1–24. https://doi.org/10.1002/aepp.13488