Qualitative evaluation of a quasi-experiment
Creators: name: Leplaa, Hidde
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Experimental Analysis of Behavior
subject: Social and Behavioral Sciences
subject: Psychology
subject: FOS: Psychology

description: Introduction In psychology experiments are regularly conducted in which interventions are being compared with a control group using quantitative measurements. Similar methods are used for quasi-experiments, for instance in field studies. Qualitative data might offer a different view on the effectiveness of an intervention (Glaser, 1978; Charmaz, 2006). There are some studies in which qualitative methods are used to evaluate (field-)experiments in the field of psychology (e.g. Tahiroglu, et al., 2012; Vors et al., 2018; Palsola, et al., 2020), and there are some methodological papers on using qualitative methods to evaluate (field-)experiments (e.g. Gerring, 2011; Naber, 2015; Prowse, 2013). But to the best of our knowledge no methodological literature exists on using qualitative methods to evaluate (field-)experiments in the field of psychology. This is a field that could highly benefit from qualitative evaluation, since in Psychology the context of behavior could support the understanding of (changes in) behavior. Qualitative methods enable researchers to see a more complete picture, which can result in a better understanding than a quantitative study. In this study a case study is conducted in which the effects of an intervention are measured using qualitative methods. The result of this methodological study should be a set of guidelines for the qualitative evaluation of intervention effects in a (quasi-) experiment. The used case-study is part of a larger research program. First, we will describe the larger context of the overall research, next the goals and approach of our study will be explained. A field experiment into compliance with COVID-19 regulations In the overall study it is tested how interventions influence the compliance to COVID-19-related measurements, specifically the rule to keep 1.5-meter distance from others. Intervention effects are measured in a A-B-A-B design (per week, where A = control and B = intervention, 8 weeks total) university campus in Utrecht, the Netherlands. At multiple locations interventions are placed: 1) large and small screens displaying videos (with sound) are repeated of people stating for whom they keep distance. The catch phrase of each short video is: "For whom do you keep distance”. Furthermore, 2) a robot that welcomes people and reminds them of the current regulations, and 3) spots are placed at the ground to indicate where you can safely stand to keep 1.5 meters distance. The effectiveness of the study is measured by quantitative and qualitative measures. First, at all locations stills are taken from a helicopter perspective using a camera that is placed there for the purpose of this study. The camera is visible, but is placed too high to be detected easily. Using these stills distances between people can be determined quantitatively. Second, qualitative interviews are conducted to determine the effect of the interventions. These interviews will be used in this study to evaluate the effectiveness of the field-experiment. In the next section we will discuss how these interviews are conducted. Evaluating intervention effects using qualitative data Interviews are conducted starting in the third week of the study. These interviews are conducted at two locations in intervention and control weeks. We distinguish three conditions: first, interviews are conducted at a control location: in one of university buildings in the city center. All interviews conducted here are part of the “control group”. Second, interviews are conducted at one of the buildings located at the main square at the university campus. Interviews conducted here are part of one of two groups: during interventions weeks this is the “intervention group”, during control weeks this is the “control -at intervention location- group”. The aim is to reach a total of 60 interviews, equally distributed among the three groups. Interviews are conducted by a total of five different interviewers. All interviewers are student-assistants trained by the principal investigator (hereafter: PI) to conduct these interviews, with some experience in conducting qualitative research and/or interviews. The respondents are students who happen to be at the interview location, or who are via via made aware of the research and are invited to come to the interview location. Most interviews will be conducted in Dutch, since this is the mother tongue for both the interviewers and respondents. If the respondent is not a native Dutch speaker and/or prefers to conduct the interview in English, this request will be met. During the interview a topic list is used, see Appendix 1 (in Dutch). The goal of the interviews is twofold. First, we expect to be able to identify “work-around strategies”: ways to live with the measurement by sometimes breaking rules, but under self-imposed criteria. Second, we ask respondents about the motives to employ these strategies. Our expectation is that, perhaps, the groups will not differ in the work-around strategies, but differences are expected with respect to their motives to apply these strategies. The data from the interviews are analyzed following the procedure of coding used in the Grounded Theory (Glaser, 1978; Charmaz, 2006). First, the data are open coded by the interviewer. The start of the open coding process is to create a memo in which in a couple of sentences the main message from the interview, according to the interviewer, is captured. In a later phase these memos will be used to verify whether the main message is still covered in the theoretical model created based on these interviews. Then, the PI and two of the interviewers continue with axial and selective coding, where empathy is used a sensitizing concept during the axial coding phase. We aim to develop a typology (Glaser &amp; Strauss, 1965; Boeije, 2010) of used work-around strategies. Furthermore, we will make queries to compare work-around strategies and motivations between the three groups. Aims of this study Goal of the study is to identify the steps needed to evaluate an (quasi-) experiment using qualitative methods. In the intended (methodological) paper we will formulate the steps needed to execute the research described above. Guidelines will be provided for important considerations in all phases of a study, from formulating the research aim to drawing conclusions. The end product serves as a tutorial for researchers who aim to evaluate an (quasi-)experiment using qualitative data. Psychological researchers conducting experiments should be able to use this paper to set up their own study. References: Boeije, H. (2010). Analysis in Qualitative Research. Sage publications, London. Charmaz, K. (2006). Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis. London: Sage. Gerring, J. (2011). How good is good enough? A multidimensional, best-possible standard for research design. Political Research Quarterly, 64, 625-636. DOI: 10.1177/1065912910361221 Glaser, B. G. (1978). Theoretical sensitivity. Mill Valley, CA: The Sociology Press. Glaser, B. G., &amp; Strauss, A. L. (1978). Awareness of dying. Chicago, IL: Aldine. Naber, A. (2015). Qualitative experiment as a participating method in innovation research. Historical Social Research, 40, 233-257. Retrieved March 20, 2021, from http://www.jstor.org/stable/24583154 Palsola, M., Renko, E., Kostamo, K., Lorencatto, F., &amp; Hankonen, N. (2020). Thematic analysis of acceptability and fidelity of engagement for behaviour change interventions: The let’s Move It intervention interview study. British Journal of Health Psychology, 25, 772-789. https://doi.org/10.1111/bjhp.12433 Prowse, M., &amp; Camfield, L. (2013). Improving the quality of development assistance: What role for qualitative methods in randomized experiments? Progress in Development Studies, 13, 51-61. DOI: 10.1177/146499341201300104 Tahiroǧlu, M., &amp; Çetin, T. (2012). The Attitude of the Students towards the Value of “Paying Attention to Being Healthy” in 4th Grade Elementary Social Sciences Course. Educational Sciences: Theory &amp; Practice, 12, 2, 1647-1651. Retrieved from https://eric.ed.gov/?id=EJ987865. Vors, O., Marqueste, T., &amp; Mascret, N. (2018). The trier social stress test and the trier social stress test for groups: qualitative investigations. PLoS One. Article e0195722. https://doi.org/10.1371/journal.pone.0195722