Ethnography and Evaluation: Their Relationship and Three Anthropological Models of Evaluation

This paper examines the relationship between ethnographic research methods and evaluation theory and methodology. It is divided into two main sections: (a) ethnography in evaluation and (b) anthropological models of evaluation. Three levels of the leading anthropological models of evaluation are summarized, which include responsive evaluation, goal-free evaluation, and constructivist evaluation. In conclusion, (a) there is no consensual definition of ethnography; (b) in many circumstances, ethnographic evaluation models may be beneficial; and (c) ethnography can be used in evaluation but requires a high level of analysis to transform ethnographic data into useful information for eliciting an evaluative conclusion. *The author would like to thank Daniela C. Schröter, Chris L. S. Coryn, and Elizabeth K. Caldwell for editing this paper and for their extremely useful comments and suggestions. Journal of MultiDisciplinary Evaluation (JMDE:3) ISSN 1556-8180 113 http://evaluation.wmich. edu/jmde/ Global Review: Publications Introduction Ethnography, an applied qualitative social science research method, is often employed in program evaluation. Ethnography, alone and according to its pure anthropological definition, is not a research method capable of being the sole method implemented in an evaluation. Ethnography may prove advantageous to evaluators as an additional method to be employed or considered. However, sound evaluation typically requires multiple data collection methods and a higher level of analysis than ethnography alone can provide. Evaluation synthesizes the narrative and develops an evaluative conclusion. There are various instances when the implementation of an evaluation model that relies heavily on qualitative methods based in the tradition of anthropological research is beneficial. As an evaluator, at minimum, familiarity with these models should be in one's repertoire. The paper is divided into two main sections: (1) Ethnography and Evaluation and (2) Anthropological Models of Evaluation. The first section presents a summary definition of ethnography, its theories, concepts, and benefits; and the difference between ethnography and anthropology. The author then provides a brief definition of evaluation and discusses the relationship between ethnography and evaluation. There are three anthropological models of evaluation in which the author summarizes, discusses the strengths and limitations, and reflects on their 1 AUTHOR'S NOTE: The author of this paper uses the terms “ethnography,” “ethnographic techniques,” and often “qualitative research methods” interchangeably. Additionally, the term “program” is used generically, to refer to the evaluand*. Ethnography in the context of this paper is primarily in regards to program and policy evaluations. Ethnography may also be used in product, personnel, and performance evaluations. —* “Evaluand: That which is being evaluated (e.g., program, policy, project, product, service, organization)” (Davidson, 2005, p. 240). Journal of MultiDisciplinary Evaluation (JMDE:3) ISSN 1556-8180 114 http://evaluation.wmich. edu/jmde/ Global Review: Publications relationship with ethnography. The paper concludes with a synopsis of the author’s main impressions and key points. Ethnography and Evaluation Ethnography is an applied research method most often associated with anthropology, where it was developed to study cultural interpretation. Ethnography, also called field research, is the process of describing a culture or way of life from a folk peoples’ perspective. Anthropologist Clifford Geertz described the ethnographic method as “thick description. ” It provides detailed notes and descriptions of everything that occurs without attempting to summarize, generalize, or hypothesize. In fact, with traditional ethnography, as a rule of thumb, for every half hour of observation a researcher writes for two hours. The researcher focuses on factual description to allow for multiple interpretations to later infer cultural meaning. To obtain this description of a population’s perception, the principle of ‘naturalism’ is assumed. Thus, trust and rapport are essential between the researcher and the population being studied. Ethnographers, if following the constructivist philosophy, believe that pure 2 Alternative definitions: Ethnography is “a descriptive study of an intact cultural or social group or an individual or individuals within the group based primarily on participant observation and open-ended interviews. Ethnography is based on learning from people as opposed to studying people” (Beebe, n.d.). Ethnographic research “involves the study of groups and people as they go about their everyday lives” (Emerson, Fretz, & Shaw, 1995). “Ethnography is the art and science of describing a group or culture” (Fetterman, 1989, p. 11). 3 Naturalism: Leave natural phenomenon alone. 4 Constructivist philosophy maintains that the researcher manufactures knowledge through her interaction in the field and that there is no objective truth to be uncovered (ontological Journal of MultiDisciplinary Evaluation (JMDE:3) ISSN 1556-8180 115 http://evaluation.wmich. edu/jmde/ Global Review: Publications objectivity is impossible as: (A) ethnography is an interpretive endeavor by fallible human beings; (B) not all field sites are “foreign” for ethnographers in the same way; (C) ethnography is not replicable; and (D) ethnography is not based on a large number of cases. The epistemology of ethnography is typically a model based on a phenomenologically oriented paradigm, which focuses on multiple perspectives and multiple realities of a phenomenon. Phenomenological inquiry seeks to answer the question: “What is the structure and essence of experience of this phenomenon for these people? ” (Patton, 1990, p. 69). Constructivists take a heuristic approach to answering this phenomenological question. According to Fetterman (1989), most ethnographers subscribe to ideational theory, which suggests that change is the result of mental activity—thoughts or ideas—versus materialists who believe that “material conditions—ecological resources, money modes of production-are the prime movers” (Fetterman, 1989, p. 16). The most popular ideational theory is cognitive theory, which assumes we can infer peoples’ thoughts from hearing what they tell us. While many theories, concepts, and methods (e.g., in-depth, open-ended interviews, direct observation, written documents, triangulation) resulting in narrative description commonly recur in the literature, consensus on any one set of fundamental principles of ethnography cannot be found (Genzuk, 2001; Patton, 1990; Payne, 1994). For example, ethnographic theories, concepts, and data collection techniques are also used in non-ethnographic qualitative research and relativism) (Maxwell, 1998 in Bickman & Rog, 1998). 5 Heuristics is a form of phenomenological inquiry focusing on the personal experiences and insights of the researcher—it considers researcher’s experience in addition to other observers that experience the phenomenon. Journal of MultiDisciplinary Evaluation (JMDE:3) ISSN 1556-8180 116 http://evaluation.wmich. edu/jmde/ Global Review: Publications distinctions between ethnography and other qualitative theories, concepts, principles, and methods is not clearly evident. Instead, there are copious combinations of varying concepts considered fundamental to ethnography from researchers and anthropologists alike. The key in understanding the differences between ethnography and other qualitative social science research methods is to understand the multiple combinations of techniques, concepts, and data collection methodologies encompassed under the term “ethnography. ” As with all research methodologies, each philosophical and theoretical decision is located on a spectrum or continuum. Thus, the definition of ethnography and what it entails is idiosyncratic to the ethnographer or researcher depending on her degree of commitment to a hodgepodge of “fundamental” concepts. Past and current literature presents definitions and concepts of ethnography differing by technique, values emphasized, time allotted, data analysis procedures, and commitment to the purist practice of anthropological ethnography. Therefore, ethnographic techniques are qualitative in nature but distinct. Below are a few of the reoccurring concepts specific to ethnography (Fetterman in Bickman and Rog, 1998; Genzuk , 2001; Hall, n.d.): The focus is on culture and cultural interpretation. There is an emphasis on an emic perspective. The holistic perspective is often of greater depth than other qualitative research methods. Sampling measures are conducted over a longer period of time. 6 Emic perspective is that of the insider and includes the acceptance of multiple realities. Journal of MultiDisciplinary Evaluation (JMDE:3) ISSN 1556-8180 117 http://evaluation.wmich. edu/jmde/ Global Review: Publications The researcher herself is the primary tool for data collection. In contrast to ethnographic methodology, evaluation methodology commonly comprises the use of both qualitative and quantitative techniques. Commonly defined as the systematic determination of something’s merit, worth, or significance (Davidson, 2005; Sanders, 1994; Scriven, 1991). Scriven (1991) claims that evaluation is not only a methodology, but a distinct multiand transdisciplinary field of study not to be regarded as merely a as a branch of applied social science. As an independent discipline, evaluation may utilize applied social science research methodology, but it is distinct by its unique purpose and methodology (e.g., ranking, grading, and scoring). The determination of merit, worth, and significance of an evaluand requires evaluators to consider the relevant values and to make judgments based on those values. Autonomous subspheres of evaluation are program, policy, product, personnel, performance, and proposal evaluations as well as metaevaluation