----

##  Reliability, Validity, and Authenticity in American Indian and                  Alaska Native Research. ERIC Digest.                                         

The constructs used by scholars across the physical, natural, and               social sciences to evaluate research quality--as objective,                     reliable, valid, generalizable, randomized, accurate, authentic--are            not value-free or apolitical. They all require the application of               human judgment, which is inevitably affected by cultural norms and              values (Westmeyer, 1981). Consequently, the use of these constructs             to assess research focused on minority, marginalized populations                requires extraordinary judgment. Such assessments must be based not             only on Western notions of scientific quality but also, in the case             of American Indians and Alaska Natives (AI/ANs), on a separate set              of criteria prescribed in the interest of sovereignty. Sovereignty              refers to the inherent, as well as constitutionally recognized,                 rights of tribes to self-government, self-determination, and                    self-education. Of course no contemporary polity--whether American              Indian tribe or U.S. federal government--exercises unlimited                    sovereignty: "In the real world, sovereignty operates within                    constraints" (Wilkins & Lomawaima, 2001, p. 5). This Digest                     addresses these issues as researchers take on the challenges set                forth by the "American Indian and Alaska Native Education Research              Agenda" (Research Agenda Working Group, Strang, & von Glatz, 2001).             The perspectives in this Digest are those of the authors and do not             necessarily represent the positions of the U.S. Department of                   Education. DEFINITION OF TERMSIn the sciences, the                        terms "accuracy," "reliability," and "validity" are used to make                evaluative judgments of measurement instruments (e.g., tests and                surveys), or of data collection methods. An "accurate" measurement              is one that is precise and exact. Although commonly applied to the              mechanical measurements of the physical sciences, accuracy is also              an issue in social science, including qualitative research, where               researchers seek to understand complex, multiple perceptions of                 reality. A "reliable" method or instrument gives consistent results             in different applications. Consistency, however, "says nothing about            being right or wrong" (Kelly, 1996, p. 43). The "rightness"                     or "truth value" of research is an issue of "validity." Validity is             always subject to human judgment: it asks the question, "Does this              device [or method] measure what it is said [or claims] to                       measure?" (Westmeyer, 1981, p.42). 
The concept of validity has                been contested and is subject to further refinement. In experimental            research, it is common to distinguish "internal validity" (whether a            proposed cause is actually the reason for an effect) from "external             validity" (whether research results are generalizable across                    settings or populations) (Miller & Salkind, 2002). However, some                qualitative researchers reject validity as a criterion for judging              research, arguing that understanding is a more proper                           concern (Wolcott, 1994). Maxwell (1992) argues that validity in                 qualitative research refers not to data or methods but to                       accounts. "Descriptive validity" tests the factual accuracy of                  accounts; "interpretive validity" asks whether an account represents            research participants' perspectives; "theoretical validity" refers              to the explanatory power of accounts; and "evaluative validity"                 questions the usefulness or applicability of accounts. Parallels can            be found in Banks' (1997) assessment of special education needs                 among AI/AN children. She distinguishes "construct validity" (the               meaningfulness of research constructs or ideas), "treatment                     validity" (whether research participants judge its objectives,                  instruments, and procedures acceptable), and "ecological                        validity" (the extent to which research looks beyond the individual             to social-historical contexts). THE REQUIREMENTS OF SOVEREIGNTY               AND SELF-DETERMINATIONJudging research adequacy within the                  larger social-historical context of AI/AN education involves                    answering questions critical to the exercise of sovereignty: Why do             the research? What factors motivate the researcher? Who has set the             research goals? Who has the "disciplinary authority" (Page, 2000, p.            23) to do the research? Who will be involved in conducting the                  research? Whom does the research serve? How will it benefit the                 local community? 
Until recently, Native communities were rarely               involved in raising these questions or in formulating, implementing,            or evaluating studies. In the past few decades, however, tribal                 governments have increasingly taken control of research site access,            instituting protocols and approval processes for research                       projects (Lomawaima, 2000; see Romero, 1994, for an excellent                   example of research working within protocols). For example, both the            Tohono O'odham and Pascua Yaqui language policies stipulate that                tribal authorities must approve all research about their people. The            Yaqui policy lays out further requirements for copyright of                     publications growing out of such research (Zepeda, 1990). Additional            sovereignty safeguards include requirements that tribal                         representatives be involved in setting research standards and                   evaluating projects against those standards. All these measures have            grown out of the recognition that local communities have as great a             stake as outside scholars in establishing what constitutes                      high-quality, useful research. 
There are viable alternatives to               conventional, colonizing research paradigms (Smith, 1999). As a                 means for improving post-World War II intergroup relations,                     sociologist Kurt Lewin (1946) introduced an action research cycle of            problem identification, fact finding, execution, and evaluation. A              decade later, Corey (1953) and Taba and Noel (1957) operationalized             this cycle in education. Now treated as synonymous with teacher                 research, action research is a "reemerging tradition" in which those            previously designated as subjects actively participate in research              processes that benefit them directly (Stringer, 1996, p. 7). Recent             examples of teacher/action research in AI/AN education include                  Lipka, Mohatt, and the Ciulistet (teacher-leaders) Group's work on              incorporating Yup'ik knowledge into science and math                            instruction (Lipka et al., 1998); collaborative research to                     revitalize Indigenous languages (McCarty, Watahomigie, & Yamamoto,              1999); and research on alternative literacy assessment and                      curriculum reform undertaken by Navajo                                          teacher-researchers (McCarty & Dick, 2003; McCarty, 2002; for an                overview of teacher research methods, see McCarty,                              1997). AUTHENTICITYAre the quality and authenticity of                    research guaranteed by the involvement of tribal members in design              and evaluation? Some say it takes even more. Swisher (1996) argues              for the primacy of Native scholars in conducting research on Native             peoples and issues, attributing to them the benefits of an insider's            view and experiences, enhanced passion and commitment, and                      the "authority to ask new and different questions" (p. 93). She                 calls on non-Native researchers and mentors, whose expertise and                high-quality work she readily acknowledges, to step aside in favor              of Native authors. Many would agree with Swisher's call to increase             the authority of, recognition of, and publication by Native scholars            and researchers, but not all are willing to ask non-Native allies to            step aside (see Mihesuah, 1996 for a range of opinions). These                  observers suggest that the most productive research will result from            respectful collaboration that does not dichotomize researchers as               Native or non-Native but does make the contributions of Native                  colleagues integral to the design and conduct of studies. 
Even                those who adhere to the "Native-only" requirement run up against                debates over who is "really Indian." Snipp (2000) productively                  discusses the many criteria used to identify and                                assign "Indian-ness" and tribal identity, and the political and                 other agendas attached to them. Snipp concludes that contemporary               criteria for tribal membership--however historically situated and               flawed--are the prerogatives of sovereign tribes. 
Local                       evaluations of authenticity, whether dependent on research subject              or research investigator, should be respected in accord with the                principle of self-determination, even as researchers struggle to                work out their often anomalous positions as "insider," "outsider,"              or some combination of both (Brayboy & Deyhle, 2000). POLITICS AND            SCHOLARSHIPThere have been many calls over the years for                    improved research in AI/AN education. The most recent is Executive              Order 13096 (Clinton, 1998), which mandates the development and                 implementation of a national research agenda in Indian education.               Consultations and meetings with Native peoples across the country               culminated in the "American Indian and Alaska Native Education                  Research Agenda" (Research Agenda Working Group, Strang, & von                  Glatz, 2001). The agenda identifies high-priority research interests            and outcomes for Native communities and clearly articulates the                 central roles that Native scholars, parents, teachers, schools, and             communities must play to ensure high-quality research. The promise              embedded in the research agenda, however, may be threatened if                  limited notions of what constitutes high-quality research, using                criteria such as those outlined earlier, overtake the powers of                 Native peoples to evaluate research in their communities and                    schools. 
Science as practiced by many Western researchers is often            invoked as the answer to educational woes. This view tends to                   dismiss the power of all human inquiry--including Indigenous                    knowledge systems--to observe, analyze, theorize, and generate                  creative solutions. Research and development projects designed                  within the Western mold--especially those meant to provide guidance             for classroom practice--often rely on standardized scripts to                   instruct and standardized tests to assess effectiveness (Metcalf,               2002). Using standardized tests with minority populations is a                  dubious practice (Padilla & Lindholm, 1995, p. 97). 
A much more               promising line of research draws upon funds of knowledge in Native              communities to derive and apply research insights in                            education. "Funds of knowledge" refers to the repositories of                   knowledge residing in communities that are frequently overlooked or             dismissed in conventional Western science and school curricula (Moll            et al., 1992). In a study of the Yupiit (Alaska) Nation and school              system, Yupiaq researcher Oscar Kawagley shows how local scientific,            mathematical, and technological knowledge can be used to improve the            school curriculum. Situating science within a Yupiaq worldview and              ecology of sustainability, Kawagley provides a rigorous and                     authentic framework for research and "for rethinking what we teach              in schools and how we teach it, particularly as it relates to                   science" (1995, p.138). 
Research is needed that attends to the                great diversity among Nations and the variability within local                  sites. Such research will most often be qualitative in its                      methodology. Page (2000) points out that "Qualitative research has              challenged the 'science' in the social sciences, and in the natural             sciences, too, even though science has long been the gold standard              for knowledge and a source of disciplinary authority" (p. 23). She              warns against "nostalgia for the large-scale, randomized experiments            thought to provide unbiased and exact answers" (p. 24). The quest               for scientific truth in such endeavors will likely prove                        elusive (Cassell, 2002) and could pose a threat to AI/AN educational            self-determination and sovereignty. 
The political context of                  educational research can make the difference between a project that             serves Native community and strengthens sovereignty or actively                 undermines both. None of the parties involved in educational                    research is apolitical--not tribal communities, tribal governments,             federal education agencies, or academicians. We all work from                   particular positions, but certain positions cannot continue to be               privileged over others. Ultimately, judgments about research quality            in Indian education most productively belong in Indian                          country. REFERENCESBanks, S. R. (1997). Caregiver and                     professional perceptions of assessment practices and validity for               American Indian/Alaska Native families. Journal of American Indian              Education, 37(1), 16-44. 
Brayboy, B. M., & Deyhle, D. (2000).                 Insider-outsider: Researchers in American Indian communities. Theory            into Practice, 39(3), 163-169. 
Cassell, J. (2002). Perturbing the             system: "Hard science," "soft science," and social science: The                 anxiety and madness of method. Human Organization, 61(2),                       177-185. 
Clinton, W. J. (1998, August 6). American Indian and                 Alaska Native education: Executive order 13096. Federal Register,               63(154), 42681-84. Retrieved December 20, 2002, from                            http://www.niea.org/indedeo.htm. 
Corey, S. M. (1953). Action                  research to improve school practices. New York: Teachers College                Press, Columbia University. 
Kawagley, A. O. (1995). A Yupiaq                  worldview: A pathway to ecology and spirit. Prospect Heights, IL:               Waveland Press. 
Kelly, K. T. (1996). The logic of reliable                    inquiry. New York: Oxford University Press. 
Lewin, K. (1946).                 Action research and minority problems. Journal of Social Issues,                2(4), 34-46. 
Lipka, J., with Mohatt, G. V., & the Ciulistet                   Group. (1998). Transforming the culture of schools: Yup'ik Eskimo               examples. Sociocultural, political, and historical studies in                   education. Mahwah, NJ: Lawrence Erlbaum Associates. 
Lomawaima, K.             T. (2000). Tribal sovereigns: Reframing research in American Indian             education. Harvard Educational Review, 70(1), 1-21. 
Maxwell, J.               A. (1992). Understanding and validity in qualitative research.                  Harvard Educational Review, 62(3), 279-300. 
McCarty, T. L. (1997).            Teacher research methods in language and education. In N. H.                    Hornberger & D. Corson (Eds.), Encyclopedia of language and                     education: Vol. 8. Research methods in language and education (pp.              227-237). Norwell, MA: Kluwer Academic Publishers. 
McCarty, T.                L. (2002). A place to be Navajo: Rough Rock and the struggle for                self-determination in Indigenous schooling. Sociocultural,                      political, and historical studies in education. Mahwah, NJ: Lawrence            Erlbaum. 
McCarty, T. L., & Dick, G. S. (2003). Telling The                    People's stories: Literacy practices and processes in a Navajo                  community school. In A. I. Willis et al. (Eds.), Multicultural                  issues in literacy research and practice. Mahwah, NJ: Lawrence                  Erlbaum Associates. 
McCarty, T. L., Watahomigie, L. J., &                     Yamamoto, A. Y. (1999). Introduction: Reversing language shift in               Indigenous America: Collaborations and views from the field [Theme              issue], Practicing Anthropology, 21(2), 2-4. 
Metcalf, S. (2002,               January 28). Reading between the lines. The Nation. Retrieved June              19, 2002, from                                                                  http://www.thenation.com/doc.mhtml?I=20020128&x=metcalf. 
Mihesuah,            D. A. (Ed.) (1996). Writing about (writing about) American                      Indians [Special issue]. American Indian Quarterly, 20(1). 
Miller,            D. C., & Salkind, N. J. (2002). Handbook of research design and                 social measurement (6th ed.). Thousand Oaks & London: Sage                      Publications. 
Moll, L. C., Amanti, C., Neff, D., & Gonzalez,                  N. (1992). Funds of knowledge for teaching: Using a qualitative                 approach to connect homes and classrooms. Theory into Practice,                 31(1), 132-141. 
Padilla, A. M., & Lindholm, K. J. (1995).                     Quantitative educational research with ethnic minorities. In J. A.              Banks & C. A. Banks (Eds.), Handbook of research on multicultural               education (pp. 97-113). New York: Macmillan Publishing. (ERIC                   Document Reproduction Service No. ED 382 701) 
Page, R. N. (2000).             The turn inward in qualitative research. Harvard Educational Review,            70(1), 23-38. 
Research Agenda Working Group, Strang, W., & von                Glatz, A. (2001). American Indian and Alaska Native education                   research agenda. Washington, DC: U.S. Department of Education. (ERIC            Document Reproduction Service No. ED 462 216) Retrieved December 20,            2002, from http://www.indianeduresearch.net 
Romero, M. K. (1994).             Identifying giftedness among Keresan Pueblo Indians: The Keres                  study. Journal of American Indian Education, 34(1), 35-58. 
Smith,             L. T. (1999). Decolonizing methodologies: Research and Indigenous               peoples. New York: Zed Books. 
Snipp, C. M. (2000, May-June). Some             alternative approaches to the classification of American Indians and            Alaska Natives. Paper presented at the American Indian and Alaska               Native Research Agenda Conference, Albuquerque, NM. Retrieved June              19, 2002, from                                                                  http://www.indianeduresearch.net/MSnipp.pdf. 
Stringer, E.                     T. (1996). Action research: A handbook for practitioners. Thousand              Oaks: Sage Publications. 
Swisher, K. G. (1996). Why Indian people             should be the ones to write about Indian education. American Indian             Quarterly, 20(1), 83-90. 
Taba, H., & Noel, E. (1957). Action                  research: A case study. Washington, DC: Association for Supervision             and Curriculum Development (NEA). 
Westmeyer, P. (1981). A guide               for use in planning and conducting research projects. Springfield,              IL: Charles C. Thomas. 
Wilkins, D., & Lomawaima, K. T. (2001).                Uneven ground: American Indian sovereignty and federal law. Norman:             University of Oklahoma Press. 
Wolcott, H. F. (1994). On                       seeking--and rejecting--validity in qualitative research. In H.                 Wolcott, Transforming qualitative data: Description, analysis, and              interpretation (pp. 337-373). Thousand Oaks, CA: Sage                           Publications. 
Zepeda, O. (1990). American Indian language policy.             In K. L. Adams & D. T. Brink (Eds.), Perspectives on official                   English: The campaign for English as the official language of the               USA (pp. 247-256). Hawthorne, NY: Walter de Gruyter &                           Co. 
----- 
K. Tsianina Lomawaima is a professor of American                  Indian studies at the University of Arizona. She teaches and writes             about the history of American Indian experiences in the late 19th               and 20th centuries, focusing on Native self-determination, federal              policy development, and impacts on Native domestic life and                     education. 
Teresa L. McCarty is a professor of language, reading              and culture and is interim dean of the College of Education at the              University of Arizona. She teaches courses and conducts qualitative             research on American Indian and bilingual/multicultural education,              Indigenous language education, and language planning and                        policy. 
This publication was prepared with funding from the Office            of Educational Research and Improvement, U.S. Department of                     Education, under contract no. ED-99-CO-0027. The opinions expressed             herein do not necessarily reflect the positions or policies of OERI,            the Department, or AEL.                                                      **Title:** Reliability, Validity, and Authenticity in American Indian and                  Alaska Native Research. ERIC Digest.                                         **Note:** Online version includes annotated bibliography with some full-text              links.                                                                       **Document Type:** Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073); **Available From:** ERIC/CRESS, P.O. Box 1348, Charleston, WV 25325-1348. Tel:                      800-624-9120 (Toll Free). For full text:                                        http://www.ael.org/ERIC/digests/edorc02-4.pdf.                               **Descriptors:** Action Research,  American Indian Education,  Educational Research,             Indigenous Knowledge, Participatory Research,  Reliability,                     Standards,  Tribal Sovereignty, Tribally Controlled Education,                   Validity                                                                    **Identifiers:**  Authenticity, ERIC Digests, Local Control, Politics of Research,                Research Quality                                                            
###
[Return to ERIC Digest Search Page]