ED470951 2002-12-00 Reliability,

Validity, and Authenticity in American

Indian and Alaska Native Research.

ERIC Digest.

ERIC Development Team

www.eric.ed.gov

Table of Contents

If you're viewing this document online, you can click any of the topics below to link directly to that section.

Reliability, Validity, and Authenticity in American Indian and Alaska

Native Research. ERIC Digest............................................. 1

DEFINITION OF TERMS...................................................... 2

THE REQUIREMENTS OF SOVEREIGNTY AND

SELF-DETERMINATION................................................ 3

AUTHENTICITY................................................................. 4

POLITICS AND SCHOLARSHIP............................................. 4

REFERENCES.................................................................. 5

ERIC Identifier: ED470951

Publication Date: 2002-12-00

Author: Lomawaima, K. Tsianina - McCarty, Teresa L.

Source: ERIC Clearinghouse on Rural Education and Small Schools Charleston WV.

Reliability, Validity, and Authenticity in

American Indian and Alaska Native Research.

ERIC Digest.

ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.

Page 1 of 9
THIS DIGEST WAS CREATED BY ERIC, THE EDUCATIONAL RESOURCES

INFORMATION CENTER. FOR MORE INFORMATION ABOUT ERIC, CONTACT

ACCESS ERIC 1-800-LET-ERIC

The constructs used by scholars across the physical, natural, and social sciences to

evaluate research quality--as objective, reliable, valid, generalizable, randomized,

accurate, authentic--are not value-free or apolitical. They all require the application of

human judgment, which is inevitably affected by cultural norms and values (Westmeyer,

1981). Consequently, the use of these constructs to assess research focused on

minority, marginalized populations requires extraordinary judgment. Such assessments

must be based not only on Western notions of scientific quality but also, in the case of

American Indians and Alaska Natives (AI/ANs), on a separate set of criteria prescribed

in the interest of sovereignty. Sovereignty refers to the inherent, as well as

constitutionally recognized, rights of tribes to self-government, self-determination, and

self-education. Of course no contemporary polity--whether American Indian tribe or U.S.

federal government--exercises unlimited sovereignty: "In the real world, sovereignty

operates within constraints" (Wilkins & Lomawaima, 2001, p. 5). This Digest addresses

these issues as researchers take on the challenges set forth by the "American Indian

and Alaska Native Education Research Agenda" (Research Agenda Working Group,

Strang, & von Glatz, 2001). The perspectives in this Digest are those of the authors and

do not necessarily represent the positions of the U.S. Department of Education.

DEFINITION OF TERMS

In the sciences, the terms "accuracy," "reliability," and "validity" are used to make

evaluative judgments of measurement instruments (e.g., tests and surveys), or of data

collection methods. An "accurate" measurement is one that is precise and exact.

Although commonly applied to the mechanical measurements of the physical sciences,

accuracy is also an issue in social science, including qualitative research, where

researchers seek to understand complex, multiple perceptions of reality. A "reliable"

method or instrument gives consistent results in different applications. Consistency,

however, "says nothing about being right or wrong" (Kelly, 1996, p. 43). The "rightness"

or "truth value" of research is an issue of "validity." Validity is always subject to human

judgment: it asks the question, "Does this device [or method] measure what it is said [or

claims] to measure?" (Westmeyer, 1981, p.42).

The concept of validity has been contested and is subject to further refinement. In

experimental research, it is common to distinguish "internal validity" (whether a

proposed cause is actually the reason for an effect) from "external validity" (whether

research results are generalizable across settings or populations) (Miller & Salkind,

2002). However, some qualitative researchers reject validity as a criterion for judging

research, arguing that understanding is a more proper concern (Wolcott, 1994). Maxwell

(1992) argues that validity in qualitative research refers not to data or methods but to

accounts. "Descriptive validity" tests the factual accuracy of accounts; "interpretive

validity" asks whether an account represents research participants' perspectives;

"theoretical validity" refers to the explanatory power of accounts; and "evaluative

www.eric.ed.gov ERIC Custom Transformations Team

Page 2 of 9 ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.
validity" questions the usefulness or applicability of accounts. Parallels can be found in

Banks' (1997) assessment of special education needs among AI/AN children. She

distinguishes "construct validity" (the meaningfulness of research constructs or ideas),

"treatment validity" (whether research participants judge its objectives, instruments, and

procedures acceptable), and "ecological validity" (the extent to which research looks

beyond the individual to social-historical contexts).

THE REQUIREMENTS OF SOVEREIGNTY AND

SELF-DETERMINATION

Judging research adequacy within the larger social-historical context of AI/AN education

involves answering questions critical to the exercise of sovereignty: Why do the

research? What factors motivate the researcher? Who has set the research goals? Who

has the "disciplinary authority" (Page, 2000, p. 23) to do the research? Who will be

involved in conducting the research? Whom does the research serve? How will it benefit

the local community?

Until recently, Native communities were rarely involved in raising these questions or in

formulating, implementing, or evaluating studies. In the past few decades, however,

tribal governments have increasingly taken control of research site access, instituting

protocols and approval processes for research projects (Lomawaima, 2000; see

Romero, 1994, for an excellent example of research working within protocols). For

example, both the Tohono O'odham and Pascua Yaqui language policies stipulate that

tribal authorities must approve all research about their people. The Yaqui policy lays out

further requirements for copyright of publications growing out of such research (Zepeda,

1990). Additional sovereignty safeguards include requirements that tribal

representatives be involved in setting research standards and evaluating projects

against those standards. All these measures have grown out of the recognition that

local communities have as great a stake as outside scholars in establishing what

constitutes high-quality, useful research.

There are viable alternatives to conventional, colonizing research paradigms (Smith,

1999). As a means for improving post-World War II intergroup relations, sociologist Kurt

Lewin (1946) introduced an action research cycle of problem identification, fact finding,

execution, and evaluation. A decade later, Corey (1953) and Taba and Noel (1957)

operationalized this cycle in education. Now treated as synonymous with teacher

research, action research is a "reemerging tradition" in which those previously

designated as subjects actively participate in research processes that benefit them

directly (Stringer, 1996, p. 7). Recent examples of teacher/action research in AI/AN

education include Lipka, Mohatt, and the Ciulistet (teacher-leaders) Group's work on

incorporating Yup'ik knowledge into science and math instruction (Lipka et al., 1998);

collaborative research to revitalize Indigenous languages (McCarty, Watahomigie, &

Yamamoto, 1999); and research on alternative literacy assessment and curriculum

reform undertaken by Navajo teacher-researchers (McCarty & Dick, 2003; McCarty,

ERIC Resource Center www.eric.ed.gov

ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.

Page 3 of 9
2002; for an overview of teacher research methods, see McCarty, 1997).

AUTHENTICITY

Are the quality and authenticity of research guaranteed by the involvement of tribal

members in design and evaluation? Some say it takes even more. Swisher (1996)

argues for the primacy of Native scholars in conducting research on Native peoples and

issues, attributing to them the benefits of an insider's view and experiences, enhanced

passion and commitment, and the "authority to ask new and different questions" (p. 93).

She calls on non-Native researchers and mentors, whose expertise and high-quality

work she readily acknowledges, to step aside in favor of Native authors. Many would

agree with Swisher's call to increase the authority of, recognition of, and publication by

Native scholars and researchers, but not all are willing to ask non-Native allies to step

aside (see Mihesuah, 1996 for a range of opinions). These observers suggest that the

most productive research will result from respectful collaboration that does not

dichotomize researchers as Native or non-Native but does make the contributions of

Native colleagues integral to the design and conduct of studies.

Even those who adhere to the "Native-only" requirement run up against debates over

who is "really Indian." Snipp (2000) productively discusses the many criteria used to

identify and assign "Indian-ness" and tribal identity, and the political and other agendas

attached to them. Snipp concludes that contemporary criteria for tribal

membership--however historically situated and flawed--are the prerogatives of

sovereign tribes.

Local evaluations of authenticity, whether dependent on research subject or research

investigator, should be respected in accord with the principle of self-determination, even

as researchers struggle to work out their often anomalous positions as "insider,"

"outsider," or some combination of both (Brayboy & Deyhle, 2000).

POLITICS AND SCHOLARSHIP

There have been many calls over the years for improved research in AI/AN education.

The most recent is Executive Order 13096 (Clinton, 1998), which mandates the

development and implementation of a national research agenda in Indian education.

Consultations and meetings with Native peoples across the country culminated in the

"American Indian and Alaska Native Education Research Agenda" (Research Agenda

Working Group, Strang, & von Glatz, 2001). The agenda identifies high-priority research

interests and outcomes for Native communities and clearly articulates the central roles

that Native scholars, parents, teachers, schools, and communities must play to ensure

high-quality research. The promise embedded in the research agenda, however, may

be threatened if limited notions of what constitutes high-quality research, using criteria

such as those outlined earlier, overtake the powers of Native peoples to evaluate

research in their communities and schools.

Science as practiced by many Western researchers is often invoked as the answer to

www.eric.ed.gov ERIC Custom Transformations Team

Page 4 of 9 ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.
educational woes. This view tends to dismiss the power of all human inquiry--including

Indigenous knowledge systems--to observe, analyze, theorize, and generate creative

solutions. Research and development projects designed within the Western

mold--especially those meant to provide guidance for classroom practice--often rely on

standardized scripts to instruct and standardized tests to assess effectiveness (Metcalf,

2002). Using standardized tests with minority populations is a dubious practice (Padilla

& Lindholm, 1995, p. 97).

A much more promising line of research draws upon funds of knowledge in Native

communities to derive and apply research insights in education. "Funds of knowledge"

refers to the repositories of knowledge residing in communities that are frequently

overlooked or dismissed in conventional Western science and school curricula (Moll et

al., 1992). In a study of the Yupiit (Alaska) Nation and school system, Yupiaq

researcher Oscar Kawagley shows how local scientific, mathematical, and technological

knowledge can be used to improve the school curriculum. Situating science within a

Yupiaq worldview and ecology of sustainability, Kawagley provides a rigorous and

authentic framework for research and "for rethinking what we teach in schools and how

we teach it, particularly as it relates to science" (1995, p.138).

Research is needed that attends to the great diversity among Nations and the variability

within local sites. Such research will most often be qualitative in its methodology. Page

(2000) points out that "Qualitative research has challenged the 'science' in the social

sciences, and in the natural sciences, too, even though science has long been the gold

standard for knowledge and a source of disciplinary authority" (p. 23). She warns

against "nostalgia for the large-scale, randomized experiments thought to provide

unbiased and exact answers" (p. 24). The quest for scientific truth in such endeavors

will likely prove elusive (Cassell, 2002) and could pose a threat to AI/AN educational

self-determination and sovereignty.

The political context of educational research can make the difference between a project

that serves Native community and strengthens sovereignty or actively undermines both.

None of the parties involved in educational research is apolitical--not tribal communities,

tribal governments, federal education agencies, or academicians. We all work from

particular positions, but certain positions cannot continue to be privileged over others.

Ultimately, judgments about research quality in Indian education most productively

belong in Indian country.

REFERENCES

Banks, S. R. (1997). Caregiver and professional perceptions of assessment practices

and validity for American Indian/Alaska Native families. Journal of American Indian

Education, 37(1), 16-44.

Brayboy, B. M., & Deyhle, D. (2000). Insider-outsider: Researchers in American Indian

communities. Theory into Practice, 39(3), 163-169.

ERIC Resource Center www.eric.ed.gov

ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.

Page 5 of 9
Cassell, J. (2002). Perturbing the system: "Hard science," "soft science," and social

science: The anxiety and madness of method. Human Organization, 61(2), 177-185.

Clinton, W. J. (1998, August 6). American Indian and Alaska Native education:

Executive order 13096. Federal Register, 63(154), 42681-84. Retrieved December 20,

2002, from http://www.niea.org/indedeo.htm.

Corey, S. M. (1953). Action research to improve school practices. New York: Teachers

College Press, Columbia University.

Kawagley, A. O. (1995). A Yupiaq worldview: A pathway to ecology and spirit. Prospect

Heights, IL: Waveland Press.

Kelly, K. T. (1996). The logic of reliable inquiry. New York: Oxford University Press.

Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4),

34-46.

Lipka, J., with Mohatt, G. V., & the Ciulistet Group. (1998). Transforming the culture of

schools: Yup'ik Eskimo examples. Sociocultural, political, and historical studies in

education. Mahwah, NJ: Lawrence Erlbaum Associates.

Lomawaima, K. T. (2000). Tribal sovereigns: Reframing research in American Indian

education. Harvard Educational Review, 70(1), 1-21.

Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard

Educational Review, 62(3), 279-300.

McCarty, T. L. (1997). Teacher research methods in language and education. In N. H.

Hornberger & D. Corson (Eds.), Encyclopedia of language and education: Vol. 8.

Research methods in language and education (pp. 227-237). Norwell, MA: Kluwer

Academic Publishers.

McCarty, T. L. (2002). A place to be Navajo: Rough Rock and the struggle for

self-determination in Indigenous schooling. Sociocultural, political, and historical studies

in education. Mahwah, NJ: Lawrence Erlbaum.

McCarty, T. L., & Dick, G. S. (2003). Telling The People's stories: Literacy practices and

processes in a Navajo community school. In A. I. Willis et al. (Eds.), Multicultural issues

in literacy research and practice. Mahwah, NJ: Lawrence Erlbaum Associates.

McCarty, T. L., Watahomigie, L. J., & Yamamoto, A. Y. (1999). Introduction: Reversing

language shift in Indigenous America: Collaborations and views from the field [Theme

issue], Practicing Anthropology, 21(2), 2-4.

www.eric.ed.gov ERIC Custom Transformations Team

Page 6 of 9 ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.
Metcalf, S. (2002, January 28). Reading between the lines. The Nation. Retrieved June

19, 2002, from http://www.thenation.com/doc.mhtml?I=20020128&x=metcalf.

Mihesuah, D. A. (Ed.) (1996). Writing about (writing about) American Indians [Special

issue]. American Indian Quarterly, 20(1).

Miller, D. C., & Salkind, N. J. (2002). Handbook of research design and social

measurement (6th ed.). Thousand Oaks & London: Sage Publications.

Moll, L. C., Amanti, C., Neff, D., & Gonzalez, N. (1992). Funds of knowledge for

teaching: Using a qualitative approach to connect homes and classrooms. Theory into

Practice, 31(1), 132-141.

Padilla, A. M., & Lindholm, K. J. (1995). Quantitative educational research with ethnic

minorities. In J. A. Banks & C. A. Banks (Eds.), Handbook of research on multicultural

education (pp. 97-113). New York: Macmillan Publishing. (ERIC Document

Reproduction Service No. ED 382 701)

Page, R. N. (2000). The turn inward in qualitative research. Harvard Educational

Review, 70(1), 23-38.

Research Agenda Working Group, Strang, W., & von Glatz, A. (2001). American Indian

and Alaska Native education research agenda. Washington, DC: U.S. Department of

Education. (ERIC Document Reproduction Service No. ED 462 216) Retrieved

December 20, 2002, from http://www.indianeduresearch.net

Romero, M. K. (1994). Identifying giftedness among Keresan Pueblo Indians: The Keres

study. Journal of American Indian Education, 34(1), 35-58.

Smith, L. T. (1999). Decolonizing methodologies: Research and Indigenous peoples.

New York: Zed Books.

Snipp, C. M. (2000, May-June). Some alternative approaches to the classification of

American Indians and Alaska Natives. Paper presented at the American Indian and

Alaska Native Research Agenda Conference, Albuquerque, NM. Retrieved June 19,

2002, from http://www.indianeduresearch.net/MSnipp.pdf.

Stringer, E. T. (1996). Action research: A handbook for practitioners. Thousand Oaks:

Sage Publications.

Swisher, K. G. (1996). Why Indian people should be the ones to write about Indian

education. American Indian Quarterly, 20(1), 83-90.

Taba, H., & Noel, E. (1957). Action research: A case study. Washington, DC:

Association for Supervision and Curriculum Development (NEA).

ERIC Resource Center www.eric.ed.gov

ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.

Page 7 of 9
Westmeyer, P. (1981). A guide for use in planning and conducting research projects.

Springfield, IL: Charles C. Thomas.

Wilkins, D., & Lomawaima, K. T. (2001). Uneven ground: American Indian sovereignty

and federal law. Norman: University of Oklahoma Press.

Wolcott, H. F. (1994). On seeking--and rejecting--validity in qualitative research. In H.

Wolcott, Transforming qualitative data: Description, analysis, and interpretation (pp.

337-373). Thousand Oaks, CA: Sage Publications.

Zepeda, O. (1990). American Indian language policy. In K. L. Adams & D. T. Brink

(Eds.), Perspectives on official English: The campaign for English as the official

language of the USA (pp. 247-256). Hawthorne, NY: Walter de Gruyter & Co.

-----

K. Tsianina Lomawaima is a professor of American Indian studies at the University of

Arizona. She teaches and writes about the history of American Indian experiences in

the late 19th and 20th centuries, focusing on Native self-determination, federal policy

development, and impacts on Native domestic life and education.

Teresa L. McCarty is a professor of language, reading and culture and is interim dean of

the College of Education at the University of Arizona. She teaches courses and

conducts qualitative research on American Indian and bilingual/multicultural education,

Indigenous language education, and language planning and policy.

This publication was prepared with funding from the Office of Educational Research and

Improvement, U.S. Department of Education, under contract no. ED-99-CO-0027. The

opinions expressed herein do not necessarily reflect the positions or policies of OERI,

the Department, or AEL.

Title: Reliability, Validity, and Authenticity in American Indian and Alaska Native

Research. ERIC Digest.

Note: Online version includes annotated bibliography with some full-text links.

Document Type: Information Analyses---ERIC Information Analysis Products (IAPs)

(071); Information Analyses---ERIC Digests (Selected) in Full Text (073);

Available From: ERIC/CRESS, P.O. Box 1348, Charleston, WV 25325-1348. Tel:

800-624-9120 (Toll Free). For full text: http://www.ael.org/ERIC/digests/edorc02-4.pdf.

Descriptors: Action Research, American Indian Education, Educational Research,

Indigenous Knowledge, Participatory Research, Reliability, Standards, Tribal

Sovereignty, Tribally Controlled Education, Validity

Identifiers: Authenticity, ERIC Digests, Local Control, Politics of Research, Research

Quality

###

www.eric.ed.gov ERIC Custom Transformations Team

Page 8 of 9 ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.
[Return to ERIC Digest Search Page]

ERIC Resource Center www.eric.ed.gov

ED470951 2002-12-00 Reliability, Validity, and Authenticity in American Indian and

Alaska Native Research. ERIC Digest.

Page 9 of 9