Assigning Grade Levels to Textbooks: Is It Just Readability?

Assigning Grade Levels to Textbooks: Is it just Readability? David F. Dufty (d.dufty@mail.psyc.memphis.edu) Arthur C. Graesser (a-graesser@memphis.edu) Max M. Louwerse (max@mail.psyc.memphis.edu) Danielle S. McNamara (d.mcnamara@mail.psyc.memphis.edu) Department of Psychology / Institute for Intelligent Systems Memphis, TN 38152 USA Abstract We evaluated the effectiveness of new indices of text cohesion to determine the appropriate human assigned grade level of a text. In particular, we investigated the efficacy of automated text indices produced by the online tool Coh- Metrix in predicting the grade level assigned by publishers to their own textbooks. To do this, we sampled 311 school textbooks from a large database, choosing roughly equal numbers of science, narrative, and social science texts. Publisher-assigned grade levels were found to be moderately predictable by traditional approaches such as the Flesch- Kincaid Grade Level. Prediction of grade level was significantly improved by the inclusion of cohesion indices obtained by Coh-Metrix. Implications for the improvement of textbook selection are discussed. Introduction The problem of providing material of appropriate difficulty to a student is an old one, with a venerable research tradition. Selecting the right textbook for a student is perhaps more achievable in a personalized, one-on-one, tutoring context, because the teacher can accurately assess the learning needs of the student. The task is considerably more difficult when such context is lacking, such as when a school is planning a curriculum or when a publisher produces a new textbook. There, the writer, editor, or decision-maker is removed from the learner and must make an educated guess as to the learners’ needs. And unlike the one-on-one tutor, a decision about a book by a school board, an education department, or a publisher is not made for a single student, but for a class or even hundreds of classes at once. In practice, the decision to assign a particular textbook to a particular curriculum (and therefore, to the students enrolled in that curriculum) is a complex one. Schools, often via a school board, decide on priorities and subject areas. Teachers frequently have input into the process, contributing knowledge about the needs of the particular student population. Legislators impose constraints according to the wishes of the state. Publishers also play a role in providing the available books for consideration. As a guide in this process, books are commonly assigned a grade level for which they are considered appropriate. This may be a precise level or a range of grade levels. The manner in which such a grade level is assigned, either by a publisher, an educational institution, or a third party, has been the subject of much research and debate (e.g., Chall, Conard, & Harris-Sharples, 1991; DuBay, 2004), and is the central issue of this paper. Despite the existence of numerous systems for assigning grade level, in practice this decision comes down to human judgment, with factors such as the author’s intentions, the needs of the education system, and the opinion of teachers, educators and publishers, all playing a part. While editors may pay some attention to automated scores, such scores are but one of many factors that are taken into consideration. The task of automated grade level detection is to predict the judgment of editors, not the other way around. Traditional approaches. There are at least 200 readability measures for texts, all of which primarily measure two aspects of the sentence: word difficulty, usually through word length or syllable count, and sentence difficulty, typically through sentence length (DuBay, 2004). One of the oldest and best-known methods for determining the appropriate grade level of a textbook is the Flesch-Kincaid Grade Level formula (Klare, 1974-1975). Grade level is based on a formula that computes the average number of syllables per word and the average length of all the sentences. The formula is computed as: FK = (.39 x ASL) + (11.8 x ASW) – 15.59 FK is the Flesch-Kincaid Grade Level, ASL is the average sentence length (the number of words divided by the number of sentences), and ASW is the average number of syllables per word (the number of syllables divided by the number of words). This formula is elegant in its simplicity. It takes superficial characteristics of the text and provides an index of difficulty, expressed as a grade level. Furthermore, it is effective. Sentence length correlates with variables that impact the effort required to read the sentence such as syntactic complexity. Number of syllables gives an indicator