The Problem of Literacy
By: Ryan Clason
Tags: Education, Education Reform, Teaching

(For greater context regarding the problem, ref. my article on gerrymandering)

In the age of data who knew that collecting it was one of the toughest efforts in education. We collect reams of data on students; socio-economic status, EL status, cognitive ability, but one metric that tapers off into the ether seems to be data on student literacy. If I were to ask what the Lexile/ literacy level of Concord High students is, could this question be answered. And that can be problematic if we are to track one of the more important — if not the most important metric tied to a student: literacy.

Literacy is measured, but it is measured in the lower grades with AR tests and new computerized literacy programs like iReady. But these programs seem to only tune themselves to the lower grades (iReady is used for upper-level EL students for testing purposes). Standardized tests have been decried as invalid measures of literacy (1,2,3), but over a decade’s worth of data shows really strong trends. Income level seems to be the strongest correlate to literacy (4). Data from SAT has been one of the longest longitudinal metric sources for education. Before the redesign SAT scores were flat as a pancake (5), and other standardized test measures have showed a consistent gap between race (6). On the whole, standardized tests have been reliable measures of student progress at a macro level.

The Data

One standardized test that seems to have more of a relevance of student literacy have been college entrance exams. While the SAT or ACT may affect entrance into a school, it’s superfluous in the face of a college entrance exam. While one may get you in the door, the other determines the length of the journey to your goal. One such test has been the Early Assessment Program (EAP).

The EAP was attached to the STAR test; the former NCLB standardized test. After measuring the sample test, the grade level of the questions seems to be close to the 11th grade level. When students take the test (11th grade) it tests how proficient they are in 11th grade content (7). The following is a chart that compares EAP scores of a random selection of California schools set against their graduation rates.

There is clearly a substantial gap between the graduation rates of students and their math and English scores. As a note, this is a global issue — a random sample of California schools. But this isn’t necessarily troublesome in and of itself. Students will become more proficient as they become more educated. They’ll reach an English and math ceiling in their studies and maybe not need to use the skills demanded of them outside of academia.

More than 60 percent of the nearly 40,000 first-time freshmen admitted to the CSU require remedial education in English, mathematics or both. These 25,000 freshmen all have taken the required college preparatory curriculum and earned at least a B grade point average in high school. The cost in time and money to these students and to the state is substantial. Moreover, these students are confused by seemingly having done the right things in high school only to find out after admission to the CSU that they need further preparation. (8)

If we were to conclude that students will become more proficient after high school due to some catalyst affecting their progress, we’d be mistaken. For Mt. Diablo High School students their EAP results are 14% for English and 0% for Math. (9) The grades we give students provide a signal as to their performance. In our effort to support students with their feelings of self-efficacy and accomplishment through clear grade inflation we shield them to the reality that they may be illiterate by several degrees. This is something that must be managed. And you cannot manage what you cannot measure.

The new Common Core test, CAASP, doesn’t have the responsiveness or consistency (it’s not given with frequency), so it becomes problematic tracking student literacy. From a pragmatic standpoint the EAP and other college entrance exams are the more relevant tests. Consider the amount of student loans one must take to finish college; with the student loan debt total for the nation being $1.48 Trillion (10) you get into college, but how much capital do you have to lay down for remediation.

With a dropout rate of 30% (11) a prioritized correlate may be tough to find, but you can probably attribute literacy as a key component of college attendance after high school and Freshman year dropout rates. You can trace all of this need for remediation back to high school, then middle school, and elementary.

Measuring literacy in middle and high school becomes imperative to ensuring student success beyond the confines of k-12 education. To do so, a control group needs to be created. The data must be specific to CHS and include some of the testing measures mentioned previously. Unfortunately, some of the data is dated. With the switch to Common Core many of the standardized measures have become invalid continuing forward. However, the data has been collected longitudinally so it’s integrity lies within the trend and it can serve as a predictor.

This graph represents a sample Community College entrance exam results of second semester seniors at Concord High School in the 2015–2016 school year. Students were given a sample of 10 English questions and 10 math questions.

The results are very telling. No students passed the test — the passing goal was set at 85%. This indicates that students will need remediation when entering college. With a majority of students hovering around the 30th to the 60th percentile, students may need greater remediation. There is no way to really say how much remediation they need — while they take remedial classes they may be also take general education classes. But students who need heavy remediation may find themselves struggling in college level general education courses. It is important to note that the test is leveled at the 11th grade. While the argument that not every student should go to college (alternatives are never mentioned after the argument is made), the test indicates how proficient students are with 11th grade material.

The results in the above chart does correlate to the data below. (12) This is a chart from Diablo Valley College. Although the sample is relatively small for the report, it shows similar non-passing results for the entrance Exam.

In larger studies student literacy in California has not been good as assessed by the CSUs and UCs. In 2009, CSU Freshmen were tracked at 58% for being unready for college. Seven CSU campuses had rates as high as 66%. At the UC level, 26% of Freshman were unprepared for college level writing. (13)

At the other end of the literacy spectrum, here are the results from a sample California High School Exit Exam (CAHSEE) taken by a sample of 140 sophomores and juniors. Because the sample was low, aggregate data can only be disclosed.

The CAHSEE is set to an 8th grade curriculum and is no longer used in the state of California as a prerequisite for high school graduation. Scores are much better, but if this were used as a college entrance exam, no students passed and would still need remediation.

With this data a baseline can be created literacy rate can be determined. But this isn’t enough. Student literacy and literacy growth is unknown outside of what standardized tests tell us. The responsiveness of these tests seems isn’t enough to recover or assist students with their growth. Flexible assessments might be a more responsive system. The important components of literacy are word fluency and comprehension. Each play into each other, i.e. being fluent in grade level language helps with comprehension simply by allowing the student to decode what they are reading, and comprehension can help students use context clues to decode words. Here are a few proposed samples of how literacy can be measured.

Solution 1

One can measure literacy on very general terms — perhaps modeling a college entrance exam for grades 8,9,10 would be appropriate. This would be very simple: have teachers sit down and create grade-level common assessments.

Solution 2

One method of testing literacy may be using the CLOZE method (14). Developed in the late 1970’s, CLOZE reading has become a revived method of teaching and testing literacy.

Design

The design was pretty straight-forward: write a paragraph that the students would easily identify content wise. If they were studying the industrial revolution, the paragraph should be related to that topic. Next, find grade-level vocabulary. For this study eighth, tenth, and twelfth grade words were found. In all, three paragraphs were written with grade-level vocabulary imbedded. The vocabulary is then omitted, and the paragraph is set up in a “match from the word bank” design.

Here is a sample:

Under the Homestead Act, people [ 1 ] on a journey to find wealth on the open plains. Many were not [ 2 ] in farming, so the government passed the Morrill Act to ensure people knew how to farm. With the rise of the steel plow and other technologies, farming become a more [ 3 ] [ 4 ] as technology began to replace manual labor. As more farmers [ 5 ] prices began to sink for grain and farming for profit became [ 6 ]. With little [ 7 ], farmers began to organize to affect change.

A..endeavor B. pioneered C. recourse D. embarked E. unfeasible AB. formidable AC. proficient

Results

This paragraph is constructed with 8th grade vocabulary. Both 10th and 11th grade history students were tested (n=134). The results (ref. graph) show students scored at about a 60% at 8th grade vocabulary, 45% at 10th grade vocabulary, and 25% at 12th grade vocabulary.

Debrief

An alternative to this assessment would be to allow students to fill in the blank manually and grade it by the words they use. It might be a better measure of how literate they are as they are using their own repository to complete the assessment. This becomes highly problematic on the grading end as there is no real efficient way to match the word they use and assess the grade level of the word.

Solution 3

There isn’t really an established name or peer reviewed design for this next method of assessment, but it models other methods of assessment regarding reading comprehension. For the paper, the method is referred to as the NEWSELA assessment.

Design

The design relies heavily on the content found on NEWSELA. NEWSELA is a site that takes current news stories and writes/ re-writes them at various grade levels and Lexile levels. One story can be written at various degrees of difficulty.

The design itself involved finding articles relating back to the student’s curricular content. Three different stories were chosen at increasing levels of difficulty. Vocabulary was prechosen from the articles with the intention of having students define the words. Students were then instructed to read the article, define the prechosen words, and give a short summary of the article.

Because of the low sample (n=145), only aggregate data can be reported. The assessment was a manual process. The sample of sophomores were given articles rated at a grade level measure and the juniors were given articles rated at a Lexile level. Essentially, task was a combination of students summarizing the article and defining preselected key words. Scoring was based on judging where students missed key components of the article in the summary and incorrectly defining words. If a student correctly summarized the article at a grade level of 10, but missed 4/5 of the definitions, they would be given a score of 9.5. If a student did not correctly summarize a level 12 article, but correctly summarized a level 10 article and key word summaries were around 5/10, the student was scored at level 11.

Results

A third of sophomores scored above a 10th grade level. The remaining sophomores scored between a 7th and 10th grade level.

Juniors did roughly as well as the sophomores. Lexile is a little trickier to convert to a grade level. A Lexile of 1150 to 1200 is close to a 11th grade level.

Debrief

This is one method that does not rely on a multiple-choice format. Perhaps this allows students to use their own free expression to answer a question. Students seemed to have more trouble with defining the words rather than completing the summary.

There are a few drawbacks to this method. One is ease of grading. Grading this assessment takes a lot of judgment on the part of the scorer. The defining of terms is straight forward enough, but the summaries require arbitrary scoring — it is up to the scorer. Perhaps having the scorers look at samples and independently scoring the samples to compare and having many scorers grade the same assessment and aggregate the score.

Another drawback is the time it takes to score versus how much time teachers must dedicate to their regular instruction.

Finally, the results from this sample do not match previous scoring on the CLOZE reading sample or the control group. This puts the validity of the measure to question.

Conclusions

The control group scores of student literacy is troubling to say the least. It behooves us to take the time to measure student literacy gains in any way possible with frequency. Short of AR testing (requires outside reading) and an iReady system, other methods must be developed to track student proficiency with literacy and being able to respond with supports. Frequency becomes important to obtain a sample with confidence.

It’s important to note that no one measure can indicate a literacy. Students may not take the measures seriously unless there is sort of some consequence to their progress. If the possibility of leveled classes or mandatory tutoring do not exist or if we cannot tether these assessments to a grade (the issue of A_F grading is problematic in and of itself) students may not take the assessments seriously.

We can conclude that certain at-risk populations are going to score lower than the mean and more proficient students are going to score higher than the mean. It is induced that by the label these groups are going to perform as expected. English Learners are extensively tested for their literacy level, but other at-risk populations are not. If a teacher has a 30% mix of special education students and 504 students, there is no indication as to where they need their curriculum leveled at. There is a lot of inductive guess-work at play. But perhaps a 504 student doesn’t need as much scaffolding as their cohort. A teacher has no indication of this being the case until a few months into the year. A few months wasted on try to assemble a picture of that student’s literacy level.

It becomes imperative something is done to test for literacy to track student progress, respond to literacy needs on a prompt timeline, and to create a system of transparency for all stakeholders — especially for the students themselves.

Sources:

1. http://www.ascd.org/publications/educational-leadership/mar99/vol56/num06/Why-Standardized-Tests-Don%27t-Measure-Educational-Quality.aspx

2. https://www.usnews.com/opinion/articles/2013/04/11/why-excessive-standardized-testing-is-causing-american-schools-to-fail

3. https://www.edweek.org/ew/articles/2014/07/09/36jouriles.h33.html

4. http://www.ascd.org/publications/books/109074/chapters/How-Poverty-Affects-Behavior-and-Academic-Performance.aspx

5. https://reports.collegeboard.org/pdf/total-group-2016.pdf

6. https://www.brookings.edu/articles/the-black-white-test-score-gap-why-it-persists-and-what-can-be-done/

7. https://eap.ets.org/Previous.htm

8. http://www.calstate.edu/eap/

9. Mt. Diablo SARC

10. https://studentloanhero.com/student-loan-debt-statistics/

11. http://www.slate.com/blogs/moneybox/2014/11/19/u_s_college_dropouts_rates_explained_in_4_charts.html

12. http://dq.cde.ca.gov/dataquest

13. http://lao.ca.gov/sections/higher_ed/FAQs/Higher_Education_Issue_02.pdf

14. https://files.eric.ed.gov/fulltext/EJ387352.pdf