### Attachment A

- 9

. Technical Working Group Suggestions

In June 2014, Southwest REL researchers obtained comments from two TWG reviewers: Dan Goldhaber and Geoff Borman . This Appendix summarizes in bullet form the conceptual suggestions that Southwest REL researchers received. Line edits are omitted from this summary. The researchers response s are noted under each bullet prefaced by “RESPONSE.” 

### Description of Topic and Articulation of Need
Comment s:

 This requires a stronger argument. “We believe that” and “there is little reason to expect” are not good arguments. Instead, describe what events/trends/developments could mess things up and how your study will address these . Maybe you are doing this later, but it’s worth doing in the introduction too.

 This statement may be a bit too strong, in my opinion. This sounds like something one might say about the experimental manipulation of such a policy, which is clearly not the case here.
RESPONSE: We’ve added trend data for the past five years showing the percentage of students who completed the pre

- policy degree plans that required algebra II. There is very little change across time in the percentage of students completing alg ebra II. We have also altered the wording to make it sound less like we are using a quasi

experimental design.



### Research Questions

 I think it would be much better to focus on Enrollment and Completion rather than Completion and Failure. The problem with fa ilure is that you only observe it for students who choose to enroll. That creates a huge potential selection bias. Enrollment and Completion are both observed for everyone and by differencing them you can back out effects on Failure without calling them th at.
RESPONSE: We have added enrollment in algebra II as an outcome, and we have changed the failure outcome. We are now looking at the percentage of students who fail their third mathematics course, regardless of course content

— so we will be looking at th e mathematics failure rates for all students, not just the ones who enroll in algebra
II. As we discussed at the TWG meeting, if there is a way to assess whether students are

taking more “career

- and vocational

- type” classes after HB5, that would seem like a key policy

- relevant hypothesis to test.
RESPONSE: This is what we are planning to do as part of our descriptive analysis looking at the types of mathematics courses that students are taking post

- policy

. We’ve added some language to make this clearer.



### Literature Review

 This entire section is overly biased in favor of Algebra II. While I agree that HB5 is silly and incomprehensible, you have to try to at least come up with some potential counterarguments. Why did people want to get rid of Algebra II and dumb down their kids? What are these other mathematics courses that kids can now take to meet the requirements and how potentially beneficial are they? Are there any serious education researchers who agree with Texas? Are there any potentia l positive effects of HB5? (Students less likely to drop out or become discouraged, fewer under

prepared students qualifying for UT, career students being better prepared through specific courses they now have the freedom to take). Are there any states or countries that have done something like this? What were their experiences?
RESPONSE: We have reconstructed the literature review to present better present both sides of the algebra II debate and provide a more objective review of previous research.

 I don ’t think you can say that. The NUMBER of courses they take may not be affected but the TYPE of courses could still be affected even for students who take more courses than required.
RESPONSE: The graduation requirements at this time were simply increased; students were required to take more courses in order to graduate. Students already taking the required number of courses were not affected by the policy change.



### Power

 This would seem to be a tiny effect. What would be the practical policy relevance of such an effect?
RESPONSE: We do not believe that an effect size this small would have practical policy relevance. However, it is the minimum detectable effect size for this study, which we are required to calculate per IES requirements. We have added langu age specifying what we believe would be a policy relevant difference.

### Outcomes Analyses

 I see that success must mean attempted and passed (would any grade other than an “F” qualify as a success?), but does failure mean not taken, or taken and not passed , or both? I would think that taken and not passed rates would be higher pre

- HB5, because a larger and more diverse group, presumably, took the course when it was required. How would this affect your analysis?
RESPONSE: Failure refers to not passing the course, which in Texas means earning a grade of “F”. We will retain this definition of failure for our analyses; however, we will look at failure rates for all students’ third mathematics course, rather than just the failure rates for those who enroll in a lgebra II.



### Limitations and Possible Solutions

 Why use only one year? If you are going to do an interrupted time series study like this you need to have at least a couple of pre

- HB5 data points to account for history and maturation bias problems. What if Algebra II enrollment was on an upward trend independently already? That could cause you to draw completely the wrong conclusion.
RESPONSE: We are not conducting an interrupted time series analysis.