Domain: Educational Services
Question:
Given that the current instruments are largely direct and internal, what comparative (e.g., between sections/instructors/modalities) or external measures would most improve validity, and what minimum accompanying reporting (e.g., N/cohort size, section/instructor, modality, prerequisite completion, rubric version) would be needed to interpret any differences as learning-related rather than cohort-related?

Expected points in a correct answer:
States that the current methods of assessment (tied to the final exam) are appropriate for the stated goal
States that the targets lack justification
States that it is unclear whether the targets are attainable
States that more historical date is needed to justify targets.
States that data from prerequisite courses would be useful for interpreting results and telling the whole picture
States that employment or doctoral outcomes may help determine whether the exam reflects long term success.
States that the performance target should be tied to the school's mission statement.
States that results from subsequent courses should be reviewed to test whether the final exam predicts future success.
States that if final exam results do not align with later outcomes, the assessment method and goal should be reevaluated.
States that while additional data is generally helpful, external assessment from industry or cross-department comparisons may skew the data
States that industry input may introduce bias and have an outsized influence on goals and data analysis related to student improvement
States that external comparisons may reflect group differences rather than learning.
States that life events (e.g., pregnancies, health interruptions, bankruptcies) could skew external data more than in a controlled classroom environment
States that cross department comparisons may be misleading because student populations differ.
States that the student sample size must be reported to interpret performance differences.
States that modality or format must be reported alongside the results.
States that the rubric or instrument version must be reported to ensure comparability across terms.
States that comparisons must include student learning modality.
States that comparisons must include class instructor.
States that comparisons must include student prerequisites.
States that comparisons must include rubric version.
States that comparative analysis must remain within the existing learning outcome and goal framework
States that comparative results are not meaningful without accurate reporting.
States that external results are not meaningful if identified differences are not noted.
States that methods tied to the final exam are appropriate when predictive.
States that if students who score high on the final exam also perform well in the program and future classes, this indicates the assessment method is on track.
States that historical data and expanded context can demonstrate why the goal is important to the overall development of the program.
States that life events such as pregnancies, health interruptions, bankruptcies, or marriages are specific confounding factors in external data.
States that external data from industry is more susceptible to confounds than data from a controlled environment of specific classes within a department.