Domain: Educational Services
Question:
The university is implementing a new policy requiring rubrics for all major graded assignments (final exams, capstone projects, research papers, presentations) worth 20% or more of course grade. The Center for Excellence in Teaching prepared a comprehensive guide to support faculty during this transition. As Associate Provost for Academic Affairs, the Provost asks you to perform a rigorous review of the guide before university-wide distribution to ensure it will enable reliable, equitable and discipline-appropriate adoption across all programs.

Your review must address the guide's section on using generative AI to draft rubrics, which currently states: "Generative AI is a powerful tool that can support faculty in designing customized rubrics. AI can be a time-saving tool, but it does not have your contextual or pedagogical expertise, and outputs and responses should be reviewed to align with your specific needs." The guide includes three example prompts and brief notes on iterative refinement. One example prompt reads: "Create an analytic rubric for a persuasive essay in a writing class. The essay should focus on the importance of voter education and convince the reader of a particular viewpoint."
Please explain whether the current AI guidance is adequate for faculty who are novices with AI/LLM tools, and identify specific gaps that could lead to poor rubric validity, inconsistent grading or academic integrity risks. Then produce a revised section that does the following:

Provides an example prompt template (including required fields and optional fields) that faculty may complete before using any public or institutional LLM to draft a rubric. Required fields must include items that ensure alignment to learning outcomes, observable performance indicators, assessment reliability, accessibility and data privacy.

Supply three annotated example prompts (one each for humanities, STEM, and studio/performing arts) that demonstrate how to capture discipline-specific criteria, reproducibility or technical accuracy, creative process, and multimodal artifacts. For each example, list two common AI failure modes and a short mitigation checklist faculty should apply to the AI output.
Outline a practical, staged implementation plan (pilot → scale) that includes training, peer calibration, inter-rater reliability checks, timeline, resource estimates, and success metrics (quantitative and qualitative). Include a short protocol for a 2-week pilot in three departments that tests rubric clarity, grading consistency, and student perceptions.
Recommend mandatory policy language to add to the guide about student data handling, prohibition or limits on pasting student work into public LLMs, disclosure when rubrics were AI-assisted, and procedures for addressing AI-generated bias or exclusionary language.

Expected points in a correct answer:
States that the current AI usage guidance for rubrics is inadequate for faculty who are novices with AI and LLM tools.
Identifies at least one specific gap (e.g. does not explain how a teacher can ask an LLM to generate specific rubric types) in the current AI section that could lead to poor outcomes for faculty in terms of rubric validity
Identifies that AI-generated rubrics with vague or generic criteria could lead to unreliable scoring if not reviewed and refined by faculty
Produces a revised prompt template that faculty can complete before using an LLM to draft a rubric
Recommends the template include guidance like "be specific about your learning outcomes, the student level, and the skills being measured"
Provides an annotated example prompt with fields for class, level, assignment prompt given to students, and expected learning outcomes for an assignment
Recommends mandatory policy language addressing student data handling
Recommends mandatory policy language requiring disclosure when rubrics were drafted with AI assistance by providing example statements for policy to include in syllabi
Lists free LLM applications available to faculty suited for educational assistance, including Anthropic Claude, Open AI ChatGPT, Google Gemini, Microsoft Copilot
Identifies common failure modes of LLMs (e.g. "hallucinating sources", generating simulated data unprompted) for faculty to pay attention to during the rubric-design process
Evaluates the document's guidance on the LLM iterative refinement process by discussing how users can provide more specific instructions to AI applications than those offered in the source document
Defines LLM "iterative refinement" process as the repetitive procedure of improving the model's output by providing detailed feedback on the model's output over time
States that large language models require increasingly specific instructions from the user to improve quality of generated rubrics.
Explains the risks of LLM/AI usage by faculty in rubric design as false information, misleading sources, and vague criteria.
Demonstrates how to capture discipline-specific criteria within an AI-generated rubric by providing prompts with discipline specific instructions
Suggests strict restrictions on pasting student work into public LLMs by discussing the risks of using public LLMs to grade student work
Identifies gaps in document (e.g. does not offer assistance in troubleshooting steps) in the current AI section
Recommends showing correct and incorrect (or good and bad) examples in the template
Recommends that faculty test AI-generated rubrics on a sample of student work before using them for actual grading.
Recommends including a worked example showing the end-to-end process of iteratively refining an AI-generated rubric, including before-and-after comparisons.
Suggests providing a downloadable assignment package or practice materials for faculty to use when learning to create AI-assisted rubrics.
Recommends including a short rubric checklist to help new AI users feel confident without overwhelming them.
Suggests including a link to an AI Agent that could help teachers draft effective rubrics and help the university consolidate knowledge.
Notes that the existing sample prompts in the document feel static and lack a real worked example showing the iterative process.
Provides a concrete example of a vague AI-generated criterion (e.g., "Shows understanding") contrasted with an improved specific version (e.g., "Explained the main idea clearly").