Repository: PrairieLearn/PrairieLearn
Pull request #11956: AI question generation usage tracking
Author: login: miguelaenlle
Labels: name: feature: question editor
Created: 2025-05-10T14:12:51Z | Merged: 2025-05-21T18:50:41Z

Related to #11451. 

- AI question generation costs are now tracked in the `course_instances_usage` table and query: 

![image](https://github.com/user-attachments/assets/deead03c-640e-4fc4-9ab7-c94ca44d2d94)

- Question generation and question revisions are tracked in this table.


Diff:
diff --git a/apps/prairielearn/src/admin_queries/course_instances_usage.sql b/apps/prairielearn/src/admin_queries/course_instances_usage.sql
index 3fcc512e488..a9afef24744 100644
--- a/apps/prairielearn/src/admin_queries/course_instances_usage.sql
+++ b/apps/prairielearn/src/admin_queries/course_instances_usage.sql
@@ -136,7 +136,8 @@ WITH
       total.total_submissions,
       term.term_submissions::float / greatest(1, total.total_submissions::float) AS submission_term_ratio,
       term.external_grading_hours,
-      term.workspace_hours
+      term.workspace_hours,
+      NULL::double precision AS cost_ai_question_generation
     FROM
       student_active_data AS active
       JOIN student_term_data AS term ON (
@@ -179,7 +180,8 @@ WITH
             WHERE
               ciu.type = 'Workspace'
           )
-      ) / 3600 AS workspace_hours
+      ) / 3600 AS workspace_hours,
+      sum(ciu.cost_ai_question_generation) AS cost_ai_question_generation
     FROM
       term_course_instance_usages AS ciu
     WHERE
@@ -235,7 +237,9 @@ SELECT
   -- running duration of workspaces (in hours)
   cd.workspace_hours,
   -- total compute time for both workspaces and external grading jobs (in hours)
-  coalesce(cd.external_grading_hours, 0) + coalesce(cd.workspace_hours, 0) AS total_compute_hours
+  coalesce(cd.external_grading_hours, 0) + coalesce(cd.workspace_hours, 0) AS total_compute_hours,
+  -- cost of AI question generation (in USD)
+  cd.cost_ai_question_generation
 FROM
   combined_data AS cd
   LEFT JOIN course_instances AS ci ON (ci.id = cd.course_instance_id)
diff --git a/apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql b/apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
index ae7618ae82d..331641ed4ff 100644
--- a/apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
+++ b/apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
@@ -63,4 +63,6 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
diff --git a/apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts b/apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
index 9cac2f1bcac..cf7022561f1 100644
--- a/apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
+++ b/apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
@@ -5,9 +5,14 @@ import { loadSqlEquiv, queryAsync, queryRow, queryRows } from '@prairielearn/pos
 
 import * as b64Util from '../../lib/base64-util.js';
 import { getCourseFilesClient } from '../../lib/course-files-api.js';
-import { type Issue, QuestionGenerationContextEmbeddingSchema } from '../../lib/db-types.js';
+import {
+  IdSchema,
+  type Issue,
+  QuestionGenerationContextEmbeddingSchema,
+} from '../../lib/db-types.js';
 import { getAndRenderVariant } from '../../lib/question-render.js';
 import { type ServerJob, createServerJob } from '../../lib/server-jobs.js';
+import { updateCourseInstanceUsagesForAiQuestionGeneration } from '../../models/course-instance-usages.js';
 import { selectCourseById } from '../../models/course.js';
 import { selectQuestionById, selectQuestionByQid } from '../../models/question.js';
 import { selectUserById } from '../../models/user.js';
@@ -232,6 +237,14 @@ export async function generateQuestion({
    * to the LLM.
    */
   context: string | undefined;
+  /**
+   * The number of tokens in the prompt provided to the LLM.
+   */
+  promptTokens: number | undefined;
+  /**
+   * The number of completion tokens generated as output by the LLM.
+   */
+  completionTokens: number | undefined;
 }> {
   const serverJob = await createServerJob({
     courseId,
@@ -306,22 +319,37 @@ Keep in mind you are not just generating an example; you are generating an actua
       creator_id: authnUserId,
     });
 
-    await queryAsync(sql.insert_ai_question_generation_prompt, {
-      question_id: saveResults.question_id,
-      prompting_user_id: authnUserId,
-      prompt_type: 'initial',
-      user_prompt: prompt,
-      system_prompt: sysPrompt,
-      response: completion.choices[0].message.content,
-      html: results?.html,
-      python: results?.python,
-      errors,
-      completion,
-      job_sequence_id: serverJob.jobSequenceId,
-    });
+    const ai_question_generation_prompt_id = await queryRow(
+      sql.insert_ai_question_generation_prompt,
+      {
+        question_id: saveResults.question_id,
+        prompting_user_id: authnUserId,
+        prompt_type: 'initial',
+        user_prompt: prompt,
+        system_prompt: sysPrompt,
+        response: completion.choices[0].message.content,
+        html: results?.html,
+        python: results?.python,
+        errors,
+        completion,
+        job_sequence_id: serverJob.jobSequenceId,
+      },
+      IdSchema,
+    );
+
     job.data['questionId'] = saveResults.question_id;
     job.data['questionQid'] = saveResults.question_qid;
 
+    job.data['promptTokens'] = completion.usage?.prompt_tokens;
+    job.data['completionTokens'] = completion.usage?.completion_tokens;
+
+    await updateCourseInstanceUsagesForAiQuestionGeneration({
+      promptId: ai_question_generation_prompt_id,
+      authnUserId,
+      promptTokens: completion.usage?.prompt_tokens,
+      completionTokens: completion.usage?.completion_tokens,
+    });
+
     job.data.html = html;
     job.data.python = results?.python;
     job.data.context = context;
@@ -361,6 +389,8 @@ Keep in mind you are not just generating an example; you are generating an actua
     htmlResult: jobData.data.html,
     pythonResult: jobData.data.python,
     context: jobData.data.context,
+    promptTokens: jobData.data.promptTokens,
+    completionTokens: jobData.data.completionTokens,
   };
 }
 
@@ -495,19 +525,23 @@ Keep in mind you are not just generating an example; you are generating an actua
     errors = validateHTML(html, false, !!python);
   }
 
-  await queryAsync(sql.insert_ai_question_generation_prompt, {
-    question_id: questionId,
-    prompting_user_id: authnUserId,
-    prompt_type: isAutomated ? 'auto_revision' : 'human_revision',
-    user_prompt: revisionPrompt,
-    system_prompt: sysPrompt,
-    response: completion.choices[0].message.content,
-    html,
-    python,
-    errors,
-    completion,
-    job_sequence_id: jobSequenceId,
-  });
+  const ai_question_generation_prompt_id = await queryRow(
+    sql.insert_ai_question_generation_prompt,
+    {
+      question_id: questionId,
+      prompting_user_id: authnUserId,
+      prompt_type: isAutomated ? 'auto_revision' : 'human_revision',
+      user_prompt: revisionPrompt,
+      system_prompt: sysPrompt,
+      response: completion.choices[0].message.content,
+      html,
+      python,
+      errors,
+      completion,
+      job_sequence_id: jobSequenceId,
+    },
+    IdSchema,
+  );
 
   const files: Record<string, string> = {};
   if (results?.html) {
@@ -534,6 +568,16 @@ Keep in mind you are not just generating an example; you are generating an actua
     return;
   }
 
+  job.data['promptTokens'] = completion.usage?.prompt_tokens;
+  job.data['completionTokens'] = completion.usage?.completion_tokens;
+
+  await updateCourseInstanceUsagesForAiQuestionGeneration({
+    promptId: ai_question_generation_prompt_id,
+    authnUserId,
+    promptTokens: completion.usage?.prompt_tokens,
+    completionTokens: completion.usage?.completion_tokens,
+  });
+
   job.data.html = html;
   job.data.python = python;
 
@@ -590,6 +634,8 @@ export async function regenerateQuestion(
   jobSequenceId: string;
   htmlResult: string | undefined;
   pythonResult: string | undefined;
+  promptTokens: number | undefined;
+  completionTokens: number | undefined;
 }> {
   const serverJob = await createServerJob({
     courseId,
@@ -625,5 +671,7 @@ export async function regenerateQuestion(
     jobSequenceId: serverJob.jobSequenceId,
     htmlResult: jobData.data.html,
     pythonResult: jobData.data.python,
+    promptTokens: jobData.data.promptTokens,
+    completionTokens: jobData.data.completionTokens,
   };
 }
diff --git a/apps/prairielearn/src/lib/ai.ts b/apps/prairielearn/src/lib/ai.ts
new file mode 100644
index 00000000000..0d3a7f7dc27
--- /dev/null
+++ b/apps/prairielearn/src/lib/ai.ts
@@ -0,0 +1,18 @@
+import { config } from './config.js';
+
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
+  promptTokens,
+  completionTokens,
+}: {
+  promptTokens: number;
+  completionTokens: number;
+}) {
+  return (
+    (config.costPerMillionPromptTokens * promptTokens +
+      config.costPerMillionCompletionTokens * completionTokens) /
+    1e6
+  );
+}
diff --git a/apps/prairielearn/src/lib/config.ts b/apps/prairielearn/src/lib/config.ts
index 6fb7dbcb26e..799335fec47 100644
--- a/apps/prairielearn/src/lib/config.ts
+++ b/apps/prairielearn/src/lib/config.ts
@@ -571,6 +571,11 @@ const ConfigSchema = z.object({
    * Will be resolved relative to the repository root.
    */
   pythonVenvSearchPaths: z.string().array().default(['.venv']),
+  /**
+   * For the GPT-4o model as of 5/1/2025, in US dollars. Prices obtained from https://openai.com/api/pricing/.
+   */
+  costPerMillionPromptTokens: z.number().default(3.75),
+  costPerMillionCompletionTokens: z.number().default(15),
 });
 
 export type Config = z.infer<typeof ConfigSchema>;
diff --git a/apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql b/apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql
new file mode 100644
index 00000000000..bd649cff6a0
--- /dev/null
+++ b/apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql
@@ -0,0 +1,2 @@
+ALTER TABLE course_instance_usages
+ADD COLUMN IF NOT EXISTS cost_ai_question_generation DOUBLE PRECISION NOT NULL DEFAULT 0;
diff --git a/apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql b/apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql
new file mode 100644
index 00000000000..918fbce1cf6
--- /dev/null
+++ b/apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql
@@ -0,0 +1,4 @@
+ALTER TYPE enum_course_instance_usages_type
+-- Cost, in USD, of an AI question generation request.
+ADD VALUE 'AI question generation'
+AFTER 'Workspace';
diff --git a/apps/prairielearn/src/models/course-instance-usages.sql b/apps/prairielearn/src/models/course-instance-usages.sql
index 0a9c3572c4e..137500925db 100644
--- a/apps/prairielearn/src/models/course-instance-usages.sql
+++ b/apps/prairielearn/src/models/course-instance-usages.sql
@@ -88,3 +88,41 @@ ON CONFLICT (
 ) DO UPDATE
 SET
   duration = course_instance_usages.duration + EXCLUDED.duration;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  date_trunc('day', now() AT TIME ZONE 'UTC'),
+  $authn_user_id,
+  false
+FROM
+  ai_question_generation_prompts AS p
+  JOIN questions AS q ON (q.id = p.question_id)
+  JOIN pl_courses AS c ON (c.id = q.course_id)
+  JOIN institutions AS i ON (i.id = c.institution_id)
+WHERE
+  p.id = $prompt_id
+ON CONFLICT (
+  type,
+  course_id,
+  course_instance_id,
+  date,
+  user_id
+) DO UPDATE
+SET
+  cost_ai_question_generation = course_instance_usages.cost_ai_question_generation + EXCLUDED.cost_ai_question_generation
diff --git a/apps/prairielearn/src/models/course-instance-usages.ts b/apps/prairielearn/src/models/course-instance-usages.ts
index 833e925fac5..f932b00cecb 100644
--- a/apps/prairielearn/src/models/course-instance-usages.ts
+++ b/apps/prairielearn/src/models/course-instance-usages.ts
@@ -25,6 +25,8 @@
 
 import { loadSqlEquiv, queryAsync } from '@prairielearn/postgres';
 
+import { computeCompletionCost } from '../lib/ai.js';
+
 const sql = loadSqlEquiv(import.meta.url);
 
 /**
@@ -60,3 +62,29 @@ export async function updateCourseInstanceUsagesForGradingJob({
     grading_job_id,
   });
 }
+
+/**
+ * Update the course instance usages for an AI question generation prompt.
+ *
+ * @param param.prompt_id The ID of the AI question generation prompt.
+ */
+export async function updateCourseInstanceUsagesForAiQuestionGeneration({
+  promptId,
+  authnUserId,
+  promptTokens = 0,
+  completionTokens = 0,
+}: {
+  promptId: string;
+  authnUserId: string;
+  promptTokens?: number;
+  completionTokens?: number;
+}) {
+  await queryAsync(sql.update_course_instance_usages_for_ai_question_generation, {
+    prompt_id: promptId,
+    authn_user_id: authnUserId,
+    cost_ai_question_generation: computeCompletionCost({
+      promptTokens,
+      completionTokens,
+    }),
+  });
+}
diff --git a/database/enums/enum_course_instance_usages_type.pg b/database/enums/enum_course_instance_usages_type.pg
index d05df7e7bcc..a8dd99eabd9 100644
--- a/database/enums/enum_course_instance_usages_type.pg
+++ b/database/enums/enum_course_instance_usages_type.pg
@@ -1 +1 @@
-Submission, External grading, Workspace
+Submission, External grading, Workspace, AI question generation
diff --git a/database/tables/course_instance_usages.pg b/database/tables/course_instance_usages.pg
index cc0c249c608..33637d58b1d 100644
--- a/database/tables/course_instance_usages.pg
+++ b/database/tables/course_instance_usages.pg
@@ -1,4 +1,5 @@
 columns
+    cost_ai_question_generation: double precision not null default 0
     course_id: bigint not null
     course_instance_id: bigint
     date: timestamp with time zone not null


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>♻️ Duplicate comments (1)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts (1)</summary><blockquote>

`562-567`: **Duplicated token update logic.**

This is the second instance of the token update logic, which confirms my earlier suggestion about extracting a helper function to reduce code duplication.

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (7)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql (1)</summary><blockquote>

`70-98`: **LGTM - Well-structured usage tracking for AI question generation**

The implementation correctly inserts a usage record into the course_instance_usages table, with proper joins to associate the usage with the right institution, course, and user. The cost parameter is properly passed through to the database.


One suggestion to consider: Add a comment explaining why `include_in_statistics` is set to false. Is this to exclude AI generation costs from student usage statistics?

</blockquote></details>
<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts (2)</summary><blockquote>

`315-327`: **Consider handling potential SQL errors.**

The code retrieves the prompt_id from the SQL query but doesn't have explicit error handling if the insert operation fails.

Consider adding try/catch around the database operations to gracefully handle any potential SQL errors:

```diff
-    const prompt_id = await queryRow(sql.insert_ai_question_generation_prompt, {
-      question_id: saveResults.question_id,
-      prompting_user_id: authnUserId,
-      prompt_type: 'initial',
-      user_prompt: prompt,
-      system_prompt: sysPrompt,
-      response: completion.choices[0].message.content,
-      html: results?.html,
-      python: results?.python,
-      errors,
-      completion,
-      job_sequence_id: serverJob.jobSequenceId,
-    }, IdSchema);
+    let prompt_id;
+    try {
+      prompt_id = await queryRow(sql.insert_ai_question_generation_prompt, {
+        question_id: saveResults.question_id,
+        prompting_user_id: authnUserId,
+        prompt_type: 'initial',
+        user_prompt: prompt,
+        system_prompt: sysPrompt,
+        response: completion.choices[0].message.content,
+        html: results?.html,
+        python: results?.python,
+        errors,
+        completion,
+        job_sequence_id: serverJob.jobSequenceId,
+      }, IdSchema);
+    } catch (error) {
+      job.fail(`Failed to record AI question generation prompt: ${error.message}`);
+      return;
+    }
```

---

`341-346`: **Consider extracting token data update to a helper function.**

This code pattern for updating token counts in job data appears twice in the file (here and in the regeneration function). Consider extracting it to a helper function to avoid duplication.

```diff
+ function updateJobTokenCounts(job: ServerJob, completion: OpenAI.Chat.Completions.ChatCompletion) {
+   if (completion?.usage?.prompt_tokens) {
+     job.data['promptTokens'] = completion?.usage?.prompt_tokens;
+   }
+   if (completion?.usage?.completion_tokens) {
+     job.data['completionTokens'] = completion?.usage?.completion_tokens;
+   }
+ }

  // Then replace both instances with:
+ updateJobTokenCounts(job, completion);
```

</blockquote></details>
<details>
<summary>apps/prairielearn/src/lib/ai-grading.ts (4)</summary><blockquote>

`76-91`: **Token estimation assumptions should be documented or configurable.**

The function uses static assumptions about token counts (like 4 characters per token, 3750 system tokens, 250 completion tokens) which might not be accurate in all cases.

Consider making these assumptions configurable via the config system or at least adding more detailed comments about their source and potential limitations:

```diff
 export function approximatePromptCost(prompt: string) {
-  // There are approximately 4 characters per token (source: https://platform.openai.com/tokenizer),
-  // so we divide the length of the prompt by 4 to approximate the number of prompt tokens.
-  // Also, on average, we generate 3750 system tokens per prompt.
+  // Token estimation constants - consider moving these to config
+  const CHARS_PER_TOKEN = 4; // Approximation based on https://platform.openai.com/tokenizer
+  const AVG_SYSTEM_TOKENS = 3750; // Average system tokens per prompt based on historical data
+  const AVG_COMPLETION_TOKENS = 250; // Average completion tokens generated per prompt
+
   const approxPromptAndSystemTokenCost =
-    ((prompt.length / 4 + 3750) * config.costPerMillionPromptTokens) / 1e6;
+    ((prompt.length / CHARS_PER_TOKEN + AVG_SYSTEM_TOKENS) * config.costPerMillionPromptTokens) / 1e6;

-  // On average, we generate 250 completion tokens per prompt.
   const approxCompletionTokenCost = (250 * config.costPerMillionCompletionTokens) / 1e6;
```

---

`101-102`: **Consider making interval length configurable.**

The interval length is hard-coded to 1 hour (3600 * 1000 ms), but this might need to be adjusted in the future.

Consider making this value configurable via the config system:

```diff
-// 1 hour in milliseconds
-const intervalLengthMs = 3600 * 1000;
+// Default: 1 hour in milliseconds
+const intervalLengthMs = config.aiQuestionGenerationIntervalLengthMs ?? 3600 * 1000;
```

---

`107-115`: **Consider adding fallback value type annotation.**

The function properly handles the case when no usage data is found by returning 0, but the type annotation could be more explicit.

```diff
-export async function getIntervalUsage({
+export async function getIntervalUsage({
   aiQuestionGenerationCache,
   userId,
 }: {
   aiQuestionGenerationCache: Cache;
   userId: number;
-}) {
+}): Promise<number> {
   return (await aiQuestionGenerationCache.get(getIntervalUsageKey(userId))) ?? 0;
 }
```

---

`138-164`: **Consider error handling for cache operations.**

The function doesn't handle potential errors from the cache operations, which could occur if Redis is unavailable.

Consider adding try/catch to handle potential cache errors:

```diff
-  aiQuestionGenerationCache.set(
-    getIntervalUsageKey(userId),
-    intervalCost + completionCost,
-    timeRemainingInInterval,
-  );
+  try {
+    await aiQuestionGenerationCache.set(
+      getIntervalUsageKey(userId),
+      intervalCost + completionCost,
+      timeRemainingInInterval,
+    );
+  } catch (error) {
+    // Log the error but don't fail the operation, as this is non-critical functionality
+    console.error('Failed to update AI question generation usage cache:', error);
+  }
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 8752125fbbad338913924ea3e3d0e36f0928b987 and 1281035ce6b065ece2b67356a81bddd3a08d35d0.

</details>

<details>
<summary>📒 Files selected for processing (10)</summary>

* `apps/prairielearn/src/admin_queries/course_instances_usage.sql` (3 hunks)
* `apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql` (1 hunks)
* `apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts` (8 hunks)
* `apps/prairielearn/src/ee/pages/instructorAiGenerateDrafts/instructorAiGenerateDrafts.sql` (1 hunks)
* `apps/prairielearn/src/lib/ai-grading.ts` (2 hunks)
* `apps/prairielearn/src/lib/config.ts` (3 hunks)
* `apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql` (1 hunks)
* `apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql` (1 hunks)
* `database/tables/course_instance_usages.pg` (1 hunks)
* `database/tables/questions.pg` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧬 Code Graph Analysis (1)</summary>

<details>
<summary>apps/prairielearn/src/lib/ai-grading.ts (1)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/lib/config.ts (1)</summary>

* `config` (589-589)

</details>

</blockquote></details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (4)</summary>

* GitHub Check: build-core-images / build-images (ubuntu-24.04-arm)
* GitHub Check: build-core-images / build-images (ubuntu-24.04)
* GitHub Check: native-checks-js
* GitHub Check: test-prairielearn

</details>

<details>
<summary>🔇 Additional comments (22)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/lib/config.ts (3)</summary>

`73-74`: **Well-structured configuration for non-volatile Redis.**

The addition of a `nonVolatileRedisUrl` configuration parameter is a good approach for separating cache storage concerns. This allows for using a more persistent Redis instance for data that should survive restarts or deployments.

---

`307-307`: **Good configuration flexibility for non-volatile caching.**

Adding the `nonVolatileCacheType` enum with options for 'none', 'redis', and 'memory' provides flexible caching strategy for non-volatile data. This aligns well with the existing caching pattern in the application.

---

`573-582`: **Well-documented AI pricing and rate limit configuration.**

The implementation of AI question generation cost tracking with configurable rate limits and token pricing is well-structured. The defaults match current OpenAI GPT-4o pricing with:
- $1/hour rate limit to prevent unexpected high costs
- $3.75 per million prompt tokens
- $15 per million completion tokens

Including clear comments with the pricing source (OpenAI's website) is helpful for future maintenance.

</details>
<details>
<summary>apps/prairielearn/src/ee/pages/instructorAiGenerateDrafts/instructorAiGenerateDrafts.sql (1)</summary>

`39-39`: **No functional change detected.**

The only change appears to be the removal of a trailing newline at the end of the file, which doesn't affect functionality.

</details>
<details>
<summary>database/tables/course_instance_usages.pg (1)</summary>

`4-4`: **Good addition of AI cost tracking column.**

The new `cost_ai_question_generation` column is properly defined as a non-nullable double precision field with a default of 0. This is appropriate for tracking monetary costs and aligns with the broader AI question generation cost tracking implementation.

</details>
<details>
<summary>apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql (1)</summary>

`1-2`: **Well-written migration for the AI cost tracking column.**

This migration properly adds the `cost_ai_question_generation` column to the `course_instance_usages` table. Using `IF NOT EXISTS` makes the migration idempotent, which is a good practice for database migrations.

</details>
<details>
<summary>database/tables/questions.pg (1)</summary>

`9-9`: <details>
<summary>❌ Incorrect review comment</summary>

**Verify the impact of removing the default value for external_grading_enabled**

The default value for `external_grading_enabled` has been removed. This will change its default behavior from `false` to `NULL` when not explicitly set, which could impact queries that check for `external_grading_enabled = false` rather than `external_grading_enabled IS NULL`.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if there are queries that might be affected by the change from default false to NULL

# Find code that references external_grading_enabled without NULL checks
echo "Searching for potential affected queries..."
rg -A 5 -B 5 "external_grading_enabled\s*=\s*(true|false)" --type ts --type js --type sql

# Check if there are existing rows with NULL values for this field
echo "Checking if the column already contains NULL values:"
cat << EOF | node
const { Pool } = require('pg');
const pool = new Pool();

async function checkColumn() {
  try {
    const res = await pool.query(
      "SELECT COUNT(*) as null_count FROM questions WHERE external_grading_enabled IS NULL"
    );
    console.log(\`Number of questions with NULL external_grading_enabled: \${res.rows[0].null_count}\`);
    
    const totalRes = await pool.query("SELECT COUNT(*) as total FROM questions");
    console.log(\`Total questions: \${totalRes.rows[0].total}\`);
    
    pool.end();
  } catch (err) {
    console.error('Error executing query', err);
    pool.end();
  }
}

checkColumn();
EOF
```

Length of output: 1081

---

🏁 Script executed:

```shell
#!/bin/bash
echo "Inspecting database/tables/questions.pg around the column definition:"
sed -n '1,30p' database/tables/questions.pg

echo
echo "Searching for all occurrences of external_grading_enabled in the repository with context:"
rg -n -C 3 "external_grading_enabled"
```

Length of output: 7336

---



</details>

**No impact from removing the default for external_grading_enabled**

We’ve searched the codebase and found:

- No SQL queries comparing `external_grading_enabled = FALSE`
- Only one SQL use of `external_grading_enabled = TRUE`
- All JS/TS checks use truthiness (`if (!question.external_grading_enabled)`) which treats `NULL` as `false`

Since there are no direct `= FALSE` comparisons, removing the `DEFAULT FALSE` does not change any existing query or control‐flow behavior. You can disregard the original concern. 

> Likely an incorrect or invalid review comment.

</details>
<details>
<summary>apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql (1)</summary>

`1-4`: **LGTM!**

The migration correctly adds a new enum value 'AI question generation' to track costs associated with AI question generation requests.

</details>
<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql (1)</summary>

`66-68`: **LGTM - Added RETURNING clause to capture the inserted prompt ID**

The RETURNING clause is correctly added to get the ID of the newly inserted prompt, which will be used in the usage tracking logic.

</details>
<details>
<summary>apps/prairielearn/src/admin_queries/course_instances_usage.sql (3)</summary>

`139-140`: **LGTM - Added AI question generation cost to student usage data**

The implementation correctly adds a placeholder for AI question generation costs in the student usage data CTE.

---

`183-184`: **LGTM - Added aggregation for AI question generation costs in staff usage data**

The implementation correctly sums the AI question generation costs for staff usage.

---

`240-242`: **LGTM - Added AI question generation cost to the output**

The implementation correctly includes the AI question generation cost in the final output with a descriptive comment.

</details>
<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts (6)</summary>

`6-6`: **Good addition of cost estimation functionality.**

Nice integration with the shared cost estimation utility from the ai-grading module.

---

`233-240`: **Well-documented return type extensions.**

The return type has been properly extended with token usage information, and each new field is clearly documented with JSDoc comments.

---

`329-336`: **Good implementation of cost tracking integration.**

The code properly updates the course instance usages table with AI question generation cost data using the `estimateCompletionCost` function.

---

`387-388`: **Good propagation of token information to callers.**

The changes properly return the token count information to the API caller, allowing for transparent usage tracking throughout the system.

---

`625-626`: **Consistent return type updates.**

The return type of `regenerateQuestion` has been consistently updated to include the same token information as `generateQuestion`, which is good for API consistency.

---

`662-663`: **Consistent return value updates.**

The return object of `regenerateQuestion` properly includes the token counts, maintaining consistency with the `generateQuestion` function.

</details>
<details>
<summary>apps/prairielearn/src/lib/ai-grading.ts (4)</summary>

`3-6`: **Good import organization.**

The imports have been properly organized to support the new caching and configuration dependencies.

---

`60-74`: **Well-implemented singleton cache pattern.**

The singleton pattern for the AI question generation cache is implemented correctly, ensuring that only one instance is created regardless of how many times the initialization function is called. The function also properly uses configuration values from the system config.

---

`96-99`: **Good interval-based usage tracking approach.**

The interval-based usage tracking approach is a good solution for rate limiting. Using the modulo operation to align with consistent one-hour intervals is an efficient approach.

---

`117-133`: **Good implementation of cost estimation function.**

The `estimateCompletionCost` function is well implemented and properly handles potentially undefined token counts by defaulting to 0. The function correctly uses the configured cost rates for prompt and completion tokens.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T14:16:25Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-10T14:42:23Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-10T14:58:20Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-10T15:03:34Z
user: login: mwest1066
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T15:52:50Z
user: login: mwest1066
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T15:54:31Z
user: login: mwest1066
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T16:08:37Z
user: login: mwest1066
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T03:19:23Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/lib/ai.ts (1)</summary><blockquote>

`27-46`: **Consider adding error handling.**

The `updateCourseInstanceUsagesForAiCompletion` function interacts with the database but lacks error handling. Consider adding try/catch blocks to handle potential database errors gracefully.

```diff
export async function updateCourseInstanceUsagesForAiCompletion({
  promptId,
  authnUserId,
  promptTokens = 0,
  completionTokens = 0,
}: {
  promptId: string;
  authnUserId: string;
  promptTokens?: number;
  completionTokens?: number;
}) {
+  try {
    await queryAsync(sql.update_course_instance_usages_for_ai_question_generation, {
      prompt_id: promptId,
      authn_user_id: authnUserId,
      cost_ai_question_generation: computeCompletionCost({
        promptTokens,
        completionTokens,
      }),
    });
+  } catch (error) {
+    console.error('Failed to update course instance usages for AI completion:', error);
+    throw error; // Re-throw to allow caller to handle or log as needed
+  }
}
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**
**Cache: Disabled due to data retention organization setting**
**Knowledge Base: Disabled due to data retention organization setting**


<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 4eb9002e75eabc59f528ff58ea9f30bc11d1aaa6 and 10f9ccde330a265affa27ca6166dea76b27fd9ea.

</details>

<details>
<summary>📒 Files selected for processing (3)</summary>

* `apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts` (8 hunks)
* `apps/prairielearn/src/lib/ai.ts` (1 hunks)
* `apps/prairielearn/src/lib/config.ts` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (2)</summary>

* apps/prairielearn/src/lib/config.ts
* apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (3)</summary>

* GitHub Check: build-core-images / build-images (ubuntu-24.04-arm)
* GitHub Check: build-core-images / build-images (ubuntu-24.04)
* GitHub Check: test-prairielearn

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/lib/ai.ts (4)</summary>

`10-22`: **Function implementation looks good.**

The `computeCompletionCost` function correctly calculates the cost in USD based on prompt and completion token counts using the provided formula.

---

`30-31`: **Default values are appropriate.**

Using default values of 0 for optional token counts is appropriate as it ensures the function works even when token counts aren't provided, resulting in zero cost for those components.

---

`38-45`: <details>
<summary>✅ Verification successful</summary>

**Verify database schema compatibility.**

Ensure that the database schema has been updated with the necessary column `cost_ai_question_generation` in the appropriate table to store this cost data.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check for migrations that might have added the necessary column
echo "Checking for database migrations related to cost_ai_question_generation:"
fd -t f -e sql | xargs grep -l "cost_ai_question_generation"

echo "\nChecking for table definition files that include this column:"
fd -t f -e pg | xargs grep -l "cost_ai_question_generation"
```

Length of output: 680

---



</details>

**Database schema includes `cost_ai_question_generation`**  
Verified that the new column has been added and is ready to store cost data:

- apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql  
- database/tables/course_instance_usages.pg  
- apps/prairielearn/src/admin_queries/course_instances_usage.sql  
- apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql  

No further changes needed.

---

`3-3`: <details>
<summary>✅ Verification successful</summary>

**Verify config properties exist.**

This code relies on `config.costPerMillionPromptTokens` and `config.costPerMillionCompletionTokens`, but we need to confirm these properties are defined in the config file.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if the config file contains the required properties
CONFIG_FILE="apps/prairielearn/src/lib/config.js"
if [ -f "$CONFIG_FILE" ]; then
  echo "Config file exists. Checking for required properties:"
  grep -E "costPerMillionPromptTokens|costPerMillionCompletionTokens" "$CONFIG_FILE" -A 1 -B 1
else
  echo "Config file not found at: $CONFIG_FILE"
  echo "Searching for config file with these properties:"
  fd -t f -e js -e ts | xargs grep -l -E "costPerMillionPromptTokens|costPerMillionCompletionTokens"
fi
```

Length of output: 507

---

🏁 Script executed:

```shell
#!/bin/bash
# Verify that the required properties are defined in config.ts
CONFIG_TS="apps/prairielearn/src/lib/config.ts"
grep -n -C 2 -E "costPerMillionPromptTokens|costPerMillionCompletionTokens" "$CONFIG_TS"
```

Length of output: 421

---



</details>

**Config properties verified**

The `costPerMillionPromptTokens` and `costPerMillionCompletionTokens` properties are defined in `apps/prairielearn/src/lib/config.ts` (lines 577–578) with suitable defaults—no further action required.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-17T04:35:34Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>♻️ Duplicate comments (1)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql (1)</summary><blockquote>

`70-107`: **Centralize this SQL in the shared model layer**  
As noted in earlier reviews, it’s better to place `update_course_instance_usages_for_ai_question_generation` into `models/course-instance-usages.ts` as a helper alongside other usage-tracking queries. This promotes reuse and consistency across features.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**
**Cache: Disabled due to data retention organization setting**
**Knowledge Base: Disabled due to data retention organization setting**


<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 5a0ad124a2b2d09ee984c300538bc67a60d8a9dd and e0a9297a5a2767ab1f1417c7332f5dc2a25b00ee.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql` (1 hunks)

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (4)</summary>

* GitHub Check: build-core-images / build-images (ubuntu-24.04)
* GitHub Check: build-core-images / build-images (ubuntu-24.04-arm)
* GitHub Check: native-checks-js
* GitHub Check: test-prairielearn

</details>

<details>
<summary>🔇 Additional comments (3)</summary><blockquote>

<details>
<summary>apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql (3)</summary>

`66-68`: **Return inserted prompt ID for downstream operations**  
Adding `RETURNING id` here correctly surfaces the new prompt’s primary key so that it can be used immediately in the subsequent usages update. This aligns with the TypeScript implementation that expects the prompt ID.

---

`70-78`: **Ensure parameters are passed correctly to the upsert**  
You’ve introduced the `INSERT INTO course_instance_usages … SELECT … FROM ai_question_generation_prompts …` block to record the new cost. Please verify that the calling function always supplies non-null values for `$cost_ai_question_generation` and `$authn_user_id`, and that the prompt insertion’s returned `id` is bound to `$prompt_id`.

---

`99-107`: **Validate `ON CONFLICT` target covers the unique index**  
The conflict clause is on `(type, course_id, course_instance_id, date, user_id)`. Please confirm this matches the actual unique constraint on `course_instance_usages`—in particular, check that `institution_id` isn’t part of the index. Mismatched conflict targets can lead to silent duplicates or failed upserts.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-17T14:39:36Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-17T14:48:05Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-17T14:56:58Z
user: login: miguelaenlle
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-17T14:59:06Z
user: login: mwest1066
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-19T19:31:35Z

Review comments:
diff_hunk: @@ -179,7 +180,8 @@ WITH
             WHERE
               ciu.type = 'Workspace'
           )
-      ) / 3600 AS workspace_hours
+      ) / 3600 AS workspace_hours,
+      sum(ciu.cost_ai_question_generation) AS cost_ai_question_generation
path: apps/prairielearn/src/admin_queries/course_instances_usage.sql
user: login: miguelaenlle
body: Is this necessary?
created_at: 2025-05-10T14:38:38Z
author_association: COLLABORATOR
diff_hunk: @@ -36,4 +36,4 @@ WHERE
   q.course_id = $course_id
 ORDER BY
   q.id ASC,
-  p.id ASC;
+  p.id ASC;
path: apps/prairielearn/src/ee/pages/instructorAiGenerateDrafts/instructorAiGenerateDrafts.sql
user: login: miguelaenlle
body: Bunch of unnecessary changes here.
created_at: 2025-05-10T14:41:30Z
author_association: COLLABORATOR
diff_hunk: @@ -70,6 +70,8 @@ const ConfigSchema = z.object({
   coursesRoot: z.string().default('/data1/courses'),
   /** Set to null or '' to disable Redis. */
   redisUrl: z.string().nullable().default('redis://localhost:6379/'),
+  /** For the non-volatile Redis cluster. */
path: apps/prairielearn/src/lib/config.ts
user: login: miguelaenlle
body: Remove anything that wasn't directly used in this PR
created_at: 2025-05-10T14:41:49Z
author_association: COLLABORATOR
diff_hunk: @@ -6,7 +6,7 @@ columns
     directory: text
     draft: boolean not null default false
     external_grading_enable_networking: boolean
-    external_grading_enabled: boolean default false
path: database/tables/questions.pg
user: login: miguelaenlle
body: Revert this change
created_at: 2025-05-10T14:42:12Z
author_association: COLLABORATOR
diff_hunk: @@ -194,6 +199,47 @@ function extractFromCompletion(
   return out;
 }
 
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: miguelaenlle
body: I implemented this in a separate function since it will be reused once #11851 is merged to compute `completionCost`:

https://github.com/PrairieLearn/PrairieLearn/blob/0c228c6bad8436816ad56703d307310c5d92c7a9/apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts#L256-L285
created_at: 2025-05-10T14:58:20Z
author_association: COLLABORATOR
diff_hunk: @@ -63,4 +63,35 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  now(),
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
user: login: miguelaenlle
body: In other `course_instance_usages_` queries (e.g. `update_course_instance_usages_for_external_grading`), `date_trunc('day', _, 'UTC')` is used instead of `now()`. I used `now()` to ensure that condition for `course_instance_usages_date_type_course_id_course_instance__key` remains true:

https://github.com/PrairieLearn/PrairieLearn/blob/522c8b80e4bbb16fb547bbfca18072522b61a79f/database/tables/course_instance_usages.pg#L13-L17

Is using `now()` here okay, or should this instead be `date_trunc('day', ___)`? 
created_at: 2025-05-10T15:03:23Z
author_association: COLLABORATOR
diff_hunk: @@ -63,4 +63,35 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  now(),
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
user: login: mwest1066
body: This does need to be `date_trunc()`. We are storing this data as rollups over each day to make the eventual usage queries faster (only one row per student per day). See https://github.com/PrairieLearn/PrairieLearn/blob/eabb5ee4a4bbb247c8a81f499c6cebe0368e2391/apps/prairielearn/src/models/course-instance-usages.sql#L40 for an example of how this query should look. Basically, we do an upsert where the update adds to the existing row if the insert fails.

Rather than putting this query in `aiQuestionGeneration.sql`, can you add it to `models/course-instance-usages.ts` as a helper function?

Note that the query referenced above has a bug in it (https://github.com/PrairieLearn/PrairieLearn/issues/11849). To properly handle question sharing, we should not get the course via the chain variant -> question -> course, because this will return the _source_ course that contains the question, not the _destination_ course where the student is working. I think that we should be using `variants.course_id` to get the destination course, but I haven't verified this yet (that's why I haven't fixed #11849).
created_at: 2025-05-12T15:52:50Z
author_association: MEMBER
diff_hunk: @@ -194,6 +199,47 @@ function extractFromCompletion(
   return out;
 }
 
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: mwest1066
body: We'll want to re-use this for other AI tasks as well (e.g., AI grading) so let's shift this into a generic lib file (maybe `lib/ai.ts`?).
created_at: 2025-05-12T15:54:31Z
author_association: MEMBER
diff_hunk: @@ -194,6 +199,47 @@ function extractFromCompletion(
   return out;
 }
 
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
+  promptTokens,
+  completionTokens,
+}: {
+  promptTokens: number;
+  completionTokens: number;
+}) {
+  return (
+    (config.costPerMillionPromptTokens * (promptTokens ?? 0) +
+      config.costPerMillionCompletionTokens * (completionTokens ?? 0)) /
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: mwest1066
body: Do we need the ?? given that the arguments must be numbers?
created_at: 2025-05-12T16:08:30Z
author_association: MEMBER
diff_hunk: @@ -63,4 +63,35 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  now(),
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
user: login: mwest1066
body: As @nwalters512 pointed out in the meeting today, the query for this use case will be simpler because we already know the `course_id` we are currently inside and we don't need to worry about `course_instance_id`.
created_at: 2025-05-13T03:19:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,46 @@
+import { loadSqlEquiv, queryAsync } from '@prairielearn/postgres';
+
+import { config } from './config.js';
+
+const sql = loadSqlEquiv(import.meta.url);
path: apps/prairielearn/src/lib/ai.ts
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>🧩 Analysis chain</summary>

**Verify SQL file existence.**

This line loads SQL queries from an equivalent SQL file, but we should ensure the corresponding SQL file exists with the required query.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if the corresponding SQL file exists with the required query
SQL_FILE=$(echo $(dirname $0)/ai.sql)
if [ -f "$SQL_FILE" ]; then
  echo "SQL file exists:"
  grep -A 5 "update_course_instance_usages_for_ai_question_generation" "$SQL_FILE"
else
  echo "SQL file does not exist: $SQL_FILE"
  echo "Searching for the query in other SQL files:"
  fd -t f -e sql | xargs grep -l "update_course_instance_usages_for_ai_question_generation"
fi
```

Length of output: 450

---



</details>

**Ensure the SQL file is in the expected location**

`apps/prairielearn/src/lib/ai.ts` calls  
```ts
const sql = loadSqlEquiv(import.meta.url);
```
which will look for  
```
apps/prairielearn/src/lib/ai.sql
```  
—but no such file exists. The only matching SQL is here:  
```
apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
```

Please address this by one of the following:

- Move or copy `aiQuestionGeneration.sql` → `apps/prairielearn/src/lib/ai.sql`, ensuring it contains the `update_course_instance_usages_for_ai_question_generation` query.
- OR update your code to point to that file, for example:  
  ```diff
  - const sql = loadSqlEquiv(import.meta.url);
  + const sql = loadSqlEquiv(new URL('../ee/lib/aiQuestionGeneration.sql', import.meta.url).href);
  ```

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
In apps/prairielearn/src/lib/ai.ts at line 5, the code attempts to load SQL
queries from apps/prairielearn/src/lib/ai.sql, but this file does not exist. To
fix this, either move or copy the existing aiQuestionGeneration.sql from
apps/prairielearn/src/ee/lib/ to apps/prairielearn/src/lib/ai.sql ensuring it
contains the required query, or modify the code to load the SQL from the correct
existing path apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql instead of
using loadSqlEquiv(import.meta.url).
```

</details>

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-05-17T04:35:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,4 +63,44 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  date_trunc('day', now(), 'UTC'),
+  $authn_user_id,
+  false
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Fix invalid `date_trunc` signature and enforce UTC**  
PostgreSQL’s `date_trunc` only takes two arguments. The current `date_trunc('day', now(), 'UTC')` will error. To correctly roll up in UTC, use either:

```sql
date_trunc('day', now() AT TIME ZONE 'UTC')
```

or

```sql
(timezone('UTC', now()))::date
```

This ensures daily metrics align to UTC boundaries.

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
In apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql around lines 82 to 90,
the usage of date_trunc with three arguments is invalid in PostgreSQL. Replace
date_trunc('day', now(), 'UTC') with date_trunc('day', now() AT TIME ZONE 'UTC')
or alternatively use (timezone('UTC', now()))::date to ensure the date is
truncated correctly to the UTC day boundary.
```

</details>

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-05-17T14:39:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -194,6 +199,47 @@ function extractFromCompletion(
   return out;
 }
 
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: miguelaenlle
body: I moved `computeCompletionCost` to `lib/ai.ts` 👍 

https://github.com/PrairieLearn/PrairieLearn/blob/237c095c6f6e24af92f51de111bac700d7db62cc/apps/prairielearn/src/lib/ai.ts#L1-L18
created_at: 2025-05-17T14:48:05Z
author_association: COLLABORATOR
diff_hunk: @@ -63,4 +63,35 @@ VALUES
     to_jsonb($errors::text[]),
     $completion,
     $job_sequence_id
-  );
+  )
+RETURNING
+  id;
+
+-- BLOCK update_course_instance_usages_for_ai_question_generation
+INSERT INTO
+  course_instance_usages (
+    type,
+    institution_id,
+    course_id,
+    course_instance_id,
+    cost_ai_question_generation,
+    date,
+    user_id,
+    include_in_statistics
+  )
+SELECT
+  'AI question generation',
+  i.id,
+  c.id,
+  NULL,
+  $cost_ai_question_generation,
+  now(),
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql
user: login: miguelaenlle
body: Thank you for the feedback, @mwest1066 ! I adjusted the `update_course_instance_usages_for_ai_question_generation` prompt to add the completion cost to the existing `course_instance_usage_record` if there is a conflict:

https://github.com/PrairieLearn/PrairieLearn/blob/0b284e066ebbf23222d9137bc9e6a68c102726b3/apps/prairielearn/src/models/course-instance-usages.sql#L92-L128

I also moved the function to `models/course-instance-usages.ts`. 
created_at: 2025-05-17T14:56:58Z
author_association: COLLABORATOR
diff_hunk: @@ -194,6 +199,47 @@ function extractFromCompletion(
   return out;
 }
 
+/**
+ * Compute the cost of a completion, in US dollars.
+ */
+export function computeCompletionCost({
+  promptTokens,
+  completionTokens,
+}: {
+  promptTokens: number;
+  completionTokens: number;
+}) {
+  return (
+    (config.costPerMillionPromptTokens * (promptTokens ?? 0) +
+      config.costPerMillionCompletionTokens * (completionTokens ?? 0)) /
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: miguelaenlle
body: The `??` aren't needed; I removed them:

https://github.com/PrairieLearn/PrairieLearn/blob/0b284e066ebbf23222d9137bc9e6a68c102726b3/apps/prairielearn/src/lib/ai.ts#L3-L18
created_at: 2025-05-17T14:59:06Z
author_association: COLLABORATOR
diff_hunk: @@ -306,22 +319,37 @@ Keep in mind you are not just generating an example; you are generating an actua
       creator_id: authnUserId,
     });
 
-    await queryAsync(sql.insert_ai_question_generation_prompt, {
-      question_id: saveResults.question_id,
-      prompting_user_id: authnUserId,
-      prompt_type: 'initial',
-      user_prompt: prompt,
-      system_prompt: sysPrompt,
-      response: completion.choices[0].message.content,
-      html: results?.html,
-      python: results?.python,
-      errors,
-      completion,
-      job_sequence_id: serverJob.jobSequenceId,
-    });
+    const promptId = await queryRow(
path: apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts
user: login: mwest1066
body: We're trying to stick to the convention of singular table name + "_id", so this would be `ai_question_generation_prompt_id`. (And similarly below.)
created_at: 2025-05-19T19:28:24Z
author_association: MEMBER

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-05-10T14:12:57Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

<details>
<summary>📝 Walkthrough</summary>

## Walkthrough

This change adds tracking and reporting of AI question generation costs within PrairieLearn. It introduces a new cost column and enum value in the database, updates SQL queries and migrations, extends configuration with token cost rates, and modifies TypeScript code to compute, record, and return token usage and associated costs from OpenAI completions.

## Changes

| Files/Paths                                                                                           | Change Summary                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| apps/prairielearn/src/admin_queries/course_instances_usage.sql                                      | Extended SQL query to include a new `cost_ai_question_generation` column in CTEs and final SELECT, aggregating AI question generation costs for students and staff in course instance usage reports.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| apps/prairielearn/src/ee/lib/aiQuestionGeneration.sql                                               | Modified INSERT statement to include `RETURNING id` clause for inserted prompt ID.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts                                                | Updated `generateQuestion` and `regenerateQuestion` functions to return prompt and completion token counts, propagate token data, and record associated costs asynchronously via new helper function. Added new function `updateCourseInstanceUsagesForAiQuestionGeneration`.                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| apps/prairielearn/src/lib/ai.ts                                                                     | Added new module with function `computeCompletionCost` to compute AI completion cost from token counts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| apps/prairielearn/src/lib/config.ts                                                                 | Added new configuration fields `costPerMillionPromptTokens` and `costPerMillionCompletionTokens` with default values reflecting GPT-4o pricing as of May 2025.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| apps/prairielearn/src/migrations/20250508234914_course_instance_usages__cost_ai_question_generation__add.sql | Added migration to add a non-nullable `cost_ai_question_generation` column with default 0 to `course_instance_usages` table.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| apps/prairielearn/src/migrations/20250510130225_enum_course_instance_usages_type__ai_question_generation__add.sql | Added migration to insert `'AI question generation'` enum value into `enum_course_instance_usages_type` after `'Workspace'`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| database/tables/course_instance_usages.pg                                                           | Updated schema to add a non-nullable `cost_ai_question_generation` column with default 0 to `course_instance_usages` table.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| database/enums/enum_course_instance_usages_type.pg                                                  | Extended `enum_course_instance_usages_type` to include new value `AI question generation`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| apps/prairielearn/src/models/course-instance-usages.sql                                            | Added new SQL block `update_course_instance_usages_for_ai_question_generation` to insert or update AI question generation cost usage records with conflict handling to increment existing costs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| apps/prairielearn/src/models/course-instance-usages.ts                                             | Added new function `updateCourseInstanceUsagesForAiQuestionGeneration` to calculate and record AI question generation costs based on prompt and completion tokens.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |

</details>

<!-- walkthrough_end -->

<!-- announcements_start -->

> [!NOTE]
> <details>
> <summary>⚡️ AI Code Reviews for VS Code, Cursor, Windsurf</summary>
> 
> CodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback.
> Learn more [here](http://coderabbit.ai/ide).
> 
> </details>

---

> [!NOTE]
> <details>
> <summary>⚡️ Faster reviews with caching</summary>
> 
> CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 30th. To opt out, configure `Review - Disable Cache` at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the `Data Retention` setting under your Organization Settings.
> Enjoy the performance boost—your workflow just got faster.
> 
> </details>

<!-- announcements_end -->

---

<details>
<summary>📜 Recent review details</summary>

**Configuration used: .coderabbit.yaml**
**Review profile: CHILL**
**Plan: Pro**
**Cache: Disabled due to data retention organization setting**
**Knowledge Base: Disabled due to data retention organization setting**


<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 70143370e212f8d297538795d2f45308cea0ab0a and b87c6bbe69b82104545f25f46a22e5832fb9166b.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts` (8 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* apps/prairielearn/src/ee/lib/aiQuestionGeneration.ts

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (5)</summary>

* GitHub Check: native-checks
* GitHub Check: test-prairielearn
* GitHub Check: native-checks-js
* GitHub Check: build-core-images / build-images (ubuntu-24.04)
* GitHub Check: build-core-images / build-images (ubuntu-24.04-arm)

</details>

</details>
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpETZWaCrKNxU3bABsvkCiQBHbGlcABpIcVwvOkgAIgBBAElIIJD4fCxScipxDMhsRDRSCKoGAGt4DCJQ2MgAdzRkBwFmdRp6OQjYEnzESkhWomCvNDIfHsbrOwxHAX6ARnmATgBWADYNSETcPxIRmmQCFEQHHrNFgBYV+fQMekrcCkVsBmkStHLKon4AM0gklLBRC5TJkZQghTA5B1dQILC4bqQAAGTGwFD6AH1KsDMK9EBiCkUSEjILRqCpGj1cCpops4D1sNwyTQUBgGF5sEpDqUKlVIAJ8AjAWk8llweksJh6KlgRLdhIkBLkJUuqhqQJaf97Ax/GRELBBcceE8FUp7vDEVydfBuBCjvAfAVHtQqYjUi5+HwHpQfh8ev59jEjgi1TyvpAfiRqGiSHTEZ4KNx8H1fl1pD0GLBMKRlYd8JAyDSeswMup8BRw/g/kliGCcjEZRDCcUpegTvgGPAXfQmFD6rCVQieohZMCSMxNpBzBZIABVJnd/jCUTiKRNRzMZyuNzqaJcAGNuVinJy5tUsN8qcAOVmlC4i1Wa0gU9nfQoXEGw1GGHGRn0xnAKAyHoKscAIWtskXJhWHYLheCXEQxEkN5OiYJQqFUdQtB0f8TCgdxc1uMDCGPKCWDYDBOD8NA6nsDct35eQ0OUTDNG0XQwEMADTAMNBuG4RAAHpeG0Cs9ijCgMEExAKAYQS0FoVoMAxd14GkQTUXREgsQwHE2WkAlClIDREACLwOAMWIrIMGca1I9o6KcD1QMzbNpDcREAGUAEUABlhQ9BpkBIAAPGg7iDfNKnZTlzw+XlvlAodIR2RpEA7LsHJhIUDyBCF7IlTZPO4UR4B+eAGDQHxZHCNBIHIWimA5ZgsBRZNcAxbQVLyiUMQKjISSS2QSuRWh8GwDUel4UrEAlEkgvQWgzTbdAeBGV4DS8dD+yFK9Z18/yJCqoFWTTZFgViyiMU0zFsWpfT8TPDFmTQEkAGFoAAUU2RILR6JEcR+H5rvGrTHqM7SXver7whDZAmscLAFqTTxAw6eQHFYcNkra4FOvgbqRWU/qMBJcq9noH4nmYM6kRoChmBBtFbt0+7XkMolEBJdVogjct7GpIHeiJXYmAoWhEDjHpyowKqFGYNRyHoF6AvkBbGWZSLWRipRVWQBqUoUZqME2eIlvUCUqq8GqzpluXPM+3zPo+gWXQonYMHwWjxtwTxcEORFcY6rrDwyPq62oOb0C8DIiFm3XQqQcQ+TYR4KvXTMVrqcsykQbg/UgA1mduehQvp2XfCIKhaHDIv0TjVBXKqHoHieWgXjeOqDd7HZU4rBh0CIKuSCIRc+nz+trb5vgLqUSiS9doWzwX6KOWWwdEXwKQqB8BRmZb1ncQZCHdiTChNCMad/i8enI4yPMzqUdlnDv3TU1Cs+HP5zwNQqgtKItu5AwUArz5ibjmCI+YP7li/nwH+Xg/4/GwGyCE38nivHbv4ew8AiCy1wDGRAABuR+ZU/gZCniWGu5MA49B8v5d08hOgKRrnyLuJBvZ4D9kbRGyIe740JrKMOJMSQb2lpUe2jtnbQBXpRSgbAa4ukgB9T6ktLLWWATxPiAlhJUHgGJaIzgpIyTkiQEggkEECHkvAbyPUMgAHEI4ghMmZCyVlYg2UsHZRxMRMabmcn8cBQD6SMCzM3RaEtVpIhsJ9aAs4bBXkSFeOxKBaAkmfgUKk+ZkqJ1lHyJE2JKDBwJqHYmjjeq8BYLaEkdD+Qx3KOEQsv8+TJQYZA3Y+DJJnUSAAEVTMlBqU8CnnxiP4MWtANCXw8dfW+IIH7JSfiME89934hU/jEb+E0EED3YIAxAf4YCByGUUgRIJw6QVORU5gVTIA1I1B2Mo9RJiULKmpeg9o2RrwmMiaJsT4mJOSfAVJjARgZPQD8emtMABq8RfKzk+p5ERulCmFTUe4jRYAjBaKEiJPRakDGSWkrJQSpjzHwEsdoGxRMHHnMKv7Vx1lbKJAguCHx9F/EhLcnsgwwTAnIGYe8T4rC7ii3LCwxKfwCBlDIMLYoAAKS5toF7QW4NEO0+ApW6QAJQLyHHots6VOyvx2rAf4yQSmQBJsaSp/tToCiFOay1rZ/CWuzhQMoPwY51Elgc/6iRaCeUzOOV6KArkwKNMwrWR0EGaytVcnYPTkD+A6UrCM1NSTkgEJSVkb5ZlS2zcivIoFcpEwtWUvICqbWe1ohk5ASIGE2C9vNWEyI/UBu6Jubm+ZBDUlEfm4Z9AK1bG6Q3ZAg6em1RjnybKJrkpoDwN0ABlUHIZL4D0nV6rpWokosFMKpQHJUxYGdAA8iVDAAJlWqtPBDcIHg0pazYbRJBKC5RIg1i6N6oM+i/T0q8V8nMABi5Z4jWNsRgal4oBptLfSyG6B8f3HxFqM0V3rgmSs3eNbd6AsFVXSvYNAUhzTBkREIQQ6bqSQFlUiEjAgNBQ21T8fmdqTWVAtnLR1wrnVlqwB6r2KHERJrRPCYabxQJInsiQSlgjSYLyRJx85EnQNk2QUhZZzgehl2AlrVesVkQVugBu3SJJWxtSuZejI+mNVc3CHUBAmc1M8HwKjRcB6abJWo2RtAeaxY9BjkQP+oECpCvoP4BUs0+QlMTUG7E+Q2ShNILQcIAg8DZseI4dgRqGN8DQ1gZerYe4CoSkab0VdFzUDOkmB4yBQKDqOXKVs7mVZIfFl8CZUz4g3wg2/IjPQFkv1mSstZIE4GbL/js8QQCoD/uUxCCNtAuBTmfK+hcNAP372/WzEgf6cyAYoMByTIJwNLIwLKgA3rG20fquDAgrFUYhc6EQYFfJQS7AsbtEGIXpgziAAD8XAZgK0oMQi9JAQQWb1L9+qt4KDEIAL6aq4FYamSASDAAkPgQFehEXIixTo0SeKJJGKJSSixVj9sSkO6/DQ/skT7Km8+vIs1cHRiwdBug82DCLfE2TyDKokQ45xfognhKTFmJJxS0DFOnHU+fBzyANgQeCYiMJgsYVNNvKih82K7PFtIjpns58n3LN/ah5AAAPjFpQMs6CENl8D0HX3jcA74Ob5BlvKjW9l7rmnwDIB05U1gRneCYz5GW2zmXi25Msu59J3n/PdGC8McL4louyWk4l1xqnXNw9y4V503AyuNMRXV9rT52vkS6/9rLw3epHdzGdxbkgVvaA2+fHbiUYPdK1/6C7iKTeW/l+9zOaARRp4q/CjsUCrO9luL/Jo/i2L4/48T8Y0llimAYHKkQTP9L3GMuZfWegviGIuTi0E7O9V2GQ7YP3BQG+cFogy3iiJdRKATCWlrbJIUk7hnX5vh/EJEBA1NwuAg4rBKAABZB0BBDIRHa1DvLPYzHuMAigSAnwCUD9UzEHdvL7JETYAAIUFBNXJi2mQCzCkFJEbznRvkgCjROj6B2COAAGYNAAB2FYBeeYdg/wPOFcZCa2cIfwaaPoABZpREfLFUWcTyUkfAHwZwUdfoVoNAvIbLZATLc7VKYVNvPIZeJKREOxKwaAMAC4fMShPYFaUCcAtAeQeYcIAAJgAAZbCVgpYUxiCIltNdY6poJ3ZdhIx/A2RsZER0o0RXhUxeAKpwxKhMtNwIQXNIAT0yAkgAByZAeIKwZIcIzsPkfOYyK+drGZJUNpeZUQRZV+KrP4aBftT0HgEbbZABcbLlKAM2ZaA2Nwh+JED9O/IgNtINLHPnefXHXFcSZfIlEnX/HBTPJECyZ8KAUAiAqAiUWAuNeArgAALw0H+zr1lU1Vo0oO8FwFlWYLYM1RpxmN4XamQNQOgIwAwJVSwPMwd0gHWM2MoG2N2N9H2NlU4JONRVnwxV4gGIFyXwJRX0GCOyEgcKcPsJWHsIAA5bDGCLglh5gLgmYtIdJ4MOYcwMQQY8YQ5QMzlOscTmFnFzJfi99xND82V5AT9OUPJUAwSjVACKxFVmE+UL9GoZCeFZZ5Fzi8TikCThEijA5YMMSNssTpBuYiw81u4uSWojQhoRokRukj1Zw8CnYphnZEhPJEgj0rwkRwhKoMBPYdg5hIcfBaphUsx2TLcqCdhaCehQJ7C81mELZtC+gaEW1/1IArwj1pFPoAANbU6ABFYFOdFMI4fDdHUuCgJ4dEFACVMQuUyULwfwBSeQHJf2VrWyDrcE4UnrEovrQo0CSo2BGo3+OoyINSRo/4d/HsZMvk45EpQko7QaCVZXZU1U9Uz6TUt6bU3Uq8H0v0n0/afybpT6f9eIfaaRewztWmUUu6I+CUrPHmA+R+DNLNQA9tTzX49FTFQExfYYkEolRk2ZQSSEmE64ew+YRgxwpwjEMgRwNElmTEs8fEfPEqYkgUomFs1+YkpaUknfNrJlSkxyPxGkgJU/LlAiAYHBI7bUFk3uRQF5N4ZKKwdqYeGpR81zDs7C587SRc/SZcjED84kRicJcMB9f+RwGg46f6JI4tKTUtGlDIJI3An1Dk2ijkFuZUJFKo+AGCBRGgKeNAcFfoT/b/PkbCri4IZEJIgAdRzjzj9DYq83InYFZBriXVQqzAYIQH1kv2kodNPm4PYE9J7hvSwEkO6XCFAkwFNWFCYstX8EbE2FAX4CHD4F5WqMoDjMLilAQVYSwQ8JiA3gZLgspzyNzLKPzIoOfjzJLNWRgXWWGwrP/irIm1rOWiMrorksYvyi4zYtivQuBEwr8mopwqVLwoXMPiIrfJIuEyx2SlPLlGZJtB2CRAvOhPmGvNvNsPvOqs/QItqvZnqtIq/JOV6hJn/PGVMi8FwN3IgH+JekzT6EElXKEhqtfIhklm4CICAopO8SpKcggo5Wbi5XiE4oRnlJ5JiCDn4WbKFMVP+jGk2SmlGUVEgwWlm1iran3jFKXLfKlMmhHW4XlNQE9gwDABmFkMmgXmtNWltP2JksdL+GdJ9K7U8texeA6Tln5nX2u20Hnm8pfywU3CUGzM8Wiv626zitKP60SsG2qPgVG3qOrP2Q/WNnCTD3uvxJ/JJi4Ferhumk7FmjyBNPNN8CRuoNnNOlXPOP+sItGp2r6JWspHWqLE2qGoBrqp2o0D2ppxnz3IMDVrWuwqEkGsVpGu0jGuE31v2vJM8RAqOrAuP0grpO5URGkqREtvRKVptp2oapKnmkmEL2WiYXNlYU4uXlIq4ASDNVA2Ys6w0FqBgrZNWm7hdCIHLHkFpszPDHoNTBjuE0QGs1sxNWmgVHGkQEGU12WnIXkFiE8gmlaBOAlBqDiE+lV0kjY2ri+A7tbFiEUrdWUteBTsmRzIKOWVpt6wSoqKSqqI2TSrG3Zp90+n+2RF9pfPFNtuDpD01mLxCozsvwdJAPyqPC41Vo3LNv+wtv+3wp1uVs5iDtjANsWrn20SBKPMJzkjF3gG30dtNX3zHmpNTF5SMEuoNkoW8H+m0EmKK0ojbg7jeV0qtVNE7nyHEAQXzwjGmzlADEXCOHPUqTMywHy0qi8AYG8Epy2B2AEs/nhgyD/3godL5WFVLPZLCyIF5ifX9wsh9xMz9hIFuNIY/WBCmIOVQF4YhAoaocDE9JLHIGpA9HyzsqwGIcwIhBcryn5EpBAj+ivzrzCOpkVRUKVRIfuPhC+16DoB+h2BrVv2YZiHy3rHkL4EUOuMgUs1HyY3UPMc0blBUMNMqTwB0syWpF8HywqU/ktmqktOC1zy6yTOBFOkkOkNkPrg8gzAyEeBkIjE9SNEJpwVgFwEywaHFksrRy8AVCjtmkwKw1hFTj/lkeobVQoPpiUldGyeBClmCv1igVWS2XUALFjP5lCVoECsSj4EaFkFiyeE9gKH5BIDIPSC9AtCkbwYyEpumU6zmUtELLnpVyZqXq2XSt2X2WaJiGkZfWVVCZEcsbEYOLO2r070McB3ljuPtyNzeeh0gDhy7woD6Ljzx2/qTz/smPfv+OBaGPxR/sElMK8C1v3jAH9rADfMAsAa8Xk2OvArAagogc4tuTqQeVuvoCW01gfv9uIsywesFMvseT5TrLaSOWqNZxlX9FEGQ12DRjaXPtFC4xSj41QDuXKD7RtTqieFogeCyRFO1qpaBoiCLGNTOlIvsBBzaQYoTpLRJiSPCECsK2Shjplbg0iDwDq00KGooIrAI1TUPRaUTrSgykXGnVWj8ykHUYyJMfjWHQ4r+r9utqxCBVQELqOD2gOlhmSboaaHVbiLqgnjQFTn6CRAABI+E+apNfyQQDT1yWRg31XaaqHYyNLZxoA3p00bZWxDW3wh1CmY200U37tYBlIV1A32Lgl8k67hqMQcRxBZQGAs8PUR882GD8xfREXYx4iyGmGtkdhZUYjMxwxOhSLgnFaEs95/XMTAVwhNZ4mbGKBA3NUI31Mv9clvgmtaAUjGzaX+b6XUBop/B3YYgl3ERZSUmHTQbXSIR43xhEWVdJLvgRXc4ugytnRRXCD+hi6SpkAEEpU6IWgkAxa35WwNMe7K4+6qhEtkseDOxyoKGp5QOgO+XQQWKp2+xp0fZ9UnXk5vghwaYXWvDLX/btn8jdnYrZ6YrGbkqhtyzTmV7Mr3LvKjhSyUqePEFNmsA0EOw6Bg9A9mdpBiFkpAlw1JRzZUE/gqKakjlBI2XAPjVe1Cg2AIwHQSBwgJbRLIwkIhV5BZ6Yg+PVEjalr9zP7DzYWk8EWkWtIUXra0W9a6VMXnbsXXb2VwGDBIHL9rm8hyX30hr1sj4ttpAds9t08SOQ7GXlojg2Xl5Gs9hCH8wiOk74KK0hXVDxP0AGBXhbRR0X5E34y1Dkox0bKwIF1xBtL6AV0h1d3HMQQ5Zss95MM1DB08sLGnFaHGAqo5GXRzL2pUx8vLVOhKg0dCtBG7nhv0D2om0hR6uTRAVIoNU+vxXhUhwsBQpRBQn2Sak2XWkjhz29ZDYqpY5lWtvrUF52uduAFyZ64OKlOcMNcdY3h6GYFAj/pbmVtVuMhHmSQzXsH5A4jkoAQLEqAXA3KBn/3IBfN/M+BfKxmArwwFpnlyZxkJ6qap6utjW6aizllOPF7UreO2bMq/cZs6yQCZm2RcH6cctQ9VstJYv9J4vEBEuQMqUuNTt1CXtrsvg7t51Hs3wxe05bt1D4CIcXjfmtCMBFeAXYcTjTp+jnOQXXPQTFA9gPO+gvP4MfPOYIWHPOJuIgJhU7K8ASIXbvDYJqJaIj8PRUJDeMI1A2IcJre8J5ZWgOpAV8QQs1IX9aBu3lGdh/wDAA/5hbDYSerGCVhXg1gBB7C1gVhRASBbCBA1gWCU+1g0Ak+BAlpaBGC0A4SK+VhaB7COJDAA+loWCGAfhaALg1haAM/7DTFaAYS0A1hYSSB7CK+0BGClBGCSAlhYTK+SABAVhZ+G+4/AJIBlhU+0AnDaA1gSB5haB5hM00AVhYSlo1glgfgGB9+GBa+fgLgfhYSBB5gfg8+nCl+A+Lg5+lh7DHCSA2CowBAr+z+KwRPkDCP5Rgz+t5f/osD35oAYBT4XCCvx6o/Alg5XCfreQ35Z9zOG/FvgP3mBrBO+UYFgun1sIsE2+SwKMK/xX4rAq+CkBPhcA34CBbCdfMgVP1hIXAGAt5aEowXv4MAC+A/EfrCTQBLAloFAiAAWHsKCDbCSwFgofywG8CH+PwFEvMBb6F9GCthH4LXwYC2EN+KwbvqYhEFQAx+ggi4B3wYAXA8+2ghgGwQv6Z8BAsJewnQKWAXAv+CJFQI3hID999BkAFgteQuCMFGC3g3Pgnzv60BJBbBLgSwVWAhCb+C/OEq8GoEZ86o8A0QbYJb7p85gp/WwbYR6pXArgT/FYDf2L79V3BM/NQQIGRJ4CBAS/ZfqIO8LqBA2ofEgAqHYR0AHywqfQEAA -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details open="true">
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=PrairieLearn/PrairieLearn&utm_content=11956):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Explain this complex logic.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai explain this code block.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and explain its main purpose.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

### Support

Need help? Create a ticket on our [support page](https://www.coderabbit.ai/contact-us/support) for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai generate sequence diagram` to generate a sequence diagram of the changes in this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-05-10T14:14:24Z
body: ## [Codecov](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) Report
:x: Patch coverage is `8.13953%` with `79 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.28%. Comparing base ([`d3cdc83`](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/commit/d3cdc8367b0b42841d73d06aba77182f6cb27d1c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn)) to head ([`b87c6bb`](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/commit/b87c6bbe69b82104545f25f46a22e5832fb9166b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn)).
:warning: Report is 401 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) | Patch % | Lines |
|---|---|---|
| [...ps/prairielearn/src/ee/lib/aiQuestionGeneration.ts](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&filepath=apps%2Fprairielearn%2Fsrc%2Fee%2Flib%2FaiQuestionGeneration.ts&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn#diff-YXBwcy9wcmFpcmllbGVhcm4vc3JjL2VlL2xpYi9haVF1ZXN0aW9uR2VuZXJhdGlvbi50cw==) | 3.57% | [54 Missing :warning: ](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) |
| [.../prairielearn/src/models/course-instance-usages.ts](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&filepath=apps%2Fprairielearn%2Fsrc%2Fmodels%2Fcourse-instance-usages.ts&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn#diff-YXBwcy9wcmFpcmllbGVhcm4vc3JjL21vZGVscy9jb3Vyc2UtaW5zdGFuY2UtdXNhZ2VzLnRz) | 5.88% | [16 Missing :warning: ](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) |
| [apps/prairielearn/src/lib/ai.ts](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&filepath=apps%2Fprairielearn%2Fsrc%2Flib%2Fai.ts&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn#diff-YXBwcy9wcmFpcmllbGVhcm4vc3JjL2xpYi9haS50cw==) | 18.18% | [9 Missing :warning: ](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) |

<details><summary>Additional details and impacted files</summary>



```diff
@@            Coverage Diff             @@
##           master   #11956      +/-   ##
==========================================
- Coverage   67.31%   67.28%   -0.04%     
==========================================
  Files         704      705       +1     
  Lines       72070    72129      +59     
  Branches     7681     7678       -3     
==========================================
+ Hits        48515    48532      +17     
- Misses      23389    23431      +42     
  Partials      166      166              
```

| [Flag](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) | Coverage Δ | |
|---|---|---|
| [javascript](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) | `66.84% <8.13%> (-0.04%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn) | `81.04% <ø> (ø)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/PrairieLearn/PrairieLearn/pull/11956?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PrairieLearn).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-10T14:16:35Z
body: ## All images

| Image | Platform | Old Size | New Size | Change |
| --- | --- | --- | --- | --- |
| [prairielearn/executor:b87c6bbe69b82104545f25f46a22e5832fb9166b](https://hub.docker.com/layers/prairielearn/executor/b87c6bbe69b82104545f25f46a22e5832fb9166b/images/sha256-e9c6ff05a21ae9aee22d7f333ed581e27bdc7d90730d10c94d0b00d562fe855b?context=explore) | linux/arm64 | 1237.55 MB | 1237.57 MB | 0.00% |
| [prairielearn/executor:b87c6bbe69b82104545f25f46a22e5832fb9166b](https://hub.docker.com/layers/prairielearn/executor/b87c6bbe69b82104545f25f46a22e5832fb9166b/images/sha256-e25ab9930e1f73547398b505cb51765b9daf0fc77236e30c79d5a504a2ab9ea3?context=explore) | linux/amd64 | 1229.75 MB | 1229.76 MB | 0.00% |
| [prairielearn/prairielearn:b87c6bbe69b82104545f25f46a22e5832fb9166b](https://hub.docker.com/layers/prairielearn/prairielearn/b87c6bbe69b82104545f25f46a22e5832fb9166b/images/sha256-79fbd925ff09a6e194f9d689f924f90bd36d96894df5e25528a1c05d0330d941?context=explore) | linux/amd64 | 1229.74 MB | 1229.75 MB | 0.00% |
| [prairielearn/prairielearn:b87c6bbe69b82104545f25f46a22e5832fb9166b](https://hub.docker.com/layers/prairielearn/prairielearn/b87c6bbe69b82104545f25f46a22e5832fb9166b/images/sha256-59a60cf412dc5d32bddb71f1e1e0147c45a0e59b9ab01bf65eb4de3ecb04aefa?context=explore) | linux/arm64 | 1237.54 MB | 1237.57 MB | 0.00% |
author_association: CONTRIBUTOR